* [sup-talk] sup-sync and xapian memory usage
@ 2009-09-07 17:04 Andrew Pimlott
2009-09-07 18:14 ` Ben Walton
` (2 more replies)
0 siblings, 3 replies; 12+ messages in thread
From: Andrew Pimlott @ 2009-09-07 17:04 UTC (permalink / raw)
Last time I tried to use sup[1], I posted about sup-sync crashing with
various symptoms of memory exhaustion[2]. I've tried again (using git
mainline), with similar results, but now I have a bit more to say about
it.
I am running on a fairly low-memory virtual machine. I think some of
the variability in what I was seeing before had to do with what other
things were running. Sorry about not being aware of this before. In
the following, I have pretty well controlled for other system memory
use. All of these tests are done on the same mbox with all indices
cleared.
Some of the failures I see are out of memory when running gpg
("Errno::ENOMEM Exception: Cannot allocate memory - /usr/bin/gpg" ...).
I stopped at that point in the debugger and found that at this point,
the ruby backtick operator fails the same way on any command. Using
strace, I saw a failure in clone:
20528 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7cb3908) = -1 ENOMEM (Cannot allocate memory)
top reports 2.5M of physical and 30M of swap free, so I don't really
know why clone fails, but I guess there's not much you can do about
that.
Other failures were the result of sup blowing up on messages with large
attachments. sup's memory use is many times the attachment size.
Testing on an mbox with a single message with 4 ~5M (encoded size)
attachments (total file size ~21M), sup goes up to ~150M. Accounting
for baseline, that's about 6 times the file size. I hope that can be
improved. FWIW, mutt never seems to try to load a whole attachment into
memory.
Using the ferret index, the memory (virtual as reported by Linux)
behaviour of sup-sync is pretty good. It starts out 25M and levels out
around 31M. It spikes from time to time, presumably because of large
messages, but it comes right back. After processing the last message,
it uses another 10M to finish up.
Using the xapian index, things are different. It starts at 32M and
steadily climbs to 77M after ~3500 messages, or around 1M every 100
messages. It does seem to climb faster at first and then more slowly.
Either xapian is keeping a cache (but some searches suggest it doesn't),
it's leaking memory, or it's allocating memory in a way that the the
allocator can't reclaim the VM space. Any ideas?
BTW, is there really no way to ask for ruby's heap size with (unpatched)
ruby 1.8?
Andrew
[1] This is about the fourth time. I seem to be easily discouraged.
[2] http://rubyforge.org/pipermail/sup-talk/2009-May/002171.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-07 17:04 [sup-talk] sup-sync and xapian memory usage Andrew Pimlott
@ 2009-09-07 18:14 ` Ben Walton
2009-09-07 18:33 ` Rich Lane
2009-09-08 13:15 ` William Morgan
2 siblings, 0 replies; 12+ messages in thread
From: Ben Walton @ 2009-09-07 18:14 UTC (permalink / raw)
Excerpts from Andrew Pimlott's message of Mon Sep 07 13:04:50 -0400 2009:
> I am running on a fairly low-memory virtual machine. I think some of
> the variability in what I was seeing before had to do with what
> other
I'm running sup on real hardware w/1GB physical ram.
> things were running. Sorry about not being aware of this before. In
> the following, I have pretty well controlled for other system memory
> use. All of these tests are done on the same mbox with all indices
> cleared.
>
> Some of the failures I see are out of memory when running gpg
> ("Errno::ENOMEM Exception: Cannot allocate memory - /usr/bin/gpg" ...).
> I stopped at that point in the debugger and found that at this point,
> the ruby backtick operator fails the same way on any command. Using
> strace, I saw a failure in clone:
I found sup crashed this morning too with an ENOMEM on a gpg
operation. I thought I may actually have been at the memory
exhaustion point normally though as I run screen and tend to leave
lots of sessions going all the time.
> Using the xapian index, things are different. It starts at 32M and
> steadily climbs to 77M after ~3500 messages, or around 1M every 100
> messages. It does seem to climb faster at first and then more
> slowly.
I've never paid attention to this before, but currently, my sup
process (running for about 6 hours now) looks like it's using ~77M
virtual with 59 of that resident.
This has only happened to me once, but since you reported it, I
thought I'd chime in with some extra data.
HTH.
-Ben
--
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302
GPG Key Id: 8E89F6D2; Key Server: pgp.mit.edu
Contact me to arrange for a CAcert assurance meeting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20090907/4318eb38/attachment.bin>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-07 17:04 [sup-talk] sup-sync and xapian memory usage Andrew Pimlott
2009-09-07 18:14 ` Ben Walton
@ 2009-09-07 18:33 ` Rich Lane
2009-09-07 19:11 ` Ben Walton
2009-09-07 19:26 ` Andrew Pimlott
2009-09-08 13:15 ` William Morgan
2 siblings, 2 replies; 12+ messages in thread
From: Rich Lane @ 2009-09-07 18:33 UTC (permalink / raw)
Excerpts from Andrew Pimlott's message of Mon Sep 07 13:04:50 -0400 2009:
> Using the xapian index, things are different. It starts at 32M and
> steadily climbs to 77M after ~3500 messages, or around 1M every 100
> messages. It does seem to climb faster at first and then more slowly.
> Either xapian is keeping a cache (but some searches suggest it doesn't),
> it's leaking memory, or it's allocating memory in a way that the the
> allocator can't reclaim the VM space. Any ideas?
Xapian keeps writes buffered in memory. Try setting the environment
variable XAPIAN_FLUSH_THRESHOLD to a smaller value (the default is 10000
documents) and see if that helps.
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-07 18:33 ` Rich Lane
@ 2009-09-07 19:11 ` Ben Walton
2009-09-07 19:15 ` Rich Lane
2009-09-07 19:26 ` Andrew Pimlott
1 sibling, 1 reply; 12+ messages in thread
From: Ben Walton @ 2009-09-07 19:11 UTC (permalink / raw)
Excerpts from Rich Lane's message of Mon Sep 07 14:33:06 -0400 2009:
> Xapian keeps writes buffered in memory. Try setting the environment
> variable XAPIAN_FLUSH_THRESHOLD to a smaller value (the default is 10000
> documents) and see if that helps.
Does this explain the lag at shutdown? Xapian is flushing writes to
disk?
-Ben
--
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302
GPG Key Id: 8E89F6D2; Key Server: pgp.mit.edu
Contact me to arrange for a CAcert assurance meeting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20090907/b8fec298/attachment.bin>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-07 19:11 ` Ben Walton
@ 2009-09-07 19:15 ` Rich Lane
2009-09-08 13:39 ` William Morgan
0 siblings, 1 reply; 12+ messages in thread
From: Rich Lane @ 2009-09-07 19:15 UTC (permalink / raw)
Excerpts from Ben Walton's message of Mon Sep 07 15:11:58 -0400 2009:
> Excerpts from Rich Lane's message of Mon Sep 07 14:33:06 -0400 2009:
> > Xapian keeps writes buffered in memory. Try setting the environment
> > variable XAPIAN_FLUSH_THRESHOLD to a smaller value (the default is 10000
> > documents) and see if that helps.
>
> Does this explain the lag at shutdown? Xapian is flushing writes to
> disk?
Yep.
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-07 18:33 ` Rich Lane
2009-09-07 19:11 ` Ben Walton
@ 2009-09-07 19:26 ` Andrew Pimlott
1 sibling, 0 replies; 12+ messages in thread
From: Andrew Pimlott @ 2009-09-07 19:26 UTC (permalink / raw)
On Mon, Sep 07, 2009 at 02:33:06PM -0400, Rich Lane wrote:
> Xapian keeps writes buffered in memory. Try setting the environment
> variable XAPIAN_FLUSH_THRESHOLD to a smaller value (the default is 10000
> documents) and see if that helps.
Thanks--it was hard for me to find that kind of information. I first
tried setting XAPIAN_FLUSH_THRESHOLD to 1, and sup-sync ran slowly and
just kept getting slower:
## read 139m (about 7%) @ 9.2m/s. 0:00:15 elapsed, about 0:03:21 remaining
...
## read 1238m (about 35%) @ 3.1m/s. 0:06:36 elapsed, about 0:12:08 remaining
I stopped at this point because it was taking too long. The memory use
seemed stable, but that could have been because it was making such slow
progress. I guess xapian gets a lot slower writing as the db grows?
That's a bit discouraging. Using ferret, sup-sync only dropped from
28.1m/s to 27.3m/s during its run. For reference, when I didn't set
XAPIAN_FLUSH_THRESHOLD, I was getting 35-36m/s until it ran out of
memory.
I then set XAPIAN_FLUSH_THRESHOLD to 100 and got more reasonable
results. It started at 25.6m/s and slowed to 17.8m/s. It stabilized at
around 41M virtual memory used and finished successfuly. I also note
that the memory use didn't jump during the finish-up phase ("Deleting
missing messages") as it had with ferret.
Finally, I set XAPIAN_FLUSH_THRESHOLD to 1000. It started at 34.6m/s
and dropped to 29.8m/s., stabilized at around 51M virtual memory, and
finished successfully. In this case, it stays faster than ferret, but
it sill bugs me that xapian still slows down while ferret doesn't.
So I conclude... I don't know what I conclude. Letting xapian use a lot
of memory sure helps its performance. And a big sup-sync should only
have to be done rarely. So maybe just document that those on low-memory
systems should consider using XAPIAN_FLUSH_THRESHOLD during sup-sync.
Thanks again for your help!
Andrew
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-07 17:04 [sup-talk] sup-sync and xapian memory usage Andrew Pimlott
2009-09-07 18:14 ` Ben Walton
2009-09-07 18:33 ` Rich Lane
@ 2009-09-08 13:15 ` William Morgan
2 siblings, 0 replies; 12+ messages in thread
From: William Morgan @ 2009-09-08 13:15 UTC (permalink / raw)
Reformatted excerpts from Andrew Pimlott's message of 2009-09-07:
> Last time I tried to use sup[1], I posted about sup-sync crashing with
> various symptoms of memory exhaustion[2].
Excellent resolution to that thread.
> sup's memory use is many times the attachment size. Testing on an
> mbox with a single message with 4 ~5M (encoded size) attachments
> (total file size ~21M), sup goes up to ~150M. Accounting for
> baseline, that's about 6 times the file size. I hope that can be
> improved.
Sup uses the unmaintained RubyMail for MIME handling. I'm sure this
behavior can be improved if you can find someone who wants to write some
MIME handling code. Personally that's near the bottom of my list.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-07 19:15 ` Rich Lane
@ 2009-09-08 13:39 ` William Morgan
2009-09-08 13:58 ` Ben Walton
2009-09-08 14:27 ` Richard Heycock
0 siblings, 2 replies; 12+ messages in thread
From: William Morgan @ 2009-09-08 13:39 UTC (permalink / raw)
Reformatted excerpts from Rich Lane's message of 2009-09-07:
> Excerpts from Ben Walton's message of Mon Sep 07 15:11:58 -0400 2009:
> > Excerpts from Rich Lane's message of Mon Sep 07 14:33:06 -0400 2009:
> > > Xapian keeps writes buffered in memory. Try setting the
> > > environment variable XAPIAN_FLUSH_THRESHOLD to a smaller value
> > > (the default is 10000 documents) and see if that helps.
> >
> > Does this explain the lag at shutdown? Xapian is flushing writes to
> > disk?
>
> Yep.
Good to know. Is there a way to force a flush in Xapian? Just curious.
The delay is a little scary sometimes. Maybe it just needs an
appropriate message somewhere.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-08 13:39 ` William Morgan
@ 2009-09-08 13:58 ` Ben Walton
2009-09-08 14:27 ` Richard Heycock
1 sibling, 0 replies; 12+ messages in thread
From: Ben Walton @ 2009-09-08 13:58 UTC (permalink / raw)
Excerpts from William Morgan's message of Tue Sep 08 09:39:12 -0400 2009:
> Good to know. Is there a way to force a flush in Xapian? Just curious.
> The delay is a little scary sometimes. Maybe it just needs an
> appropriate message somewhere.
...even just a message indicating that xapian is flushing the buffered
writes would be a good thing.
-Ben
--
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302
GPG Key Id: 8E89F6D2; Key Server: pgp.mit.edu
Contact me to arrange for a CAcert assurance meeting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20090908/fa461b0e/attachment.bin>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-08 13:39 ` William Morgan
2009-09-08 13:58 ` Ben Walton
@ 2009-09-08 14:27 ` Richard Heycock
2009-09-08 15:14 ` Nicolas Pouillard
1 sibling, 1 reply; 12+ messages in thread
From: Richard Heycock @ 2009-09-08 14:27 UTC (permalink / raw)
Excerpts from William Morgan's message of Tue Sep 08 23:39:12 +1000 2009:
> Reformatted excerpts from Rich Lane's message of 2009-09-07:
> > Excerpts from Ben Walton's message of Mon Sep 07 15:11:58 -0400 2009:
> > > Excerpts from Rich Lane's message of Mon Sep 07 14:33:06 -0400 2009:
> > > > Xapian keeps writes buffered in memory. Try setting the
> > > > environment variable XAPIAN_FLUSH_THRESHOLD to a smaller value
> > > > (the default is 10000 documents) and see if that helps.
> > >
> > > Does this explain the lag at shutdown? Xapian is flushing writes to
> > > disk?
> >
> > Yep.
>
> Good to know. Is there a way to force a flush in Xapian? Just curious.
> The delay is a little scary sometimes. Maybe it just needs an
> appropriate message somewhere.
Xapian, by default, flushes every 10 000 documents which in email terms
is a pretty long time! You can force a flush but it does increase your
IO significantly. Having said that emails are fairly small so flushing
every 10 or 20 emails might not be too bad.
Maybe it could be dynamic, if there is a high volume of incoming emails
flushing could be done after 50 or 100 emails or a timer based flush
might be easier.
rgh
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-08 14:27 ` Richard Heycock
@ 2009-09-08 15:14 ` Nicolas Pouillard
2009-09-09 6:18 ` Rich Lane
0 siblings, 1 reply; 12+ messages in thread
From: Nicolas Pouillard @ 2009-09-08 15:14 UTC (permalink / raw)
Excerpts from Richard Heycock's message of Tue Sep 08 16:27:10 +0200 2009:
> Excerpts from William Morgan's message of Tue Sep 08 23:39:12 +1000 2009:
> > Reformatted excerpts from Rich Lane's message of 2009-09-07:
> > > Excerpts from Ben Walton's message of Mon Sep 07 15:11:58 -0400 2009:
> > > > Excerpts from Rich Lane's message of Mon Sep 07 14:33:06 -0400 2009:
> > > > > Xapian keeps writes buffered in memory. Try setting the
> > > > > environment variable XAPIAN_FLUSH_THRESHOLD to a smaller value
> > > > > (the default is 10000 documents) and see if that helps.
> > > >
> > > > Does this explain the lag at shutdown? Xapian is flushing writes to
> > > > disk?
> > >
> > > Yep.
> >
> > Good to know. Is there a way to force a flush in Xapian? Just curious.
> > The delay is a little scary sometimes. Maybe it just needs an
> > appropriate message somewhere.
>
> Xapian, by default, flushes every 10 000 documents which in email terms
> is a pretty long time! You can force a flush but it does increase your
> IO significantly. Having said that emails are fairly small so flushing
> every 10 or 20 emails might not be too bad.
>
> Maybe it could be dynamic, if there is a high volume of incoming emails
> flushing could be done after 50 or 100 emails or a timer based flush
> might be easier.
Should the '$' command, force a write of the Xapian database?
--
Nicolas Pouillard
http://nicolaspouillard.fr
^ permalink raw reply [flat|nested] 12+ messages in thread
* [sup-talk] sup-sync and xapian memory usage
2009-09-08 15:14 ` Nicolas Pouillard
@ 2009-09-09 6:18 ` Rich Lane
0 siblings, 0 replies; 12+ messages in thread
From: Rich Lane @ 2009-09-09 6:18 UTC (permalink / raw)
Excerpts from Nicolas Pouillard's message of Tue Sep 08 11:14:22 -0400 2009:
> Excerpts from Richard Heycock's message of Tue Sep 08 16:27:10 +0200 2009:
> > Excerpts from William Morgan's message of Tue Sep 08 23:39:12 +1000 2009:
> > > Reformatted excerpts from Rich Lane's message of 2009-09-07:
> > > > Excerpts from Ben Walton's message of Mon Sep 07 15:11:58 -0400 2009:
> > > > > Excerpts from Rich Lane's message of Mon Sep 07 14:33:06 -0400 2009:
> > > > > > Xapian keeps writes buffered in memory. Try setting the
> > > > > > environment variable XAPIAN_FLUSH_THRESHOLD to a smaller value
> > > > > > (the default is 10000 documents) and see if that helps.
> > > > >
> > > > > Does this explain the lag at shutdown? Xapian is flushing writes to
> > > > > disk?
> > > >
> > > > Yep.
> > >
> > > Good to know. Is there a way to force a flush in Xapian? Just curious.
> > > The delay is a little scary sometimes. Maybe it just needs an
> > > appropriate message somewhere.
> >
> > Xapian, by default, flushes every 10 000 documents which in email terms
> > is a pretty long time! You can force a flush but it does increase your
> > IO significantly. Having said that emails are fairly small so flushing
> > every 10 or 20 emails might not be too bad.
> >
> > Maybe it could be dynamic, if there is a high volume of incoming emails
> > flushing could be done after 50 or 100 emails or a timer based flush
> > might be easier.
>
> Should the '$' command, force a write of the Xapian database?
I think once we're saving label changes to the index immediately we
could re-purpose '$' for this. I've also been considering a timer, as
Richard suggests, which would trigger a flush once the user has been
idle for some time.
A fatal exception in Sup is still a clean exit as far as SWIG is
concerned and the Xapian database destructor will do the flush for us in
that case, so don't worry about random Sup exceptions causing data loss.
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2009-09-09 6:18 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-09-07 17:04 [sup-talk] sup-sync and xapian memory usage Andrew Pimlott
2009-09-07 18:14 ` Ben Walton
2009-09-07 18:33 ` Rich Lane
2009-09-07 19:11 ` Ben Walton
2009-09-07 19:15 ` Rich Lane
2009-09-08 13:39 ` William Morgan
2009-09-08 13:58 ` Ben Walton
2009-09-08 14:27 ` Richard Heycock
2009-09-08 15:14 ` Nicolas Pouillard
2009-09-09 6:18 ` Rich Lane
2009-09-07 19:26 ` Andrew Pimlott
2009-09-08 13:15 ` William Morgan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox