From mboxrd@z Thu Jan  1 00:00:00 1970
From: rlane@club.cc.cmu.edu (Rich Lane)
Date: Wed, 09 Sep 2009 02:18:09 -0400
Subject: [sup-talk] sup-sync and xapian memory usage
In-Reply-To: <1252422827-sup-805@peray>
References: <20090907170450.GO14010@pimlott.net> <1252348258-sup-415@zyrg.net>
	<1252350686-sup-8800@ntdws12.chass.utoronto.ca>
	<1252350896-sup-5026@zyrg.net> <1252415772-sup-7288@masanjin.net>
	<1252419716-sup-3031@roughage.com.au> <1252422827-sup-805@peray>
Message-ID: <1252427486-sup-8093@zyrg.net>

Excerpts from Nicolas Pouillard's message of Tue Sep 08 11:14:22 -0400 2009:
> Excerpts from Richard Heycock's message of Tue Sep 08 16:27:10 +0200 2009:
> > Excerpts from William Morgan's message of Tue Sep 08 23:39:12 +1000 2009:
> > > Reformatted excerpts from Rich Lane's message of 2009-09-07:
> > > > Excerpts from Ben Walton's message of Mon Sep 07 15:11:58 -0400 2009:
> > > > > Excerpts from Rich Lane's message of Mon Sep 07 14:33:06 -0400 2009:
> > > > > > Xapian keeps writes buffered in memory. Try setting the
> > > > > > environment variable XAPIAN_FLUSH_THRESHOLD to a smaller value
> > > > > > (the default is 10000 documents) and see if that helps.
> > > > > 
> > > > > Does this explain the lag at shutdown?  Xapian is flushing writes to
> > > > > disk?
> > > > 
> > > > Yep.
> > > 
> > > Good to know. Is there a way to force a flush in Xapian? Just curious.
> > > The delay is a little scary sometimes. Maybe it just needs an
> > > appropriate message somewhere.
> > 
> > Xapian, by default, flushes every 10 000 documents which in email terms
> > is a pretty long time! You can force a flush but it does increase your
> > IO significantly. Having said that emails are fairly small so flushing
> > every 10 or 20 emails might not be too bad.
> > 
> > Maybe it could be dynamic, if there is a high volume of incoming emails
> > flushing could be done after 50 or 100 emails or a timer based flush
> > might be easier.
> 
> Should the '$' command, force a write of the Xapian database?

I think once we're saving label changes to the index immediately we
could re-purpose '$' for this. I've also been considering a timer, as
Richard suggests, which would trigger a flush once the user has been
idle for some time.

A fatal exception in Sup is still a clean exit as far as SWIG is
concerned and the Xapian database destructor will do the flush for us in
that case, so don't worry about random Sup exceptions causing data loss.