From: rgh@roughage.com.au (Richard Heycock)
Subject: [sup-talk] sup-sync and xapian memory usage
Date: Wed, 09 Sep 2009 00:27:10 +1000 [thread overview]
Message-ID: <1252419716-sup-3031@roughage.com.au> (raw)
In-Reply-To: <1252415772-sup-7288@masanjin.net>
Excerpts from William Morgan's message of Tue Sep 08 23:39:12 +1000 2009:
> Reformatted excerpts from Rich Lane's message of 2009-09-07:
> > Excerpts from Ben Walton's message of Mon Sep 07 15:11:58 -0400 2009:
> > > Excerpts from Rich Lane's message of Mon Sep 07 14:33:06 -0400 2009:
> > > > Xapian keeps writes buffered in memory. Try setting the
> > > > environment variable XAPIAN_FLUSH_THRESHOLD to a smaller value
> > > > (the default is 10000 documents) and see if that helps.
> > >
> > > Does this explain the lag at shutdown? Xapian is flushing writes to
> > > disk?
> >
> > Yep.
>
> Good to know. Is there a way to force a flush in Xapian? Just curious.
> The delay is a little scary sometimes. Maybe it just needs an
> appropriate message somewhere.
Xapian, by default, flushes every 10 000 documents which in email terms
is a pretty long time! You can force a flush but it does increase your
IO significantly. Having said that emails are fairly small so flushing
every 10 or 20 emails might not be too bad.
Maybe it could be dynamic, if there is a high volume of incoming emails
flushing could be done after 50 or 100 emails or a timer based flush
might be easier.
rgh
next prev parent reply other threads:[~2009-09-08 14:27 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-09-07 17:04 Andrew Pimlott
2009-09-07 18:14 ` Ben Walton
2009-09-07 18:33 ` Rich Lane
2009-09-07 19:11 ` Ben Walton
2009-09-07 19:15 ` Rich Lane
2009-09-08 13:39 ` William Morgan
2009-09-08 13:58 ` Ben Walton
2009-09-08 14:27 ` Richard Heycock [this message]
2009-09-08 15:14 ` Nicolas Pouillard
2009-09-09 6:18 ` Rich Lane
2009-09-07 19:26 ` Andrew Pimlott
2009-09-08 13:15 ` William Morgan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1252419716-sup-3031@roughage.com.au \
--to=rgh@roughage.com.au \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox