From mboxrd@z Thu Jan  1 00:00:00 1970
From: rlane@club.cc.cmu.edu (Rich Lane)
Date: Mon, 07 Sep 2009 14:33:06 -0400
Subject: [sup-talk] sup-sync and xapian memory usage
In-Reply-To: <20090907170450.GO14010@pimlott.net>
References: <20090907170450.GO14010@pimlott.net>
Message-ID: <1252348258-sup-415@zyrg.net>

Excerpts from Andrew Pimlott's message of Mon Sep 07 13:04:50 -0400 2009:
> Using the xapian index, things are different.  It starts at 32M and
> steadily climbs to 77M after ~3500 messages, or around 1M every 100
> messages.  It does seem to climb faster at first and then more slowly.
> Either xapian is keeping a cache (but some searches suggest it doesn't),
> it's leaking memory, or it's allocating memory in a way that the the
> allocator can't reclaim the VM space.  Any ideas?

Xapian keeps writes buffered in memory. Try setting the environment
variable XAPIAN_FLUSH_THRESHOLD to a smaller value (the default is 10000
documents) and see if that helps.