Subject: Re: [sup-talk] sup with 120K mail

From: Tero Tilus <[email-masked]>
Date: Mon, 28 Dec 2009 21:34:07 +0200

Arvid Picciani, 2009-12-28 10:32:
> On 12/28/2009 04:12 PM, William Morgan wrote:
>> Can you be more specific about what's slow? I used Sup with a 160k
>> index for a long time without trouble.
> 
> "everything"  sums it up.  it takes forever to start, to quit, to show 
> the index, to show a single message, etc..

I've got 103k messages from 172 sources (137 imap sources, including
all sources receiving new mail).  I run sup on 512 MB Bytemark virtual
machine.  Startup is instantaneous, it takes 2-3 seconds to list first
screenfull of threads.  Practically all searches are instantaneous (I
remember hitting "slow queries" a couple of times).  Thread is
displayed in about messagecount/5 seconds.  Pretty much nothing feels
slow.  But there is that "pretty much nothing".

Quit is slow.  If I "just check" mail (launch, first poll adds 10-15
messages, archive them and quit) sup exits in a couple of seconds
(just long enough that you start to wonder if you pressed Q or not)
but if sup has been running days quit takes easily minutes.

-- 
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Tue, 29 Dec 2009 15:33:46 -0800

Reformatted excerpts from William Morgan's message of 2009-12-28:
> Reformatted excerpts from Tero Tilus's message of 2009-12-28:
> > William Morgan, 2009-12-28 22:17:
> > > Is this with Xapian?
> > 
> > Goodness!  Forgot the most important part...
> > 
> > Yes.  Xapian and git next.
> 
> Yeah, Xapian is the cause of that delay on quit. I think there was a
> workaround posted at some point about an environment variable or
> something...

Ah yes, try setting XAPIAN_FLUSH_THRESHOLD=10 when running sup (but not
sup-sync!) and see if that reduces the delay on quit.

Unfortunately there's no way (that I know of) to force a Xapian flush
from within Ruby. Correction from Xapian experts welcome.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Mon, 28 Dec 2009 09:34:38 -0800

> the index, to show a single message, etc..

Are you using IMAP?

---

From: Tero Tilus <[email-masked]>
Date: Mon, 28 Dec 2009 22:59:19 +0200

Steve Goldman, 2009-12-28 22:24:
> Is Xapian sup fully functional?

I've been on Xapian since september.  It has given me smooth and fully
functional ride.  I'd recommed trying it out.

---

From: Eric Sherman <[email-masked]>
Date: Mon, 28 Dec 2009 11:45:01 -0500

Excerpts from Arvid Picciani's message of Mon Dec 28 03:32:29 -0500 2009:
> > Reformatted excerpts from Arvid Picciani's message of 2009-12-27:
> >> are there any plans to support large amounts of mails?
> >> It doesn't appear to me much changed during the last months.
> >> Last time i tried it was unusable slow.
> >

I imagine [offlineimap][1] will make your life easier.  sup is speedy on my 
index of 35000 emails in maildir, while it was relatively unusable with raw 
imap.

[1]:http://software.complete.org/software/projects/show/offlineimap
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Mon, 28 Dec 2009 13:06:39 -0800

Reformatted excerpts from Steve Goldman's message of 2009-12-28:
> Is Xapian sup fully functional?  Last I read, there were some critical
> functions that were supported.

Functional enough that it's going to be the default in the next release
of Sup, and you're going to get yelled at for Ferret.

A few things don't work or are weird, but it could break everything and
still provide a better user experience than Ferret.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Mon, 28 Dec 2009 13:07:24 -0800

---

From: Tero Tilus <[email-masked]>
Date: Mon, 28 Dec 2009 22:24:13 +0200

---

From: Arvid Picciani <[email-masked]>
Date: Mon, 28 Dec 2009 06:51:01 +0100

hey,
Arvid
Asgaard Technologies

---

From: Gaute Hope <[email-masked]>
Date: Mon, 28 Dec 2009 20:57:11 +0100

> but if sup has been running days quit takes easily minutes.

Quit is slow for me too - but seldom above one minute, but I normally
don't quit (just suspending my machine) - so either my machine has
crashed or sup has crashed. So I hit $ occasionally to save the index,
otherwise I need to re-do everything I did manually since my last
'crash'. It would help if this is done when there is a sufficient amount
of stuff to save or on regular intervals.

- gaute

---

From: William Morgan <[email-masked]>
Date: Tue, 29 Dec 2009 15:54:13 -0800

Reformatted excerpts from Rich Lane's message of 2009-12-29:
> Xapian::WritableDatabase#flush
> 
> This should really be called from XapianIndex#save_index, along with a
> friendly message explaining what's going on.

Calling it from #save_index will have the advantage of us being able to
present a message, but what about calling it when "$" is pressed, or
periodically in the background? Does it block search?
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Steve Goldman <[email-masked]>
Date: Mon, 28 Dec 2009 15:24:18 -0500

Steve Goldman
[email-masked]

T: 212.219.6014
F: 212.219.6007

Tower Research Capital, LLC
377 Broadway, 11th Fl.
New York, NY 10013
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Steve Goldman <[email-masked]>
Date: Mon, 28 Dec 2009 11:45:58 -0500

> the index, to show a single message, etc..

I'm in the same boat.  Starting up, it takes over a minute before it
begins to show new messages.  My workaround is to run sup in screen.

There's a lot of variance in how long it takes to show a message once
I press enter on it.  Sometimes it's upwards of thirty seconds, other
times instant.

I always assumed this is because my mail server is on a remote host
and highly utilized.

Saving messages happens at the rate of about two messages per second.

Quitting happens quickly.

I have almost 60k messages in my index.

Thanks.
-- 

Steve Goldman

T: 212.219.6014
F: 212.219.6007

---

From: Arvid Picciani <[email-masked]>
Date: Mon, 28 Dec 2009 09:32:29 +0100

---

From: Rich Lane <[email-masked]>
Date: Tue, 29 Dec 2009 18:42:10 -0500

---

From: William Morgan <[email-masked]>
Date: Mon, 28 Dec 2009 07:12:12 -0800

---

From: "Edward Z. Yang" <[email-masked]>
Date: Mon, 28 Dec 2009 15:30:58 -0500

> functions that were supported.

I've been using Xapian sup for a few weeks now, and the primary difficulty
is that you have to specify from_email/from_to instead of just from.  But
it's very snappy, I haven't noticed any other problems and I haven't had
any major index corruption (which is what prompted the switch in the first
place).

Cheers,
Edward

---

From: William Morgan <[email-masked]>
Date: Mon, 28 Dec 2009 12:17:39 -0800

Is this with Xapian?