* Re: [sup-talk] sup with 120K mail
2009-12-28 8:32 ` Arvid Picciani
@ 2009-12-28 16:45 ` Eric Sherman
2009-12-28 16:45 ` Steve Goldman
` (2 subsequent siblings)
3 siblings, 0 replies; 18+ messages in thread
From: Eric Sherman @ 2009-12-28 16:45 UTC (permalink / raw)
To: sup-talk
Excerpts from Arvid Picciani's message of Mon Dec 28 03:32:29 -0500 2009:
> On 12/28/2009 04:12 PM, William Morgan wrote:
> > Reformatted excerpts from Arvid Picciani's message of 2009-12-27:
> >> are there any plans to support large amounts of mails?
> >> It doesn't appear to me much changed during the last months.
> >> Last time i tried it was unusable slow.
> >
> > Can you be more specific about what's slow? I used Sup with a 160k
> > index for a long time without trouble.
>
> "everything" sums it up. it takes forever to start, to quit, to show
> the index, to show a single message, etc..
I imagine [offlineimap][1] will make your life easier. sup is speedy on my
index of 35000 emails in maildir, while it was relatively unusable with raw
imap.
[1]:http://software.complete.org/software/projects/show/offlineimap
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [sup-talk] sup with 120K mail
2009-12-28 8:32 ` Arvid Picciani
2009-12-28 16:45 ` Eric Sherman
@ 2009-12-28 16:45 ` Steve Goldman
2009-12-28 17:34 ` William Morgan
2009-12-28 19:34 ` Tero Tilus
3 siblings, 0 replies; 18+ messages in thread
From: Steve Goldman @ 2009-12-28 16:45 UTC (permalink / raw)
To: sup-talk
Excerpts from Arvid Picciani's message of Mon Dec 28 03:32:29 -0500 2009:
> On 12/28/2009 04:12 PM, William Morgan wrote:
> > Reformatted excerpts from Arvid Picciani's message of 2009-12-27:
> >> are there any plans to support large amounts of mails?
> >> It doesn't appear to me much changed during the last months.
> >> Last time i tried it was unusable slow.
> >
> > Can you be more specific about what's slow? I used Sup with a 160k
> > index for a long time without trouble.
>
> "everything" sums it up. it takes forever to start, to quit, to show
> the index, to show a single message, etc..
>
I'm in the same boat. Starting up, it takes over a minute before it
begins to show new messages. My workaround is to run sup in screen.
There's a lot of variance in how long it takes to show a message once
I press enter on it. Sometimes it's upwards of thirty seconds, other
times instant.
I always assumed this is because my mail server is on a remote host
and highly utilized.
Saving messages happens at the rate of about two messages per second.
Quitting happens quickly.
I have almost 60k messages in my index.
Thanks.
--
Steve Goldman
sgoldman@tower-research.com
T: 212.219.6014
F: 212.219.6007
Tower Research Capital, LLC
377 Broadway, 11th Fl.
New York, NY 10013
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 8:32 ` Arvid Picciani
2009-12-28 16:45 ` Eric Sherman
2009-12-28 16:45 ` Steve Goldman
@ 2009-12-28 17:34 ` William Morgan
2009-12-28 19:34 ` Tero Tilus
3 siblings, 0 replies; 18+ messages in thread
From: William Morgan @ 2009-12-28 17:34 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from Arvid Picciani's message of 2009-12-28:
> "everything" sums it up. it takes forever to start, to quit, to show
> the index, to show a single message, etc..
Are you using IMAP?
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 8:32 ` Arvid Picciani
` (2 preceding siblings ...)
2009-12-28 17:34 ` William Morgan
@ 2009-12-28 19:34 ` Tero Tilus
2009-12-28 19:57 ` Gaute Hope
2009-12-28 20:17 ` William Morgan
3 siblings, 2 replies; 18+ messages in thread
From: Tero Tilus @ 2009-12-28 19:34 UTC (permalink / raw)
To: sup-talk
Arvid Picciani, 2009-12-28 10:32:
> On 12/28/2009 04:12 PM, William Morgan wrote:
>> Can you be more specific about what's slow? I used Sup with a 160k
>> index for a long time without trouble.
>
> "everything" sums it up. it takes forever to start, to quit, to show
> the index, to show a single message, etc..
I've got 103k messages from 172 sources (137 imap sources, including
all sources receiving new mail). I run sup on 512 MB Bytemark virtual
machine. Startup is instantaneous, it takes 2-3 seconds to list first
screenfull of threads. Practically all searches are instantaneous (I
remember hitting "slow queries" a couple of times). Thread is
displayed in about messagecount/5 seconds. Pretty much nothing feels
slow. But there is that "pretty much nothing".
Quit is slow. If I "just check" mail (launch, first poll adds 10-15
messages, archive them and quit) sup exits in a couple of seconds
(just long enough that you start to wonder if you pressed Q or not)
but if sup has been running days quit takes easily minutes.
--
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [sup-talk] sup with 120K mail
2009-12-28 19:34 ` Tero Tilus
@ 2009-12-28 19:57 ` Gaute Hope
2009-12-28 20:17 ` William Morgan
1 sibling, 0 replies; 18+ messages in thread
From: Gaute Hope @ 2009-12-28 19:57 UTC (permalink / raw)
To: sup-talk
[-- Attachment #1: Type: text/plain, Size: 735 bytes --]
Excerpts from Tero Tilus's message of 2009-12-28 20:34:07 +0100:
> Quit is slow. If I "just check" mail (launch, first poll adds 10-15
> messages, archive them and quit) sup exits in a couple of seconds
> (just long enough that you start to wonder if you pressed Q or not)
> but if sup has been running days quit takes easily minutes.
Quit is slow for me too - but seldom above one minute, but I normally
don't quit (just suspending my machine) - so either my machine has
crashed or sup has crashed. So I hit $ occasionally to save the index,
otherwise I need to re-do everything I did manually since my last
'crash'. It would help if this is done when there is a sufficient amount
of stuff to save or on regular intervals.
- gaute
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 19:34 ` Tero Tilus
2009-12-28 19:57 ` Gaute Hope
@ 2009-12-28 20:17 ` William Morgan
2009-12-28 20:24 ` Tero Tilus
2009-12-28 20:24 ` Steve Goldman
1 sibling, 2 replies; 18+ messages in thread
From: William Morgan @ 2009-12-28 20:17 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from Tero Tilus's message of 2009-12-28:
> Quit is slow. If I "just check" mail (launch, first poll adds 10-15
> messages, archive them and quit) sup exits in a couple of seconds
> (just long enough that you start to wonder if you pressed Q or not)
> but if sup has been running days quit takes easily minutes.
Is this with Xapian?
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 20:17 ` William Morgan
@ 2009-12-28 20:24 ` Tero Tilus
2009-12-28 21:07 ` William Morgan
2009-12-28 20:24 ` Steve Goldman
1 sibling, 1 reply; 18+ messages in thread
From: Tero Tilus @ 2009-12-28 20:24 UTC (permalink / raw)
To: sup-talk
William Morgan, 2009-12-28 22:17:
> Is this with Xapian?
Goodness! Forgot the most important part...
Yes. Xapian and git next.
--
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 20:24 ` Tero Tilus
@ 2009-12-28 21:07 ` William Morgan
2009-12-29 23:33 ` William Morgan
0 siblings, 1 reply; 18+ messages in thread
From: William Morgan @ 2009-12-28 21:07 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from Tero Tilus's message of 2009-12-28:
> William Morgan, 2009-12-28 22:17:
> > Is this with Xapian?
>
> Goodness! Forgot the most important part...
>
> Yes. Xapian and git next.
Yeah, Xapian is the cause of that delay on quit. I think there was a
workaround posted at some point about an environment variable or
something...
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 21:07 ` William Morgan
@ 2009-12-29 23:33 ` William Morgan
2009-12-29 23:42 ` Rich Lane
0 siblings, 1 reply; 18+ messages in thread
From: William Morgan @ 2009-12-29 23:33 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from William Morgan's message of 2009-12-28:
> Reformatted excerpts from Tero Tilus's message of 2009-12-28:
> > William Morgan, 2009-12-28 22:17:
> > > Is this with Xapian?
> >
> > Goodness! Forgot the most important part...
> >
> > Yes. Xapian and git next.
>
> Yeah, Xapian is the cause of that delay on quit. I think there was a
> workaround posted at some point about an environment variable or
> something...
Ah yes, try setting XAPIAN_FLUSH_THRESHOLD=10 when running sup (but not
sup-sync!) and see if that reduces the delay on quit.
Unfortunately there's no way (that I know of) to force a Xapian flush
from within Ruby. Correction from Xapian experts welcome.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-29 23:33 ` William Morgan
@ 2009-12-29 23:42 ` Rich Lane
2009-12-29 23:54 ` William Morgan
0 siblings, 1 reply; 18+ messages in thread
From: Rich Lane @ 2009-12-29 23:42 UTC (permalink / raw)
To: William Morgan; +Cc: sup-talk
Excerpts from William Morgan's message of Tue Dec 29 18:33:46 -0500 2009:
> Reformatted excerpts from William Morgan's message of 2009-12-28:
> > Reformatted excerpts from Tero Tilus's message of 2009-12-28:
> > > William Morgan, 2009-12-28 22:17:
> > > > Is this with Xapian?
> > >
> > > Goodness! Forgot the most important part...
> > >
> > > Yes. Xapian and git next.
> >
> > Yeah, Xapian is the cause of that delay on quit. I think there was a
> > workaround posted at some point about an environment variable or
> > something...
>
> Ah yes, try setting XAPIAN_FLUSH_THRESHOLD=10 when running sup (but not
> sup-sync!) and see if that reduces the delay on quit.
>
> Unfortunately there's no way (that I know of) to force a Xapian flush
> from within Ruby. Correction from Xapian experts welcome.
Xapian::WritableDatabase#flush
This should really be called from XapianIndex#save_index, along with a
friendly message explaining what's going on.
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-29 23:42 ` Rich Lane
@ 2009-12-29 23:54 ` William Morgan
0 siblings, 0 replies; 18+ messages in thread
From: William Morgan @ 2009-12-29 23:54 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from Rich Lane's message of 2009-12-29:
> Xapian::WritableDatabase#flush
>
> This should really be called from XapianIndex#save_index, along with a
> friendly message explaining what's going on.
Calling it from #save_index will have the advantage of us being able to
present a message, but what about calling it when "$" is pressed, or
periodically in the background? Does it block search?
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 20:17 ` William Morgan
2009-12-28 20:24 ` Tero Tilus
@ 2009-12-28 20:24 ` Steve Goldman
2009-12-28 20:30 ` Edward Z. Yang
` (2 more replies)
1 sibling, 3 replies; 18+ messages in thread
From: Steve Goldman @ 2009-12-28 20:24 UTC (permalink / raw)
To: sup-talk
Excerpts from William Morgan's message of Mon Dec 28 15:17:39 -0500 2009:
> Reformatted excerpts from Tero Tilus's message of 2009-12-28:
> > Quit is slow. If I "just check" mail (launch, first poll adds 10-15
> > messages, archive them and quit) sup exits in a couple of seconds
> > (just long enough that you start to wonder if you pressed Q or not)
> > but if sup has been running days quit takes easily minutes.
>
> Is this with Xapian?
Is Xapian sup fully functional? Last I read, there were some critical
functions that were supported.
--
Steve Goldman
sgoldman@tower-research.com
T: 212.219.6014
F: 212.219.6007
Tower Research Capital, LLC
377 Broadway, 11th Fl.
New York, NY 10013
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [sup-talk] sup with 120K mail
2009-12-28 20:24 ` Steve Goldman
@ 2009-12-28 20:30 ` Edward Z. Yang
2009-12-28 20:59 ` Tero Tilus
2009-12-28 21:06 ` William Morgan
2 siblings, 0 replies; 18+ messages in thread
From: Edward Z. Yang @ 2009-12-28 20:30 UTC (permalink / raw)
To: Steve Goldman; +Cc: sup-talk
Excerpts from Steve Goldman's message of Mon Dec 28 15:24:18 -0500 2009:
> Is Xapian sup fully functional? Last I read, there were some critical
> functions that were supported.
I've been using Xapian sup for a few weeks now, and the primary difficulty
is that you have to specify from_email/from_to instead of just from. But
it's very snappy, I haven't noticed any other problems and I haven't had
any major index corruption (which is what prompted the switch in the first
place).
Cheers,
Edward
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 20:24 ` Steve Goldman
2009-12-28 20:30 ` Edward Z. Yang
@ 2009-12-28 20:59 ` Tero Tilus
2009-12-28 21:06 ` William Morgan
2 siblings, 0 replies; 18+ messages in thread
From: Tero Tilus @ 2009-12-28 20:59 UTC (permalink / raw)
To: sup-talk
Steve Goldman, 2009-12-28 22:24:
> Is Xapian sup fully functional?
I've been on Xapian since september. It has given me smooth and fully
functional ride. I'd recommed trying it out.
--
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [sup-talk] sup with 120K mail
2009-12-28 20:24 ` Steve Goldman
2009-12-28 20:30 ` Edward Z. Yang
2009-12-28 20:59 ` Tero Tilus
@ 2009-12-28 21:06 ` William Morgan
2 siblings, 0 replies; 18+ messages in thread
From: William Morgan @ 2009-12-28 21:06 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from Steve Goldman's message of 2009-12-28:
> Is Xapian sup fully functional? Last I read, there were some critical
> functions that were supported.
Functional enough that it's going to be the default in the next release
of Sup, and you're going to get yelled at for Ferret.
A few things don't work or are weird, but it could break everything and
still provide a better user experience than Ferret.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 18+ messages in thread