* [sup-talk] Slow import of messages
@ 2009-07-13 9:15 Nico Schottelius
2009-07-13 9:26 ` Michael Stapelberg
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Nico Schottelius @ 2009-07-13 9:15 UTC (permalink / raw)
Hello!
I'm looking at sup as a replacement for mutt.
Following the idea of sup, I took a snapshot of
my Mails from 2008-2009 (~56k mails) and run
sup-sync --all-sources
Now the problem is that sup gets about 3 mails per
second into its index. That means it takes about
56k/3 ~= 15k; 15000/60 = 250 minutes
That import is already from a local Maildir (synced with
offlineimap).
For me, this is incredibly slow. Using mutt it's way faster,
not even comparing the fact the speed improvement you get
when dividing into folders.
What's your impression? Am I totally wrong or is sup just
the wrong tool for handling a lot of messages?
Sincerly,
Nico
^ permalink raw reply [flat|nested] 4+ messages in thread* [sup-talk] Slow import of messages
2009-07-13 9:15 [sup-talk] Slow import of messages Nico Schottelius
@ 2009-07-13 9:26 ` Michael Stapelberg
2009-07-13 21:22 ` Jim Cheetham
2009-07-28 19:10 ` William Morgan
2 siblings, 0 replies; 4+ messages in thread
From: Michael Stapelberg @ 2009-07-13 9:26 UTC (permalink / raw)
Hi Nico,
Excerpts from Nico Schottelius's message of Mo Jul 13 11:15:16 +0200 2009:
> I'm looking at sup as a replacement for mutt.
> Following the idea of sup, I took a snapshot of
> my Mails from 2008-2009 (~56k mails) and run
I have about 33k emails on my IMAP server in my LAN.
> Now the problem is that sup gets about 3 mails per
> second into its index. That means it takes about
I got around 10 MB/s performance using sup-sync. Maybe
your mails are all very large, I don?t know. Maybe maildir is
just a lot slower than IMAP in sup-sync.
> What's your impression? Am I totally wrong or is sup just
> the wrong tool for handling a lot of messages?
My impression is that sup-sync can definitely be improved regarding
its speed. I?m not sure if it is actually doable or if it is some ruby
library (Ferret?) which is the limiting factor here. However, it was
not as bad for me as it is for you ;-).
Best regards,
Michael
^ permalink raw reply [flat|nested] 4+ messages in thread
* [sup-talk] Slow import of messages
2009-07-13 9:15 [sup-talk] Slow import of messages Nico Schottelius
2009-07-13 9:26 ` Michael Stapelberg
@ 2009-07-13 21:22 ` Jim Cheetham
2009-07-28 19:10 ` William Morgan
2 siblings, 0 replies; 4+ messages in thread
From: Jim Cheetham @ 2009-07-13 21:22 UTC (permalink / raw)
On Mon, Jul 13, 2009 at 9:15 PM, Nico
Schottelius<nico-sup at schottelius.org> wrote:
> What's your impression? Am I totally wrong or is sup just
> the wrong tool for handling a lot of messages?
No, it's just slow to index for the first time. Unless you receive
1000s of messages per hour, you should be OK ...
-jim
^ permalink raw reply [flat|nested] 4+ messages in thread
* [sup-talk] Slow import of messages
2009-07-13 9:15 [sup-talk] Slow import of messages Nico Schottelius
2009-07-13 9:26 ` Michael Stapelberg
2009-07-13 21:22 ` Jim Cheetham
@ 2009-07-28 19:10 ` William Morgan
2 siblings, 0 replies; 4+ messages in thread
From: William Morgan @ 2009-07-28 19:10 UTC (permalink / raw)
Hi Nico,
Reformatted excerpts from Nico Schottelius's message of 2009-07-13:
> Now the problem is that sup gets about 3 mails per second into its
> index.
As others have pointed out, messages only have to be indexed once, so
this isn't ultimately that big of a deal. But 3m/s seems very slow for a
Maildir source.
With my mbox on my reasonably-fast laptop, I start with around 50m/s
using Ferret, and at 80m/s using Xapian. (These numbers degrade slightly
as the index gets fuller.) I would expect similar numbers from Maildir.
So something weird is happening for you. Do you have a notion of whether
it's disk or CPU that's the bottleneck?
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2009-07-28 19:10 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-07-13 9:15 [sup-talk] Slow import of messages Nico Schottelius
2009-07-13 9:26 ` Michael Stapelberg
2009-07-13 21:22 ` Jim Cheetham
2009-07-28 19:10 ` William Morgan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox