* [sup-talk] [PATCH] maildir speedups @ 2008-04-30 18:50 Ben Walton 2008-05-20 3:29 ` William Morgan 0 siblings, 1 reply; 11+ messages in thread From: Ben Walton @ 2008-04-30 18:50 UTC (permalink / raw) Hi All, The attached small patch improves maildir performance by making the assumption that nothing will be modifying the underlying files themselves. It uses the mtimes of cur/ and new/ to mark whether or not to poll files in the directory. This sees the initial poll be a somewhat slow process (nfs for me again), but subsequent polls are much faster. I'm caching the mtimes in sources.yaml too, but currently this doesn't save anything across loads of sup since the id's (and id -> fn map) are not cached across restarts. I hope you find this useful. -Ben -- --------------------------------------------------------------------------------------------------------------------------- Ben Walton <bdwalton at gmail.com> When one person suffers from a delusion, it is called insanity. When many people suffer from a delusion it is called Religion. Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance --------------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-maildir-speed-improvements.patch Type: text/x-patch Size: 4256 bytes Desc: not available URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080430/1ba57542/attachment-0001.bin> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-04-30 18:50 [sup-talk] [PATCH] maildir speedups Ben Walton @ 2008-05-20 3:29 ` William Morgan [not found] ` <f96e0240805200924u1c991392oe36de853e208bab8@mail.gmail.com> 0 siblings, 1 reply; 11+ messages in thread From: William Morgan @ 2008-05-20 3:29 UTC (permalink / raw) Reformatted excerpts from Ben Walton's message of 2008-04-30: > The attached small patch improves maildir performance by making the > assumption that nothing will be modifying the underlying files > themselves. It uses the mtimes of cur/ and new/ to mark whether or > not to poll files in the directory. This sounds great. My only comment is that the Logger::log stuff should probably be Redwood::log. If you make that change I'm happy to apply. -- William <wmorgan-sup at masanjin.net> ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <f96e0240805200924u1c991392oe36de853e208bab8@mail.gmail.com>]
* [sup-talk] [PATCH] maildir speedups [not found] ` <f96e0240805200924u1c991392oe36de853e208bab8@mail.gmail.com> @ 2008-05-21 1:08 ` Ben Walton 2008-05-25 2:34 ` William Morgan 0 siblings, 1 reply; 11+ messages in thread From: Ben Walton @ 2008-05-21 1:08 UTC (permalink / raw) Ok, the attached patch should address the use of Logger::log vs Redwood::log. Additionally, I changed the names of the cached mtimes and corrected a bug that saw new sources not work correctly. I hope you find this acceptable (and useful). -Ben -- --------------------------------------------------------------------------------------------------------------------------- Ben Walton <bdwalton at gmail.com> When one person suffers from a delusion, it is called insanity. When many people suffer from a delusion it is called Religion. Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance --------------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-maildir-speedups.patch Type: text/x-diff Size: 4549 bytes Desc: not available URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080520/648ed94d/attachment.bin> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-05-21 1:08 ` Ben Walton @ 2008-05-25 2:34 ` William Morgan 2008-05-29 2:49 ` Grant Hollingworth 0 siblings, 1 reply; 11+ messages in thread From: William Morgan @ 2008-05-25 2:34 UTC (permalink / raw) Published on 'maildir-speedups' and merged into next. Thanks! -- William <wmorgan-sup at masanjin.net> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-05-25 2:34 ` William Morgan @ 2008-05-29 2:49 ` Grant Hollingworth 2008-05-29 3:05 ` Grant Hollingworth 2008-05-29 13:34 ` Ben Walton 0 siblings, 2 replies; 11+ messages in thread From: Grant Hollingworth @ 2008-05-29 2:49 UTC (permalink / raw) * William Morgan [2008-05-24 22:34 -0400]: > Published on 'maildir-speedups' and merged into next. Thanks! Sup had my CPU working overtime after I updated. I ran ruby-prof and found that the line @ids_to_fns.delete_if { |k, v| !@ids.include?(k) } in Maildir#scan_mailbox was the culprit. Those id lists are pretty long and include? means comparing each id (a Bignum) in @ids_to_fns with every id in @ids. A faster method is @ids_to_fns = @ids.inject({}) do |hash, i| hash[i] = @ids_to_fns[i] hash end Or (less pretty but faster and probably clearer) new_ids_to_fns = {} @ids.each {|i| new_ids_to_fns[i] = @ids_to_fns[i] } @ids_to_fns = new_ids_to_fns But I guess the real question is whether the line is even needed. There probably won't be a big difference between @ids_to_fns.keys and @ids, so why not leave some extra values in the hash? ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-05-29 2:49 ` Grant Hollingworth @ 2008-05-29 3:05 ` Grant Hollingworth 2008-05-29 3:06 ` Grant Hollingworth 2008-05-29 13:34 ` Ben Walton 1 sibling, 1 reply; 11+ messages in thread From: Grant Hollingworth @ 2008-05-29 3:05 UTC (permalink / raw) * Grant Hollingworth [2008-05-28 22:49 -0400]: > Or (less pretty but faster and probably clearer) > [...] Best: @ids_to_fns = @ids_to_fns.values_at(@ids) Next time I should check the docs first.... ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-05-29 3:05 ` Grant Hollingworth @ 2008-05-29 3:06 ` Grant Hollingworth 0 siblings, 0 replies; 11+ messages in thread From: Grant Hollingworth @ 2008-05-29 3:06 UTC (permalink / raw) * Grant Hollingworth [2008-05-28 23:05 -0400]: > Best: > > @ids_to_fns = @ids_to_fns.values_at(@ids) > > Next time I should check the docs first.... Ugh. Never mind. Next time I should get some sleep before posting crap to the list. ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-05-29 2:49 ` Grant Hollingworth 2008-05-29 3:05 ` Grant Hollingworth @ 2008-05-29 13:34 ` Ben Walton 2008-05-30 16:26 ` William Morgan 1 sibling, 1 reply; 11+ messages in thread From: Ben Walton @ 2008-05-29 13:34 UTC (permalink / raw) On Wed, May 28, 2008 at 10:49 PM, Grant Hollingworth <grant at antiflux.org> wrote: > * William Morgan [2008-05-24 22:34 -0400]: >> Published on 'maildir-speedups' and merged into next. Thanks! > > Sup had my CPU working overtime after I updated. I ran ruby-prof and found > that the line > > @ids_to_fns.delete_if { |k, v| !@ids.include?(k) } > > in Maildir#scan_mailbox was the culprit. > > Those id lists are pretty long and include? means comparing each id (a Bignum) > in @ids_to_fns with every id in @ids. > > A faster method is > > @ids_to_fns = @ids.inject({}) do |hash, i| > hash[i] = @ids_to_fns[i] > hash > end > > Or (less pretty but faster and probably clearer) > > new_ids_to_fns = {} > @ids.each {|i| new_ids_to_fns[i] = @ids_to_fns[i] } > @ids_to_fns = new_ids_to_fns > > But I guess the real question is whether the line is even needed. There > probably won't be a big difference between @ids_to_fns.keys and @ids, so why > not leave some extra values in the hash? In hindsight, that seems rather obvious, although I've not noticed the cpu consumption on my box...Either of the above to fixes would work. I had a reason for explicitly purging old records from the mapping when I created this, but it escapes me now...dropping that line may be acceptable unless there is a reason that the keys in @ids_to_fns should be a 1:1 mapping to values in @ids. Really, there shouldn't be 'extra' keys in @ids_to_fns unless a message disappears from the Maildir anyway, which should cause an OutOfSync exception, no? Thanks -Ben -- --------------------------------------------------------------------------------------------------------------------------- Ben Walton <bdwalton at gmail.com> When one person suffers from a delusion, it is called insanity. When many people suffer from a delusion it is called Religion. Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance --------------------------------------------------------------------------------------------------------------------------- ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-05-29 13:34 ` Ben Walton @ 2008-05-30 16:26 ` William Morgan 2008-06-04 0:39 ` Ben Walton 0 siblings, 1 reply; 11+ messages in thread From: William Morgan @ 2008-05-30 16:26 UTC (permalink / raw) Reformatted excerpts from Ben Walton's message of 2008-05-29: > Really, there shouldn't be 'extra' keys in @ids_to_fns unless a > message disappears from the Maildir anyway, which should cause an > OutOfSync exception, no? I believe this is the case. I'm happy to take a patch when you guys (who actually use Maildir!) are ready. :) -- William <wmorgan-sup at masanjin.net> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-05-30 16:26 ` William Morgan @ 2008-06-04 0:39 ` Ben Walton 2008-06-04 2:49 ` William Morgan 0 siblings, 1 reply; 11+ messages in thread From: Ben Walton @ 2008-06-04 0:39 UTC (permalink / raw) Ok, I took Grant's suggestion and just dropped the cleanup of the id -> filename mapping hash. -Ben -- --------------------------------------------------------------------------------------------------------------------------- Ben Walton <bdwalton at gmail.com> When one person suffers from a delusion, it is called insanity. When many people suffer from a delusion it is called Religion. Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance --------------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-maildir-cpu-usage-regression-fix.patch Type: text/x-diff Size: 1048 bytes Desc: not available URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20080603/adff285d/attachment-0001.bin> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [sup-talk] [PATCH] maildir speedups 2008-06-04 0:39 ` Ben Walton @ 2008-06-04 2:49 ` William Morgan 0 siblings, 0 replies; 11+ messages in thread From: William Morgan @ 2008-06-04 2:49 UTC (permalink / raw) Reformatted excerpts from Ben Walton's message of 2008-06-03: > Ok, I took Grant's suggestion and just dropped the cleanup of the id > -> filename mapping hash. Merged right in. Thanks! -- William <wmorgan-sup at masanjin.net> ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2008-06-04 2:49 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-04-30 18:50 [sup-talk] [PATCH] maildir speedups Ben Walton
2008-05-20 3:29 ` William Morgan
[not found] ` <f96e0240805200924u1c991392oe36de853e208bab8@mail.gmail.com>
2008-05-21 1:08 ` Ben Walton
2008-05-25 2:34 ` William Morgan
2008-05-29 2:49 ` Grant Hollingworth
2008-05-29 3:05 ` Grant Hollingworth
2008-05-29 3:06 ` Grant Hollingworth
2008-05-29 13:34 ` Ben Walton
2008-05-30 16:26 ` William Morgan
2008-06-04 0:39 ` Ben Walton
2008-06-04 2:49 ` William Morgan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox