* [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.
@ 2008-01-18 8:58 Jeff Balogh
2008-01-22 3:03 ` William Morgan
0 siblings, 1 reply; 9+ messages in thread
From: Jeff Balogh @ 2008-01-18 8:58 UTC (permalink / raw)
The size needs to be <= 7 digits to preserve sup's increasing id
requirement. Otherwise, large messages (probably w/ attachments) have
an id that is a magnitude larger than small messages.
---
lib/sup/maildir.rb | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb
index ba9da00..7b85383 100644
--- a/lib/sup/maildir.rb
+++ b/lib/sup/maildir.rb
@@ -144,7 +144,7 @@ private
def make_id fn
# use 7 digits for the size. why 7? seems nice.
- sprintf("%d%07d", File.mtime(fn), File.size(fn)).to_i
+ sprintf("%d%07d", File.mtime(fn), File.size(fn) % 10000000).to_i
end
def with_file_for id
--
1.5.3.7
^ permalink raw reply [flat|nested] 9+ messages in thread* [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.
2008-01-18 8:58 [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits Jeff Balogh
@ 2008-01-22 3:03 ` William Morgan
2008-01-22 14:01 ` Jeff Balogh
2008-01-22 20:44 ` Marcus Williams
0 siblings, 2 replies; 9+ messages in thread
From: William Morgan @ 2008-01-22 3:03 UTC (permalink / raw)
Reformatted excerpts from its.jeff.balogh's message of 2008-01-18:
> The size needs to be <= 7 digits to preserve sup's increasing id
> requirement. Otherwise, large messages (probably w/ attachments) have
> an id that is a magnitude larger than small messages.
God, I was hoping no one would ever see this code. And I apparently
forgot how printf works when writing it.
Did this solve the problem you were having with Sup not noticing new
Maildir messages?
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.
2008-01-22 3:03 ` William Morgan
@ 2008-01-22 14:01 ` Jeff Balogh
2008-01-23 5:20 ` William Morgan
2008-01-22 20:44 ` Marcus Williams
1 sibling, 1 reply; 9+ messages in thread
From: Jeff Balogh @ 2008-01-22 14:01 UTC (permalink / raw)
William Morgan wrote:
> Reformatted excerpts from its.jeff.balogh's message of 2008-01-18:
> > The size needs to be <= 7 digits to preserve sup's increasing id
> > requirement. Otherwise, large messages (probably w/ attachments) have
> > an id that is a magnitude larger than small messages.
>
> God, I was hoping no one would ever see this code. And I apparently
> forgot how printf works when writing it.
>
> Did this solve the problem you were having with Sup not noticing new
> Maildir messages?
Yes, Sup works for me with Maildir.
-- jeff
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.
2008-01-22 3:03 ` William Morgan
2008-01-22 14:01 ` Jeff Balogh
@ 2008-01-22 20:44 ` Marcus Williams
2008-01-23 5:20 ` William Morgan
1 sibling, 1 reply; 9+ messages in thread
From: Marcus Williams @ 2008-01-22 20:44 UTC (permalink / raw)
On 22.1.2008, William Morgan wrote:
> God, I was hoping no one would ever see this code. And I apparently
> forgot how printf works when writing it.
:) Why do you use mtime + filesize rather than just the filename
(before the colon) as the id? Other than the flags on files (after the
colon) does this ever change?
Marcus
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.
2008-01-22 20:44 ` Marcus Williams
@ 2008-01-23 5:20 ` William Morgan
2008-01-23 9:59 ` Marcus Williams
0 siblings, 1 reply; 9+ messages in thread
From: William Morgan @ 2008-01-23 5:20 UTC (permalink / raw)
Reformatted excerpts from Marcus Williams's message of 2008-01-22:
> :) Why do you use mtime + filesize rather than just the filename
> (before the colon) as the id? Other than the flags on files (after the
> colon) does this ever change?
Exactly the type of question man was not meant to ask.
The short answer is: because maildir filenames, while unique, are not
integers, and Sup needs integers.
The long answer is: in order to have a mailstore-format-indepedent way
of keeping track of messages that haven't been added to the index (as
opposed to messages which the mailstore thinks are unread, which is a
different question entirely), Sup uses message identifiers that are
unique, increasing, but not necessarily continguous, integers. Then Sup
can keep track of the last id that was added to the index for each
source, and polling consists of simply adding messages for which the id
is greater than the last id that Sup added from that source.
The other way to do this would be to have each source type maintain its
own logic for returning which messages hadn't been added to the index.
That could have been implemented by either pawing through the index upon
startup for each message (unacceptably slow for large mailstores), or by
maintaining a separate listing of added messages (probably very large
and seems a bit silly).
The terrible part about this solution, of course, is that Sup can't
uniquely represent two messages that have the same mtime and the same
size. One day I'm sure this will cause pain.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.
2008-01-23 5:20 ` William Morgan
@ 2008-01-23 9:59 ` Marcus Williams
2008-01-25 5:14 ` William Morgan
0 siblings, 1 reply; 9+ messages in thread
From: Marcus Williams @ 2008-01-23 9:59 UTC (permalink / raw)
On 23.1.2008, William Morgan wrote:
> Reformatted excerpts from Marcus Williams's message of 2008-01-22:
> > :) Why do you use mtime + filesize rather than just the filename
> > (before the colon) as the id? Other than the flags on files (after the
> > colon) does this ever change?
>
> Exactly the type of question man was not meant to ask.
:)
> The terrible part about this solution, of course, is that Sup can't
> uniquely represent two messages that have the same mtime and the same
> size. One day I'm sure this will cause pain.
If you store the id in the ferret index, is it quick enough just to
use ferret to tell you if its there? That way an id could be a string
and could be tracked by the source via ferret (a source would just
have to guarantee its unique when combined with the source id).
Off on a slight tangent - I cant get any maildir working with sup
here. I added a maildir to my sources list and it gets scanned as
expected but nothing gets into the index. I'm wondering whether this
is as a result of the id function.
Marcus
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.
2008-01-23 9:59 ` Marcus Williams
@ 2008-01-25 5:14 ` William Morgan
2008-01-27 20:07 ` Marcus Williams
0 siblings, 1 reply; 9+ messages in thread
From: William Morgan @ 2008-01-25 5:14 UTC (permalink / raw)
Reformatted excerpts from Marcus Williams's message of 2008-01-23:
> If you store the id in the ferret index, is it quick enough just to
> use ferret to tell you if its there? That way an id could be a string
> and could be tracked by the source via ferret (a source would just
> have to guarantee its unique when combined with the source id).
I don't think so. I haven't tested this empirically, and Ferret is
pretty darn fast, but it comes down to this: if we have to perform an
operation a number of times that scales linearly with the number of
messages in a mailstore (which unfortunately we do), that operation
shouldn't be something like "do a search on a search engine". In fact,
if the make_id call /is/ actually slower than a Ferret search, and
that's the bottleneck, we can rewrite that with RubyInline or something
and there's no way Ferret will be faster than that.
> Off on a slight tangent - I cant get any maildir working with sup
> here. I added a maildir to my sources list and it gets scanned as
> expected but nothing gets into the index. I'm wondering whether this
> is as a result of the id function.
Is this with Jeff's fix for the id?
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.
2008-01-25 5:14 ` William Morgan
@ 2008-01-27 20:07 ` Marcus Williams
0 siblings, 0 replies; 9+ messages in thread
From: Marcus Williams @ 2008-01-27 20:07 UTC (permalink / raw)
On 25.1.2008, William Morgan wrote:
> I don't think so. I haven't tested this empirically, and Ferret is
> pretty darn fast, but it comes down to this: if we have to perform an
> operation a number of times that scales linearly with the number of
> messages in a mailstore (which unfortunately we do), that operation
> shouldn't be something like "do a search on a search engine".
Yeah I was forgetting that things like imap have to regenerate them
for the whole source. Would be nice if that could be solved somehow,
that way making an id would be a one time thing.
> > Off on a slight tangent - I cant get any maildir working with sup
> > here. I added a maildir to my sources list and it gets scanned as
> > expected but nothing gets into the index. I'm wondering whether this
> > is as a result of the id function.
>
> Is this with Jeff's fix for the id?
Dont know, will follow up with a new thread when I get back to testing
it again.
Marcus
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2008-01-27 20:07 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-01-18 8:58 [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits Jeff Balogh
2008-01-22 3:03 ` William Morgan
2008-01-22 14:01 ` Jeff Balogh
2008-01-23 5:20 ` William Morgan
2008-01-22 20:44 ` Marcus Williams
2008-01-23 5:20 ` William Morgan
2008-01-23 9:59 ` Marcus Williams
2008-01-25 5:14 ` William Morgan
2008-01-27 20:07 ` Marcus Williams
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox