Subject: [sup-talk] [PATCH] modulo the file size in a maildir, so it's <= 7 digits.

From: [email-masked] (Jeff Balogh)
Date: Fri, 18 Jan 2008 03:58:33 -0500

The size needs to be <= 7 digits to preserve sup's increasing id
requirement.  Otherwise, large messages (probably w/ attachments) have
an id that is a magnitude larger than small messages.
---
 lib/sup/maildir.rb |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb
index ba9da00..7b85383 100644
--- a/lib/sup/maildir.rb
+++ b/lib/sup/maildir.rb
@@ -144,7 +144,7 @@ private
 
   def make_id fn
     # use 7 digits for the size. why 7? seems nice.
-    sprintf("%d%07d", File.mtime(fn), File.size(fn)).to_i
+    sprintf("%d%07d", File.mtime(fn), File.size(fn) % 10000000).to_i
   end
 
   def with_file_for id
-- 
1.5.3.7

---

From: [email-masked] (William Morgan)
Date: Mon, 21 Jan 2008 19:03:38 -0800

> an id that is a magnitude larger than small messages.

God, I was hoping no one would ever see this code. And I apparently
forgot how printf works when writing it.

Did this solve the problem you were having with Sup not noticing new
Maildir messages?

-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Jeff Balogh)
Date: Tue, 22 Jan 2008 09:01:37 -0500

> Did this solve the problem you were having with Sup not noticing new

Yes, Sup works for me with Maildir.

-- jeff

---

From: [email-masked] (Marcus Williams)
Date: Tue, 22 Jan 2008 20:44:47 +0000

> forgot how printf works when writing it.

:) Why do you use mtime + filesize rather than just the filename
(before the colon) as the id? Other than the flags on files (after the
colon) does this ever change?

Marcus

---

From: [email-masked] (William Morgan)
Date: Tue, 22 Jan 2008 21:20:16 -0800

> colon) does this ever change?

Exactly the type of question man was not meant to ask.

The short answer is: because maildir filenames, while unique, are not
integers, and Sup needs integers.

The long answer is: in order to have a mailstore-format-indepedent way
of keeping track of messages that haven't been added to the index (as
opposed to messages which the mailstore thinks are unread, which is a
different question entirely), Sup uses message identifiers that are
unique, increasing, but not necessarily continguous, integers. Then Sup
can keep track of the last id that was added to the index for each
source, and polling consists of simply adding messages for which the id
is greater than the last id that Sup added from that source.

The other way to do this would be to have each source type maintain its
own logic for returning which messages hadn't been added to the index.
That could have been implemented by either pawing through the index upon
startup for each message (unacceptably slow for large mailstores), or by
maintaining a separate listing of added messages (probably very large
and seems a bit silly).

The terrible part about this solution, of course, is that Sup can't
uniquely represent two messages that have the same mtime and the same
size. One day I'm sure this will cause pain.

---

From: [email-masked] (William Morgan)
Date: Tue, 22 Jan 2008 21:20:38 -0800

> Yes, Sup works for me with Maildir.

Ok, this patch is in both master and next.

---

From: [email-masked] (Marcus Williams)
Date: Wed, 23 Jan 2008 09:59:54 +0000

> Exactly the type of question man was not meant to ask.

:)

> size. One day I'm sure this will cause pain.

If you store the id in the ferret index, is it quick enough just to
use ferret to tell you if its there? That way an id could be a string
and could be tracked by the source via ferret (a source would just
have to guarantee its unique when combined with the source id).

Off on a slight tangent - I cant get any maildir working with sup
here. I added a maildir to my sources list and it gets scanned as
expected but nothing gets into the index. I'm wondering whether this
is as a result of the id function.

Marcus

---

From: [email-masked] (William Morgan)
Date: Thu, 24 Jan 2008 21:14:01 -0800

> have to guarantee its unique when combined with the source id).

I don't think so. I haven't tested this empirically, and Ferret is
pretty darn fast, but it comes down to this: if we have to perform an
operation a number of times that scales linearly with the number of
messages in a mailstore (which unfortunately we do), that operation
shouldn't be something like "do a search on a search engine". In fact,
if the make_id call /is/ actually slower than a Ferret search, and
that's the bottleneck, we can rewrite that with RubyInline or something
and there's no way Ferret will be faster than that.

> is as a result of the id function.

Is this with Jeff's fix for the id?

---

From: [email-masked] (Marcus Williams)
Date: Sun, 27 Jan 2008 20:07:31 +0000

On 25.1.2008, William Morgan wrote:
> shouldn't be something like "do a search on a search engine". 

Yeah I was forgetting that things like imap have to regenerate them
for the whole source. Would be nice if that could be solved somehow,
that way making an id would be a one time thing.

> Is this with Jeff's fix for the id?

Dont know, will follow up with a new thread when I get back to testing
it again.

Marcus