Subject: Re: [sup-devel] [PATCH 2/2] Use nanosecond resolution of mtime for generating the unique id for each message

From: Ben Walton <[email-masked]>
Date: Fri, 19 Mar 2010 16:05:06 -0400

Excerpts from Michael Stapelberg's message of Fri Mar 19 15:47:53 -0400 2010:

> There is: Rich mentioned that proper move/delete handling for
> maildirs also solves the problem. However that doesn’t seem to be
> planned for the very near future, so I posted my patch which fixes
> the problem right now (given that you use ruby 1.9 and a decent
> filesystem).

Ok.  I'd rather work with you towards implementing this handling than
force an upgrade on myself that would be both ruby and filesystem
(incl servers and nfs).

Thanks
-Ben
-- 
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302

_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Rich Lane <[email-masked]>
Date: Fri, 19 Mar 2010 17:17:50 -0400

Excerpts from Mark Alexander's message of 2010-03-19 16:35:38 -0400:
> > Excerpts from Ben Walton's message of 2010-03-19 20:35:31 +0100:
> > > Isn't there a more portable solution?
> > There is: Rich mentioned that proper move/delete handling for maildirs also
> > solves the problem.
> 
> I must have missed that message.  What is the proposed solution?
> How does it solve the problem of out-of-order maildir IDs?

Notmuch implements this correctly. They essentially keep a sorted list
of mail filenames in the index and then iterate over that and the sorted
list of the filenames that are currently in the maildir to detect
inserts, deletes, and renames. 

I just rebased and pushed a multiple-locations branch I've had sitting
around to mainline. It's probably buggy. It's been a while since I
thought about this problem, but I think keeping track of multiple
locations (source/source info) for a single message (identified by
message-id) should make the maildir logic simpler.
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Ben Walton <[email-masked]>
Date: Fri, 19 Mar 2010 15:35:31 -0400

Excerpts from Michael Stapelberg's message of Fri Mar 19 15:27:36 -0400 2010:

Hi Michael,

> Looks like you are right. The change won’t break on ext3, though, the
> nanoseconds will just be zero. Time to upgrade to ext4, I’d say ;-).

I like the idea here, but requiring ruby 1.9 _and_ a filesystem
upgrade for a _mail client_ is a bit of a stretch, don't you think?
Isn't there a more portable solution?

Thanks
-Ben

---

From: Michael Stapelberg <[email-masked]>
Date: Fri, 19 Mar 2010 16:36:37 +0100

Hi,

the attached patch uses nanosecond resolution to generate the unique id for
each message of a maildir. This is necessary because I have about 2000 messages
which have the same mtime and size. I am not sure on how to properly check for
ruby 1.9 and do the right thing. Also, the user probably needs to run sup-sync
after this change. Maybe it should be implemented as a config option and be
enabled by default on new configs by sup-config? Thoughts?

Best regards,
Michael

---

From: Michael Stapelberg <[email-masked]>
Date: Mon, 22 Mar 2010 12:30:15 +0100

Hi,

attached to this mail you can find an updated version of the patch. It now
uses a fixed length in sprintf for the nanoseconds so that the IDs will be
increasing.

Best regards,
Michael

---

From: Mark Alexander <[email-masked]>
Date: Fri, 19 Mar 2010 15:17:01 -0400

Excerpts from Michael Stapelberg's message of Fri Mar 19 11:36:37 -0400 2010:
> which have the same mtime and size.

This a great idea, but if I understand correctly, it requires a file
system that supports nanosecond resolution on timestamps, like ext4.
Those of us on ext3 are out of luck, apparently:

  https://ext4.wiki.kernel.org/index.php/Ext4_Howto#Inode-related_features
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Michael Stapelberg <[email-masked]>
Date: Fri, 19 Mar 2010 20:27:36 +0100

Hi Mark,

Best regards,
Michael

---

From: Michael Stapelberg <[email-masked]>
Date: Fri, 19 Mar 2010 20:47:53 +0100

Hi Ben,

There is: Rich mentioned that proper move/delete handling for maildirs also
solves the problem. However that doesn’t seem to be planned for the very near
future, so I posted my patch which fixes the problem right now (given that
you use ruby 1.9 and a decent filesystem).

Best regards,
Michael

---

From: Eric Sherman <[email-masked]>
Date: Fri, 19 Mar 2010 16:40:46 -0400

Excerpts from Michael Stapelberg's message of Fri Mar 19 15:46:29 -0400 2010:
> Excerpts from Eric Sherman's message of 2010-03-19 20:34:32 +0100:
> > Would inserting an SHA1 into the id solve these problems, or does it just 
> > introduce more problems?
> The latter. The ID needs to be linear, not only unique.

Oh I see.

Is it fair to say that if a set of messages are can only be disambiguated 
by adding nanosecond precision to their timestamps, the ordering across 
such a set may as well be arbitrary?

In looking at make_id, it seems like we should be able to simply append 
more numeric noise at the end of the id.  The file size is already appended 
to the mtime, potentially making the ordering arbitrary for equivalent 
mtimes.

Would appending x-digits of numeric noise, probably derived from SHA1, be 
suitable?
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Mark Alexander <[email-masked]>
Date: Fri, 19 Mar 2010 22:23:00 -0400

Excerpts from Mark Alexander's message of Fri Mar 19 17:04:11 -0400 2010:
> Excerpts from Eric Sherman's message of Fri Mar 19 16:40:46 -0400 2010:
> I was thinking the same thing.  For performance, perhaps only the SHA1
> of the file name, not the file contents, would be adequate.

Never mind.  This won't work because it violates the requirement
that IDs must always increase.
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Tero Tilus <[email-masked]>
Date: Thu, 25 Mar 2010 08:25:21 +0200

Ben Walton, 2010-03-22 18:25:
> So we're still needing ruby 1.9 and ext4 then?  That's a deal
> breaker for me.

I'm with Ben in this.  I'll need to be able to do without them both
for quite a while.

-- 
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Mark Alexander <[email-masked]>
Date: Fri, 19 Mar 2010 17:04:11 -0400

---

From: Mark Alexander <[email-masked]>
Date: Fri, 19 Mar 2010 16:35:38 -0400

---

From: Ben Walton <[email-masked]>
Date: Mon, 22 Mar 2010 12:25:47 -0400

Excerpts from Michael Stapelberg's message of Mon Mar 22 07:30:15 -0400 2010:

> attached to this mail you can find an updated version of the
> patch. It now uses a fixed length in sprintf for the nanoseconds so
> that the IDs will be increasing.

So we're still needing ruby 1.9 and ext4 then?  That's a deal breaker
for me.  1.9 I can live with, but there is no way my environment will
be using ext4 any time soon.

Thanks
-Ben

---

From: Eric Sherman <[email-masked]>
Date: Fri, 19 Mar 2010 15:34:32 -0400

---

From: Michael Stapelberg <[email-masked]>
Date: Fri, 19 Mar 2010 20:46:29 +0100

Hi Eric,

Best regards,
Michael

---

From: Rich Lane <[email-masked]>
Date: Fri, 19 Mar 2010 17:02:44 -0400

> force an upgrade on myself that would be both ruby and filesystem

I'd also much rather see proper maildir handling. My objections to the
current patch are Ruby 1.8 compatibility - not just the missing method,
but that you'd need to reindex whenever switching between Ruby 1.8 and
1.9.
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Rich Lane <[email-masked]>
Date: Mon, 22 Mar 2010 13:00:47 -0400

> uses a fixed length in sprintf for the nanoseconds so that the IDs will be

If you add a config option (defaulting to false) I'll accept the patch. 

I'm going to restart looking at implementing proper maildir support.
IIRC the tough part is making it fit within sup's existing source
interfaces. Part of my reason for nuking IMAP/mbox+ssh sources was to
make changes to the source API easier, so hopefully this will be more
tractable now.
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel