Re: [sup-devel] Query for largest msg_id?

From: Horacio Sanson <hsanson@gmail.com>
To: Sup developer discussion <sup-devel@rubyforge.org>
Subject: Re: [sup-devel] Query for largest msg_id?
Date: Tue, 17 May 2011 00:02:39 +0900	[thread overview]
Message-ID: <BANLkTin5eRZw44Je+==V=K4pyVw2T7S3+A@mail.gmail.com> (raw)
In-Reply-To: <1305471101-sup-6655@masanjin.net>

[-- Attachment #1: Type: text/plain, Size: 3224 bytes --]

On Mon, May 16, 2011 at 12:01 AM, William Morgan
<wmorgan-sup@masanjin.net> wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-05-10:
>> Is there a way to query Heliotrope what is the largest msg_id
>> currently in the index?
>
> Sort of---it's a hack, but if you search for e.g. "a OR -a" you'll get
> every message in the index, and the first result will be the message
> with the highest id thanks to Whistlepig's search semantics.
>

Indeed I have been trying to wrap my head around the IMAP spec and
still don't get a lot of things. For now I will just keep the max UID
read from IMAP server somewhere on disk.

>> I am trying to improve the imap-dumper.rb so
>> it does not download all my emails every time but only the new ones.
>
> Sounds great. Unfortunately the Heliotrope message id and the IMAP
> message id / message uid are completely different things, and
> maintaining a cross-session mapping of them is impossible for generic
> IMAP servers, because the uid of every message can change every time you
> connect to an IMAP server---see the section on IMAP's 'uidvalidity'
> variable. So you'll have to rescan the inbox every time and rebuild the
> mapping. Welcome to hell.
>

When UIDVALIDITY differs I will simply re-scan the whole mailbox and
feed it to Heliotrope. I trust Heliotrope won't add duplicates.

>> Also while looking at the code I see that messages are stored in the
>> index using the msg_id as parsed by RMail. There is no further
>> association with the source or mailbox from where the messages were
>> downloaded. This I think may cause collisions if we use one Heliotrope
>> server with more than one email account. Not sure what is the
>> probability of two messages from two different IMAP servers having the
>> same msg_id but nothing in the standard rules out that possibility.
>
> This is yet another id: the Message-Id header of the email. This is only
> needed to build up the thread structure and should otherwise be ignored.

I am attaching my first small hack for GMail <-> Heliotrope
synchronization. For now it only downloads mail from GMail and injects
them to Heliotrope just as the imap-dumper.rb does. The difference is
that I keep track of the last message UID and UIDVALIDITY values to
avoid re-scanning the whole folder every time.

Now I wan't to take advantage of GMail IMAP extensions (e.g.
X-GM-LABELS, X-GM-THRID) to allow labels/threads synchronization. But
have some doubts about how to correctly use the Heliotrope REST API.
For example in the Heliotrope::Index the add_message method allows to
insert a message and assign it labels, flags and extra parameters at
the same time. How can I do this with the REST API? The only example I
see only adds a message body.

    RestClient.post "http://localhost:8042/message", :message => body

Also for what purpose are the ext array used for? Can I use it to add
an account/mailbox property to each message so I can latter retrieve
all messages associated to a mailbox/account pair?

regards,
Horacio

> --
> William <wmorgan-sup@masanjin.net>
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>

[-- Attachment #2: gmail.rb --]
[-- Type: application/x-ruby, Size: 7797 bytes --]

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel