From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 10.142.241.17 with SMTP id o17cs15327wfh; Tue, 17 May 2011 08:16:11 -0700 (PDT) Received: by 10.224.176.200 with SMTP id bf8mr561067qab.181.1305645370165; Tue, 17 May 2011 08:16:10 -0700 (PDT) Return-Path: Received: from rubyforge.org (rubyforge.org [205.234.109.19]) by mx.google.com with ESMTP id r38si1117639qcs.103.2011.05.17.08.16.09; Tue, 17 May 2011 08:16:10 -0700 (PDT) Received-SPF: pass (google.com: domain of sup-devel-bounces@rubyforge.org designates 205.234.109.19 as permitted sender) client-ip=205.234.109.19; Authentication-Results: mx.google.com; spf=pass (google.com: domain of sup-devel-bounces@rubyforge.org designates 205.234.109.19 as permitted sender) smtp.mail=sup-devel-bounces@rubyforge.org; dkim=neutral (body hash did not verify) header.i=@gmail.com Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 53729185838A for ; Tue, 17 May 2011 11:16:09 -0400 (EDT) Received: from mail-vw0-f50.google.com (mail-vw0-f50.google.com [209.85.212.50]) by rubyforge.org (Postfix) with ESMTP id EF422185838A for ; Tue, 17 May 2011 11:15:49 -0400 (EDT) Received: by vws14 with SMTP id 14so561657vws.23 for ; Tue, 17 May 2011 08:15:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=87sMkTqZEVAwNx6S26kdZ4Q6WbR+gO00r8UBtkb0iZc=; b=CsKhkbYYZBnrjd3Zttr2F7/NtaMm+gFE8Dr+BXWho+3s+2ydkgPmuocmLSRIslqWvl gL/SXJolremzlJ9Wb9wNdYkv9VD1iDpUohC7pmZhi/hIoUbcepOnoL7a7XzCwAZgzEJA DPA43BfBhDbUxzEgM+5lDjWL/mnRwvac6sJw8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=MDxhwPBztGPmOeHufvMTkCPyZ22GhGBZkVh2OuA3WDl0VcchAaoqNmR+bCOqm2xL6M cXEd75+frXp/lphe+bMF20gMiHXy3XhjT8VmtkYTV2RLyKbJeUIKInLzMOFuaYt/FP4D VHgRix2U6JIQk0+6q3ZH79k9n932CJYGrbw8A= MIME-Version: 1.0 Received: by 10.52.93.166 with SMTP id cv6mr1045645vdb.121.1305645348946; Tue, 17 May 2011 08:15:48 -0700 (PDT) Received: by 10.52.112.100 with HTTP; Tue, 17 May 2011 08:15:48 -0700 (PDT) In-Reply-To: References: <1305471101-sup-6655@masanjin.net> Date: Wed, 18 May 2011 00:15:48 +0900 Message-ID: From: Horacio Sanson To: Sup developer discussion Content-Type: multipart/mixed; boundary=bcaec50161513a027f04a37a3f28 Subject: Re: [sup-devel] Query for largest msg_id? X-BeenThere: sup-devel@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Sup developer discussion List-Id: Sup developer discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: sup-devel-bounces@rubyforge.org Errors-To: sup-devel-bounces@rubyforge.org --bcaec50161513a027f04a37a3f28 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Sorry the patch I sent has a small problem. Use this patch instead. On Tue, May 17, 2011 at 11:59 PM, Horacio Sanson wrote: > I implemented a new version of the GMail -> Heliotrope sync script and > attach it here in hopes > someone will test it and provide some feedback/comments. This is by no > means ready for general > use so don't use for anything else than experimentation. But since > this script only opens mailboxes > in read only mode (examine) there should not be any problems with you > emails on the server. > > This script is GMail specific and has these features: > > =A0- Downloads emails from all mailboxes (except All, Trash and Spam) > automatically using > =A0 =A0the XLIST GMail IMAP extension and feeds them to Heliotrope via > REST interface. > =A0- Remembers the last email downloaded so it does not start from the be= ginning > =A0 =A0every time. > =A0- Synchronizes GMail labels using the X-GM-LABELS IMAP extension. > =A0- Synchronizes GMail flags with Heliotrope state flags. > =A0- Adds a new mailbox property to messages. This may allow later to imp= lement > =A0 =A0Heliotrope -> GMail synchronization. > > Things to check and do: > =A0- I am seeing some negative thread_id's in the response. Need to check= if > =A0 =A0this is normal or a bug in Heliotrope or my script. > =A0- GMail inbox is with a capital "I" (e.g. Inbox) while heliotrope > uses a small "i". > =A0 =A0Shall I down case all labels? or make a special treatment for Inbo= x? > =A0- Refactor the script into something more modular and elegant. > > To use this script I had to modify heliotrope-server.rb to allow > setting labels and states when > posting new messages (see attached patch). > > regards, > Horacio > > On Tue, May 17, 2011 at 12:02 AM, Horacio Sanson wrot= e: >> On Mon, May 16, 2011 at 12:01 AM, William Morgan >> wrote: >>> Reformatted excerpts from Horacio Sanson's message of 2011-05-10: >>>> Is there a way to query Heliotrope what is the largest msg_id >>>> currently in the index? >>> >>> Sort of---it's a hack, but if you search for e.g. "a OR -a" you'll get >>> every message in the index, and the first result will be the message >>> with the highest id thanks to Whistlepig's search semantics. >>> >> >> Indeed I have been trying to wrap my head around the IMAP spec and >> still don't get a lot of things. For now I will just keep the max UID >> read from IMAP server somewhere on disk. >> >>>> I am trying to improve the imap-dumper.rb so >>>> it does not download all my emails every time but only the new ones. >>> >>> Sounds great. Unfortunately the Heliotrope message id and the IMAP >>> message id / message uid are completely different things, and >>> maintaining a cross-session mapping of them is impossible for generic >>> IMAP servers, because the uid of every message can change every time yo= u >>> connect to an IMAP server---see the section on IMAP's 'uidvalidity' >>> variable. So you'll have to rescan the inbox every time and rebuild the >>> mapping. Welcome to hell. >>> >> >> When UIDVALIDITY differs I will simply re-scan the whole mailbox and >> feed it to Heliotrope. I trust Heliotrope won't add duplicates. >> >>>> Also while looking at the code I see that messages are stored in the >>>> index using the msg_id as parsed by RMail. There is no further >>>> association with the source or mailbox from where the messages were >>>> downloaded. This I think may cause collisions if we use one Heliotrope >>>> server with more than one email account. Not sure what is the >>>> probability of two messages from two different IMAP servers having the >>>> same msg_id but nothing in the standard rules out that possibility. >>> >>> This is yet another id: the Message-Id header of the email. This is onl= y >>> needed to build up the thread structure and should otherwise be ignored= . >> >> I am attaching my first small hack for GMail <-> Heliotrope >> synchronization. For now it only downloads mail from GMail and injects >> them to Heliotrope just as the imap-dumper.rb does. The difference is >> that I keep track of the last message UID and UIDVALIDITY values to >> avoid re-scanning the whole folder every time. >> >> Now I wan't to take advantage of GMail IMAP extensions (e.g. >> X-GM-LABELS, X-GM-THRID) to allow labels/threads synchronization. But >> have some doubts about how to correctly use the Heliotrope REST API. >> For example in the Heliotrope::Index the add_message method allows to >> insert a message and assign it labels, flags and extra parameters at >> the same time. How can I do this with the REST API? The only example I >> see only adds a message body. >> >> =A0 =A0RestClient.post "http://localhost:8042/message", :message =3D> bo= dy >> >> Also for what purpose are the ext array used for? Can I use it to add >> an account/mailbox property to each message so I can latter retrieve >> all messages associated to a mailbox/account pair? >> >> regards, >> Horacio >> >>> -- >>> William >>> _______________________________________________ >>> Sup-devel mailing list >>> Sup-devel@rubyforge.org >>> http://rubyforge.org/mailman/listinfo/sup-devel >>> >> > --bcaec50161513a027f04a37a3f28 Content-Type: text/x-patch; charset=US-ASCII; name="0001-Implement-post-message.json.patch" Content-Disposition: attachment; filename="0001-Implement-post-message.json.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gnsziu5b2 RnJvbSAyNjViMjVkYmM0MGIyMGU1NDE2M2FjMzY4ODEyNWM2ZGNjMTI4NjdjIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBIb3JhY2lvIFNhbnNvbiA8aHNhbnNvbkBnbWFpbC5jb20+CkRh dGU6IFR1ZSwgMTcgTWF5IDIwMTEgMjM6MzQ6MTQgKzA5MDAKU3ViamVjdDogW1BBVENIXSBJbXBs ZW1lbnQgcG9zdCAvbWVzc2FnZS5qc29uLgoKTm93IHdlIGNhbiAgYWRkIG1lc3NhZ2VzIHRvIGhl bGlvdHJvcGUgd2l0aCBsYWJlbHMsIHN0YXRlIGFuZCBtYWlsYm94CmluZm9ybWF0aW9uIGluIGEg c2luZ2xlIFBPU1QgcmVxdWVzdC4gVGhlIHJlcXVlc3QgYm9keSBoYXMgdG8gYmUgSlNPTgp3aXRo IGEgZm9ybWF0IGxpa2U6Cgp7CiAgYm9keTogPHJmYzgyMiByYXcgbWVzc2FnZSBzdHJpbmc+LAog IGxhYmVsczogWyJpbmJveCIsICJzY2hvb2wiLCAiaG9tZSJdLAogIHN0YXRlOiBbInNlZW4iLCAi b2xkIl0sCiAgbWFpbGJveDogImluYm94Igp9Ci0tLQogYmluL2hlbGlvdHJvcGUtc2VydmVyIHwg ICAzMCArKysrKysrKysrKysrKysrKysrKysrKysrKystLS0KIDEgZmlsZXMgY2hhbmdlZCwgMjcg aW5zZXJ0aW9ucygrKSwgMyBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9iaW4vaGVsaW90cm9w ZS1zZXJ2ZXIgYi9iaW4vaGVsaW90cm9wZS1zZXJ2ZXIKaW5kZXggZDcxOTg5ZC4uYmZhOGZkZCAx MDA2NDQKLS0tIGEvYmluL2hlbGlvdHJvcGUtc2VydmVyCisrKyBiL2Jpbi9oZWxpb3Ryb3BlLXNl cnZlcgpAQCAtMzAyLDEwICszMDIsMzQgQEAgY2xhc3MgSGVsaW90cm9wZVNlcnZlciA8IFNpbmF0 cmE6OkJhc2UKICAgZW5kCiAKICAgcG9zdCAiL21lc3NhZ2UuanNvbiIgZG8KLSAgICBib2R5ID0g cGFyYW1zWyJib2R5Il0gb3IgcmFpc2UgUmVxdWVzdEVycm9yLCAibmVlZCBhICdib2R5JyBwYXJh bSIKLSAgICBwdXRzIGJvZHkKKyAgICBjb250ZW50X3R5cGUgOmpzb24KKyAgICBiZWdpbgorICAg ICAgcmF3Ym9keSA9IHBhcmFtc1siYm9keSJdIG9yIHJhaXNlIFJlcXVlc3RFcnJvciwgIm5lZWQg YSAnYm9keScgcGFyYW0iCisgICAgICByYXdib2R5LmZvcmNlX2VuY29kaW5nICJiaW5hcnkiIGlm IHJhd2JvZHkucmVzcG9uZF90bz8oOmZvcmNlX2VuY29kaW5nKSAjIHNpZ2guLi4KIAotICAgIHs6 c3RhdHVzID0+ICJvayJ9LnRvX2pzb24KKyAgICAgIG1lc3NhZ2UgPSBIZWxpb3Ryb3BlOjpNZXNz YWdlLm5ldyhyYXdib2R5KS5wYXJzZSEKKworICAgICAgbWJveCA9IHBhcmFtc1sibWFpbGJveCJd IHx8ICJpbmJveCIKKyAgICAgIGRvY19pZCA9IG5pbAorICAgICAgdGhyZWFkX2lkID0gbmlsCisg ICAgICBzdGF0ZSA9IG5pbAorICAgICAgbGFiZWxzID0gW21ib3hdCisgICAgICBpZiBAaW5kZXgu Y29udGFpbnNfbXNnaWQ/IG1lc3NhZ2UubXNnaWQKKyAgICAgICAgbWVzc2FnZWluZm8gPSBnZXRf bWVzc2FnZV9zdW1tYXJ5IG1lc3NhZ2UubXNnaWQKKyAgICAgICAgZG9jX2lkID0gIG1lc3NhZ2Vp bmZvWzptZXNzYWdlX2lkXQorICAgICAgICB0aHJlYWRfaWQgPSBtZXNzYWdlaW5mb1s6dGhyZWFk X2lkXQorICAgICAgICB7IDpyZXNwb25zZSA9PiA6b2ssIDpzdGF0dXMgPT4gOnNlZW4sIDpkb2Nf aWQgPT4gZG9jX2lkLCA6dGhyZWFkX2lkID0+IHRocmVhZF9pZCB9CisgICAgICBlbHNlCisgICAg ICAgICMgU2V0IHN0YXRlCisgICAgICAgIHN0YXRlID0gcGFyYW1zWyJzdGF0ZSJdIHx8IFsidW5z ZWVuIl0KKyAgICAgICAgbGFiZWxzLnB1c2gocGFyYW1zWyJsYWJlbHMiXSkuZmxhdHRlbiEudW5p cSEgaWYgcGFyYW1zWyJsYWJlbHMiXQorICAgICAgICBsb2MgPSBAc3RvcmUuYWRkIHJhd2JvZHkK KyAgICAgICAgZG9jX2lkLCB0aHJlYWRfaWQgPSBAaW5kZXguYWRkX21lc3NhZ2UgbWVzc2FnZSwg c3RhdGUsIGxhYmVscywgeyA6bG9jID0+IGxvYywgOm1ib3ggPT4gbWJveCB9CisgICAgICAgIHsg OnJlc3BvbnNlID0+IDpvaywgOnN0YXR1cyA9PiBzdGF0ZSwgOmRvY19pZCA9PiBkb2NfaWQsIDp0 aHJlYWRfaWQgPT4gdGhyZWFkX2lkLCA6bGFiZWxzID0+IGxhYmVscyB9CisgICAgICBlbmQKKyAg ICByZXNjdWUgSGVsaW90cm9wZTo6SW52YWxpZE1lc3NhZ2VFcnJvciA9PiBlCisgICAgICB7IDpy ZXNwb25zZSA9PiA6ZXJyb3IsIDplcnJvcl9tZXNzYWdlID0+IGUubWVzc2FnZSB9CisgICAgZW5k LnRvX2pzb24KICAgZW5kCiAKIHByaXZhdGUKLS0gCjEuNy40LjEKCg== --bcaec50161513a027f04a37a3f28 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel --bcaec50161513a027f04a37a3f28--