From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 10.52.177.71 with SMTP id co7csp273vdc; Tue, 14 May 2013 19:10:23 -0700 (PDT) X-Received: by 10.60.62.198 with SMTP id a6mr18513085oes.22.1368583822847; Tue, 14 May 2013 19:10:22 -0700 (PDT) Return-Path: Received: from mail-ob0-x22c.google.com (mail-ob0-x22c.google.com [2607:f8b0:4003:c01::22c]) by mx.google.com with ESMTPS id fi9si479177obb.6.2013.05.14.19.10.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 14 May 2013 19:10:22 -0700 (PDT) Received-SPF: pass (google.com: domain of hsanson@gmail.com designates 2607:f8b0:4003:c01::22c as permitted sender) client-ip=2607:f8b0:4003:c01::22c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of hsanson@gmail.com designates 2607:f8b0:4003:c01::22c as permitted sender) smtp.mail=hsanson@gmail.com; dkim=pass header.i=@gmail.com Received: by mail-ob0-f172.google.com with SMTP id tb18so1368162obb.17 for ; Tue, 14 May 2013 19:10:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=bKhEA0QbaKyiHLE4U+j80D+i3YwyYDFBZZ+87iC2bDw=; b=pkFnQsIyYdreuPOeIk6qUXehJoHr03kM9GVGmn2OqRpbbqwHwSdXon2av5G1jJqcwY E1k0kMwtLAXNxkJPVgA8OKguFdJg/aRdh8dE61Od/LEcQo4ZliEX8setGPmRBOQsCMlb uz9Lm3gELTGTx/HkT02tgL233QmQLhESXvL3KJC1IUB3n6T54MlEkpZ6XzqljEI/7HeS 4LbamC3wDgKYhCVVHUHcANkkEaJ4R1oEVzEOsbt0T860Rj0aSQYdPHo/DT82IDrhCTkf B/87JayUliz+GLvS9GSQV+Ae44XZzpJ7CQ16d2nQWpgURo4jpoOYttfpPfU0Fg60d1pO njng== MIME-Version: 1.0 X-Received: by 10.182.45.197 with SMTP id p5mr15569970obm.92.1368583822115; Tue, 14 May 2013 19:10:22 -0700 (PDT) Received: by 10.182.123.2 with HTTP; Tue, 14 May 2013 19:10:22 -0700 (PDT) In-Reply-To: References: <518E1A2B.2080903@gaute.vetsj.com> Date: Wed, 15 May 2013 11:10:22 +0900 Message-ID: Subject: Re: [sup-devel] Experimental Gmail Source From: Horacio Sanson To: Gaute Hope Cc: Sup developer discussion Content-Type: multipart/alternative; boundary=047d7b67637a900a9704dcb84097 --047d7b67637a900a9704dcb84097 Content-Type: text/plain; charset=ISO-8859-1 Hello all, I am still trying to implement sync-back functionality to my GMail source but not going anywhere. The problem I have is that the Sup index keeps the messages ids provided by RMail (e.g. 20cf301af801a2aa8b04dc6e9931@google.com) and not the id I get from the source that is the X-MSG-ID provided by Gmail (e.g. 1434541737393941768). I need a way to query the index using the Gmail provided id and get a message back or at least it's labels so I can compare and update them on the server side if required. This is what I have found so far: - There is no way to query the index for a message using the source (Gmail) provided id. - The only places I can see where the source message ids are stored is in a locations array that keeps the source own id and the source message id. - Sup index provides methods to query messages by id (build_message, contains_id?, etc) but the id they accept is the message id provided my RMail, not the id of the source. Questions: - What is the purpose of the locations array? To allow a single message to exists on multiple sources? - Is there an easy way to query the index to return a message/messages using the source provided id? - Is a good idea to add a term to the index that keeps the source id along with the message id? regards, Horacio On Sun, May 12, 2013 at 3:18 AM, Horacio Sanson wrote: > > > > On Sat, May 11, 2013 at 7:15 PM, Gaute Hope wrote: > >> >> >> On 09. mai 2013 11:28, Horacio Sanson wrote: >> > I am trying to implement a new source for Gmail accounts. This is >> > copied from my efforts to do the same in Heliotrope. >> > >> > Here is an experimental implementation that can read the email from >> > Gmail and add it to the Sup index: >> > >> > https://github.com/hsanson/sup/tree/gmail_source >> > >> > To use: >> > >> > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail >> > source: sup-add gmail://username@gmail.com - Start sup and see how >> > it syncs your emails. >> > >> > Warnings: >> > >> > - This is experimental - This always syncs only the All mailbox so >> > make sure to use an account with not too many emails for testing. - >> > All email data and headers are stored in a LevelDB database at: >> > ~/.sup/gmail/account >> > >> >> > - For some reason I get duplicate "Inbox" and "Sent" labels in the >> > list of labels and I am not sure why. >> > - I still have no clue on how to handle sync-back. That is how to >> propagate >> > changes made in sup >> > back to Gmail. Any tips on how the maildir source does it would be >> > appreciated. >> >> Hi Horacio, >> >> nice work. Working directly with GMail labels is probably a good idea >> (the other option is to move messages between IMAP folders). I have a >> design question though: >> >> Should remote sources be part of regular sup? Or should rather the >> fetching and syncing be put in a separate script which creates a LevelDB >> setup like you have it with a Gmail source in sup working directly on it? >> > > > I don't like the current two step sync setup that Sup uses now (IMAP -> > Maildir -> Sup). It requires external programs (offlineimap) and > synchronization is one way only. I know about the sync-back branch but > AFAIK it is still limited to flags only. Also Maildir is an old storage > format that doesn't work well with current email workflows. For example I > have tons of duplicate emails on several folders due to the use of > mailboxes rather than labels. > > I would prefer Sup to take care of the mail storage/indexing (as > heliotrope does) and the sources be in charge of syncing the Sup > storage/index with the remote servers. I understand that this can be > difficult due to the difference in paradigm between IMAP/POP and Sup but > GMail offers extensions that map directly to Sup workflow. GMail gives each > email a unique 64bit ascending indentifier to all mail messages and adds > labels them. I don't even need the LevelDB database as I could easily > implement the source so it fetches the mail headers and body from the Gmail > server directly when requested. I only added the LevelDB storage as a cache > to speed up the message lookups and for offline use. > > >> Recall that the IMAP source was removed in 52e29ba [1] (discussion >> probably on the mailinglist somewhere). >> >> > Yes I know this and reading the commits and source code of this source I > can tell that William hated the IMAP protocol. This is understandable as I > myself have dealt with this protocol and know first hand how broken it is. > But again with the extensions supported by GMail servers the implementation > is far easier to do. > > >> I like this approach for GMail, but I would like to see it for regular >> IMAP sources as well with folders as labels.. I briefly experimented >> with a maildir-root folder approach [2] which treats all underlying >> maildirs as sources which correspond to a label (do not use, >> incomplete). It of course presents a plethora of questions on how to >> sync messages between labels, but implementing it is is probably >> relatively straight forward. >> >> > IMAP is a horrible protocol and implementing it requires herculean effort. > Still once the Gmail source is finished it can become a starting point for > a more complete IMAP source. > > >> > Help: >> > >> > - How do I stop the source poll when I quit sup? If I have a large >> > amount of emails when polling is running and I quit sup the process >> > hangs there. >> >> This normally runs in a separate thread, I don't think maildir really >> stops the polling - so I sometimes get an error if I quit sup while the >> polling is running and various stuff just disappears underneath the >> poller. >> >> > I see that sup simply kills the threads... would be better if each source > had a stop method that Sup could invoke to stop the polling. I will try to > look into this issue as it is problematic for remote sources with large > amounts of emails. > > I don't have the chance to get into your other questions at the moment. >> >> > Thanks for answering my inquiries. If you have a chance I would really > like to know how to get the labels for a specific email from the index. I > need this to implement the sync-back part of the Gmail source. > > regards, > Horacio > > >> Regards, Gaute >> >> [1] https://github.com/sup-heliotrope/sup/commit/52e29ba >> [2] https://github.com/gauteh/sup/tree/maildir-root > > --047d7b67637a900a9704dcb84097 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hello all,

I am still trying to impleme= nt sync-back functionality to my GMail source but not going anywhere.
=
The problem I have is that the Sup index keeps the messages ids = provided by RMail (e.g.=A020cf301af801a2aa8b04dc6e9931@google.com) and not the id I get= from the source that is the X-MSG-ID provided by Gmail (e.g.=A014345417373= 93941768). I need a way to query the index using the Gmail provided id and = get a message back or at least it's labels so I can compare and update = them on the server side if required.

This is what I have found so far:

=A0 - There is no way to query the index for a = message using the source (Gmail) provided id.
=A0 - The only = places I can see where the source message ids are stored is in a locations = array
=A0 =A0 =A0that keeps the source own id and the source message id.
=A0 - Sup index provides methods to query messages by id (buil= d_message, contains_id?, etc)
=A0 =A0 but the id they accep= t is the message id provided my RMail, not the id of the source.

Questions:

=A0 - What is the purpose of the locations array? To allow a single me= ssage to exists on
=A0 =A0 multiple sources?
=A0 - Is there an easy way to query the index to return a message/messages = using the=A0
=A0 =A0 source provided id?
= =A0 - Is a good idea to add a term to the index that keeps the source id al= ong with the message id?

regards,
Horacio



<= br>
On Sun, May 12, 2013 at 3:18 AM, Horacio Sans= on <hsanson@gmail.com> wrote:



On Sat, May 1= 1, 2013 at 7:15 PM, Gaute Hope <eg@gaute.vetsj.com> wrote:<= br>


On 09. mai 2013 11:28, Horacio Sanson wrote:
> I am trying to implement a new source for Gmail accounts. This is
> copied from my efforts to do the same in Heliotrope.
>
> Here is an experimental implementation that can read the email from > Gmail and add it to the Sup index:
>
> https://github.com/hsanson/sup/tree/gmail_source
>
> To use:
>
> - Install leveldb gem "sudo gem install leveldb-ruby" - Add = a gmail
> source: =A0 sup-add gmail://username@gmail.com - Start sup and see how
> it syncs your emails.
>
> Warnings:
>
> - This is experimental - This always syncs only the All mailbox so
> make sure to use an account with not too many emails for testing. - > All email data and headers are stored in a LevelDB database at:
> ~/.sup/gmail/account
>

> - For some reason I get duplicate "Inbox" and &qu= ot;Sent" labels in the
> list of labels and I am not sure why.
> =A0- I still have no clue on how =A0to handle sync-back. That is how t= o
propagate
> changes made in sup
> back to Gmail. Any tips on how the maildir source does it would be
> appreciated.

Hi Horacio,

nice work. Working directly with GMail labels is probably a good idea
(the other option is to move messages between IMAP folders). I have a
design question though:

Should remote sources be part of regular sup? Or should rather the
fetching and syncing be put in a separate script which creates a LevelDB setup like you have it with a Gmail source in sup working directly on it?


I don't l= ike the current two step sync setup that Sup uses now (IMAP -> Maildir -= > Sup). It requires external programs (offlineimap) and synchronization = is one way only. I know about the sync-back branch but AFAIK it is still li= mited to flags only. Also Maildir is an old storage format that doesn't= work well with current email workflows. For example I have tons of duplica= te emails on several folders due to the use of mailboxes rather than labels= .

I would prefer Sup to take care of the mail storage/ind= exing (as heliotrope does) and the sources be in charge of syncing the Sup = storage/index with the remote servers. I understand that this can be diffic= ult due to the difference in paradigm between IMAP/POP and Sup but GMail of= fers extensions that map directly to Sup workflow. GMail gives each email a= unique 64bit ascending indentifier to all mail messages and adds labels th= em. I don't even need the LevelDB database as I could easily implement = the source so it fetches the mail headers and body from the Gmail server di= rectly when requested. I only added the LevelDB storage as a cache to speed= up the message lookups and for offline use.=A0
=A0
Recall that the IMAP source was removed in 52e29ba [1] (discussion
probably on the mailinglist somewhere).


Yes I know this and reading the = commits and source code of this source I can tell that William hated the IM= AP protocol. This is understandable as I myself have dealt with this protoc= ol and know first hand how broken it is. But again with the extensions supp= orted by GMail servers the implementation is far easier to do.
=A0
I like this approach for GMail, but I would like to see it for regular
IMAP sources as well with folders as labels.. I briefly experimented
with a maildir-root folder approach [2] which treats all underlying
maildirs as sources which correspond to a label (do not use,
incomplete). It of course presents a plethora of questions on how to
sync messages between labels, but implementing it is is probably
relatively straight forward.

=A0
IMAP is a horrible pro= tocol and implementing it requires herculean effort. Still once the Gmail s= ource is finished it can become a starting point for a more complete IMAP s= ource.=A0
=A0
> Help:
>
> - How do I stop the source poll when I quit sup? If I have a large
> amount of emails when polling is running and I quit sup the process > hangs there.

This normally runs in a separate thread, I don't think maildir re= ally
stops the polling - so I sometimes get an error if I quit sup while the
polling is running and various stuff just disappears underneath the poller.=


I see that sup simply kills the = threads... would be better if each source had a stop method that Sup could = invoke to stop the polling. I will try to look into this issue as it is pro= blematic for remote sources with large amounts of emails.

I don't have the chance to get into your other questions at the moment.=


Thanks for answering my inquirie= s. If you have a chance I would really like to know how to get the labels f= or a specific email from the index. I need this to implement the sync-back = part of the Gmail source.

regards,
Horacio
= =A0
Regards, Gaute

[1] https://github.com/sup-heliotrope/sup/commit/52e29ba
[2] https://github.com/gauteh/sup/tree/maildir-root

--047d7b67637a900a9704dcb84097--