From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 10.52.177.71 with SMTP id co7csp92367vdc; Sat, 11 May 2013 11:18:47 -0700 (PDT) X-Received: by 10.60.134.106 with SMTP id pj10mr9516255oeb.3.1368296327047; Sat, 11 May 2013 11:18:47 -0700 (PDT) Return-Path: Received: from mail-oa0-f51.google.com (mail-oa0-f51.google.com [209.85.219.51]) by mx.google.com with ESMTPS id rx8si8161785oeb.53.2013.05.11.11.18.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 11 May 2013 11:18:47 -0700 (PDT) Received-SPF: pass (google.com: domain of hsanson@gmail.com designates 209.85.219.51 as permitted sender) client-ip=209.85.219.51; Authentication-Results: mx.google.com; spf=pass (google.com: domain of hsanson@gmail.com designates 209.85.219.51 as permitted sender) smtp.mail=hsanson@gmail.com; dkim=pass header.i=@gmail.com Received: by mail-oa0-f51.google.com with SMTP id f4so6085727oah.24 for ; Sat, 11 May 2013 11:18:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=zK06OuxLLm9iRYw6Mwg9Tfa5hLYuiWS90NH78PAB9OE=; b=vo9Z946vDr+bMbHhKkFg0YnpAITkrSoc5aWczdfiS0H1R44L/H25fLLrI0WgK/nUri Ev7ArvEbwGzL1Fl17tE0Mpvw+kfKITBUu7IxaU0OsL5tjb8dUtsMu5o59w4tvIG836dG 8bfCDs0LqxI2YP18ccbsSug6Fgs4kmKHMpT4lG5UIMEiddABK7B+qse3Tu6fEE1gGC5x AkNgvIm+HKliQRBMIOCRSSR7u+YUkJNLx96X53npJxdYSp6t1HkaPfYSt/upxsnexTez SmJT4a/MC5FgD4tEfvzjs2xOpu5FhwezuaOcF1UZlv+ck+O8idcn9u3hm1cHfSoAl6Vn Ghvw== MIME-Version: 1.0 X-Received: by 10.60.121.106 with SMTP id lj10mr7092146oeb.128.1368296326596; Sat, 11 May 2013 11:18:46 -0700 (PDT) Received: by 10.182.123.2 with HTTP; Sat, 11 May 2013 11:18:46 -0700 (PDT) In-Reply-To: <518E1A2B.2080903@gaute.vetsj.com> References: <518E1A2B.2080903@gaute.vetsj.com> Date: Sun, 12 May 2013 03:18:46 +0900 Message-ID: Subject: Re: [sup-devel] Experimental Gmail Source From: Horacio Sanson To: Gaute Hope Cc: Sup developer discussion Content-Type: multipart/alternative; boundary=047d7b5d9c8f7e8d3204dc75506a --047d7b5d9c8f7e8d3204dc75506a Content-Type: text/plain; charset=ISO-8859-1 On Sat, May 11, 2013 at 7:15 PM, Gaute Hope wrote: > > > On 09. mai 2013 11:28, Horacio Sanson wrote: > > I am trying to implement a new source for Gmail accounts. This is > > copied from my efforts to do the same in Heliotrope. > > > > Here is an experimental implementation that can read the email from > > Gmail and add it to the Sup index: > > > > https://github.com/hsanson/sup/tree/gmail_source > > > > To use: > > > > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail > > source: sup-add gmail://username@gmail.com - Start sup and see how > > it syncs your emails. > > > > Warnings: > > > > - This is experimental - This always syncs only the All mailbox so > > make sure to use an account with not too many emails for testing. - > > All email data and headers are stored in a LevelDB database at: > > ~/.sup/gmail/account > > > > > - For some reason I get duplicate "Inbox" and "Sent" labels in the > > list of labels and I am not sure why. > > - I still have no clue on how to handle sync-back. That is how to > propagate > > changes made in sup > > back to Gmail. Any tips on how the maildir source does it would be > > appreciated. > > Hi Horacio, > > nice work. Working directly with GMail labels is probably a good idea > (the other option is to move messages between IMAP folders). I have a > design question though: > > Should remote sources be part of regular sup? Or should rather the > fetching and syncing be put in a separate script which creates a LevelDB > setup like you have it with a Gmail source in sup working directly on it? > I don't like the current two step sync setup that Sup uses now (IMAP -> Maildir -> Sup). It requires external programs (offlineimap) and synchronization is one way only. I know about the sync-back branch but AFAIK it is still limited to flags only. Also Maildir is an old storage format that doesn't work well with current email workflows. For example I have tons of duplicate emails on several folders due to the use of mailboxes rather than labels. I would prefer Sup to take care of the mail storage/indexing (as heliotrope does) and the sources be in charge of syncing the Sup storage/index with the remote servers. I understand that this can be difficult due to the difference in paradigm between IMAP/POP and Sup but GMail offers extensions that map directly to Sup workflow. GMail gives each email a unique 64bit ascending indentifier to all mail messages and adds labels them. I don't even need the LevelDB database as I could easily implement the source so it fetches the mail headers and body from the Gmail server directly when requested. I only added the LevelDB storage as a cache to speed up the message lookups and for offline use. > Recall that the IMAP source was removed in 52e29ba [1] (discussion > probably on the mailinglist somewhere). > > Yes I know this and reading the commits and source code of this source I can tell that William hated the IMAP protocol. This is understandable as I myself have dealt with this protocol and know first hand how broken it is. But again with the extensions supported by GMail servers the implementation is far easier to do. > I like this approach for GMail, but I would like to see it for regular > IMAP sources as well with folders as labels.. I briefly experimented > with a maildir-root folder approach [2] which treats all underlying > maildirs as sources which correspond to a label (do not use, > incomplete). It of course presents a plethora of questions on how to > sync messages between labels, but implementing it is is probably > relatively straight forward. > > IMAP is a horrible protocol and implementing it requires herculean effort. Still once the Gmail source is finished it can become a starting point for a more complete IMAP source. > > Help: > > > > - How do I stop the source poll when I quit sup? If I have a large > > amount of emails when polling is running and I quit sup the process > > hangs there. > > This normally runs in a separate thread, I don't think maildir really > stops the polling - so I sometimes get an error if I quit sup while the > polling is running and various stuff just disappears underneath the poller. > > I see that sup simply kills the threads... would be better if each source had a stop method that Sup could invoke to stop the polling. I will try to look into this issue as it is problematic for remote sources with large amounts of emails. I don't have the chance to get into your other questions at the moment. > > Thanks for answering my inquiries. If you have a chance I would really like to know how to get the labels for a specific email from the index. I need this to implement the sync-back part of the Gmail source. regards, Horacio > Regards, Gaute > > [1] https://github.com/sup-heliotrope/sup/commit/52e29ba > [2] https://github.com/gauteh/sup/tree/maildir-root --047d7b5d9c8f7e8d3204dc75506a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable



On Sat, May 11, 2013 at 7:15 PM, Gaute Hope <<= a href=3D"mailto:eg@gaute.vetsj.com" target=3D"_blank">eg@gaute.vetsj.com> wrote:


On 09. mai 2013 11:28, Horacio Sanson wrote:
> I am trying to implement a new source for Gmail accounts. This is
> copied from my efforts to do the same in Heliotrope.
>
> Here is an experimental implementation that can read the email from > Gmail and add it to the Sup index:
>
> https://github.com/hsanson/sup/tree/gmail_source
>
> To use:
>
> - Install leveldb gem "sudo gem install leveldb-ruby" - Add = a gmail
> source: =A0 sup-add gmail://username@gmail.com - Start sup and see how
> it syncs your emails.
>
> Warnings:
>
> - This is experimental - This always syncs only the All mailbox so
> make sure to use an account with not too many emails for testing. - > All email data and headers are stored in a LevelDB database at:
> ~/.sup/gmail/account
>

> - For some reason I get duplicate "Inbox" and &qu= ot;Sent" labels in the
> list of labels and I am not sure why.
> =A0- I still have no clue on how =A0to handle sync-back. That is how t= o
propagate
> changes made in sup
> back to Gmail. Any tips on how the maildir source does it would be
> appreciated.

Hi Horacio,

nice work. Working directly with GMail labels is probably a good idea
(the other option is to move messages between IMAP folders). I have a
design question though:

Should remote sources be part of regular sup? Or should rather the
fetching and syncing be put in a separate script which creates a LevelDB setup like you have it with a Gmail source in sup working directly on it?


I don't like the curr= ent two step sync setup that Sup uses now (IMAP -> Maildir -> Sup). I= t requires external programs (offlineimap) and synchronization is one way o= nly. I know about the sync-back branch but AFAIK it is still limited to fla= gs only. Also Maildir is an old storage format that doesn't work well w= ith current email workflows. For example I have tons of duplicate emails on= several folders due to the use of mailboxes rather than labels.

I would prefer Sup to take care of the mail storage/ind= exing (as heliotrope does) and the sources be in charge of syncing the Sup = storage/index with the remote servers. I understand that this can be diffic= ult due to the difference in paradigm between IMAP/POP and Sup but GMail of= fers extensions that map directly to Sup workflow. GMail gives each email a= unique 64bit ascending indentifier to all mail messages and adds labels th= em. I don't even need the LevelDB database as I could easily implement = the source so it fetches the mail headers and body from the Gmail server di= rectly when requested. I only added the LevelDB storage as a cache to speed= up the message lookups and for offline use.=A0
=A0
Recall that the IMAP source was removed in 52e29ba [1] (discussion
probably on the mailinglist somewhere).


Yes I know this and readin= g the commits and source code of this source I can tell that William hated = the IMAP protocol. This is understandable as I myself have dealt with this = protocol and know first hand how broken it is. But again with the extension= s supported by GMail servers the implementation is far easier to do.
=A0
I like this approach for GMail, but I would like to see it for regular
IMAP sources as well with folders as labels.. I briefly experimented
with a maildir-root folder approach [2] which treats all underlying
maildirs as sources which correspond to a label (do not use,
incomplete). It of course presents a plethora of questions on how to
sync messages between labels, but implementing it is is probably
relatively straight forward.

=A0
IMAP is a horrible pro= tocol and implementing it requires herculean effort. Still once the Gmail s= ource is finished it can become a starting point for a more complete IMAP s= ource.=A0
=A0
> Help:
>
> - How do I stop the source poll when I quit sup? If I have a large
> amount of emails when polling is running and I quit sup the process > hangs there.

This normally runs in a separate thread, I don't think maildir re= ally
stops the polling - so I sometimes get an error if I quit sup while the
polling is running and various stuff just disappears underneath the poller.=


I see that sup simply kills the = threads... would be better if each source had a stop method that Sup could = invoke to stop the polling. I will try to look into this issue as it is pro= blematic for remote sources with large amounts of emails.

I don't have the chance to get into your other questions at the moment.=


Thanks for answering my inquirie= s. If you have a chance I would really like to know how to get the labels f= or a specific email from the index. I need this to implement the sync-back = part of the Gmail source.

regards,
Horacio
=A0
Regards, Gaute

[1] https://github.com/sup-heliotrope/sup/commit/52e29ba
[2] https://github.com/gauteh/sup/tree/maildir-root
--047d7b5d9c8f7e8d3204dc75506a--