From: Horacio Sanson <hsanson@gmail.com>
To: Gaute Hope <eg@gaute.vetsj.com>
Cc: Sup developer discussion <sup-devel@rubyforge.org>
Subject: Re: [sup-devel] Experimental Gmail Source
Date: Wed, 15 May 2013 11:10:22 +0900 [thread overview]
Message-ID: <CAHWBo_ZY7bWTxfqbQ1H3hu-q6mzKZm-YYe-+t0b-0M7xwUDfYQ@mail.gmail.com> (raw)
In-Reply-To: <CAHWBo_ZMce1GV=ChkLrSNbz9Lf-DtPb4W479kChXmUFMQf5qpw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 6474 bytes --]
Hello all,
I am still trying to implement sync-back functionality to my GMail source
but not going anywhere.
The problem I have is that the Sup index keeps the messages ids provided by
RMail (e.g. 20cf301af801a2aa8b04dc6e9931@google.com) and not the id I get
from the source that is the X-MSG-ID provided by Gmail
(e.g. 1434541737393941768). I need a way to query the index using the Gmail
provided id and get a message back or at least it's labels so I can compare
and update them on the server side if required.
This is what I have found so far:
- There is no way to query the index for a message using the source
(Gmail) provided id.
- The only places I can see where the source message ids are stored is in
a locations array
that keeps the source own id and the source message id.
- Sup index provides methods to query messages by id (build_message,
contains_id?, etc)
but the id they accept is the message id provided my RMail, not the id
of the source.
Questions:
- What is the purpose of the locations array? To allow a single message
to exists on
multiple sources?
- Is there an easy way to query the index to return a message/messages
using the
source provided id?
- Is a good idea to add a term to the index that keeps the source id
along with the message id?
regards,
Horacio
On Sun, May 12, 2013 at 3:18 AM, Horacio Sanson <hsanson@gmail.com> wrote:
>
>
>
> On Sat, May 11, 2013 at 7:15 PM, Gaute Hope <eg@gaute.vetsj.com> wrote:
>
>>
>>
>> On 09. mai 2013 11:28, Horacio Sanson wrote:
>> > I am trying to implement a new source for Gmail accounts. This is
>> > copied from my efforts to do the same in Heliotrope.
>> >
>> > Here is an experimental implementation that can read the email from
>> > Gmail and add it to the Sup index:
>> >
>> > https://github.com/hsanson/sup/tree/gmail_source
>> >
>> > To use:
>> >
>> > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail
>> > source: sup-add gmail://username@gmail.com - Start sup and see how
>> > it syncs your emails.
>> >
>> > Warnings:
>> >
>> > - This is experimental - This always syncs only the All mailbox so
>> > make sure to use an account with not too many emails for testing. -
>> > All email data and headers are stored in a LevelDB database at:
>> > ~/.sup/gmail/account
>> >
>>
>> > - For some reason I get duplicate "Inbox" and "Sent" labels in the
>> > list of labels and I am not sure why.
>> > - I still have no clue on how to handle sync-back. That is how to
>> propagate
>> > changes made in sup
>> > back to Gmail. Any tips on how the maildir source does it would be
>> > appreciated.
>>
>> Hi Horacio,
>>
>> nice work. Working directly with GMail labels is probably a good idea
>> (the other option is to move messages between IMAP folders). I have a
>> design question though:
>>
>> Should remote sources be part of regular sup? Or should rather the
>> fetching and syncing be put in a separate script which creates a LevelDB
>> setup like you have it with a Gmail source in sup working directly on it?
>>
>
>
> I don't like the current two step sync setup that Sup uses now (IMAP ->
> Maildir -> Sup). It requires external programs (offlineimap) and
> synchronization is one way only. I know about the sync-back branch but
> AFAIK it is still limited to flags only. Also Maildir is an old storage
> format that doesn't work well with current email workflows. For example I
> have tons of duplicate emails on several folders due to the use of
> mailboxes rather than labels.
>
> I would prefer Sup to take care of the mail storage/indexing (as
> heliotrope does) and the sources be in charge of syncing the Sup
> storage/index with the remote servers. I understand that this can be
> difficult due to the difference in paradigm between IMAP/POP and Sup but
> GMail offers extensions that map directly to Sup workflow. GMail gives each
> email a unique 64bit ascending indentifier to all mail messages and adds
> labels them. I don't even need the LevelDB database as I could easily
> implement the source so it fetches the mail headers and body from the Gmail
> server directly when requested. I only added the LevelDB storage as a cache
> to speed up the message lookups and for offline use.
>
>
>> Recall that the IMAP source was removed in 52e29ba [1] (discussion
>> probably on the mailinglist somewhere).
>>
>>
> Yes I know this and reading the commits and source code of this source I
> can tell that William hated the IMAP protocol. This is understandable as I
> myself have dealt with this protocol and know first hand how broken it is.
> But again with the extensions supported by GMail servers the implementation
> is far easier to do.
>
>
>> I like this approach for GMail, but I would like to see it for regular
>> IMAP sources as well with folders as labels.. I briefly experimented
>> with a maildir-root folder approach [2] which treats all underlying
>> maildirs as sources which correspond to a label (do not use,
>> incomplete). It of course presents a plethora of questions on how to
>> sync messages between labels, but implementing it is is probably
>> relatively straight forward.
>>
>>
> IMAP is a horrible protocol and implementing it requires herculean effort.
> Still once the Gmail source is finished it can become a starting point for
> a more complete IMAP source.
>
>
>> > Help:
>> >
>> > - How do I stop the source poll when I quit sup? If I have a large
>> > amount of emails when polling is running and I quit sup the process
>> > hangs there.
>>
>> This normally runs in a separate thread, I don't think maildir really
>> stops the polling - so I sometimes get an error if I quit sup while the
>> polling is running and various stuff just disappears underneath the
>> poller.
>>
>>
> I see that sup simply kills the threads... would be better if each source
> had a stop method that Sup could invoke to stop the polling. I will try to
> look into this issue as it is problematic for remote sources with large
> amounts of emails.
>
> I don't have the chance to get into your other questions at the moment.
>>
>>
> Thanks for answering my inquiries. If you have a chance I would really
> like to know how to get the labels for a specific email from the index. I
> need this to implement the sync-back part of the Gmail source.
>
> regards,
> Horacio
>
>
>> Regards, Gaute
>>
>> [1] https://github.com/sup-heliotrope/sup/commit/52e29ba
>> [2] https://github.com/gauteh/sup/tree/maildir-root
>
>
[-- Attachment #2: Type: text/html, Size: 9078 bytes --]
next prev parent reply other threads:[~2013-05-15 2:10 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-09 9:28 Horacio Sanson
2013-05-09 19:13 ` Eric Weikl
2013-05-11 10:15 ` Gaute Hope
2013-05-11 18:18 ` Horacio Sanson
2013-05-15 2:10 ` Horacio Sanson [this message]
2013-05-21 21:47 ` Matthieu Rakotojaona
2013-06-02 2:45 ` Horacio Sanson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAHWBo_ZY7bWTxfqbQ1H3hu-q6mzKZm-YYe-+t0b-0M7xwUDfYQ@mail.gmail.com \
--to=hsanson@gmail.com \
--cc=eg@gaute.vetsj.com \
--cc=sup-devel@rubyforge.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox