sup

A curses threads-with-tags style email client

sup-website.git

git clone https://supmua.dev/git/sup-website/

community/pipermail-archives/sup-devel/2013-05.txt (40994B) - raw

      1 From eg@gaute.vetsj.com  Thu May  2 08:17:18 2013
      2 From: eg@gaute.vetsj.com (Gaute Hope)
      3 Date: Thu, 02 May 2013 10:17:18 +0200
      4 Subject: [sup-devel] sup 0.13
      5 In-Reply-To: <CAHWBo_aKjp__dkjuJ-tYHpKJuv2G=k_jxTWMo0QyyUtvxZbGXA@mail.gmail.com>
      6 References: <CAOxvSbeLRoCwy2THkKa4TKsNqdsx9n-M5BxrZHGfPEUjfEWjSg@mail.gmail.com>
      7 	<1367233230-sup-3593@tesla> <517E5BD8.6030803@gaute.vetsj.com>
      8 	<CAHWBo_aKjp__dkjuJ-tYHpKJuv2G=k_jxTWMo0QyyUtvxZbGXA@mail.gmail.com>
      9 Message-ID: <5182210E.4040100@gaute.vetsj.com>
     10 
     11 
     12 
     13 On 30. april 2013 11:44, Horacio Sanson wrote:
     14 > Great to see Sup getting back on track again..
     15 > 
     16 > I submitted some patches for the Gmail dumper of Heliotrope some time ago
     17 > but the lack of non alphabet languages (Japanese, Chinese) made it
     18 > impossible for me to keep using heliotrope/turnesole.
     19 > 
     20 > The main issue to support Japanese/Chinese with heliotrope was that
     21 > whistlepig (indexer) lacked the ability to tokenize these languages. Also
     22 > the half baked UTF-8 support caused several issues with these languages.
     23 > 
     24 > I would like to help in testing/implementing support for these languages,
     25 > starting with Japanese, but I would require some guidance. First I would
     26 > like to know is there is a way to configure the Xapian tokenizer
     27 > (segmenter) within sup? Please consider that I am new to both sup and to
     28 > Xapian.
     29 
     30 Hi Horacio,
     31 
     32 consider opening an issue at
     33 https://github.com/sup-heliotrope/sup/issues to make sure this doesn't
     34 disappear. Some changes will probably be made to the indexer when going
     35 to Mail (from RMail), but I hope to be able to migrate the existing
     36 index. Perhaps its time to get it right for arbitrary languages as well.
     37 I am unfamiliar with Japanes/Chinese - does UTF-8 cover the needs?
     38 
     39 Mail is better at handling UTF-8 and I think there was some fork that
     40 had some extra support for Japanese.
     41 
     42 Regards, Gaute
     43 
     44 From eg@gaute.vetsj.com  Tue May  7 11:54:22 2013
     45 From: eg@gaute.vetsj.com (Gaute Hope)
     46 Date: Tue, 07 May 2013 13:54:22 +0200
     47 Subject: [sup-devel] sup 0.13 - please test
     48 In-Reply-To: <CAOxvSbeLRoCwy2THkKa4TKsNqdsx9n-M5BxrZHGfPEUjfEWjSg@mail.gmail.com>
     49 References: <CAOxvSbeLRoCwy2THkKa4TKsNqdsx9n-M5BxrZHGfPEUjfEWjSg@mail.gmail.com>
     50 Message-ID: <5188EB6E.8070504@gaute.vetsj.com>
     51 
     52 On 28. april 2013 22:32, Hamish D wrote:
     53 > We'd like to get version 0.13 of sup out soon.  There's a fair bit of
     54 > work gone into sup since 0.12 was released (in January 2011) that it
     55 > would be good to get out there. Once that is done we can then work on
     56 > the bigger goals we have for 0.14 - see
     57 > https://github.com/sup-heliotrope/sup/wiki/Development#roadmap for
     58 > those plans.
     59 > 
     60 > So the question is, are there any serious bugs that we should fix
     61 > before releasing 0.13?
     62 
     63 Greetings sup-devel and sup-talk,
     64 
     65 
     66 Thanks to several contributors I think we should be getting ready for
     67 0.13 now soon, we've removed a lot of old and deprecated stuff and
     68 generally cleaned a bit up.
     69 
     70 There is a new web page: http://supmua.org which is getting in shape and
     71 a wiki which is quickly growing: https://github.com/sup-heliotrope/sup/wiki
     72 
     73 We have forked our own version of ncursesw (ncursesw-sup) which now is a
     74 dependency for sup [1].
     75 
     76 To be able to do the next release we need some testing of the current
     77 state of 'develop' and the new ncursesw-sup gem. For instructions on how
     78 to help out with testing the latest git version (develop) check out:
     79 https://github.com/sup-heliotrope/sup/wiki/Running-from-git
     80 
     81 If you have any problems or questions please ask.
     82 
     83 Report issues to: https://github.com/sup-heliotrope/sup/issues
     84 
     85 
     86 [1] https://rubygems.org/gems/ncursesw-sup
     87 
     88 
     89 Regards,
     90 Gaute
     91 
     92 From hsanson@gmail.com  Thu May  9 01:39:08 2013
     93 From: hsanson@gmail.com (Horacio Sanson)
     94 Date: Thu, 9 May 2013 10:39:08 +0900
     95 Subject: [sup-devel] sup 0.13
     96 In-Reply-To: <5182210E.4040100@gaute.vetsj.com>
     97 References: <CAOxvSbeLRoCwy2THkKa4TKsNqdsx9n-M5BxrZHGfPEUjfEWjSg@mail.gmail.com>
     98 	<1367233230-sup-3593@tesla> <517E5BD8.6030803@gaute.vetsj.com>
     99 	<CAHWBo_aKjp__dkjuJ-tYHpKJuv2G=k_jxTWMo0QyyUtvxZbGXA@mail.gmail.com>
    100 	<5182210E.4040100@gaute.vetsj.com>
    101 Message-ID: <CAHWBo_ZC176WjeMpPs_rJE9=41exnxxJ0ENfLpHRj1ZtOpg2Ng@mail.gmail.com>
    102 
    103 UTF-8 handles most cases but I still have to deal with emails in
    104 ISO2022-JP, Shift-JIS and EUC-JP. After some research it seems Xapian has
    105 no support for Asian languages. I will try to make some tests and open an
    106 issue if I cannot make it work.
    107 
    108 I can see in the sup configuration file that the stem language can be
    109 configured but there are no CJK stemmers for Xapian that I can find.
    110 
    111 
    112 On Thu, May 2, 2013 at 5:17 PM, Gaute Hope <eg at gaute.vetsj.com> wrote:
    113 
    114 >
    115 >
    116 > On 30. april 2013 11:44, Horacio Sanson wrote:
    117 > > Great to see Sup getting back on track again..
    118 > >
    119 > > I submitted some patches for the Gmail dumper of Heliotrope some time ago
    120 > > but the lack of non alphabet languages (Japanese, Chinese) made it
    121 > > impossible for me to keep using heliotrope/turnesole.
    122 > >
    123 > > The main issue to support Japanese/Chinese with heliotrope was that
    124 > > whistlepig (indexer) lacked the ability to tokenize these languages. Also
    125 > > the half baked UTF-8 support caused several issues with these languages.
    126 > >
    127 > > I would like to help in testing/implementing support for these languages,
    128 > > starting with Japanese, but I would require some guidance. First I would
    129 > > like to know is there is a way to configure the Xapian tokenizer
    130 > > (segmenter) within sup? Please consider that I am new to both sup and to
    131 > > Xapian.
    132 >
    133 > Hi Horacio,
    134 >
    135 > consider opening an issue at
    136 > https://github.com/sup-heliotrope/sup/issues to make sure this doesn't
    137 > disappear. Some changes will probably be made to the indexer when going
    138 > to Mail (from RMail), but I hope to be able to migrate the existing
    139 > index. Perhaps its time to get it right for arbitrary languages as well.
    140 > I am unfamiliar with Japanes/Chinese - does UTF-8 cover the needs?
    141 >
    142 > Mail is better at handling UTF-8 and I think there was some fork that
    143 > had some extra support for Japanese.
    144 >
    145 > Regards, Gaute
    146 >
    147 -------------- next part --------------
    148 An HTML attachment was scrubbed...
    149 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130509/e4a66d23/attachment.html>
    150 
    151 From hsanson@gmail.com  Thu May  9 09:28:10 2013
    152 From: hsanson@gmail.com (Horacio Sanson)
    153 Date: Thu, 9 May 2013 18:28:10 +0900
    154 Subject: [sup-devel] Experimental Gmail Source
    155 Message-ID: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
    156 
    157 I am trying to implement a new source for Gmail accounts. This is copied
    158 from my efforts to do the same in Heliotrope.
    159 
    160 Here is an experimental implementation that can read the email from Gmail
    161 and add it to the Sup index:
    162 
    163 https://github.com/hsanson/sup/tree/gmail_source
    164 
    165 To use:
    166 
    167   - Install leveldb gem "sudo gem install leveldb-ruby"
    168   - Add a gmail source:   sup-add gmail://username at gmail.com
    169   - Start sup and see how it syncs your emails.
    170 
    171 Warnings:
    172 
    173   - This is experimental
    174   - This always syncs only the All mailbox so make sure to use an
    175      account with not too many emails for testing.
    176   - All email data and headers are stored in a LevelDB database at:
    177        ~/.sup/gmail/account
    178 
    179 Help:
    180 
    181   - How do I stop the source poll when I quit sup? If I have a large amount
    182     of emails when polling is running and I quit sup the process hangs
    183 there.
    184     Does sup has a mechanism to stop the polling in progress?
    185   - For some reason I get duplicate "Inbox" and "Sent" labels in the list of
    186     labels and I am not sure why.
    187   - I still have no clue on how to handle sync-back. That is how to
    188 propagate
    189     changes made in sup back to Gmail. Any tips on how the maildir source
    190     does it would be appreciated.
    191 
    192 regards
    193 Horacio Sanson
    194 -------------- next part --------------
    195 An HTML attachment was scrubbed...
    196 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130509/b112f3f5/attachment.html>
    197 
    198 From eric.weikl@gmx.net  Thu May  9 19:13:12 2013
    199 From: eric.weikl@gmx.net (Eric Weikl)
    200 Date: Thu, 09 May 2013 21:13:12 +0200
    201 Subject: [sup-devel] Experimental Gmail Source
    202 In-Reply-To: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
    203 References: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
    204 Message-ID: <1368126620-sup-137@mint>
    205 
    206 Hi Horacio,
    207 
    208 Excerpts from Horacio Sanson's message of 2013-05-09 11:28:10 +0200:
    209 >   - I still have no clue on how to handle sync-back. That is how to
    210 >     propagate changes made in sup back to Gmail. Any tips on how the
    211 >     maildir source does it would be appreciated.
    212 
    213 The maildir branch currently only syncs the flags supported by the
    214 maildir spec (by manipulating the file name, see "What can I put in
    215 info?"[1]). These flags can be synced back to GMail or other mail
    216 providers using offlineimap.
    217 
    218 To sync arbitrary labels, we would probably have to modify the original
    219 mail headers, like GMail seems to do with the X-GM-LABELS label (I only
    220 looked at your code, I don't have a GMail account).
    221 
    222 Gaute and Matthieu started gathering some ideas in the Wiki[2].
    223 
    224 Cheers,
    225 Eric
    226 
    227 Links:
    228 [1] http://cr.yp.to/proto/maildir.html
    229 [2] https://github.com/sup-heliotrope/sup/wiki/Development%3A-Maildir-syncback
    230 
    231 From hsanson@gmail.com  Sat May 11 00:30:08 2013
    232 From: hsanson@gmail.com (Horacio Sanson)
    233 Date: Sat, 11 May 2013 09:30:08 +0900
    234 Subject: [sup-devel] How to get labels from sup for a specific message id?
    235 Message-ID: <CAHWBo_Z-Wmqdq6z+b5Qcnic8sQSLMsh=3h9_C-n0LT8urbLW3A@mail.gmail.com>
    236 
    237 For my gmail source [1] I need to find out the labels for each message in
    238 sup. This is to update the remote Gmail with the labels the user set using
    239 sup.
    240 
    241 I am not very familiar with sup source code and my attempts to extract the
    242 message labels based on the message id have failed.
    243 
    244 My last try was:
    245 
    246 index = Redwood::Index.instance
    247 msg = index.build_message(uid)
    248 
    249 but this results in an error about Range not being converted to Integer.
    250 The uid is the id I use to register the message to sup during the poll
    251 method call.
    252 
    253 Any tips are appreciated
    254 Horacio
    255 
    256 [1] https://github.com/hsanson/sup/tree/gmail_source
    257 -------------- next part --------------
    258 An HTML attachment was scrubbed...
    259 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130511/00fb9918/attachment.html>
    260 
    261 From eg@gaute.vetsj.com  Sat May 11 10:15:07 2013
    262 From: eg@gaute.vetsj.com (Gaute Hope)
    263 Date: Sat, 11 May 2013 12:15:07 +0200
    264 Subject: [sup-devel] Experimental Gmail Source
    265 In-Reply-To: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
    266 References: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
    267 Message-ID: <518E1A2B.2080903@gaute.vetsj.com>
    268 
    269 
    270 
    271 On 09. mai 2013 11:28, Horacio Sanson wrote:
    272 > I am trying to implement a new source for Gmail accounts. This is 
    273 > copied from my efforts to do the same in Heliotrope.
    274 > 
    275 > Here is an experimental implementation that can read the email from 
    276 > Gmail and add it to the Sup index:
    277 > 
    278 > https://github.com/hsanson/sup/tree/gmail_source
    279 > 
    280 > To use:
    281 > 
    282 > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail 
    283 > source:   sup-add gmail://username at gmail.com - Start sup and see how 
    284 > it syncs your emails.
    285 > 
    286 > Warnings:
    287 > 
    288 > - This is experimental - This always syncs only the All mailbox so 
    289 > make sure to use an account with not too many emails for testing. - 
    290 > All email data and headers are stored in a LevelDB database at: 
    291 > ~/.sup/gmail/account
    292 > 
    293 
    294 > - For some reason I get duplicate "Inbox" and "Sent" labels in the 
    295 > list of labels and I am not sure why. 
    296 >  - I still have no clue on how  to handle sync-back. That is how to
    297 propagate
    298 > changes made in sup
    299 > back to Gmail. Any tips on how the maildir source does it would be 
    300 > appreciated.
    301 
    302 Hi Horacio,
    303 
    304 nice work. Working directly with GMail labels is probably a good idea
    305 (the other option is to move messages between IMAP folders). I have a
    306 design question though:
    307 
    308 Should remote sources be part of regular sup? Or should rather the
    309 fetching and syncing be put in a separate script which creates a LevelDB
    310 setup like you have it with a Gmail source in sup working directly on it?
    311 
    312 Recall that the IMAP source was removed in 52e29ba [1] (discussion
    313 probably on the mailinglist somewhere).
    314 
    315 I like this approach for GMail, but I would like to see it for regular
    316 IMAP sources as well with folders as labels.. I briefly experimented
    317 with a maildir-root folder approach [2] which treats all underlying
    318 maildirs as sources which correspond to a label (do not use,
    319 incomplete). It of course presents a plethora of questions on how to
    320 sync messages between labels, but implementing it is is probably
    321 relatively straight forward.
    322 
    323 > Help:
    324 > 
    325 > - How do I stop the source poll when I quit sup? If I have a large 
    326 > amount of emails when polling is running and I quit sup the process
    327 > hangs there. 
    328 
    329 This normally runs in a separate thread, I don't think maildir really
    330 stops the polling - so I sometimes get an error if I quit sup while the
    331 polling is running and various stuff just disappears underneath the poller.
    332 
    333 I don't have the chance to get into your other questions at the moment.
    334 
    335 Regards, Gaute
    336 
    337 [1] https://github.com/sup-heliotrope/sup/commit/52e29ba
    338 [2] https://github.com/gauteh/sup/tree/maildir-root
    339 
    340 From eg@gaute.vetsj.com  Sat May 11 10:38:16 2013
    341 From: eg@gaute.vetsj.com (Gaute Hope)
    342 Date: Sat, 11 May 2013 12:38:16 +0200
    343 Subject: [sup-devel] sup for 2.0
    344 Message-ID: <518E1F98.5070308@gaute.vetsj.com>
    345 
    346 Greetings suppers,
    347 
    348 I was thinking about setting up a branch for next generation sup, I have
    349 partly switched to Mail (in stead of RMail) and started migrating to
    350 using safe-ids in the index as opposed to the slightly modified
    351 message-id and message-refs. I have put this in a sup-for-two branch
    352 [1]. It is based on the discussion in issue 22 [2].
    353 
    354 Anyway, my proposal is: we set up a sup-for-two branch on origin. The
    355 things that should go in there are:
    356 - maildir-sync
    357 - Mail
    358 - Id -> safe id change
    359 - other big changes we want for sup-0.14 or the future
    360 
    361 The only mature feature here is the maildir-sync branch (which I have
    362 also based my Mail work on). I still think that we should merge in the
    363 Mail work pretty soon because the change of id to safeid (also for all
    364 refs) will be extra work to port later - given that we move to Mail.
    365 There are some issues and unfinished things still:
    366 - performance (about half of RMail)
    367 - only implemented for maildir, not for mbox or for sending
    368 - haven't figured out all threading logic [3]
    369 - migration script for index [4] is incomplete (and it might not even be
    370 really possible)
    371 
    372 Mail is UTF-8 (but crashes on too nasty input [5]), I've started to
    373 collect nasty emails for testing purposes.. we should add those to the
    374 test-suite somehow. So far most UTF-8 issues seems to have been solved
    375 by [6] and native UTF-8 in Mail. Tested on some 25k of email of
    376 extremely varying quality.
    377 
    378 Comments are appreciated.
    379 
    380 Regards, Gaute
    381 
    382 
    383 [1] https://github.com/gauteh/sup/tree/sup-for-two
    384 [2] https://github.com/sup-heliotrope/sup/issues/22
    385 [3] https://github.com/gauteh/sup/blob/sup-for-two/lib/sup/thread.rb#L299
    386 [4] https://github.com/gauteh/sup/blob/sup-for-two/bin/sup-migrate-index
    387 [5] https://github.com/mikel/mail/issues/544
    388 [6] https://github.com/gauteh/sup/blob/sup-for-two/lib/sup/util.rb#L344
    389 
    390 From hsanson@gmail.com  Sat May 11 18:18:46 2013
    391 From: hsanson@gmail.com (Horacio Sanson)
    392 Date: Sun, 12 May 2013 03:18:46 +0900
    393 Subject: [sup-devel] Experimental Gmail Source
    394 In-Reply-To: <518E1A2B.2080903@gaute.vetsj.com>
    395 References: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
    396 	<518E1A2B.2080903@gaute.vetsj.com>
    397 Message-ID: <CAHWBo_ZMce1GV=ChkLrSNbz9Lf-DtPb4W479kChXmUFMQf5qpw@mail.gmail.com>
    398 
    399 On Sat, May 11, 2013 at 7:15 PM, Gaute Hope <eg at gaute.vetsj.com> wrote:
    400 
    401 >
    402 >
    403 > On 09. mai 2013 11:28, Horacio Sanson wrote:
    404 > > I am trying to implement a new source for Gmail accounts. This is
    405 > > copied from my efforts to do the same in Heliotrope.
    406 > >
    407 > > Here is an experimental implementation that can read the email from
    408 > > Gmail and add it to the Sup index:
    409 > >
    410 > > https://github.com/hsanson/sup/tree/gmail_source
    411 > >
    412 > > To use:
    413 > >
    414 > > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail
    415 > > source:   sup-add gmail://username at gmail.com - Start sup and see how
    416 > > it syncs your emails.
    417 > >
    418 > > Warnings:
    419 > >
    420 > > - This is experimental - This always syncs only the All mailbox so
    421 > > make sure to use an account with not too many emails for testing. -
    422 > > All email data and headers are stored in a LevelDB database at:
    423 > > ~/.sup/gmail/account
    424 > >
    425 >
    426 > > - For some reason I get duplicate "Inbox" and "Sent" labels in the
    427 > > list of labels and I am not sure why.
    428 > >  - I still have no clue on how  to handle sync-back. That is how to
    429 > propagate
    430 > > changes made in sup
    431 > > back to Gmail. Any tips on how the maildir source does it would be
    432 > > appreciated.
    433 >
    434 > Hi Horacio,
    435 >
    436 > nice work. Working directly with GMail labels is probably a good idea
    437 > (the other option is to move messages between IMAP folders). I have a
    438 > design question though:
    439 >
    440 > Should remote sources be part of regular sup? Or should rather the
    441 > fetching and syncing be put in a separate script which creates a LevelDB
    442 > setup like you have it with a Gmail source in sup working directly on it?
    443 >
    444 
    445 
    446 I don't like the current two step sync setup that Sup uses now (IMAP ->
    447 Maildir -> Sup). It requires external programs (offlineimap) and
    448 synchronization is one way only. I know about the sync-back branch but
    449 AFAIK it is still limited to flags only. Also Maildir is an old storage
    450 format that doesn't work well with current email workflows. For example I
    451 have tons of duplicate emails on several folders due to the use of
    452 mailboxes rather than labels.
    453 
    454 I would prefer Sup to take care of the mail storage/indexing (as heliotrope
    455 does) and the sources be in charge of syncing the Sup storage/index with
    456 the remote servers. I understand that this can be difficult due to the
    457 difference in paradigm between IMAP/POP and Sup but GMail offers extensions
    458 that map directly to Sup workflow. GMail gives each email a unique 64bit
    459 ascending indentifier to all mail messages and adds labels them. I don't
    460 even need the LevelDB database as I could easily implement the source so it
    461 fetches the mail headers and body from the Gmail server directly when
    462 requested. I only added the LevelDB storage as a cache to speed up the
    463 message lookups and for offline use.
    464 
    465 
    466 > Recall that the IMAP source was removed in 52e29ba [1] (discussion
    467 > probably on the mailinglist somewhere).
    468 >
    469 >
    470 Yes I know this and reading the commits and source code of this source I
    471 can tell that William hated the IMAP protocol. This is understandable as I
    472 myself have dealt with this protocol and know first hand how broken it is.
    473 But again with the extensions supported by GMail servers the implementation
    474 is far easier to do.
    475 
    476 
    477 > I like this approach for GMail, but I would like to see it for regular
    478 > IMAP sources as well with folders as labels.. I briefly experimented
    479 > with a maildir-root folder approach [2] which treats all underlying
    480 > maildirs as sources which correspond to a label (do not use,
    481 > incomplete). It of course presents a plethora of questions on how to
    482 > sync messages between labels, but implementing it is is probably
    483 > relatively straight forward.
    484 >
    485 >
    486 IMAP is a horrible protocol and implementing it requires herculean effort.
    487 Still once the Gmail source is finished it can become a starting point for
    488 a more complete IMAP source.
    489 
    490 
    491 > > Help:
    492 > >
    493 > > - How do I stop the source poll when I quit sup? If I have a large
    494 > > amount of emails when polling is running and I quit sup the process
    495 > > hangs there.
    496 >
    497 > This normally runs in a separate thread, I don't think maildir really
    498 > stops the polling - so I sometimes get an error if I quit sup while the
    499 > polling is running and various stuff just disappears underneath the poller.
    500 >
    501 >
    502 I see that sup simply kills the threads... would be better if each source
    503 had a stop method that Sup could invoke to stop the polling. I will try to
    504 look into this issue as it is problematic for remote sources with large
    505 amounts of emails.
    506 
    507 I don't have the chance to get into your other questions at the moment.
    508 >
    509 >
    510 Thanks for answering my inquiries. If you have a chance I would really like
    511 to know how to get the labels for a specific email from the index. I need
    512 this to implement the sync-back part of the Gmail source.
    513 
    514 regards,
    515 Horacio
    516 
    517 
    518 > Regards, Gaute
    519 >
    520 > [1] https://github.com/sup-heliotrope/sup/commit/52e29ba
    521 > [2] https://github.com/gauteh/sup/tree/maildir-root
    522 -------------- next part --------------
    523 An HTML attachment was scrubbed...
    524 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130512/d8ebe0a6/attachment-0001.html>
    525 
    526 From eg@gaute.vetsj.com  Mon May 13 07:46:33 2013
    527 From: eg@gaute.vetsj.com (Gaute Hope)
    528 Date: Mon, 13 May 2013 09:46:33 +0200
    529 Subject: [sup-devel] sup 0.13 - please test
    530 In-Reply-To: <5188EB6E.8070504@gaute.vetsj.com>
    531 References: <CAOxvSbeLRoCwy2THkKa4TKsNqdsx9n-M5BxrZHGfPEUjfEWjSg@mail.gmail.com>
    532 	<5188EB6E.8070504@gaute.vetsj.com>
    533 Message-ID: <51909A59.4030003@gaute.vetsj.com>
    534 
    535 
    536 
    537 On 07. mai 2013 13:54, Gaute Hope wrote:
    538 > On 28. april 2013 22:32, Hamish D wrote:
    539 >> We'd like to get version 0.13 of sup out soon.  There's a fair bit of
    540 >> work gone into sup since 0.12 was released (in January 2011) that it
    541 >> would be good to get out there. Once that is done we can then work on
    542 >> the bigger goals we have for 0.14 - see
    543 >> https://github.com/sup-heliotrope/sup/wiki/Development#roadmap for
    544 >> those plans.
    545 >>
    546 >> So the question is, are there any serious bugs that we should fix
    547 >> before releasing 0.13?
    548 > 
    549 > Greetings sup-devel and sup-talk,
    550 > 
    551 > 
    552 > Thanks to several contributors I think we should be getting ready for
    553 > 0.13 now soon, we've removed a lot of old and deprecated stuff and
    554 > generally cleaned a bit up.
    555 > 
    556 > There is a new web page: http://supmua.org which is getting in shape and
    557 > a wiki which is quickly growing: https://github.com/sup-heliotrope/sup/wiki
    558 > 
    559 > We have forked our own version of ncursesw (ncursesw-sup) which now is a
    560 > dependency for sup [1].
    561 
    562 I think we are as good as we are going to get now. Unless something big
    563 comes up we will release 0.13 on Wednesday.
    564 
    565 Regards, Gaute
    566 
    567 
    568 From matthieu.rakotojaona@gmail.com  Mon May 13 23:15:31 2013
    569 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona)
    570 Date: Tue, 14 May 2013 01:15:31 +0200
    571 Subject: [sup-devel] sup 0.13 - please test
    572 In-Reply-To: <51909A59.4030003@gaute.vetsj.com>
    573 References: <CAOxvSbeLRoCwy2THkKa4TKsNqdsx9n-M5BxrZHGfPEUjfEWjSg@mail.gmail.com>
    574 	<5188EB6E.8070504@gaute.vetsj.com>
    575 	<51909A59.4030003@gaute.vetsj.com>
    576 Message-ID: <1368485829-sup-9105@kpad>
    577 
    578 Excerpts from Gaute Hope's message of lun. mai 13 09:46:33 +0200 2013:
    579 > 
    580 > On 07. mai 2013 13:54, Gaute Hope wrote:
    581 > > On 28. april 2013 22:32, Hamish D wrote:
    582 > >> We'd like to get version 0.13 of sup out soon.  There's a fair bit of
    583 > >> work gone into sup since 0.12 was released (in January 2011) that it
    584 > >> would be good to get out there. Once that is done we can then work on
    585 > >> the bigger goals we have for 0.14 - see
    586 > >> https://github.com/sup-heliotrope/sup/wiki/Development#roadmap for
    587 > >> those plans.
    588 > >>
    589 > >> So the question is, are there any serious bugs that we should fix
    590 > >> before releasing 0.13?
    591 > > 
    592 > > Greetings sup-devel and sup-talk,
    593 > > 
    594 > > 
    595 > > Thanks to several contributors I think we should be getting ready for
    596 > > 0.13 now soon, we've removed a lot of old and deprecated stuff and
    597 > > generally cleaned a bit up.
    598 > > 
    599 > > There is a new web page: http://supmua.org which is getting in shape and
    600 > > a wiki which is quickly growing: https://github.com/sup-heliotrope/sup/wiki
    601 > > 
    602 > > We have forked our own version of ncursesw (ncursesw-sup) which now is a
    603 > > dependency for sup [1].
    604 > 
    605 > I think we are as good as we are going to get now. Unless something big
    606 > comes up we will release 0.13 on Wednesday.
    607 > 
    608 > Regards, Gaute
    609 > 
    610 
    611 Guys you are awesome. I am running sup like a breeze from ruby 1.8 with
    612 the gem built from the gemspec. I could even use the gpgme gem to
    613 sign/encrypt/decrypt/verify some emails, so it really is a huge thing to
    614 me.
    615 
    616 I do have a problem running with ruby 1.9 though, with xapian not
    617 launching correctly. 
    618 
    619 -- 
    620 Matthieu Rakotojaona
    621 
    622 From hsanson@gmail.com  Wed May 15 02:10:22 2013
    623 From: hsanson@gmail.com (Horacio Sanson)
    624 Date: Wed, 15 May 2013 11:10:22 +0900
    625 Subject: [sup-devel] Experimental Gmail Source
    626 In-Reply-To: <CAHWBo_ZMce1GV=ChkLrSNbz9Lf-DtPb4W479kChXmUFMQf5qpw@mail.gmail.com>
    627 References: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
    628 	<518E1A2B.2080903@gaute.vetsj.com>
    629 	<CAHWBo_ZMce1GV=ChkLrSNbz9Lf-DtPb4W479kChXmUFMQf5qpw@mail.gmail.com>
    630 Message-ID: <CAHWBo_ZY7bWTxfqbQ1H3hu-q6mzKZm-YYe-+t0b-0M7xwUDfYQ@mail.gmail.com>
    631 
    632 Hello all,
    633 
    634 I am still trying to implement sync-back functionality to my GMail source
    635 but not going anywhere.
    636 The problem I have is that the Sup index keeps the messages ids provided by
    637 RMail (e.g. 20cf301af801a2aa8b04dc6e9931 at google.com) and not the id I get
    638 from the source that is the X-MSG-ID provided by Gmail
    639 (e.g. 1434541737393941768). I need a way to query the index using the Gmail
    640 provided id and get a message back or at least it's labels so I can compare
    641 and update them on the server side if required.
    642 
    643 This is what I have found so far:
    644 
    645   - There is no way to query the index for a message using the source
    646 (Gmail) provided id.
    647   - The only places I can see where the source message ids are stored is in
    648 a locations array
    649      that keeps the source own id and the source message id.
    650   - Sup index provides methods to query messages by id (build_message,
    651 contains_id?, etc)
    652     but the id they accept is the message id provided my RMail, not the id
    653 of the source.
    654 
    655 Questions:
    656 
    657   - What is the purpose of the locations array? To allow a single message
    658 to exists on
    659     multiple sources?
    660   - Is there an easy way to query the index to return a message/messages
    661 using the
    662     source provided id?
    663   - Is a good idea to add a term to the index that keeps the source id
    664 along with the message id?
    665 
    666 regards,
    667 Horacio
    668 
    669 
    670 
    671 
    672 On Sun, May 12, 2013 at 3:18 AM, Horacio Sanson <hsanson at gmail.com> wrote:
    673 
    674 >
    675 >
    676 >
    677 > On Sat, May 11, 2013 at 7:15 PM, Gaute Hope <eg at gaute.vetsj.com> wrote:
    678 >
    679 >>
    680 >>
    681 >> On 09. mai 2013 11:28, Horacio Sanson wrote:
    682 >> > I am trying to implement a new source for Gmail accounts. This is
    683 >> > copied from my efforts to do the same in Heliotrope.
    684 >> >
    685 >> > Here is an experimental implementation that can read the email from
    686 >> > Gmail and add it to the Sup index:
    687 >> >
    688 >> > https://github.com/hsanson/sup/tree/gmail_source
    689 >> >
    690 >> > To use:
    691 >> >
    692 >> > - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail
    693 >> > source:   sup-add gmail://username at gmail.com - Start sup and see how
    694 >> > it syncs your emails.
    695 >> >
    696 >> > Warnings:
    697 >> >
    698 >> > - This is experimental - This always syncs only the All mailbox so
    699 >> > make sure to use an account with not too many emails for testing. -
    700 >> > All email data and headers are stored in a LevelDB database at:
    701 >> > ~/.sup/gmail/account
    702 >> >
    703 >>
    704 >> > - For some reason I get duplicate "Inbox" and "Sent" labels in the
    705 >> > list of labels and I am not sure why.
    706 >> >  - I still have no clue on how  to handle sync-back. That is how to
    707 >> propagate
    708 >> > changes made in sup
    709 >> > back to Gmail. Any tips on how the maildir source does it would be
    710 >> > appreciated.
    711 >>
    712 >> Hi Horacio,
    713 >>
    714 >> nice work. Working directly with GMail labels is probably a good idea
    715 >> (the other option is to move messages between IMAP folders). I have a
    716 >> design question though:
    717 >>
    718 >> Should remote sources be part of regular sup? Or should rather the
    719 >> fetching and syncing be put in a separate script which creates a LevelDB
    720 >> setup like you have it with a Gmail source in sup working directly on it?
    721 >>
    722 >
    723 >
    724 > I don't like the current two step sync setup that Sup uses now (IMAP ->
    725 > Maildir -> Sup). It requires external programs (offlineimap) and
    726 > synchronization is one way only. I know about the sync-back branch but
    727 > AFAIK it is still limited to flags only. Also Maildir is an old storage
    728 > format that doesn't work well with current email workflows. For example I
    729 > have tons of duplicate emails on several folders due to the use of
    730 > mailboxes rather than labels.
    731 >
    732 > I would prefer Sup to take care of the mail storage/indexing (as
    733 > heliotrope does) and the sources be in charge of syncing the Sup
    734 > storage/index with the remote servers. I understand that this can be
    735 > difficult due to the difference in paradigm between IMAP/POP and Sup but
    736 > GMail offers extensions that map directly to Sup workflow. GMail gives each
    737 > email a unique 64bit ascending indentifier to all mail messages and adds
    738 > labels them. I don't even need the LevelDB database as I could easily
    739 > implement the source so it fetches the mail headers and body from the Gmail
    740 > server directly when requested. I only added the LevelDB storage as a cache
    741 > to speed up the message lookups and for offline use.
    742 >
    743 >
    744 >> Recall that the IMAP source was removed in 52e29ba [1] (discussion
    745 >> probably on the mailinglist somewhere).
    746 >>
    747 >>
    748 > Yes I know this and reading the commits and source code of this source I
    749 > can tell that William hated the IMAP protocol. This is understandable as I
    750 > myself have dealt with this protocol and know first hand how broken it is.
    751 > But again with the extensions supported by GMail servers the implementation
    752 > is far easier to do.
    753 >
    754 >
    755 >> I like this approach for GMail, but I would like to see it for regular
    756 >> IMAP sources as well with folders as labels.. I briefly experimented
    757 >> with a maildir-root folder approach [2] which treats all underlying
    758 >> maildirs as sources which correspond to a label (do not use,
    759 >> incomplete). It of course presents a plethora of questions on how to
    760 >> sync messages between labels, but implementing it is is probably
    761 >> relatively straight forward.
    762 >>
    763 >>
    764 > IMAP is a horrible protocol and implementing it requires herculean effort.
    765 > Still once the Gmail source is finished it can become a starting point for
    766 > a more complete IMAP source.
    767 >
    768 >
    769 >> > Help:
    770 >> >
    771 >> > - How do I stop the source poll when I quit sup? If I have a large
    772 >> > amount of emails when polling is running and I quit sup the process
    773 >> > hangs there.
    774 >>
    775 >> This normally runs in a separate thread, I don't think maildir really
    776 >> stops the polling - so I sometimes get an error if I quit sup while the
    777 >> polling is running and various stuff just disappears underneath the
    778 >> poller.
    779 >>
    780 >>
    781 > I see that sup simply kills the threads... would be better if each source
    782 > had a stop method that Sup could invoke to stop the polling. I will try to
    783 > look into this issue as it is problematic for remote sources with large
    784 > amounts of emails.
    785 >
    786 > I don't have the chance to get into your other questions at the moment.
    787 >>
    788 >>
    789 > Thanks for answering my inquiries. If you have a chance I would really
    790 > like to know how to get the labels for a specific email from the index. I
    791 > need this to implement the sync-back part of the Gmail source.
    792 >
    793 > regards,
    794 > Horacio
    795 >
    796 >
    797 >> Regards, Gaute
    798 >>
    799 >> [1] https://github.com/sup-heliotrope/sup/commit/52e29ba
    800 >> [2] https://github.com/gauteh/sup/tree/maildir-root
    801 >
    802 >
    803 -------------- next part --------------
    804 An HTML attachment was scrubbed...
    805 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130515/9233e342/attachment.html>
    806 
    807 From eg@gaute.vetsj.com  Wed May 15 08:30:27 2013
    808 From: eg@gaute.vetsj.com (Gaute Hope)
    809 Date: Wed, 15 May 2013 10:30:27 +0200
    810 Subject: [sup-devel] Release 0.13.0
    811 Message-ID: <519347A3.4070500@gaute.vetsj.com>
    812 
    813 -----BEGIN PGP SIGNED MESSAGE-----
    814 Hash: SHA1
    815 
    816 Greetings fellow Suppers,
    817 
    818 We have just released version 0.13.0 at Rubygems:
    819 
    820   https://rubygems.org/gems/sup
    821 
    822 to install:
    823 
    824   gem install sup
    825 
    826 for more information check out the wiki:
    827 
    828   https://github.com/sup-heliotrope/sup/wiki
    829 
    830 or the new web site:
    831 
    832   http://supmua.org
    833 
    834 
    835 Thanks a lot to all who have contributed over the last years since the
    836 previous release, and in particular all those who have been part of
    837 effort to get Sup going again and getting this release out as well as
    838 William Morgan who has let us keep working on Sup. I hope it will last!
    839 
    840 
    841 Please report bugs too:
    842 
    843   https://github.com/sup-heliotrope/sup/issues
    844 
    845 
    846 - - The Sup maintainers
    847 
    848 
    849 Note: The gem has been signed with my key (included below), note
    850 however that Rubygems signing is still only in its infancy.
    851 
    852 - -----BEGIN CERTIFICATE-----
    853 MIIDVDCCAjygAwIBAgIBADANBgkqhkiG9w0BAQUFADBQMQswCQYDVQQDDAJlZzEV
    854 MBMGCgmSJomT8ixkARkWBWdhdXRlMRUwEwYKCZImiZPyLGQBGRYFdmV0c2oxEzAR
    855 BgoJkiaJk/IsZAEZFgNjb20wHhcNMTMwNTA4MTAzODQ3WhcNMTQwNTA4MTAzODQ3
    856 WjBQMQswCQYDVQQDDAJlZzEVMBMGCgmSJomT8ixkARkWBWdhdXRlMRUwEwYKCZIm
    857 iZPyLGQBGRYFdmV0c2oxEzARBgoJkiaJk/IsZAEZFgNjb20wggEiMA0GCSqGSIb3
    858 DQEBAQUAA4IBDwAwggEKAoIBAQC7sNc5zY4MrYB7eywE/aK2IoDqpM9lq4ZFlHzt
    859 Pmq1LG6ah2lu/HfjqxiPoqwY7QkdSOGDLSk7G8YBqDA/tODhkPPSTqxBDzYyCO46
    860 haWTtoN5tJkxIDJKp1nVXHi0Mlb4GJVKd9P0q95BeBYBfs8vyPN+y4b4Gebgx9U3
    861 KqMDbe5h9MAPZGmtiRFMb3ugmiujDm7v8fACa5EtSvK/lxMkRDglecT/knE99NYI
    862 l35SO/Bune1bxYmkwW64mQ4wRlGVeAnX+19msALfS9rdJL26dfW2LgqWi5QoVTBH
    863 KNKTl/i3fxK0mzgtnoRCWdMJQFNNonFTnPUUawi1c9Kh4AdPAgMBAAGjOTA3MAkG
    864 A1UdEwQCMAAwHQYDVR0OBBYEFJNCOxL0SWcbW2M+DIEUzAMz1bZsMAsGA1UdDwQE
    865 AwIEsDANBgkqhkiG9w0BAQUFAAOCAQEAr3QUayd0geBDExO+WwzaEPAuUZ3zWQYG
    866 G9vrplCkmJtjS/X/wVAef7Jn/V5MNkXKXsiOgXJXki+n7HulNZUf1rzr7Un96gVJ
    867 1hq/ZTuapnPpstBqqdv60RB8HNGydHQeEz6us5z3nj+KchPqJ657Dz8oX/Nm6/24
    868 7QSQpCh8xBYdSWEpoIE0zUSY77LtVTRVwIr9uDpWTTr9kCVBINBsOQNjWKruEWjV
    869 +JMuDs+iWefpF4R3BySoOc1Q4WoES3+oc0qo37MsAZyfnQIPTZkyLZCMxeL6Mha4
    870 hFc2yANBj8voaY5C74Cg2VqExtcnSaxUtW9wC4w5hOlg0AVfb1JWzg==
    871 - -----END CERTIFICATE-----
    872 -----BEGIN PGP SIGNATURE-----
    873 Version: GnuPG v2.0.19 (GNU/Linux)
    874 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
    875 
    876 iQIcBAEBAgAGBQJRk0eiAAoJEJgnp+igdJAj58oP/2APxYsJ3RWtlhFGDk5IVSeW
    877 XNxw449fyRB1pnsyUvFvtkf0IU6SzVF/lV+Iq2mNIgGlsN1wKU5zk92Ic3cm0KAh
    878 dO8YmvQP46NV0UqZWSg6cu91cMbBOUD1KIpww633fEjN7N9fHuAO1BWZlQqpTt/S
    879 Y7bkpCCQWzwcY9yDtiZDp1qXQoLvxTg8epsIV1XwgGi9G/h0IAXZIG2VqmQl/80T
    880 0VteWfGHZ/PWa+34/101Q5uN8SXo+HLW2o4UD0lPSL0RRoQE0AkilbctzOQT6TE4
    881 lYmhPmgFDBoW5XWv73JxkS8y95BcYCUl2Q3w0KuyeYJeAIYWaU8WTCebYEBMnuud
    882 Lob97x/o+YA8SQylSnJxOyT8kTqvH4ZG7X2yEeJztJQaNagUlKIuvujF/s1lCaRi
    883 2O5kcI6UaAC6yu3XUwXESsXOQ0cUQnbG6P/CZtBXDFaeYqMHq5KSbjm99h+s8l6/
    884 97VQCxa2JYaaQb5wHtk0jrPAmpbA+EXJ4LzlMZeDMRf7+J1Ag8Y6XDRP2BN2p6Q8
    885 QKNoHrrPpCPZyXDxoMYHVoGl4UYXCS0n8jTdUG62b5fJjs9A5YayT4PiXtmxs/B3
    886 HYsDc9jGsoZyduJWI36pPQgc9wb65rpRLm1NMySokzp1zIydEmYZY8+Y+zXa2xba
    887 31ciZlyNkN0TO/JmU34G
    888 =Trft
    889 -----END PGP SIGNATURE-----
    890 
    891 From eg@gaute.vetsj.com  Mon May 20 07:24:23 2013
    892 From: eg@gaute.vetsj.com (Gaute Hope)
    893 Date: Mon, 20 May 2013 09:24:23 +0200
    894 Subject: [sup-devel] sup 0.13
    895 In-Reply-To: <CAHWBo_ZC176WjeMpPs_rJE9=41exnxxJ0ENfLpHRj1ZtOpg2Ng@mail.gmail.com>
    896 References: <CAOxvSbeLRoCwy2THkKa4TKsNqdsx9n-M5BxrZHGfPEUjfEWjSg@mail.gmail.com>
    897 	<1367233230-sup-3593@tesla> <517E5BD8.6030803@gaute.vetsj.com>
    898 	<CAHWBo_aKjp__dkjuJ-tYHpKJuv2G=k_jxTWMo0QyyUtvxZbGXA@mail.gmail.com>
    899 	<5182210E.4040100@gaute.vetsj.com>
    900 	<CAHWBo_ZC176WjeMpPs_rJE9=41exnxxJ0ENfLpHRj1ZtOpg2Ng@mail.gmail.com>
    901 Message-ID: <5199CFA7.9020900@gaute.vetsj.com>
    902 
    903 Hi,
    904 
    905 There has recently been opened an issue regarding this:
    906 
    907 https://github.com/sup-heliotrope/sup/issues/60
    908 
    909 Regards, Gaute
    910 
    911 On 09. mai 2013 03:39, Horacio Sanson wrote:
    912 > UTF-8 handles most cases but I still have to deal with emails in
    913 > ISO2022-JP, Shift-JIS and EUC-JP. After some research it seems Xapian has
    914 > no support for Asian languages. I will try to make some tests and open an
    915 > issue if I cannot make it work.
    916 > 
    917 > I can see in the sup configuration file that the stem language can be
    918 > configured but there are no CJK stemmers for Xapian that I can find.
    919 > 
    920 > 
    921 > On Thu, May 2, 2013 at 5:17 PM, Gaute Hope <eg at gaute.vetsj.com> wrote:
    922 > 
    923 >>
    924 >>
    925 >> On 30. april 2013 11:44, Horacio Sanson wrote:
    926 >>> Great to see Sup getting back on track again..
    927 >>>
    928 >>> I submitted some patches for the Gmail dumper of Heliotrope some time ago
    929 >>> but the lack of non alphabet languages (Japanese, Chinese) made it
    930 >>> impossible for me to keep using heliotrope/turnesole.
    931 >>>
    932 >>> The main issue to support Japanese/Chinese with heliotrope was that
    933 >>> whistlepig (indexer) lacked the ability to tokenize these languages. Also
    934 >>> the half baked UTF-8 support caused several issues with these languages.
    935 >>>
    936 >>> I would like to help in testing/implementing support for these languages,
    937 >>> starting with Japanese, but I would require some guidance. First I would
    938 >>> like to know is there is a way to configure the Xapian tokenizer
    939 >>> (segmenter) within sup? Please consider that I am new to both sup and to
    940 >>> Xapian.
    941 >>
    942 >> Hi Horacio,
    943 >>
    944 >> consider opening an issue at
    945 >> https://github.com/sup-heliotrope/sup/issues to make sure this doesn't
    946 >> disappear. Some changes will probably be made to the indexer when going
    947 >> to Mail (from RMail), but I hope to be able to migrate the existing
    948 >> index. Perhaps its time to get it right for arbitrary languages as well.
    949 >> I am unfamiliar with Japanes/Chinese - does UTF-8 cover the needs?
    950 >>
    951 >> Mail is better at handling UTF-8 and I think there was some fork that
    952 >> had some extra support for Japanese.
    953 >>
    954 >> Regards, Gaute
    955 >>
    956 > 
    957 
    958 From matthieu.rakotojaona@gmail.com  Tue May 21 21:47:16 2013
    959 From: matthieu.rakotojaona@gmail.com (Matthieu Rakotojaona)
    960 Date: Tue, 21 May 2013 23:47:16 +0200
    961 Subject: [sup-devel] Experimental Gmail Source
    962 In-Reply-To: <CAHWBo_ZY7bWTxfqbQ1H3hu-q6mzKZm-YYe-+t0b-0M7xwUDfYQ@mail.gmail.com>
    963 References: <CAHWBo_Y6jEWCH8O+m1iw5_p4qd8ghxgvCsa1+15xR7fh+6S0nA@mail.gmail.com>
    964 	<518E1A2B.2080903@gaute.vetsj.com>
    965 	<CAHWBo_ZMce1GV=ChkLrSNbz9Lf-DtPb4W479kChXmUFMQf5qpw@mail.gmail.com>
    966 	<CAHWBo_ZY7bWTxfqbQ1H3hu-q6mzKZm-YYe-+t0b-0M7xwUDfYQ@mail.gmail.com>
    967 Message-ID: <1369172802-sup-2003@kpad>
    968 
    969 Hey Horacio,
    970 
    971 I took a stab at your gmail_source branch, and made a few
    972 fixes/improvements [0]:
    973 
    974 - Add configuration option in sup-add
    975 - Dump the LevelDB path in the sources.yaml
    976 - Add a load_from_yaml method for a source to initialize its working
    977   values (for instance, the @db cannot be serialized, it needs to be
    978   reconstructed)
    979 - Fixed the msg_att monkey-patch for imap.rb
    980 
    981 All in all, the gmail source seems to work. I tested it on my usual
    982 gmail account, I haven't tried to download it all, but I did download a
    983 few dozens of emails without a problem. I'd like to warn users about
    984 LevelDB though: it's sad to say, but as other wmorgan's stuff, it looks
    985 abandoned. There are at least 2 bugs you will encounter if you try it: a
    986 pb in configuration (fixed in [1]) and you need the `snappy` gem to make
    987 it work if your db is more than 4MB large [2]. There are some up-to-date
    988 forks, though.
    989 
    990 I see LevelDB is used mostly for storing messages and mailboxes
    991 uid{validity/last}, but if we are to use gmail (it's the only IMAP
    992 provider that makes sense for sup), I believe we would stick to the All
    993 Mail label, right ? So, no need for storing this in db, rather in the
    994 sources.yaml file. Also, if leveldb-ruby is unreliable (I did encounter
    995 some issues way back about something with glibc...), and we want to use
    996 it for caching messages, I think we can salvage heliotrope's zmbox [3]
    997 because it's so simple to use yet far better than simple mbox.
    998 
    999 Regarding your ids questions, if you want to access the sup's messages
   1000 from the gmail source, you could use the mail's Message-ID header and
   1001 apply the same logic as in Message.sanitize_message_id. Caution,
   1002 however: I've already encountered the case where multiple messages in
   1003 GMail (i.e multiple X-GM-MSGID) have the same Message-ID, so they would
   1004 be considered the same in sup/heliotrope... yeah, that's annoying as
   1005 hell, and I don't know how we can solve this in the case of multiple
   1006 sources.
   1007 
   1008 If you want to sync-back, maybe sup can call a source-level "sync_back"
   1009 method with the current known state ? Speaking of which, for general
   1010 synchronization we could reuse the elegant offlineimap's sync algorithm
   1011 [4]. The idea is basic: have each source class store a snapshot of the
   1012 state.  When a message is modified on the source, diff the change with
   1013 the known status and propagate to sup; when a message is modified in
   1014 sup, diff with the known status and propagate to the source.
   1015 
   1016 Just a brain dump.
   1017 
   1018 [0] https://github.com/rakoo/sup/tree/gmail_source
   1019 [1] https://github.com/wmorgan/leveldb-ruby/pull/27
   1020 [2] https://github.com/wmorgan/leveldb-ruby/issues/23
   1021 [3] https://github.com/sup-heliotrope/heliotrope/blob/64d4b50d5649ec616a311a4cf6955137fdaeb13d/lib/heliotrope/zmbox.rb
   1022 [4] http://offlineimap.org/howitworks.html
   1023 
   1024 Regards,
   1025 
   1026 -- 
   1027 Matthieu Rakotojaona
   1028 -------------- next part --------------
   1029 A non-text attachment was scrubbed...
   1030 Name: signature.asc
   1031 Type: application/pgp-signature
   1032 Size: 836 bytes
   1033 Desc: not available
   1034 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20130521/1fc3b5b4/attachment.bin>
   1035