Subject: Re: [sup-talk] [sup-devel] heliotrope-syncback

From: Matthieu Rakotojaona <[email-masked]>
Date: Fri, 11 Nov 2011 02:25:59 +0100

On Thu, Nov 10, 2011 at 12:18 AM, William Morgan
<[email-masked]> wrote:
> Now THIS is exciting. Thanks for starting on it!

Yes it is. I have now finished (sort of) the FETCH operations. You can
grab it here :
https://github.com/rakoo/ximapd-heliotrope

I've struggled a lot with the RFC, not understanding half of it. I
don't know if the FETCH results are exact, but they seem to be. Please
feel free to test it and tell me what are the problems.

For those of you not aware  : the previous link leads you to a IMAP
interface for Heliotrope. I've only implemented the "read-only"
commands, so your messages won't have any problems, but you should be
able with any IMAP client (again, read-only).

Cya,

-- 
Matthieu RAKOTOJAONA
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Wed, 09 Nov 2011 23:18:08 +0000

Reformatted excerpts from Matthieu Rakotojaona's message of 2011-11-09:
> Okay, forget about buggy syncback, now comes the IMAP interface !

> Considering the implementation, I've considered only UIDs, which are
> heliotrope's message_ids. Both can and must only go up as messages
> come and go, so it's cool.

That's right. Thread ids can sometimes be non-increasing, but message
ids should always be increasing. And the IMAP interface shouldn't have
anything to do with thread ids anyways.

> This leads me to the question : what is the license of Heliotrope ?
> Ximapd seems to be something BSD-like, but Heliotrope has none.

Hm, good question. I will probably GPL it. Let me think for a few days.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Wed, 9 Nov 2011 04:22:51 +0100

Okay, forget about buggy syncback, now comes the IMAP interface ! I've
begun to hack ximapd[0], a ruby IMAP server, to work with heliotrope.
Unfortunately it is not supported anymore.

So far, I can connect with it (but with no security), list "mailboxes"
(just like GMail, they are just labels) select one and search for
something, all using imap4rev1 specs and, of course, heliotrope as a
backend. The most interesting parts like FETCHing and STORing are
still a few lines of code away. Of course, the goal is to use
something like offlineimap, Because It Works, to sync both mail stores
in a reliable fashion.

So if you want to test it, go to https://github.com/rakoo/ximapd-heliotrope.
This leads me to the question : what is the license of Heliotrope ?
Ximapd seems to be something BSD-like, but Heliotrope has none.

Cya !

[0] https://github.com/shugo/ximapd
-- 
Matthieu RAKOTOJAONA
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Thu, 3 Nov 2011 00:55:40 +0100

Hello everyone,

I am playing around with heliotrope, and I've managed to writo a ruby
script to sync changes you made there in your IMAP mailbox. It is
primarily intended to be used with GMail, but nothing prevents it from
working with others.

Basically, this is how it works :
- when you make a change in heliotrope, it is written in an external file
- the script reads the file line by line and applies back any changes

At the moment, I've only logged messages and state changes. When you
send a mail with GMail, it is automatically added in GMail "Sent"
mail, so I thought heliotrope could automagically thread it, but I
haven't tested yet.

The changes in thu imap folder happen like this :
- search for a thread with the "subject" we are interested in
- label/unlabel each message in the thread
- change state of each message as written in the log file

Many problems :
- first and foremost, the link between a thread in heliotrope and the
same in GMail is done through a basic IMAP search on the entire
"subject" string. I don't know about how the threading happens in
heliotrope, but I hope both are the same
- the ruby Net::IMAP library doesn't want to send non-ascii characters
( or I'm too dumb to see how to). So I decided to do it the hard way :
remove any word (blocks of chars separated by spaces) that contains
any non-ascii characters. Well, I,ve just removed one character for
the moment, because I had an error when trying to match a UTF-8 string
with a Unicode regexp.
- "states" should be message-centered, as explained in the comments,
yet I don't know how to link messages in heliotrope and messages in
GMail. And I don't think searching through the whole body of the
messages in GMail would be very fast. So, I changed the state of all
the messages in the thread. After all, when I have an "Unseen" message
in a thread, seeing the whole thread looks interesting; same for
"Starred".
- I've heard the ruby IMAP lib is buggy; what is it exactly ?

All in all, the most important part of the process is just the writing
of the logfile. (One of) My main problem with switching to heliotrope
entirely is the fact that it doesn't sync with anything else. So now,
if I have a log of everything I've done, I can at least do the same
changes by hand. And the script is to be run apart from everything
else, because I don't think it's part of heliotrope job to sync with
other mailserver. In fact, this script is more of a "fiddle around to
learn ruby" script, so please don't care about the code quality =]

Cya,

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Thu, 17 Nov 2011 02:33:07 +0100

Hello,

I'm advancing on the IMAP interface. I can almost use OfflineIMAP, but
there are a few things :

- I needed an other method in the heliotrope-client file :
add-message. It is exactly like send_message but doesn't send it.
- I needed to have the msgid of a msg if it  was already in the store,
so a few changes

Both of these are in the patch attached.

My problem now is that the addition of messages to the store doesn't
seem to work. I use the same things as in heliotrope-add :
force_encode("binary") if ...
@heliotropeclient.add_message ...

I send the whole rawbody to heliotrope. But when I want to see the
mail, the body is empty (turnsole says 0b, the web interface displays
only the header). I even used the RestClient directly, with no luck.
So where are my mistakes ?

---

From: Michael Stapelberg <[email-masked]>
Date: Mon, 07 Nov 2011 18:56:10 +0000

Hi,

Excerpts from William Morgan's message of 2011-11-07 06:57:19 +0000:
> In my experience, I haven't found any large bugs, just hard-to-debug
> issues in the error-handling, and general slowness (though that is
> probably attributable to the IMAP protocol itself).
When I switched from ruby IMAP to offlineimap, a few "new" messages popped up,
which were not in the index before. So my suspicion is that the IMAP backend
has been eating messages. Since I was switching away from it, I did not take
any time to further investigate, so only take it as an anecdote, not actual
evidence.

Best regards,
Michael

---

From: William Morgan <[email-masked]>
Date: Mon, 07 Nov 2011 06:57:19 +0000

Hi Matthieu,

Reformatted excerpts from Matthieu Rakotojaona's message of 2011-11-02:
> script to sync changes you made there in your IMAP mailbox.

Cool!

> When you

That should work.

> - search for a thread with the "subject" we are interested in

This is problematic. Message subjects are definitely not unique. You
should match up the Message-Id fields instead, if that's at all possible
with IMAP.

If it's not possible, well, IMAP does have a notion of a per-message
ids, but it's fraught with difficulties...

> I don't know about how the threading happens in heliotrope, but I hope
> both are the same

They will mostly be the same, but it's not guaranteed and you will see
some differences.

Keep up the good work!