Subject: [sup-devel] Query for largest msg_id?

From: Horacio Sanson <[email-masked]>
Date: Wed, 11 May 2011 00:17:47 +0900

Is there a way to query Heliotrope what is the largest msg_id
currently in the index? I am trying to improve the imap-dumper.rb so
it does not download all my emails every time but only the new ones.
>From rfc4549.txt (see section 4.3.1) I learned that the way to do this
is to FETCH all the messages from [ lastknownmsg_id + 1 : * ]. This
will give the ids of all messages added to the server after the last
sync. Reading the source code I could not find a way to query this
value.

Also while looking at the code I see that messages are stored in the
index using the msg_id as parsed by RMail. There is no further
association with the source or mailbox from where the messages were
downloaded. This I think may cause collisions if we use one Heliotrope
server with more than one email account. Not sure what is the
probability of two messages from two different IMAP servers having the
same msg_id but nothing in the standard rules out that possibility.

regards
Horacio
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Tue, 24 May 2011 12:16:24 +0900

On Tue, May 24, 2011 at 6:22 AM, Hamish <[email-masked]> wrote:
> Excerpts from Horacio Sanson's message of Fri May 20 16:38:30 +0100 2011:
>> On Wed, May 18, 2011 at 2:05 PM, William Morgan wrote:
>> > Reformatted excerpts from Horacio Sanson's message of 2011-05-17:
>> >> I implemented a new version of the GMail -> Heliotrope sync script and
>> >> attach it here in hopes someone will test it and provide some
>> >> feedback/comments.
>
> This sounds awesome. I've already downloaded my gmail email using
> offlineimap (into maildir format), and I'm wondering if it's possible to
> get your script to use the local copies first, and only then get the new
> emails from gmail - so as to save redownloading several hundred
> megabytes of email ...
>

Unfortunately this script is GMail specific and I make use of some
GMail only extensions (X-GM-LABELS, XLIST, etc)  that render the
script unusable for any other mail store.  My main motivation for
creating this script was to get rid of offlineimap as this would store
each message twice, once in offlineimap folders and once in heliotrope
store.

> I'm keen to try it out, but don't want to use it on my main account as I
> imagine it would take a long time. Maybe I'll just set up a second
> account with some test email to play with it.
>

Please do not use this script with your main account,...  It won't
affect your GMail data since I use read-only mode (IMAP examine) but
it still has some problems I need to handle before I can guarantee
that Heliotrope is 100% sync with GMail after running the script. For
instance sometimes GMail fails to send me the message body or timeouts
generating some IMAP exceptions I need to handle more gracefully.

If you have a test account with some messages any testing is greatly
appreciated...

> If it's just too hard to do, I'll just have to do the whole thing
> anyway, but it won't be too bad as just a one off job.
>
> Thanks for your work though
> Hamish

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 18 May 2011 23:14:00 -0400

Excerpts from Sascha Silbe's message of mié may 18 16:09:26 -0400 2011:
> Excerpts from William Morgan's message of Wed May 18 06:59:05 +0200 2011:
> > Reformatted excerpts from Horacio Sanson's message of 2011-05-16:
> > > When UIDVALIDITY differs I will simply re-scan the whole mailbox and
> > > feed it to Heliotrope. I trust Heliotrope won't add duplicates.
> > 
> 
> > The REST api will ignore duplicates based on the Message-Id header.
> 
> This is something I don't like about sup: There's no way to access the
> "duplicate" messages, e.g. to check whether an outgoing message was
> received back fine from the ML or to check the headers.
> 
> If heliotrope permanently erases the copies (instead of just not
> exposing UI to access them), that would be a major step backwards.

Same here.  I have this in detailed-headers.rb:

headers["Message Id"] = message.id
message.locations.each_with_index { |location, idx|
  headers["Location#{idx}"] = "#{location.source} #{location.info}"
}

So I can immediately see all locations of a message; particularly useful
for messages that I send.  I don't have easy access to the headers
within the sup UI, but the one or two times I've wanted to do that, I
simply pasted the path to the file to "less" (which works fine because I
use Maildirs; on mboxes it would be a lot more complex).

-- 
Álvaro Herrera <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Sun, 15 May 2011 15:01:41 +0000

Reformatted excerpts from Horacio Sanson's message of 2011-05-10:
> currently in the index?

Sort of---it's a hack, but if you search for e.g. "a OR -a" you'll get
every message in the index, and the first result will be the message
with the highest id thanks to Whistlepig's search semantics.

> I am trying to improve the imap-dumper.rb so

Sounds great. Unfortunately the Heliotrope message id and the IMAP
message id / message uid are completely different things, and
maintaining a cross-session mapping of them is impossible for generic
IMAP servers, because the uid of every message can change every time you
connect to an IMAP server---see the section on IMAP's 'uidvalidity'
variable. So you'll have to rescan the inbox every time and rebuild the
mapping. Welcome to hell.

> same msg_id but nothing in the standard rules out that possibility.

This is yet another id: the Message-Id header of the email. This is only
needed to build up the thread structure and should otherwise be ignored.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Sun, 22 May 2011 22:05:57 +0000

> exposing UI to access them), that would be a major step backwards.

What do you suggest as a way of exposing duplicates within the UI? I
worry that just keeping all copies of a message around will make the
thread structure messy. Is there a better alternative?
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Sat, 21 May 2011 00:38:30 +0900

On Wed, May 18, 2011 at 2:05 PM, William Morgan
<[email-masked]> wrote:
> Great! This is very exciting.
>
>> - I am seeing some negative thread_id's in the response. Need to check
>> if this is normal or a bug in Heliotrope or my script.
>
> This is normal. It indicates a thread in which the root message can't be
> found (so Heliotrope must construct a pseudo-message to hold the tree
> together). Nothing to worry about, except as an indicator that you're
> missing email refefred to by something else in the thread.
>

Is there a way to use GMail supplied thread ids?? there is an
extension X-GM-THRID that
provides such information.

>>   - GMail inbox is with a capital "I" (e.g. Inbox) while heliotrope
>> uses a small "i".
>>     Shall I down case all labels? or make a special treatment for Inbox?
>
> I suggest special-casing inbox.
>

Perfect, the new script special cases inbox

>> To use this script I had to modify heliotrope-server.rb to allow
>> setting labels and states when posting new messages (see attached
>> patch).
>
> I apologize for the confusion in this, but this is actually the wrong
> endpoint to use. POST "/message" (no .json) is for adding new emails.
> This endpoint was introduced temporarily for outgoing emails, i.e.  when
> you compose or reply via turnsole. But I'm going to rename it as well as
> implement it.
>

Great I rewrote the GMail -> Heliotrope script to use the improved
POST "/message"
and works wonderfully.

I am attaching the new version that is cleaner and seems to work
without problems.

> Anyways, I've added the relevant code to /message, so your script should
> work against that.
> --

---

From: William Morgan <[email-masked]>
Date: Wed, 18 May 2011 05:05:27 +0000

---

From: William Morgan <[email-masked]>
Date: Wed, 18 May 2011 04:59:05 +0000

> For example in the Heliotrope::Index the add_message method allows to
> insert a message and assign it labels, flags and extra parameters at
> the same time. How can I do this with the REST API? The only example I
> see only adds a message body.
> 
>     RestClient.post "http://localhost:8042/message", :message => body

I've just pushed a commit to fix this. You can now send labels, state
and extra params to the POST. See heliotrope-add for an example.

> Also for what purpose are the ext array used for? Can I use it to add
> an account/mailbox property to each message so I can latter retrieve
> all messages associated to a mailbox/account pair?

Anything you put in there will come back as part of a message info block
(e.g. when you GET /thread/123.json), but it's not indexed at all, so
you won't be able to get the list of matching messages.

Probably the easiest way to accomplish that is to make a unique label
for each mailbox/account pair, but that's not ideal. We can think of
ways to make this work.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Sascha Silbe <[email-masked]>
Date: Wed, 18 May 2011 22:09:26 +0200

Sascha

-- 
http://sascha.silbe.org/
http://www.infra-silbe.de/
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Sun, 22 May 2011 22:00:51 +0000

> provides such information.

Not really. Heliotrope has to maintain its own message ids and thread ids.
When you feed it messages, it threads things different from how Gmail does,
anyways.

> Great I rewrote the GMail -> Heliotrope script to use the improved

Great!

> I am attaching the new version that is cleaner and seems to work

Very nice. I would like to include something like this when I release
Heliotrope to the world at large, since I suspect this will be a popular
migration path. So thank you for laying the foundation.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Hamish <[email-masked]>
Date: Mon, 23 May 2011 22:22:03 +0100

---

From: Horacio Sanson <[email-masked]>
Date: Tue, 17 May 2011 23:59:34 +0900

I implemented a new version of the GMail -> Heliotrope sync script and
attach it here in hopes
someone will test it and provide some feedback/comments. This is by no
means ready for general
use so don't use for anything else than experimentation. But since
this script only opens mailboxes
in read only mode (examine) there should not be any problems with you
emails on the server.

This script is GMail specific and has these features:

  - Downloads emails from all mailboxes (except All, Trash and Spam)
automatically using
    the XLIST GMail IMAP extension and feeds them to Heliotrope via
REST interface.
  - Remembers the last email downloaded so it does not start from the beginning
    every time.
  - Synchronizes GMail labels using the X-GM-LABELS IMAP extension.
  - Synchronizes GMail flags with Heliotrope state flags.
  - Adds a new mailbox property to messages. This may allow later to implement
    Heliotrope -> GMail synchronization.

Things to check and do:
  - I am seeing some negative thread_id's in the response. Need to check if
    this is normal or a bug in Heliotrope or my script.
  - GMail inbox is with a capital "I" (e.g. Inbox) while heliotrope
uses a small "i".
    Shall I down case all labels? or make a special treatment for Inbox?
  - Refactor the script into something more modular and elegant.

To use this script I had to modify heliotrope-server.rb to allow
setting labels and states when
posting new messages (see attached patch).

regards,
Horacio

On Tue, May 17, 2011 at 12:02 AM, Horacio Sanson <[email-masked]> wrote:
> On Mon, May 16, 2011 at 12:01 AM, William Morgan
> Indeed I have been trying to wrap my head around the IMAP spec and
> still don't get a lot of things. For now I will just keep the max UID
> read from IMAP server somewhere on disk.
>
> I am attaching my first small hack for GMail <-> Heliotrope
> synchronization. For now it only downloads mail from GMail and injects
> them to Heliotrope just as the imap-dumper.rb does. The difference is
> that I keep track of the last message UID and UIDVALIDITY values to
> avoid re-scanning the whole folder every time.
>
> Now I wan't to take advantage of GMail IMAP extensions (e.g.
> X-GM-LABELS, X-GM-THRID) to allow labels/threads synchronization. But
> have some doubts about how to correctly use the Heliotrope REST API.
> regards,
From 5504d5732ebe3c8b86b942b9b898221a520d283f Mon Sep 17 00:00:00 2001
From: Horacio Sanson <[email-masked]>
Date: Tue, 17 May 2011 23:34:14 +0900
Subject: [PATCH] Implement post /message.json.

Now we can  add messages to heliotrope with labels, state and mailbox
information in a single POST request. The request body has to be JSON
with a format like:

{
  body: <rfc822 raw message string>,
  labels: ["inbox", "school", "home"],
  state: ["seen", "old"],
  mailbox: "inbox"
}
---
 bin/heliotrope-server |   30 +++++++++++++++++++++++++++---
 1 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/bin/heliotrope-server b/bin/heliotrope-server
index d71989d..93b0490 100644
--- a/bin/heliotrope-server
+++ b/bin/heliotrope-server
@@ -302,10 +302,34 @@ class HeliotropeServer < Sinatra::Base
   end
 
   post "/message.json" do
-    body = params["body"] or raise RequestError, "need a 'body' param"
-    puts body
+    content_type :json
+    begin
+      rawbody = params["body"] or raise RequestError, "need a 'body' param"
+      rawbody.force_encoding "binary" if rawbody.respond_to?(:force_encoding) # sigh...
 
-    {:status => "ok"}.to_json
+      message = Heliotrope::Message.new(rawbody).parse!
+
+      mbox = params["mailbox"] || "inbox"
+      doc_id = nil
+      thread_id = nil
+      state = nil
+      labels = [mbox]
+      if @index.contains_msgid? message.msgid
+        messageinfo = get_message_summary message.msgid
+        doc_id =  messageinfo[:message_id]
+        thread_id = messageinfo[:thread_id]
+        { :response => :ok, :status => :seen, :doc_id => doc_id, :thread_id => thread_id }
+      else
+        # Set state
+        state params["state"] || ["unseen"]
+        labels.push(params["labels"]).flatten!.uniq! if params["labels"]
+        loc = @store.add rawbody
+        doc_id, thread_id = @index.add_message message, state, labels, { :loc => loc, :mbox => mbox }
+        { :response => :ok, :status => state, :doc_id => doc_id, :thread_id => thread_id, :labels => labels }
+      end
+    rescue Heliotrope::InvalidMessageError => e
+      { :response => :error, :error_message => e.message }
+    end.to_json
   end
 
 private
-- 
1.7.4.1

---

From: Gaudenz Steinlin <[email-masked]>
Date: Mon, 23 May 2011 09:52:43 +0200

> thread structure messy. Is there a better alternative?

Merging the messages is fine and actually an improvement over other
MUAs. But adding a line in the header section indicating the source
and how often the message appears in the source would be nice. Like

Sources: maildir://bla/blu (2), maildir://foo/bar (1)

Also if you display the message in the raw form (V key in sup), then
all the copies could be shown in full. This would help diagnosing
why you receive duplicates of certain messages. 

Gaudenz

-- 
Ever tried. Ever failed. No matter.
Try again. Fail again. Fail better.
~ Samuel Beckett ~
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Tue, 17 May 2011 00:02:39 +0900

regards,
Horacio

---

From: William Morgan <[email-masked]>
Date: Mon, 23 May 2011 16:14:53 +0000

Reformatted excerpts from Gaudenz Steinlin's message of 2011-05-23:
> MUAs.

I agree. I like the behavior.

> But adding a line in the header section indicating the source
> and how often the message appears in the source would be nice.

This is essentially what Alvaro's detailed-headers hook does. But unlike
Sup, heliotrope doesn't have the notion of different mailstores. There's
just one.

With some effort, we could add some kind of differentiating information
at mail addition time, but it's hard to know what would fit the general
case. A list of (from, timestamp) tuples?
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Horacio Sanson <[email-masked]>
Date: Wed, 18 May 2011 00:15:48 +0900

Sorry the patch I sent has a small problem. Use this patch instead.

>> all messages associated to a mailbox/account pair?
From 265b25dbc40b20e54163ac3688125c6dcc12867c Mon Sep 17 00:00:00 2001
From: Horacio Sanson <[email-masked]>
Date: Tue, 17 May 2011 23:34:14 +0900
Subject: [PATCH] Implement post /message.json.

{

index d71989d..bfa8fdd 100644
 
   post "/message.json" do
 
+        state = params["state"] || ["unseen"]
 
 private
-- 
1.7.4.1