* [sup-devel] Query for largest msg_id?
@ 2011-05-10 15:17 Horacio Sanson
2011-05-15 15:01 ` William Morgan
0 siblings, 1 reply; 16+ messages in thread
From: Horacio Sanson @ 2011-05-10 15:17 UTC (permalink / raw)
To: sup-devel
Is there a way to query Heliotrope what is the largest msg_id
currently in the index? I am trying to improve the imap-dumper.rb so
it does not download all my emails every time but only the new ones.
From rfc4549.txt (see section 4.3.1) I learned that the way to do this
is to FETCH all the messages from [ lastknownmsg_id + 1 : * ]. This
will give the ids of all messages added to the server after the last
sync. Reading the source code I could not find a way to query this
value.
Also while looking at the code I see that messages are stored in the
index using the msg_id as parsed by RMail. There is no further
association with the source or mailbox from where the messages were
downloaded. This I think may cause collisions if we use one Heliotrope
server with more than one email account. Not sure what is the
probability of two messages from two different IMAP servers having the
same msg_id but nothing in the standard rules out that possibility.
regards
Horacio
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-10 15:17 [sup-devel] Query for largest msg_id? Horacio Sanson
@ 2011-05-15 15:01 ` William Morgan
2011-05-16 15:02 ` Horacio Sanson
0 siblings, 1 reply; 16+ messages in thread
From: William Morgan @ 2011-05-15 15:01 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-05-10:
> Is there a way to query Heliotrope what is the largest msg_id
> currently in the index?
Sort of---it's a hack, but if you search for e.g. "a OR -a" you'll get
every message in the index, and the first result will be the message
with the highest id thanks to Whistlepig's search semantics.
> I am trying to improve the imap-dumper.rb so
> it does not download all my emails every time but only the new ones.
Sounds great. Unfortunately the Heliotrope message id and the IMAP
message id / message uid are completely different things, and
maintaining a cross-session mapping of them is impossible for generic
IMAP servers, because the uid of every message can change every time you
connect to an IMAP server---see the section on IMAP's 'uidvalidity'
variable. So you'll have to rescan the inbox every time and rebuild the
mapping. Welcome to hell.
> Also while looking at the code I see that messages are stored in the
> index using the msg_id as parsed by RMail. There is no further
> association with the source or mailbox from where the messages were
> downloaded. This I think may cause collisions if we use one Heliotrope
> server with more than one email account. Not sure what is the
> probability of two messages from two different IMAP servers having the
> same msg_id but nothing in the standard rules out that possibility.
This is yet another id: the Message-Id header of the email. This is only
needed to build up the thread structure and should otherwise be ignored.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-15 15:01 ` William Morgan
@ 2011-05-16 15:02 ` Horacio Sanson
2011-05-17 14:59 ` Horacio Sanson
2011-05-18 4:59 ` William Morgan
0 siblings, 2 replies; 16+ messages in thread
From: Horacio Sanson @ 2011-05-16 15:02 UTC (permalink / raw)
To: Sup developer discussion
[-- Attachment #1: Type: text/plain, Size: 3224 bytes --]
On Mon, May 16, 2011 at 12:01 AM, William Morgan
<wmorgan-sup@masanjin.net> wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-05-10:
>> Is there a way to query Heliotrope what is the largest msg_id
>> currently in the index?
>
> Sort of---it's a hack, but if you search for e.g. "a OR -a" you'll get
> every message in the index, and the first result will be the message
> with the highest id thanks to Whistlepig's search semantics.
>
Indeed I have been trying to wrap my head around the IMAP spec and
still don't get a lot of things. For now I will just keep the max UID
read from IMAP server somewhere on disk.
>> I am trying to improve the imap-dumper.rb so
>> it does not download all my emails every time but only the new ones.
>
> Sounds great. Unfortunately the Heliotrope message id and the IMAP
> message id / message uid are completely different things, and
> maintaining a cross-session mapping of them is impossible for generic
> IMAP servers, because the uid of every message can change every time you
> connect to an IMAP server---see the section on IMAP's 'uidvalidity'
> variable. So you'll have to rescan the inbox every time and rebuild the
> mapping. Welcome to hell.
>
When UIDVALIDITY differs I will simply re-scan the whole mailbox and
feed it to Heliotrope. I trust Heliotrope won't add duplicates.
>> Also while looking at the code I see that messages are stored in the
>> index using the msg_id as parsed by RMail. There is no further
>> association with the source or mailbox from where the messages were
>> downloaded. This I think may cause collisions if we use one Heliotrope
>> server with more than one email account. Not sure what is the
>> probability of two messages from two different IMAP servers having the
>> same msg_id but nothing in the standard rules out that possibility.
>
> This is yet another id: the Message-Id header of the email. This is only
> needed to build up the thread structure and should otherwise be ignored.
I am attaching my first small hack for GMail <-> Heliotrope
synchronization. For now it only downloads mail from GMail and injects
them to Heliotrope just as the imap-dumper.rb does. The difference is
that I keep track of the last message UID and UIDVALIDITY values to
avoid re-scanning the whole folder every time.
Now I wan't to take advantage of GMail IMAP extensions (e.g.
X-GM-LABELS, X-GM-THRID) to allow labels/threads synchronization. But
have some doubts about how to correctly use the Heliotrope REST API.
For example in the Heliotrope::Index the add_message method allows to
insert a message and assign it labels, flags and extra parameters at
the same time. How can I do this with the REST API? The only example I
see only adds a message body.
RestClient.post "http://localhost:8042/message", :message => body
Also for what purpose are the ext array used for? Can I use it to add
an account/mailbox property to each message so I can latter retrieve
all messages associated to a mailbox/account pair?
regards,
Horacio
> --
> William <wmorgan-sup@masanjin.net>
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
[-- Attachment #2: gmail.rb --]
[-- Type: application/x-ruby, Size: 7797 bytes --]
[-- Attachment #3: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-16 15:02 ` Horacio Sanson
@ 2011-05-17 14:59 ` Horacio Sanson
2011-05-17 15:15 ` Horacio Sanson
2011-05-18 5:05 ` William Morgan
2011-05-18 4:59 ` William Morgan
1 sibling, 2 replies; 16+ messages in thread
From: Horacio Sanson @ 2011-05-17 14:59 UTC (permalink / raw)
To: Sup developer discussion
[-- Attachment #1: Type: text/plain, Size: 5000 bytes --]
I implemented a new version of the GMail -> Heliotrope sync script and
attach it here in hopes
someone will test it and provide some feedback/comments. This is by no
means ready for general
use so don't use for anything else than experimentation. But since
this script only opens mailboxes
in read only mode (examine) there should not be any problems with you
emails on the server.
This script is GMail specific and has these features:
- Downloads emails from all mailboxes (except All, Trash and Spam)
automatically using
the XLIST GMail IMAP extension and feeds them to Heliotrope via
REST interface.
- Remembers the last email downloaded so it does not start from the beginning
every time.
- Synchronizes GMail labels using the X-GM-LABELS IMAP extension.
- Synchronizes GMail flags with Heliotrope state flags.
- Adds a new mailbox property to messages. This may allow later to implement
Heliotrope -> GMail synchronization.
Things to check and do:
- I am seeing some negative thread_id's in the response. Need to check if
this is normal or a bug in Heliotrope or my script.
- GMail inbox is with a capital "I" (e.g. Inbox) while heliotrope
uses a small "i".
Shall I down case all labels? or make a special treatment for Inbox?
- Refactor the script into something more modular and elegant.
To use this script I had to modify heliotrope-server.rb to allow
setting labels and states when
posting new messages (see attached patch).
regards,
Horacio
On Tue, May 17, 2011 at 12:02 AM, Horacio Sanson <hsanson@gmail.com> wrote:
> On Mon, May 16, 2011 at 12:01 AM, William Morgan
> <wmorgan-sup@masanjin.net> wrote:
>> Reformatted excerpts from Horacio Sanson's message of 2011-05-10:
>>> Is there a way to query Heliotrope what is the largest msg_id
>>> currently in the index?
>>
>> Sort of---it's a hack, but if you search for e.g. "a OR -a" you'll get
>> every message in the index, and the first result will be the message
>> with the highest id thanks to Whistlepig's search semantics.
>>
>
> Indeed I have been trying to wrap my head around the IMAP spec and
> still don't get a lot of things. For now I will just keep the max UID
> read from IMAP server somewhere on disk.
>
>>> I am trying to improve the imap-dumper.rb so
>>> it does not download all my emails every time but only the new ones.
>>
>> Sounds great. Unfortunately the Heliotrope message id and the IMAP
>> message id / message uid are completely different things, and
>> maintaining a cross-session mapping of them is impossible for generic
>> IMAP servers, because the uid of every message can change every time you
>> connect to an IMAP server---see the section on IMAP's 'uidvalidity'
>> variable. So you'll have to rescan the inbox every time and rebuild the
>> mapping. Welcome to hell.
>>
>
> When UIDVALIDITY differs I will simply re-scan the whole mailbox and
> feed it to Heliotrope. I trust Heliotrope won't add duplicates.
>
>>> Also while looking at the code I see that messages are stored in the
>>> index using the msg_id as parsed by RMail. There is no further
>>> association with the source or mailbox from where the messages were
>>> downloaded. This I think may cause collisions if we use one Heliotrope
>>> server with more than one email account. Not sure what is the
>>> probability of two messages from two different IMAP servers having the
>>> same msg_id but nothing in the standard rules out that possibility.
>>
>> This is yet another id: the Message-Id header of the email. This is only
>> needed to build up the thread structure and should otherwise be ignored.
>
> I am attaching my first small hack for GMail <-> Heliotrope
> synchronization. For now it only downloads mail from GMail and injects
> them to Heliotrope just as the imap-dumper.rb does. The difference is
> that I keep track of the last message UID and UIDVALIDITY values to
> avoid re-scanning the whole folder every time.
>
> Now I wan't to take advantage of GMail IMAP extensions (e.g.
> X-GM-LABELS, X-GM-THRID) to allow labels/threads synchronization. But
> have some doubts about how to correctly use the Heliotrope REST API.
> For example in the Heliotrope::Index the add_message method allows to
> insert a message and assign it labels, flags and extra parameters at
> the same time. How can I do this with the REST API? The only example I
> see only adds a message body.
>
> RestClient.post "http://localhost:8042/message", :message => body
>
> Also for what purpose are the ext array used for? Can I use it to add
> an account/mailbox property to each message so I can latter retrieve
> all messages associated to a mailbox/account pair?
>
> regards,
> Horacio
>
>> --
>> William <wmorgan-sup@masanjin.net>
>> _______________________________________________
>> Sup-devel mailing list
>> Sup-devel@rubyforge.org
>> http://rubyforge.org/mailman/listinfo/sup-devel
>>
>
[-- Attachment #2: gmail.rb --]
[-- Type: application/x-ruby, Size: 8704 bytes --]
[-- Attachment #3: 0001-Implement-post-message.json.patch --]
[-- Type: text/x-patch, Size: 2204 bytes --]
From 5504d5732ebe3c8b86b942b9b898221a520d283f Mon Sep 17 00:00:00 2001
From: Horacio Sanson <hsanson@gmail.com>
Date: Tue, 17 May 2011 23:34:14 +0900
Subject: [PATCH] Implement post /message.json.
Now we can add messages to heliotrope with labels, state and mailbox
information in a single POST request. The request body has to be JSON
with a format like:
{
body: <rfc822 raw message string>,
labels: ["inbox", "school", "home"],
state: ["seen", "old"],
mailbox: "inbox"
}
---
bin/heliotrope-server | 30 +++++++++++++++++++++++++++---
1 files changed, 27 insertions(+), 3 deletions(-)
diff --git a/bin/heliotrope-server b/bin/heliotrope-server
index d71989d..93b0490 100644
--- a/bin/heliotrope-server
+++ b/bin/heliotrope-server
@@ -302,10 +302,34 @@ class HeliotropeServer < Sinatra::Base
end
post "/message.json" do
- body = params["body"] or raise RequestError, "need a 'body' param"
- puts body
+ content_type :json
+ begin
+ rawbody = params["body"] or raise RequestError, "need a 'body' param"
+ rawbody.force_encoding "binary" if rawbody.respond_to?(:force_encoding) # sigh...
- {:status => "ok"}.to_json
+ message = Heliotrope::Message.new(rawbody).parse!
+
+ mbox = params["mailbox"] || "inbox"
+ doc_id = nil
+ thread_id = nil
+ state = nil
+ labels = [mbox]
+ if @index.contains_msgid? message.msgid
+ messageinfo = get_message_summary message.msgid
+ doc_id = messageinfo[:message_id]
+ thread_id = messageinfo[:thread_id]
+ { :response => :ok, :status => :seen, :doc_id => doc_id, :thread_id => thread_id }
+ else
+ # Set state
+ state params["state"] || ["unseen"]
+ labels.push(params["labels"]).flatten!.uniq! if params["labels"]
+ loc = @store.add rawbody
+ doc_id, thread_id = @index.add_message message, state, labels, { :loc => loc, :mbox => mbox }
+ { :response => :ok, :status => state, :doc_id => doc_id, :thread_id => thread_id, :labels => labels }
+ end
+ rescue Heliotrope::InvalidMessageError => e
+ { :response => :error, :error_message => e.message }
+ end.to_json
end
private
--
1.7.4.1
[-- Attachment #4: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-17 14:59 ` Horacio Sanson
@ 2011-05-17 15:15 ` Horacio Sanson
2011-05-18 5:05 ` William Morgan
1 sibling, 0 replies; 16+ messages in thread
From: Horacio Sanson @ 2011-05-17 15:15 UTC (permalink / raw)
To: Sup developer discussion
[-- Attachment #1: Type: text/plain, Size: 5282 bytes --]
Sorry the patch I sent has a small problem. Use this patch instead.
On Tue, May 17, 2011 at 11:59 PM, Horacio Sanson <hsanson@gmail.com> wrote:
> I implemented a new version of the GMail -> Heliotrope sync script and
> attach it here in hopes
> someone will test it and provide some feedback/comments. This is by no
> means ready for general
> use so don't use for anything else than experimentation. But since
> this script only opens mailboxes
> in read only mode (examine) there should not be any problems with you
> emails on the server.
>
> This script is GMail specific and has these features:
>
> - Downloads emails from all mailboxes (except All, Trash and Spam)
> automatically using
> the XLIST GMail IMAP extension and feeds them to Heliotrope via
> REST interface.
> - Remembers the last email downloaded so it does not start from the beginning
> every time.
> - Synchronizes GMail labels using the X-GM-LABELS IMAP extension.
> - Synchronizes GMail flags with Heliotrope state flags.
> - Adds a new mailbox property to messages. This may allow later to implement
> Heliotrope -> GMail synchronization.
>
> Things to check and do:
> - I am seeing some negative thread_id's in the response. Need to check if
> this is normal or a bug in Heliotrope or my script.
> - GMail inbox is with a capital "I" (e.g. Inbox) while heliotrope
> uses a small "i".
> Shall I down case all labels? or make a special treatment for Inbox?
> - Refactor the script into something more modular and elegant.
>
> To use this script I had to modify heliotrope-server.rb to allow
> setting labels and states when
> posting new messages (see attached patch).
>
> regards,
> Horacio
>
> On Tue, May 17, 2011 at 12:02 AM, Horacio Sanson <hsanson@gmail.com> wrote:
>> On Mon, May 16, 2011 at 12:01 AM, William Morgan
>> <wmorgan-sup@masanjin.net> wrote:
>>> Reformatted excerpts from Horacio Sanson's message of 2011-05-10:
>>>> Is there a way to query Heliotrope what is the largest msg_id
>>>> currently in the index?
>>>
>>> Sort of---it's a hack, but if you search for e.g. "a OR -a" you'll get
>>> every message in the index, and the first result will be the message
>>> with the highest id thanks to Whistlepig's search semantics.
>>>
>>
>> Indeed I have been trying to wrap my head around the IMAP spec and
>> still don't get a lot of things. For now I will just keep the max UID
>> read from IMAP server somewhere on disk.
>>
>>>> I am trying to improve the imap-dumper.rb so
>>>> it does not download all my emails every time but only the new ones.
>>>
>>> Sounds great. Unfortunately the Heliotrope message id and the IMAP
>>> message id / message uid are completely different things, and
>>> maintaining a cross-session mapping of them is impossible for generic
>>> IMAP servers, because the uid of every message can change every time you
>>> connect to an IMAP server---see the section on IMAP's 'uidvalidity'
>>> variable. So you'll have to rescan the inbox every time and rebuild the
>>> mapping. Welcome to hell.
>>>
>>
>> When UIDVALIDITY differs I will simply re-scan the whole mailbox and
>> feed it to Heliotrope. I trust Heliotrope won't add duplicates.
>>
>>>> Also while looking at the code I see that messages are stored in the
>>>> index using the msg_id as parsed by RMail. There is no further
>>>> association with the source or mailbox from where the messages were
>>>> downloaded. This I think may cause collisions if we use one Heliotrope
>>>> server with more than one email account. Not sure what is the
>>>> probability of two messages from two different IMAP servers having the
>>>> same msg_id but nothing in the standard rules out that possibility.
>>>
>>> This is yet another id: the Message-Id header of the email. This is only
>>> needed to build up the thread structure and should otherwise be ignored.
>>
>> I am attaching my first small hack for GMail <-> Heliotrope
>> synchronization. For now it only downloads mail from GMail and injects
>> them to Heliotrope just as the imap-dumper.rb does. The difference is
>> that I keep track of the last message UID and UIDVALIDITY values to
>> avoid re-scanning the whole folder every time.
>>
>> Now I wan't to take advantage of GMail IMAP extensions (e.g.
>> X-GM-LABELS, X-GM-THRID) to allow labels/threads synchronization. But
>> have some doubts about how to correctly use the Heliotrope REST API.
>> For example in the Heliotrope::Index the add_message method allows to
>> insert a message and assign it labels, flags and extra parameters at
>> the same time. How can I do this with the REST API? The only example I
>> see only adds a message body.
>>
>> RestClient.post "http://localhost:8042/message", :message => body
>>
>> Also for what purpose are the ext array used for? Can I use it to add
>> an account/mailbox property to each message so I can latter retrieve
>> all messages associated to a mailbox/account pair?
>>
>> regards,
>> Horacio
>>
>>> --
>>> William <wmorgan-sup@masanjin.net>
>>> _______________________________________________
>>> Sup-devel mailing list
>>> Sup-devel@rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/sup-devel
>>>
>>
>
[-- Attachment #2: 0001-Implement-post-message.json.patch --]
[-- Type: text/x-patch, Size: 2206 bytes --]
From 265b25dbc40b20e54163ac3688125c6dcc12867c Mon Sep 17 00:00:00 2001
From: Horacio Sanson <hsanson@gmail.com>
Date: Tue, 17 May 2011 23:34:14 +0900
Subject: [PATCH] Implement post /message.json.
Now we can add messages to heliotrope with labels, state and mailbox
information in a single POST request. The request body has to be JSON
with a format like:
{
body: <rfc822 raw message string>,
labels: ["inbox", "school", "home"],
state: ["seen", "old"],
mailbox: "inbox"
}
---
bin/heliotrope-server | 30 +++++++++++++++++++++++++++---
1 files changed, 27 insertions(+), 3 deletions(-)
diff --git a/bin/heliotrope-server b/bin/heliotrope-server
index d71989d..bfa8fdd 100644
--- a/bin/heliotrope-server
+++ b/bin/heliotrope-server
@@ -302,10 +302,34 @@ class HeliotropeServer < Sinatra::Base
end
post "/message.json" do
- body = params["body"] or raise RequestError, "need a 'body' param"
- puts body
+ content_type :json
+ begin
+ rawbody = params["body"] or raise RequestError, "need a 'body' param"
+ rawbody.force_encoding "binary" if rawbody.respond_to?(:force_encoding) # sigh...
- {:status => "ok"}.to_json
+ message = Heliotrope::Message.new(rawbody).parse!
+
+ mbox = params["mailbox"] || "inbox"
+ doc_id = nil
+ thread_id = nil
+ state = nil
+ labels = [mbox]
+ if @index.contains_msgid? message.msgid
+ messageinfo = get_message_summary message.msgid
+ doc_id = messageinfo[:message_id]
+ thread_id = messageinfo[:thread_id]
+ { :response => :ok, :status => :seen, :doc_id => doc_id, :thread_id => thread_id }
+ else
+ # Set state
+ state = params["state"] || ["unseen"]
+ labels.push(params["labels"]).flatten!.uniq! if params["labels"]
+ loc = @store.add rawbody
+ doc_id, thread_id = @index.add_message message, state, labels, { :loc => loc, :mbox => mbox }
+ { :response => :ok, :status => state, :doc_id => doc_id, :thread_id => thread_id, :labels => labels }
+ end
+ rescue Heliotrope::InvalidMessageError => e
+ { :response => :error, :error_message => e.message }
+ end.to_json
end
private
--
1.7.4.1
[-- Attachment #3: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-16 15:02 ` Horacio Sanson
2011-05-17 14:59 ` Horacio Sanson
@ 2011-05-18 4:59 ` William Morgan
2011-05-18 20:09 ` Sascha Silbe
1 sibling, 1 reply; 16+ messages in thread
From: William Morgan @ 2011-05-18 4:59 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-05-16:
> When UIDVALIDITY differs I will simply re-scan the whole mailbox and
> feed it to Heliotrope. I trust Heliotrope won't add duplicates.
The REST api will ignore duplicates based on the Message-Id header.
> For example in the Heliotrope::Index the add_message method allows to
> insert a message and assign it labels, flags and extra parameters at
> the same time. How can I do this with the REST API? The only example I
> see only adds a message body.
>
> RestClient.post "http://localhost:8042/message", :message => body
I've just pushed a commit to fix this. You can now send labels, state
and extra params to the POST. See heliotrope-add for an example.
> Also for what purpose are the ext array used for? Can I use it to add
> an account/mailbox property to each message so I can latter retrieve
> all messages associated to a mailbox/account pair?
Anything you put in there will come back as part of a message info block
(e.g. when you GET /thread/123.json), but it's not indexed at all, so
you won't be able to get the list of matching messages.
Probably the easiest way to accomplish that is to make a unique label
for each mailbox/account pair, but that's not ideal. We can think of
ways to make this work.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-17 14:59 ` Horacio Sanson
2011-05-17 15:15 ` Horacio Sanson
@ 2011-05-18 5:05 ` William Morgan
2011-05-20 15:38 ` Horacio Sanson
1 sibling, 1 reply; 16+ messages in thread
From: William Morgan @ 2011-05-18 5:05 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-05-17:
> I implemented a new version of the GMail -> Heliotrope sync script and
> attach it here in hopes someone will test it and provide some
> feedback/comments.
Great! This is very exciting.
> - I am seeing some negative thread_id's in the response. Need to check
> if this is normal or a bug in Heliotrope or my script.
This is normal. It indicates a thread in which the root message can't be
found (so Heliotrope must construct a pseudo-message to hold the tree
together). Nothing to worry about, except as an indicator that you're
missing email refefred to by something else in the thread.
> - GMail inbox is with a capital "I" (e.g. Inbox) while heliotrope
> uses a small "i".
> Shall I down case all labels? or make a special treatment for Inbox?
I suggest special-casing inbox.
> To use this script I had to modify heliotrope-server.rb to allow
> setting labels and states when posting new messages (see attached
> patch).
I apologize for the confusion in this, but this is actually the wrong
endpoint to use. POST "/message" (no .json) is for adding new emails.
This endpoint was introduced temporarily for outgoing emails, i.e. when
you compose or reply via turnsole. But I'm going to rename it as well as
implement it.
Anyways, I've added the relevant code to /message, so your script should
work against that.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-18 4:59 ` William Morgan
@ 2011-05-18 20:09 ` Sascha Silbe
2011-05-19 3:14 ` Alvaro Herrera
2011-05-22 22:05 ` William Morgan
0 siblings, 2 replies; 16+ messages in thread
From: Sascha Silbe @ 2011-05-18 20:09 UTC (permalink / raw)
To: sup-devel
[-- Attachment #1.1: Type: text/plain, Size: 771 bytes --]
Excerpts from William Morgan's message of Wed May 18 06:59:05 +0200 2011:
> Reformatted excerpts from Horacio Sanson's message of 2011-05-16:
> > When UIDVALIDITY differs I will simply re-scan the whole mailbox and
> > feed it to Heliotrope. I trust Heliotrope won't add duplicates.
>
> The REST api will ignore duplicates based on the Message-Id header.
This is something I don't like about sup: There's no way to access the
"duplicate" messages, e.g. to check whether an outgoing message was
received back fine from the ML or to check the headers.
If heliotrope permanently erases the copies (instead of just not
exposing UI to access them), that would be a major step backwards.
Sascha
--
http://sascha.silbe.org/
http://www.infra-silbe.de/
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]
[-- Attachment #2: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-18 20:09 ` Sascha Silbe
@ 2011-05-19 3:14 ` Alvaro Herrera
2011-05-22 22:05 ` William Morgan
1 sibling, 0 replies; 16+ messages in thread
From: Alvaro Herrera @ 2011-05-19 3:14 UTC (permalink / raw)
To: sup-devel
Excerpts from Sascha Silbe's message of mié may 18 16:09:26 -0400 2011:
> Excerpts from William Morgan's message of Wed May 18 06:59:05 +0200 2011:
> > Reformatted excerpts from Horacio Sanson's message of 2011-05-16:
> > > When UIDVALIDITY differs I will simply re-scan the whole mailbox and
> > > feed it to Heliotrope. I trust Heliotrope won't add duplicates.
> >
>
> > The REST api will ignore duplicates based on the Message-Id header.
>
> This is something I don't like about sup: There's no way to access the
> "duplicate" messages, e.g. to check whether an outgoing message was
> received back fine from the ML or to check the headers.
>
> If heliotrope permanently erases the copies (instead of just not
> exposing UI to access them), that would be a major step backwards.
Same here. I have this in detailed-headers.rb:
headers["Message Id"] = message.id
message.locations.each_with_index { |location, idx|
headers["Location#{idx}"] = "#{location.source} #{location.info}"
}
So I can immediately see all locations of a message; particularly useful
for messages that I send. I don't have easy access to the headers
within the sup UI, but the one or two times I've wanted to do that, I
simply pasted the path to the file to "less" (which works fine because I
use Maildirs; on mboxes it would be a lot more complex).
--
Álvaro Herrera <alvherre@alvh.no-ip.org>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-18 5:05 ` William Morgan
@ 2011-05-20 15:38 ` Horacio Sanson
2011-05-22 22:00 ` William Morgan
2011-05-23 21:22 ` Hamish
0 siblings, 2 replies; 16+ messages in thread
From: Horacio Sanson @ 2011-05-20 15:38 UTC (permalink / raw)
To: Sup developer discussion
[-- Attachment #1: Type: text/plain, Size: 2135 bytes --]
On Wed, May 18, 2011 at 2:05 PM, William Morgan
<wmorgan-sup@masanjin.net> wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-05-17:
>> I implemented a new version of the GMail -> Heliotrope sync script and
>> attach it here in hopes someone will test it and provide some
>> feedback/comments.
>
> Great! This is very exciting.
>
>> - I am seeing some negative thread_id's in the response. Need to check
>> if this is normal or a bug in Heliotrope or my script.
>
> This is normal. It indicates a thread in which the root message can't be
> found (so Heliotrope must construct a pseudo-message to hold the tree
> together). Nothing to worry about, except as an indicator that you're
> missing email refefred to by something else in the thread.
>
Is there a way to use GMail supplied thread ids?? there is an
extension X-GM-THRID that
provides such information.
>> - GMail inbox is with a capital "I" (e.g. Inbox) while heliotrope
>> uses a small "i".
>> Shall I down case all labels? or make a special treatment for Inbox?
>
> I suggest special-casing inbox.
>
Perfect, the new script special cases inbox
>> To use this script I had to modify heliotrope-server.rb to allow
>> setting labels and states when posting new messages (see attached
>> patch).
>
> I apologize for the confusion in this, but this is actually the wrong
> endpoint to use. POST "/message" (no .json) is for adding new emails.
> This endpoint was introduced temporarily for outgoing emails, i.e. when
> you compose or reply via turnsole. But I'm going to rename it as well as
> implement it.
>
Great I rewrote the GMail -> Heliotrope script to use the improved
POST "/message"
and works wonderfully.
I am attaching the new version that is cleaner and seems to work
without problems.
> Anyways, I've added the relevant code to /message, so your script should
> work against that.
> --
> William <wmorgan-sup@masanjin.net>
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
[-- Attachment #2: gmail.rb --]
[-- Type: application/x-ruby, Size: 11631 bytes --]
[-- Attachment #3: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-20 15:38 ` Horacio Sanson
@ 2011-05-22 22:00 ` William Morgan
2011-05-23 21:22 ` Hamish
1 sibling, 0 replies; 16+ messages in thread
From: William Morgan @ 2011-05-22 22:00 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Horacio Sanson's message of 2011-05-20:
> Is there a way to use GMail supplied thread ids?? there is an
> extension X-GM-THRID that
> provides such information.
Not really. Heliotrope has to maintain its own message ids and thread ids.
When you feed it messages, it threads things different from how Gmail does,
anyways.
> Great I rewrote the GMail -> Heliotrope script to use the improved
> POST "/message"
> and works wonderfully.
Great!
> I am attaching the new version that is cleaner and seems to work
> without problems.
Very nice. I would like to include something like this when I release
Heliotrope to the world at large, since I suspect this will be a popular
migration path. So thank you for laying the foundation.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-18 20:09 ` Sascha Silbe
2011-05-19 3:14 ` Alvaro Herrera
@ 2011-05-22 22:05 ` William Morgan
2011-05-23 7:52 ` Gaudenz Steinlin
1 sibling, 1 reply; 16+ messages in thread
From: William Morgan @ 2011-05-22 22:05 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Sascha Silbe's message of 2011-05-18:
> If heliotrope permanently erases the copies (instead of just not
> exposing UI to access them), that would be a major step backwards.
What do you suggest as a way of exposing duplicates within the UI? I
worry that just keeping all copies of a message around will make the
thread structure messy. Is there a better alternative?
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-22 22:05 ` William Morgan
@ 2011-05-23 7:52 ` Gaudenz Steinlin
2011-05-23 16:14 ` William Morgan
0 siblings, 1 reply; 16+ messages in thread
From: Gaudenz Steinlin @ 2011-05-23 7:52 UTC (permalink / raw)
To: sup-devel
Excerpts from William Morgan's message of 2011-05-23 00:05:57 +0200:
> Reformatted excerpts from Sascha Silbe's message of 2011-05-18:
> > If heliotrope permanently erases the copies (instead of just not
> > exposing UI to access them), that would be a major step backwards.
>
> What do you suggest as a way of exposing duplicates within the UI? I
> worry that just keeping all copies of a message around will make the
> thread structure messy. Is there a better alternative?
Merging the messages is fine and actually an improvement over other
MUAs. But adding a line in the header section indicating the source
and how often the message appears in the source would be nice. Like
Sources: maildir://bla/blu (2), maildir://foo/bar (1)
Also if you display the message in the raw form (V key in sup), then
all the copies could be shown in full. This would help diagnosing
why you receive duplicates of certain messages.
Gaudenz
--
Ever tried. Ever failed. No matter.
Try again. Fail again. Fail better.
~ Samuel Beckett ~
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-23 7:52 ` Gaudenz Steinlin
@ 2011-05-23 16:14 ` William Morgan
0 siblings, 0 replies; 16+ messages in thread
From: William Morgan @ 2011-05-23 16:14 UTC (permalink / raw)
To: sup-devel
Reformatted excerpts from Gaudenz Steinlin's message of 2011-05-23:
> Merging the messages is fine and actually an improvement over other
> MUAs.
I agree. I like the behavior.
> But adding a line in the header section indicating the source
> and how often the message appears in the source would be nice.
This is essentially what Alvaro's detailed-headers hook does. But unlike
Sup, heliotrope doesn't have the notion of different mailstores. There's
just one.
With some effort, we could add some kind of differentiating information
at mail addition time, but it's hard to know what would fit the general
case. A list of (from, timestamp) tuples?
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-20 15:38 ` Horacio Sanson
2011-05-22 22:00 ` William Morgan
@ 2011-05-23 21:22 ` Hamish
2011-05-24 3:16 ` Horacio Sanson
1 sibling, 1 reply; 16+ messages in thread
From: Hamish @ 2011-05-23 21:22 UTC (permalink / raw)
To: sup-devel
Excerpts from Horacio Sanson's message of Fri May 20 16:38:30 +0100 2011:
> On Wed, May 18, 2011 at 2:05 PM, William Morgan wrote:
> > Reformatted excerpts from Horacio Sanson's message of 2011-05-17:
> >> I implemented a new version of the GMail -> Heliotrope sync script and
> >> attach it here in hopes someone will test it and provide some
> >> feedback/comments.
This sounds awesome. I've already downloaded my gmail email using
offlineimap (into maildir format), and I'm wondering if it's possible to
get your script to use the local copies first, and only then get the new
emails from gmail - so as to save redownloading several hundred
megabytes of email ...
I'm keen to try it out, but don't want to use it on my main account as I
imagine it would take a long time. Maybe I'll just set up a second
account with some test email to play with it.
If it's just too hard to do, I'll just have to do the whole thing
anyway, but it won't be too bad as just a one off job.
Thanks for your work though
Hamish
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [sup-devel] Query for largest msg_id?
2011-05-23 21:22 ` Hamish
@ 2011-05-24 3:16 ` Horacio Sanson
0 siblings, 0 replies; 16+ messages in thread
From: Horacio Sanson @ 2011-05-24 3:16 UTC (permalink / raw)
To: Sup developer discussion
On Tue, May 24, 2011 at 6:22 AM, Hamish <dmishd@gmail.com> wrote:
> Excerpts from Horacio Sanson's message of Fri May 20 16:38:30 +0100 2011:
>> On Wed, May 18, 2011 at 2:05 PM, William Morgan wrote:
>> > Reformatted excerpts from Horacio Sanson's message of 2011-05-17:
>> >> I implemented a new version of the GMail -> Heliotrope sync script and
>> >> attach it here in hopes someone will test it and provide some
>> >> feedback/comments.
>
> This sounds awesome. I've already downloaded my gmail email using
> offlineimap (into maildir format), and I'm wondering if it's possible to
> get your script to use the local copies first, and only then get the new
> emails from gmail - so as to save redownloading several hundred
> megabytes of email ...
>
Unfortunately this script is GMail specific and I make use of some
GMail only extensions (X-GM-LABELS, XLIST, etc) that render the
script unusable for any other mail store. My main motivation for
creating this script was to get rid of offlineimap as this would store
each message twice, once in offlineimap folders and once in heliotrope
store.
> I'm keen to try it out, but don't want to use it on my main account as I
> imagine it would take a long time. Maybe I'll just set up a second
> account with some test email to play with it.
>
Please do not use this script with your main account,... It won't
affect your GMail data since I use read-only mode (IMAP examine) but
it still has some problems I need to handle before I can guarantee
that Heliotrope is 100% sync with GMail after running the script. For
instance sometimes GMail fails to send me the message body or timeouts
generating some IMAP exceptions I need to handle more gracefully.
If you have a test account with some messages any testing is greatly
appreciated...
> If it's just too hard to do, I'll just have to do the whole thing
> anyway, but it won't be too bad as just a one off job.
>
> Thanks for your work though
> Hamish
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2011-05-24 3:58 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-05-10 15:17 [sup-devel] Query for largest msg_id? Horacio Sanson
2011-05-15 15:01 ` William Morgan
2011-05-16 15:02 ` Horacio Sanson
2011-05-17 14:59 ` Horacio Sanson
2011-05-17 15:15 ` Horacio Sanson
2011-05-18 5:05 ` William Morgan
2011-05-20 15:38 ` Horacio Sanson
2011-05-22 22:00 ` William Morgan
2011-05-23 21:22 ` Hamish
2011-05-24 3:16 ` Horacio Sanson
2011-05-18 4:59 ` William Morgan
2011-05-18 20:09 ` Sascha Silbe
2011-05-19 3:14 ` Alvaro Herrera
2011-05-22 22:05 ` William Morgan
2011-05-23 7:52 ` Gaudenz Steinlin
2011-05-23 16:14 ` William Morgan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox