Subject: Re: [sup-devel] Update in heliotrope !

From: Steven Hum <[email-masked]>
Date: Tue, 25 Sep 2012 17:01:45 -0400

Yes! :-) (though I did try the import with the server running after the 
persistent hanging without.. just in case the documentation wasn't 
correct!).

As nothing appeared to be working, I purged the heliotrope mailstore and 
performed 2 actions: 1) the initial server startup to initialize the 
mailstore and associated files, and 2) an import of a small maildir 
(after stopping the server). I presume the logs you are referring to is 
the LOG file in the <mailstore>/store dir.

LOG from step 1) (or LOG.old after completing both steps)

2012/09/25-16:42:13.770269 7ff2345bc700 Delete type=3 #1

LOG from attempted import, step 2)

2012/09/25-16:43:12.450537 7f0c45ff8700 Recovering log #3
2012/09/25-16:43:12.450585 7f0c45ff8700 Level-0 table #5: started
2012/09/25-16:43:12.458919 7f0c45ff8700 Level-0 table #5: 344 bytes OK
2012/09/25-16:43:12.460026 7f0c45ff8700 Delete type=0 #3
2012/09/25-16:43:12.460050 7f0c45ff8700 Delete type=3 #2

Oh, the second thing you should know is that I run on debian (test 
/ wheezy) and have been using rvm 1.9.3. I know that the debian 
environment doesn'd always make it easy for ruby installs at times. In 
addition to the gems requirements listed in the README, I had to install 
the mail and thin gems to get heliotrope to run.

Regards,
Steven

Excerpts from Matthieu Rakotojaona's message of 2012-09-25 15:44:49 -0400:
> On Tue, Sep 25, 2012 at 9:15 PM, Steven Hum <[email-masked]> wrote:
> > Hello Matthieu,
> 
> Hello,
> 
> > I am able to initialize the heliotrope mailstore directory easy enough
> > with
> >
> > ruby -Ilib bin/heliotrope-server -d <mailstore>
> >
> > (stuff gets created :-) but when I try to import with
> >
> > ruby -Ilib bin/heliotrope-import -a <maildir> -d <mailstore>
> 
> Did you even read the README ? =]
> 
> It clearly says : "To bulk import mail, use heltrope-import. You must
> stop your server first."
> 
> The thing is, both heliotrope-server and heliotrope-import will try to
> access the same resources, which are not designed for
> multi-concurrency. So, you have to shut heliotrope-server down before
> using heliotrope-import.
> 
> > I tried using heliotrope-add with the server running, but it
> > hung at the same point (not surpisingly).
> 
> This is more problematic, because it should work. Can you share some
> of your logs ?
> 
> > Secondly, your note "stop all other activity with heliotrope while you
> > sync with offlinimap". Does that mean heliotrope-server should not be
> > running if offlineimap is (updating the maildirs)?
> 
> No, on the contrary, it means that while heliotrope-server is running,
> you should either access it with a traditional http client, or with an
> IMAP client, but you should never do both the same time.
> But you will need heliotrope-server running for accessing IMAP.
> 
-- 
"Truth or die."

Steven Hum  
5 - 28 Gilmour St  
Ottawa, ON K2P 0N3  
email [email-masked]  
tel 613.237.9058  
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Steven Hum <[email-masked]>
Date: Tue, 25 Sep 2012 15:15:01 -0400

Hello Matthieu,

<maildir> being the dir that contains the cur/new/tmp directories that 
contain the actual emails downloaded by offlineimap,

the import process hangs after the "Adding mail..." message and doesn't 
load any emails (I just terminate the process with a ctrl-c). What am 
I missing? I tried using heliotrope-add with the server running, but it 
hung at the same point (not surpisingly).

Thanks,
Steven

Excerpts from Matthieu Rakotojaona's message of 2012-09-24 18:13:57 -0400:
> Hello everyone,
> 
> I have updated my version of heliotrope on my imap branch
> (https://github.com/rakoo/heliotrope/tree/imap). You just have to
> clone it and checkout the branch.
> You may now synchronize your mails between heliotrope and a maildir
> with offlineimap. I tested most of  the situations that can happen
> when using it (Adding/removing a label on heliotrope side,
> adding/removing a mail on maildir side)
> 
> A few notes :
> 
> * It is not "multi-write-safe", in the sense that you should stop all
> other activity with heliotrope while you sync with offlinimap.
> Concurrent access will have unknown outcome.
> * To continue on this point, use offlineimap with the `-1` flag, which
> will turn off multithreading. It should work without it though,
> because AFAIK offlineimap threads do not share mailboxes. But I'm not
> sure.
> * There are a few hardcoded labels/dir that you should find in your
> maildir, such as INBOX/inbox and "All Mail", which has no equivalent
> in heliotrope. Fear not, this one is taken care of.
> * Syncing with another IMAP server should also work,because it is
> transparent to offlineimap (at least from the outside)
> 
> Have fun !
> 
-- 
"Truth or die."

Steven Hum  
5 - 28 Gilmour St  
Ottawa, ON K2P 0N3  
email [email-masked]  
tel 613.237.9058

---

From: Steven Hum <[email-masked]>
Date: Thu, 27 Sep 2012 12:46:01 -0400

I couldn't find any physical log files anywhere (at least any text files 
with an obvious name...). The only other file in the heliotrope 
mailstore directory I found was a 32 byte binary file named "000022.log"

Where should these be located? Looks like there could be something 
fundamentally broken on my system if these log files aren't even being 
written!

Regards,
Steven

Excerpts from Matthieu Rakotojaona's message of 2012-09-27 11:50:40 -0400:
> Sorry for the delay.
> 
> On Tue, Sep 25, 2012 at 11:01 PM, Steven Hum <[email-masked]> wrote:
> > I presume the logs you are referring to is
> Hmm no, I would like to see the heliotrope-server logs, or the
> heliotrope-import logs. These could help solve the problem.
> 
> Sorry, this is totally undocumented, but yes, you have to install the
> mail gem. I decided to part from rmail, since it isn't maintained
> anymore. Thin was just there to improve performance and is optional. I
> just found the HTTP part stable enough fore my needs, so I changed to
> it (And effectively saw some improvements)
> 
-- 
"Truth or die."

Steven Hum  
5 - 28 Gilmour St  
Ottawa, ON K2P 0N3  
email [email-masked]  
tel 613.237.9058

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Thu, 27 Sep 2012 17:50:40 +0200

Sorry for the delay.

-- 
Matthieu RAKOTOJAONA

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Tue, 25 Sep 2012 21:44:49 +0200

Hello,

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Tue, 25 Sep 2012 00:13:57 +0200

Hello everyone,

A few notes :

Have fun !