Subject: [sup-talk] problems with Maildir and IMAP offline

From: [email-masked] (Luis Cañas Díaz)
Date: Mon, 13 Jul 2009 23:20:22 +0200

Hi guys,
first things first. Your mail client looks promising, I've been testing many of them (most of them from the kde/gnome world) and yours seems to be perfect to be running in my dear mac mini ;)

This afternoon I've been trying to set up a gmail account with offlineimap and I'm having the same problem reported by this guy[1]. I run offlineimap and it creates a Maildir directory with 4hundred read mails and 2hundred unread mails (yes, I should pay more attention to my gmail account;). The problem is that sup sees all of the messages as unread which is not true according to the Maildir structure:

   [luis at gongbu INBOX]$ ls new/|wc -l;ls cur/|wc -l
   210
   412

After applying changes with sup, the state is the same. When using sup I "delete" messages and mark them as read.

The sup version is: 

   [luis at gongbu ~]$ sup -v
   [Mon Jul 13 23:14:25 +0200 2009] using character set encoding "UTF-8"
   sup v0.8.1

Last thing, I haven't seen it written by I guess it's a good idea having the files locally to use the search engine. Am I right?

Thanks for your help! :)

[1] http://osdir.com/ml/mail.sup.general/2008-08/msg00029.html

-- 
Luis Ca?as D?az          | GSyC/LibreSoft Research Lab
http://libresoft.es      | Grupo de Sistemas y Comunicaciones
Tel: (+34) 91 488 85 23  | Universidad Rey Juan Carlos

---

From: [email-masked] (Andrei Thorp)
Date: Tue, 14 Jul 2009 09:24:32 -0400

> "delete" messages and mark them as read.
I'm somewhat unhappy about this too. It seems to me that sup just plain
does not support the read features of maildirs.
 a) It doesn't import them as read
 b) sup-sync-back does not mark them as read.

The best I could find is manually marking things as read and deleting
them, and then using sup-sync-back with the deleted options which it
does understand.

On the other hand, sup provides hooks to make calling offlineimap and
sup-sync-back completely transparent and well-timed. You can call
sup-sync-back followed by offlineimap on each sup poll, so that's pretty
sweet.
-- 
Andrei Thorp, Developer: Xandros Corp. (http://www.xandros.com)

---

From: [email-masked] (Andrei Thorp)
Date: Tue, 14 Jul 2009 10:02:11 -0400

Excerpts from Tim Gray's message of Tue Jul 14 09:45:44 -0400 2009:
> On Tue 14, Jul'09 at  9:24 AM -0400, Andrei Thorp wrote:
> 

I'm forwarding a summary of your message with my replies to the mailing
list. It seems like you replied to me only by accident?

>   From my playing around with sup and from what I've read, sup just doesn't 
> play nicely with other mail clients.  Sup will read from your sources, but 
> the sources never get modified.

Again, see sup-sync-back -- it does indeed have _some_ ability to write
its state back to the source.

> I know sup is slowly moving in the direction of a more locked in design.  I 
> just wish sup added automatic sync back capabilities.

Put this in a hook and it's automatic. That's the beauty of the
hook/multiple binary design.

> As far as OfflineIMAP goes, I didn't think I would do this, but I've been 
> running it in it's full curses mode with an auto refresh set.

I'm not entirely sure why you want this. If you're worried about
failures, do they really happen? If you want to see the log, aren't you
already piping it? If you want notifications, you could hook them up to
happen in your sup hook (the one I described before) rather trivially
via notify-send.
-- 
Andrei Thorp, Developer: Xandros Corp. (http://www.xandros.com)

---

From: [email-masked] (Tim Gray)
Date: Tue, 14 Jul 2009 11:00:34 -0400

> list. It seems like you replied to me only by accident?

Whoops.  I didn't realize this list didn't have reply-to set.  Thanks.

> its state back to the source.

To me, this is a vital feature of any mail client.  I'll look into it, but I 
would really like this to be fully integrated into my mail client.  And from 
I've read, this isn't something that's going to happen.

Sometimes I need to use mutt/pine.  Sometimes webmail.  Most importantly, I 
want all my mail in a form that is easy to backup, human readable, and can 
be manipulated with standard utilities.  Maildir fits the bill.

I think it'd be fantastic if there was a mail client that allowed you to 
sort by folders, but also provided a sup-like view that laid on top if/when 
you needed.

> hook/multiple binary design.

Honestly, if it's that easy, shouldn't it be configured like that out of the 
box?

> happen in your sup hook (the one I described before) rather trivially

For use with other mail clients for one.  It's kind of nice having my IMAP 
syncing separate from the mail client.  I'm sure I could run it 
automatically via cron, but then you'd have to give up the quick refresh 
capability (minor) and the IDLE mode.  Sure, I could pipe the output 
somewhere else, but wouldn't that accomplish the same thing as just looking 
at the output already provided, either curses or one of the other ones? 
Again, it's running in screen so I don't even have it open on any windows.

Of course, that doesn't mean this is the best route to go, but it's been 
working for me for so far - I am new to it though.  It's a nice pretty 
colored interface too :)

Tim

---

From: [email-masked] (Ben Walton)
Date: Tue, 14 Jul 2009 11:07:46 -0400

> I've read, this isn't something that's going to happen.

There was some talk a while back about modifying the behaviour of
Maildir sources.  The issue was that there is an odd scanning bug with
Maildir that sometimes sees messages get skipped.  A proposed solution
was to have sup start 'playing nice' with Maildir so that when new
messages were found in new/, they'd be indexed and recorded with a
filename in cur/, presumably with the S (seen) flag appended properly.

I _think_ William was looking at that, but I'm not sure.  It's
something I'd like too (not to play nice with other clients, but
because it's "just better").  I haven't had much time lately, but if
William confirms he's not working on it, I'd be interested in
implementing it.

Thanks
-Ben
-- 
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302

GPG Key Id: 8E89F6D2; Key Server: pgp.mit.edu
Contact me to arrange for a CAcert assurance meeting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20090714/e30a5b14/attachment.bin>

---

From: [email-masked] (Andrei Thorp)
Date: Tue, 14 Jul 2009 11:14:50 -0400

> Honestly, if it's that easy, shouldn't it be configured like that out of the 

People perhaps don't want this. It's slower, and a lot of folks probably
do no syncing back since all they use is sup anyway.

> For use with other mail clients for one.

Understood. Makes sense in your use case. I don't really understand why
you'd want to use pine/mutt when you've already got sup though ;)
-- 
Andrei Thorp, Developer: Xandros Corp. (http://www.xandros.com)

---

From: [email-masked] (Tim Gray)
Date: Tue, 14 Jul 2009 11:28:06 -0400

On Tue 14, Jul'09 at 11:07 AM -0400, Ben Walton wrote:
> It's something I'd like too (not to play nice with other clients, but
> because it's "just better").

This sound like a good thing.  In my experience, 'just better' usually 
equals 'playing nice', since 'playing nice' usually equals following the 
standards.  That's how Maildirs are supposed to work - if everybody followed 
the spec, we'd be set.

Though I don't program for a living, nor do I know any Ruby, it seems like a 
doable thing.  Is there something fundamental blocking per-message sync-back 
of status as you read/delete/otherwise modify your mail?  I would assume 
there a reference to the original message location in a message's database 
entry.  If a message gets read, write an S at the end of the filename and 
move it to /cur.  If it gets deleted, delete it.  Operations on a message 
should affect the database and the mail store at the same time.  Why is 
something like sup-sync-back even necessary?

Anyway, these are just the musings of someone who doesn't have the tools to 
write something like sup, so I'll be quiet now before I embarrass myself any 
further.

---

From: [email-masked] (Andrei Thorp)
Date: Tue, 14 Jul 2009 11:39:09 -0400

> standards.  That's how Maildirs are supposed to work - if everybody followed 

I'd just like to mention that the reason we tend not to sync back in
general isn't because we hate standards and want people to get hurt :)

It's roughly because of the extra features in sup:
 - It keeps a database so it can search for stuff quickly
 - It implements its own tagging regardless of source (unlike stuff like
   mutt that just rely on "physical" folders)
 - It merges sources

So while it's possible to do some merging back, it's sometimes awkward
because sup's tags do not have to map 1:1 to IMAP folders or anything
else. It's otherwise also a bit awkward because we might have mail from
many sources in one place, so writing back isn't as simple as just
editing files. You also have to _not_ edit files.

Anyway, that all said, it's clearly possible and would be very nice to
have.
-- 
Andrei Thorp, Developer: Xandros Corp. (http://www.xandros.com)

---

From: [email-masked] (Tim Gray)
Date: Tue, 14 Jul 2009 11:45:07 -0400

> do no syncing back since all they use is sup anyway.

I would hope in this day in age we can change a database entry and move a 
file at the same time without too much of a performance hit.  I would think 
it only gets unbearable when you don't update as you go and you have to do a 
large batch of messages.  But you are right, maybe people don't want that.

Personally, and this is a side note, if I was involved in the project, I'd 
push for canning the built in IMAP capabilities all together.  Just run 
offlineimap or one of the other IMAP syncing utilities.  Add to that beefing 
up the Maildir handling.  Just worry about better and more transparent 
sync-back-to-Maildir capabilities and let offlineimap worry about the server 
side of the equation.

> you'd want to use pine/mutt when you've already got sup though ;)

As you can tell, I'm not quite sold on sup yet.  I've switched mail clients 
enough over the past few years that switching in and of itself is a giant 
pain.  I've settled on Maildir as my 'client' for now.  I can serve it via 
IMAP if I need, I can read it with a couple mail clients, and I can easily 
convert it to mbox however I want to with some simple Python code.

If I switch to sup full time, I would be comforted by the fact that if I 
choose to change in 2 years, all of my mail is marked read, deleted mail is 
actually deleted, and even though I was using sup labels for my 
organization, things are still organized in some fashion by appropriate 
folders on the file system.  The last part could be achievable by some 
combination of offlineimap + gmail/sieve filtering and fetchmail/getmail + 
procmail.  It'd still be nice to be able to move physical messages around in 
sup though.  Maybe I'm just nuts though.

---

From: [email-masked] (Tim Gray)
Date: Tue, 14 Jul 2009 11:59:16 -0400

> general isn't because we hate standards and want people to get hurt :)

Haha.  Of course.

> It's roughly because of the extra features in sup:

I got you.  Still, I envisage a mail client that has an underlying structure 
built on folders.  On top of that a database/virtual folder interface is 
built.  Of course, you'd have the ability to access the folder based 
structure when needed.  So you could move a message or a conversation to a 
given folder if you want.

For example, I have work emails for a certain project with a given label. 
It's wonderful that in sup, those currently in my physical inbox and those 
currently in an archive folder somewhere show up under the same label in 
sup.  But there's no mechanism to moving those in my physical inbox to the 
archive folder.  Some might say who cares, but I can't leave all my email in 
my physical inbox forever, which is synced by offlineimap, because people 
associated with this project have a nasty habit of sending 10-20 mb 
attachments and I don't want to clutter up my IMAP boxes with hundreds of 
megs of attachments forever.  Heck, I can't do that unless I have an IMAP 
service with 25 gigs of storage.*  Sometimes they just need to be collected 
in a totally offline box, but somewhere that is still accessible by 
searching if I need access to it.

Sup is pretty close to this already, minus the access to the underlying 
structure.

---

[*] Obviously for those of you using Gmail, this isn't an issue.  Who cares 
about deleting messages?

---

From: [email-masked] (Israel Herraiz)
Date: Tue, 14 Jul 2009 18:21:45 +0200

Excerpts from Luis's message on Jul 13, 2009 about 11 PM:
> This afternoon I've been trying to set up a gmail account with offlineimap and
> I'm having the same problem reported by this guy[1]. I run offlineimap and it
> creates a Maildir directory with 4hundred read mails and 2hundred unread mails
> (yes, I should pay more attention to my gmail account;).

I have not used IMAP with Sup, but AFAIK, Sup does not touch the
original sources of email. For instance, it does not read nor modify
messages using IMAP, it just read them using that protocol (or maildir
or whatever).

Therefore, the only way to get those messages marked as read is using
sup-sync with the --read option. That will mark the messages as read
in Sup.

However there are two problems. If all the messages (unread or read)
are in the same source, there is no way to indicate which messages are
going to be marked as read and which are not.

Moreover, if you read a message in Sup, it still will appear in Gmail
as unread.

I also use Gmail with Sup, but using POP. I use the web interface of
Gmail when I cannnot access my laptop. I use IMAP from my mobile
phone. And I download all my email to my laptop using POP.

If I get a message in my laptop, it is marked as archived and read in
GMail, so it does not appear as unread in the web nor in my phone.

However, if I read the message in the phone or the web, it will appear
as unread in Sup.

Still it is a good solution if you have a "central" node where you
store your mail, and do your main work (my laptop), and a couple of
"external" agents (phone, web) to be able of reading your email if you
do not have your laptop at hand.

Another advantage is that you keep a copy of your mail in your
laptop. GMail is not going to last forever (of course, it still has a
long live) and with this you ensure that you will not loose your
old-email in GMail is closed some day, and it is also very handy to
make backups of all your email (just copy the mbox file to another
location).

To get the email I use getmail [1], and msmtp [2] with these scripts
[3] (see last section) as a lightweight replacement for sendmail. I
can provide you my config scripts if you want.

It is not a good solution though if you read your email from several
different computers.

Cheers,
Israel

[1] http://pyropus.ca/software/getmail/
[2] http://msmtp.sourceforge.net/
[3] http://wiki.archlinux.org/index.php/Msmtp