Subject: Re: [sup-talk] current state of synching upstream?

From: Ryan Barrett <[email-masked]>
Date: Sun, 11 Apr 2010 10:52:57 -0700 (PDT)

ping? anyone successfully using sup in parallel with other client(s)?

scott, are you still working on maildir sync? any updates since january?

On Thu, 8 Apr 2010, Ryan Barrett wrote:

> hi all! i've been looking forward to trying sup since i first heard about it 
> a couple years ago, and i'm getting pretty close. one of my few remaining 
> concerns is using sup in parallel with other mail clients, including sup 
> itself.
>
> i know best practice is to just use one sup installation and no other 
> clients, but i also know that synching back upstream is a frequent topic 
> here. i've seen the discussions of sup-sync --changed, sup-sync-back, maildir 
> + offlineimap etc. i'm mostly looking to hear about the current state. is 
> anyone successfully running sup synched with another sup or other client? if 
> so, how?
>
> it seems like people have generally agreed that the best approach is to make 
> sup-sync-back support maildir, and then use offlineimap to sync from maildir 
> to the source. is that still true? cc'ing scott henson, who mentioned he was 
> working on maildir sync in january.
>
> http://rubyforge.org/pipermail/sup-talk/2010-January/003761.html
>
> more background:
>
> http://rubyforge.org/pipermail/sup-talk/2009-April/002126.html
>
> http://rubyforge.org/pipermail/sup-talk/2009-July/002567.html
>
> -Ryan
>
> --
> http://snarfed.org/
>

  -Ryan

--
http://snarfed.org/
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Ryan Barrett <[email-masked]>
Date: Thu, 8 Apr 2010 02:40:32 -0700 (PDT)

hi all! i've been looking forward to trying sup since i first heard about it a 
couple years ago, and i'm getting pretty close. one of my few remaining 
concerns is using sup in parallel with other mail clients, including sup 
itself.

i know best practice is to just use one sup installation and no other clients, 
but i also know that synching back upstream is a frequent topic here. i've seen 
the discussions of sup-sync --changed, sup-sync-back, maildir + offlineimap 
etc. i'm mostly looking to hear about the current state. is anyone 
successfully running sup synched with another sup or other client? if so, how?

more background:

-Ryan

---

From: Tero Tilus <[email-masked]>
Date: Tue, 21 Dec 2010 13:01:59 +0200

Matthias Vallentin, 2010-12-21 08:44:
>> The only downside to this is that my sources.yaml file needs manual
>> twiddling at restart to add the sources that were added during
>> runtime.
> 
> As Tero mentioned, this is weird indeed since sup calls Index.save
> which in turn should call SourceManager.save_sources. I'll look into
> this when I get a chance.
...
> Neither master nor Damien's Maildir clone contain the class
> Recoverable.  Is this code you added somewhere else?

This sources not getting saved might be because of that Recoverable
wrapper class.  YAML marshalling might not be able to digest it.

--
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Ben Walton <[email-masked]>
Date: Sat, 18 Dec 2010 14:21:40 -0500

Excerpts from Matthias Vallentin's message of Sat Dec 18 14:04:27 -0500 2010:

> The downside appears to be that each rotation adds, in the above
> example, 10 new source entries to sources.yaml and requires
> switching of polling in the non-current sources.

Here's my current approach that I've been happy with since I
implemented it back in the summer:

1. Procmail files all mail (regardless of originating source) into
   Maildirs like .incoming.%Y.%m.  This gives me a new maildir each month
   that holds all incoming mail for that month.  The basics of the
   .procmailrc to do this are:

--snip--

MAILDIR=$HOME/Maildir/
DATEDIR=`date +%Y.%m`

:0
$MAILDIR/.incoming.$DATEDIR/

--snip--

2. I have the following hook setup as after-poll:

--snip--
s = "maildir:/path/to/Maildir/.incoming.#{Date.today.strftime("%Y.%m")}"

unless Redwood::SourceManager.source_for(s)
  Redwood::Logger.force_message "Adding new source: #{s}"
  Redwood::SourceManager.add_source Recoverable.new(Redwood::Maildir.new(s))
end
--snip--

3. I have almost all of my labelling done via the before-add-message
   hook.  This gets me the per-mailing-list tags that I would have
   applied based on source originally.  I almost never apply tags
   manually any more.

The only downside to this is that my sources.yaml file needs manual
twiddling at restart to add the sources that were added during
runtime.  I've yet to be annoyed enough by this to figure out how to
make these dynamic additions sticky across restarts.

HTH.
-Ben
--
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302

---

From: Tero Tilus <[email-masked]>
Date: Wed, 22 Dec 2010 18:27:30 +0200

Matthias Vallentin, 2010-12-22 16:42:
> Adding the with -> marked statement in Index.save printed, however, not
> the debug statement in SourceManager.save_sources! Any ideas how this is
> possible?

I think I have a hunch.  SourceManager is a singleton, which (see
util.rb) silently drops all the calls if it is in deinstantiated
state.  Could you inject some debugging in Singleton#method_missing to
see if SourceManager is already deinstantiated when #save_sources is
called.

---

From: Ben Walton <[email-masked]>
Date: Wed, 14 Apr 2010 10:16:10 -0400

Excerpts from William Morgan's message of Wed Apr 14 09:00:59 -0400 2010:

> (Assuming that the Maildir format won't simply collapse with the result
> of converting 29 million lines of mbox goodness.)

mbox2maildir is (generally) your friend.  We've used it here with some
success.  The only issue we had is when the first line in a paragraph
of a mail message starts with /From /.  As you've seen from dealing
with mboxes in sup, this isn't necessarily an easy problem to fix...in
our case, we're dealing with mboxes created with the likes of ancient
pine and elm, etc, so we might be worse off than normal mutt-made
mboxes for instance.

A conversion solution that is better, if possible, is to use imapsync,
as it leaves the mbox parsing to the imap software.  It requires more
overhead to be useful though[1].

Thanks
-Ben

[1] I'll provide more (dovecot) tips if anyone is interested.
-- 
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302

---

From: Tero Tilus <[email-masked]>
Date: Mon, 12 Apr 2010 12:53:04 +0300

Rich Lane, 2010-04-12 03:57:
> Sup will automatically detect changes to the Seen flag in the
> maildir and update the message's labels.

What about Trashed, Draft and Flagged?  They have obvious counterparts
in Sup labels too.

> I haven't decided what to do about the other direction. Historically
> Sup has never modified mail sources, but changing flags in a maildir
> is safe enough that I might implement it.

For what I know maildir was designed to handle delivery, deletion and
flag changes "safely" (no mail content corruption) even without
locking.

Updating message status upstream is a major point for multi-agent
users.  And I have got the impression that there are quire a few of
them already.

---

From: Matthias Vallentin <[email-masked]>
Date: Mon, 20 Dec 2010 22:48:32 -0800

On Mon, Dec 20, 2010 at 10:44:45PM -0800, Matthias Vallentin wrote:
> As Tilo mentioned [..]

I was referring to Tero, sorry about the confusion.

   Matthias

---

From: Rich Lane <[email-masked]>
Date: Sun, 11 Apr 2010 20:57:42 -0400

> concerns is using sup in parallel with other mail clients, including sup 

My primary goal for the 0.12 release is making Sup play nice with other
clients. You can see the work in progress on the maildir branch - it's
getting more stable but it's still rough around the edges. Most of the
changes on this branch are to make Sup gracefully handle cases where
another client has moved or deleted mail out from under us.

Sup will automatically detect changes to the Seen flag in the maildir
and update the message's labels. I haven't decided what to do about the
other direction. Historically Sup has never modified mail sources, but
changing flags in a maildir is safe enough that I might implement it.
Running sup-sync-back has the problem that you can't run the UI at the
same time.

Does anyone use the current sup-sync-back that only support mboxes? If
there are no strong objections then the 0.12 sup-sync-back will only
support maildir.
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Wed, 14 Apr 2010 11:57:28 -0400

Reformatted excerpts from Ben Walton's message of 2010-04-14:
> mbox2maildir is (generally) your friend.

Oh, I'm sure the conversion is pretty straight-forward (modulo the
"From_" issues you point out). I just wonder if I'm going to hit limits
on the number of files in a directory or something weird like that.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Ben Walton <[email-masked]>
Date: Tue, 21 Dec 2010 09:11:42 -0500

> wrapper class.  YAML marshalling might not be able to digest it.

When I wrote the hook, I referenced the following from
lib/sup/source.rb:

  def load_sources fn=Redwood::SOURCE_FN
    source_array = (Redwood::load_yaml_obj(fn) || []).map { |o| Recoverable.new o }
    @source_mutex.synchronize do
      @sources = Hash[*(source_array).map { |s| [s.id, s] }.flatten]
      @sources_dirty = false
    end
  end

So, after de-marshalling the yaml file, it wraps the object in a
Recoverable.  This class is defined in lib/sup/util.rb.

I'm looking at the code again right now as the Recoverable may be the
key as to why this doesn't get saved...

Thanks
-Ben

---

From: Ben Walton <[email-masked]>
Date: Wed, 14 Apr 2010 12:08:14 -0400

Excerpts from William Morgan's message of Wed Apr 14 11:57:28 -0400 2010:

> "From_" issues you point out). I just wonder if I'm going to hit
> limits on the number of files in a directory or something weird like
> that.

You might, but I don't know what that limit would be...My inbox is
currently looking like:

--snip--
bwalton @ pinkfloyd : ~/Maildir
$ lc
total 3.1M
drwx------ 2 bwalton user 496K Apr 29  2008 cur
-rw------- 1 bwalton user 1.2K Apr 28  2008 dovecot-uidlist
drwx------ 2 bwalton user 2.6M Apr 14 12:04 new
drwx------ 2 bwalton user 4.0K Apr 14 12:04 tmp

bwalton @ pinkfloyd : ~/Maildir
$ ls -lA new/ | wc -l
36413
--snip--

Big (notice the size of the new/ directory), but not huge.  Poll times
increase linearly though, which one drawback to the current approach
of not moving mail into cur/ when it's detected.

Thanks
-Ben

---

From: Matthias Vallentin <[email-masked]>
Date: Mon, 20 Dec 2010 22:44:45 -0800

> implemented it back in the summer:

Cool, this is exactly how I expected it to look like!

>   Redwood::SourceManager.add_source Recoverable.new(Redwood::Maildir.new(s))

Neither master nor Damien's Maildir clone contain the class Recoverable.
Is this code you added somewhere else?

The only (conceptual) issue I see with this code that you miss sources
when having more than a month of idleness. Say the suffix of s is
2012.10 and the next time you check back is in December, then you'll
miss 2012.11. Now whether anyone of us has the freedom to not check
email for that long is a different question :-).

> twiddling at restart to add the sources that were added during

As Tilo mentioned, this is weird indeed since sup calls Index.save which
in turn should call SourceManager.save_sources. I'll look into this when
I get a chance.

   Matthias

---

From: William Morgan <[email-masked]>
Date: Wed, 14 Apr 2010 09:00:59 -0400

Reformatted excerpts from Rich Lane's message of 2010-04-11:
> Does anyone use the current sup-sync-back that only support mboxes?

I do, but I would consider moving to Maildir to support the common good.

---

From: Matthias Vallentin <[email-masked]>
Date: Fri, 17 Dec 2010 21:12:16 -0800

On Wed, Dec 15, 2010 at 05:06:54PM +0000, James Taylor wrote:
> For what it is worth, I've been using maildir-sync for almost six months
> every day, no problems. I exercise it pretty hard since I usually have
> at least four imap clients (mobile devices and such) accessing the
> maildir simultaneously via dovecot imap. 

This looks like the scenario I envision. Two questions:

    - Do you use the IMAP clients as read-only clients or do you invoke
      sup-sync after the devices modify the maildir (or before polling
      in sup)?

    - Do you use sup-sync-back when changing message state via sup? From
      what I understand, this would be desirable to copy messages from
      new to cur in order to avoid a linear increase in poll times with
      growing number of mails in new.

   Matthias

---

From: James Taylor <[email-masked]>
Date: Sat, 18 Dec 2010 05:25:05 +0000

>       growing number of mails in new.

Neither. The imap clients are read write, the only time this becomes a
problem is when the imap client moves a message from new to cur. Sup
will not be able to find the message until the next time it polls (but
this is mostly seamless, if I use the imap client I just poll and
refresh in sup). This does imply that sup must be polling cur as well as
new.

Similarly, sup-sync-back is not needed because messages are synced back
to the maildir immediately. 
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: James Taylor <[email-masked]>
Date: Wed, 15 Dec 2010 17:06:54 +0000

Excerpts from Matthias Vallentin's message of Wed Dec 15 08:19:55 +0000 2010:

> Does the current version of sup still keep mail in new? As a fairly new
> sup user, I am trying to understand the limitations of maildir support
> and this one seems to be one.

For what it is worth, I've been using maildir-sync for almost six months
every day, no problems. I exercise it pretty hard since I usually have
at least four imap clients (mobile devices and such) accessing the
maildir simultaneously via dovecot imap. Everything works fine. 

Currently using Damien's latest sync up with the mainline from here:

  http://git.fensalir.fr/?p=dleone/sup.git;a=shortlog;h=refs/heads/maildir-sync-master

Anybody else using this? It seems ready to integrate.

-- jt

---

From: Tero Tilus <[email-masked]>
Date: Sat, 18 Dec 2010 22:02:46 +0200

> twiddling at restart to add the sources that were added during

How come?  For what I know sup should save sources.yml (the call to
SoureManager.save_sources is in Index.save) upon graceful exit.
That's exactly how sup-add modifies sources.yml.

---

From: Daemian Mack <[email-masked]>
Date: Mon, 12 Apr 2010 08:11:01 -0400

Excerpts from Andrew Pimlott's message of Mon Apr 12 01:02:07 -0400 2010:
> Excerpts from Rich Lane's message of Sun Apr 11 17:57:42 -0700 2010:
> It wouldn't kill me to lose it, but I use it, and think on principle it
> would be better to have an overlap period.

I use sup-sync-back with mbox. I agree with Andrew that overlap would
be a good thing in principle, though personally speaking, I'll
probably take this change as a nudge to convert to maildir. My mbox
files get pretty large (~80-100Mb) between sup-sync-backs, and working with those
files on my aging mail box (1.6GHz Athlon XP) gets pretty clunky in the meantime.

Also looking forward to more interclient operability!

Any interest in kicking off a simple usage survey to find out how
people are using sup? I'd be really curious to see things like...

- the mailstore formats being used
- whether most people use sup as their sole MUA or as one of many
- how multi-MUA users handle working with multiple clients
- the various daily sup workflows in use

Might be helpful from a "what's going on out there" roadmap
perspective, too.
-- 
http://daemianmack.com/

---

From: Ben Walton <[email-masked]>
Date: Sat, 18 Dec 2010 15:12:34 -0500

> That's exactly how sup-add modifies sources.yml.

I'm not sure as I haven't had the time to look into it.  I only
discovered this after the first time sup restarted after adding this
hook (a few months later).  I rarely restart sup and when I do, I
don't usually have time to poke at the problem (work mail and
all...).  I might be able to just make the SourceManager.save_sources
call manually.

Thanks
-Ben

---

From: Matthias Vallentin <[email-masked]>
Date: Wed, 22 Dec 2010 06:42:22 -0800

> key as to why this doesn't get saved...

I did some printf-debugging at shutdown time and am now really puzzled
why SourceManager.save_source is not invoked. The function Index.save
and SourceManager.save_source look as follows:

    def save
        debug "saving index and sources..."
        FileUtils.mkdir_p @dir unless File.exists? @dir
->      debug "*** Index.save"
        SourceManager.save_sources
        save_index
    end

    def save_sources fn=Redwood::SOURCE_FN
->      debug "*** SourceManager.save_sources"
        @source_mutex.synchronize do
            ...
        end
    end

   Matthias

---

From: Andrew Pimlott <[email-masked]>
Date: Sun, 11 Apr 2010 22:02:07 -0700

Andrew

---

From: Matthias Vallentin <[email-masked]>
Date: Wed, 15 Dec 2010 00:19:55 -0800

My traditional mindest is still similar to logrotate, i.e., regularly
copying maildirs into an archive. If this is still an issue, what about
an where mail directories with more than X messages are being
automatically copied to an archive? This would bound the linear poll
times at a configurable threshold. I don't know how this would look like
(adding a new source per rotated maildir does not seem a nice solution)
and how it would interplay with Xapian. For example, I worry that
maildir rotation could involve major index rebuilding.

   Matthias

---

From: Matthias Vallentin <[email-masked]>
Date: Sat, 18 Dec 2010 11:04:27 -0800

On Sat, Dec 18, 2010 at 05:25:05AM +0000, James Taylor wrote:
> problem is when the imap client moves a message from new to cur. 

Is that the only operation the imap clients do? What happens if they
delete messages (or moved them to another source) and sup has that
message already in the index.

> Sup will not be able to find the message until the next time it polls
> (but this is mostly seamless, if I use the imap client I just poll and
> refresh in sup). This does imply that sup must be polling cur as well
> as new.

If sup polls both cur and new then there is no more benefit in keeping
new small to avoid long poll times for large maildirs. 

This brings me to my main sup question. What is the best strategy to
avoid overly large maildirs? Say I have around 10 maildirs, each of
which represent a separate source. The classic scheme that comes to mind
(and that I use with Mutt and Mairix) would be some horizontal
partitioning scheme according to time, i.e., creating a new tree of
maildirs at a certain time interval (e.g., quarterly could imply a
naming scheme for the top-level directories like mail-2010-1 to
mail-2010-4). Then, only the most recent partition needs to be polled.

The downside appears to be that each rotation adds, in the above
example, 10 new source entries to sources.yaml and requires switching
of polling in the non-current sources.

Does that sound tractable and aligned with sup's mail philosophy? 

   Matthias