Subject: [sup-talk] New User Questions

From: [email-masked] (Guarded Identity)
Date: Tue, 08 Jan 2008 21:39:18 -0600

Hello everyone,

First off.  William, thanks a lot for all the work.  The Mutt home page
has always had this quote: "All mail clients suck. This one [mutt] just
sucks less."  I know Sup has room for more features and functionality,
but it my mind, it's no where near sucking, especially with the option
of using sup-console or Ruby scripts using Redwood libraries.

Having just gotten started, I have a slew of questions.  I'll try to
organize/enumerate them below to help with in-line responses.

I'm using maildirs as opposed to mboxes.  I've always preferred maildirs
over mboxes to avoid dealing with gigantic mail files.

   1. Can sent mail only be stored in an mbox, or are other formats
      supported?

I haven't gotten around to digging deeply into Sup's source, so I'm not
sure about the details of how E-mails are indexed.  I know Sup doesn't
play well with other MUAs.  The FAQ discusses mboxes pretty well.  I was
hoping for a little more discussion of how modification of maildir mail
by other MUAs affects Sup.  For instance, the maildir specification has
mail put in $MAILDIR/new/ until read by a MUA, which should move the
mail to $MAILDIR/cur/ to mark the mail as "seen".

   2. Will migration of maildir mail from new/ to cur/ necessitate an
      execution of sup-sync?

Also, "read" flags (and others) are stored in the Maildir filename.

   3. If the filename of maildir mail changes, is a sup-sync required?

If 2. or 3. are indeed problems, maybe I could front the maildir sources
with an IMAP server.  I had done this before, actually, but I noticed
that Sup indexing was slowed down quite a bit (rss2email and mailing
lists generate a lot of new mail).  If I do go with IMAP, I guess I
could sequence a sup-sync right after my crontab call to fetchmail to
help with indexing?  I just wasn't keen on Sup being locked out
periodically by crontab.

Alternatively, I could move all my Maildirs into mboxes.  I know Sup has
a bias towards not deleting mail, but sup-sync-back is there for mboxes.

   4. How far away is sup-sync-back support for maildirs or IMAP?  Is
      the work straight-forward or are there some technical kinks to be
      ironed out?

   5. Should I consider using mboxes for any reason beyond support by
      sup-sync-back?

Also, I would really like to have time-based auto-expiry (excluding
starred or special-labeled items) for some of my mail (primarily mailing
lists and rss feeds).  With mboxes, I guess I could do some
"deleted"-labeling with a Ruby script followed by a call to
sup-sync-back.
    
   6. What are my scripting options for mail expiry with Maildirs?  The
      search to get the message objects is pretty straight forward.  How
      much further work is it to delete the message from the index and
      to get a filename of the message to delete from the maildir?

Also, I kind of miss urlview.

   7. Any proposal for easily opening urls from messages?  
    
      urlview is already coded up, so I'm all for a mechanism for piping
      messages into an external application.  But this leads to other
      design issues, I think, like how to enable user-level key
      bindings.

I have few questions regarding hooks and threading:

   8. I'm a little confused about threading interacts with labeling in
      Sup.  Labels are applied to entire threads, but it seems that
      labels are stored on a per-mail basis.  Is seems possible for
      message within a thread to have different labels, say by using a
      before-add-message.rb hook.  Is the label set for the thread a
      union of all the labels of its constituents?
    
   9. If a new message comes into a thread, is it auto-labeled with the
      labels of the thread?

  10. I tried to do some auto-archiving with the before-add-message.rb
      hook using message.remove_label("inbox"), but it didn't work.  Is
      this possible?  Or is auto-archiving only possible at the source
      level?

Finally. . .

  11. Is it possible to search for mail that has no labels without saying
      "!label:label1 AND !label:label2 AND ..."?

      I looked a little in the Ferret documentation, but didn't find a
      way.  I was trying to do this to get at some mail I archived
      accidentally before applying labels.

That's what I've got for now.  I hope I didn't overwhelm the list with a
lot of stupid questions.

Thanks for your help,
Sukant

---

From: [email-masked] (William Morgan)
Date: Wed, 09 Jan 2008 22:34:03 -0800

Excerpts from Guarded Identity's message of Tue Jan 08 19:39:18 -0800 2008:
> I know Sup has room for more features and functionality, but it my
> mind, it's no where near sucking, especially with the option of using
> sup-console or Ruby scripts using Redwood libraries.

Thanks! Always nice to hear.

> 1. Can sent mail only be stored in an mbox, or are other formats

Right now it's mbox only. There's no technical reason it couldn't be
stored in a Maildir, it's just that writing to an mbox is just the
easiest possible thing in the world.

> 2. Will migration of maildir mail from new/ to cur/ necessitate an

Hm. I don't think so. As long as the mtime and size of the file are
preserved (which I believe they are during a move) you should be ok.

> 3. If the filename of maildir mail changes, is a sup-sync required?

No, for the same reason.

> If 2. or 3. are indeed problems, maybe I could front the maildir
> sources with an IMAP server.  I had done this before, actually, but I
> noticed that Sup indexing was slowed down quite a bit (rss2email and
> mailing lists generate a lot of new mail).

If that's true, then it must be due to IMAP transmission costs. I'm
certainly willing to believe that pulling a message from IMAP is
significantly slower than a disk read.

> 4. How far away is sup-sync-back support for maildirs or IMAP?  Is the
> work straight-forward or are there some technical kinks to be ironed
> out?

The actual interactions with the sources themselves should be pretty
easy, because they both support deleting individual messages directly.
Mbox was actually the hard one, because you have to rewrite the entire
file to delete messages.

But ideally sup-sync-back would be source type agnostic, so there's some
work to be done in terms of adding deletability to the source API in
such a manner that it handles both the mbox case and the not-mbox case.
(Mbox deletions need to happen in one go, whereas not-mbox deletions can
happen one at a time.)

> 5. Should I consider using mboxes for any reason beyond support by

No. Mbox are a terrible, evil format that just happens to be
well-supported by Sup.

> starred or special-labeled items) for some of my mail (primarily
> mailing lists and rss feeds). With mboxes, I guess I could do some

That sounds like the right approach.

> much further work is it to delete the message from the index and to
> get a filename of the message to delete from the maildir?

Trivial, although the filename/IMAP id require minor API changes to
expose those functions. But with a working sup-sync-back that applies to
all source types, you won't have to do anything other than inject
:deleted labels everywhere and simply call sup-sync-back.

> messages into an external application.  But this leads to other design
> issues, I think, like how to enable user-level key bindings.

I'm not familiar with urlview. If it is a console-based interactive
program, it should be possible to just spawn it via the hooks. We may
have to add a hook, depending on the exact usage case here.

> Sup.  Labels are applied to entire threads, but it seems that labels
> are stored on a per-mail basis.

Correct. But you never really get to play directly with message-level
labels in Sup, except for a few little things like starring and unread
status, which can be applied per message in thread-view-mode.

> Is seems possible for message within a thread to have different
> labels, say by using a before-add-message.rb hook.  Is the label set
> for the thread a union of all the labels of its constituents?

Correct.

> 9. If a new message comes into a thread, is it auto-labeled with the

No.

> hook using message.remove_label("inbox"), but it didn't work.  Is this
> possible?  Or is auto-archiving only possible at the source level?

That should work. If you post your hook we can take a crack at
debugging.

> 11. Is it possible to search for mail that has no labels without
> saying "!label:label1 AND !label:label2 AND ..."?
> 
> I looked a little in the Ferret documentation, but didn't find a way.
> I was trying to do this to get at some mail I archived accidentally
> before applying labels.

Interesting question. I don't know of a better way. If there were one it
would be in the Ferret documentation. I suppose you could use DeMorgan's
law to save yourself a few characters though. :)

-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guarded Identity)
Date: Wed, 26 Mar 2008 00:57:05 -0500

Hi,

I've been using Sup for a few months now, read up a little more on Ruby, and
thought I might be in more of a position to respond to this thread.

> > Will migration of maildir mail from new/ to cur/ necessitate an execution
> > of sup-sync?
>

I was thinking about the algorithm for calculating the message's unique
"source_info" for Maildir mail (mtime appended to filesize).  I know altering
the backend might really be something you don't want to support, so maybe I'm
voicing a moot concern.  It's just that sometimes I get annoyed at people
sending me mail with large attachments (especially at work where I just can't
control this kind of thing too well).  In the past, I'd use mutt to purge mail
of their large attachments.

However, I guess if I do this en mass, I loose my labeling of these messages
when I do a sup-sync because the message's source_info changes, right?, but
have you considered using a message's "Message-Id" header as part of the
source_info?  I'm thinking you might have.  For the most part, this should be
pretty unique, right?  I know the Message-Id is sometimes missing, but we can
default to the old source_id if things are really wacky (which should be
rarely, right?).  The only reason I could think you wouldn't use the Message-Id
is because maybe it's less efficient to open up a message than look at it's
file stats.  Or maybe there's another reason?

> > I had done this before, actually, but I noticed that Sup indexing was
> > slowed down quite a bit (rss2email and mailing lists generate a lot of new
> > mail).
>

I was thinking about this too.  Is there anything we can do to pull down
headers first to fill the index (backgrounding the indexing)?  Maybe that might
make the delay with IMAP less of a nuisance?

However, I won't harp on this too much.  Because I wasn't comfortable
synchronizing (say with Unison) sup Ferret indexes across hosts, I'm ssh'ing
into the machine with the index and using "maildir" sources.  But this
unfortunately causes me a problem with viewing graphical attachments because X
forwarding is really slow.  So for now I'm saving attachments and scp'ing them
over.  I'm not sure there's much anyone can do about any of this.

> > I would really like to have time-based auto-expiry (excluding starred or
> > special-labeled items) for some of my mail (primarily mailing lists and rss
> > feeds). With mboxes, I guess I could do some "deleted"-labeling with a Ruby
> > script followed by a call to sup-sync-back.
>
> > 6. What are my scripting options for mail expiry with Maildirs?  The search
> > to get the message objects is pretty straight forward.  How much further
> > work is it to delete the message from the index and to get a filename of
> > the message to delete from the maildir?
>
> Trivial, although the filename/IMAP id require minor API changes to expose
> those functions. But with a working sup-sync-back that applies to all source
> types, you won't have to do anything other than inject :deleted labels
> everywhere and simply call sup-sync-back.

Okay,

So I've been using a helper script that does some things to help me manage my
Maildir mail (all other source types are ignored).  My primary idea was to have
a script to expire mail, but allowing me to save threads labeled "starred" or
"save".  Eventually I added in a function to manipulate labels.  Unlike
sup-tweak-labels, this script gathers it's pool of messages from the index
using a Ferret query (Chronic allowed) rather than using source URIs.

I got around some of the encapsulation inherent in the API by just brute force
breaking encapsulation (which Ruby makes kind of ridiculously easy it seems).

I've attached the script, and would like to see any feedback if people have
some.  In particular, I'm eager to know of any concerns people have with the
approach or defects people catch.  I'm still pretty new to Ruby, so if I did
anything flat out stupid with the language, please let me know.

By the way, I was thinking that there seems to be a good deal of healthy
participation in this list.  Does anybody see any value to hanging out in #sup
on Freenode?  Sometimes it's just nice to have an IRC forum.

-Sukant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sup-tweak-maildir
Type: application/octet-stream
Size: 6652 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/sup-talk/attachments/20080326/cef1c568/attachment.obj