Subject: [sup-talk] preparing for 0.7

From: [email-masked] (William Morgan)
Date: Mon, 16 Mar 2009 10:27:48 -0700

Hi all,

I've prepared master for an 0.7 release. If you get a chance, please try
it out and let me know if there's anything obviously broken. The changes
I'm most excited about are the Ferret locking (so hopefully no more
index corruption) and being able to type long answers to Sup's
questions.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Mark Alexander)
Date: Tue, 17 Mar 2009 11:29:51 -0700

On Mon, Mar 16, 2009 at 10:27 AM, William Morgan

> index corruption) and being able to type long answers to Sup's

Thanks!  I'm trying it now.

Last month I discovered sup and attempted to use it
in a serious way as a replacement for mutt.  I fed it
around 10000 messages that I had in my ~/Maildir
and tried to use it for several days.  But I kept running
into ferret-related crashes every few minutes, so I
abandoned the effort and went back to mutt.

I wasn't willing to give up completely, though, and
started to investigate the idea of replacing ferret
with Sphinx (yes, I know about the Sup-As-Service
project, but I was impatient).  I didn't get too far
with that, though (lack of time, mainly).  Sphinx
is a pain, frankly, and I was racking my brains for
ways of dealing sanely with the incremental indexing,
the matching of message IDs with Sphinx IDs, and
the XML interface.

So I'm really glad to see the new release and hope
the ferret-related bugs are gone.  So far things
look good.  It's lasted lots longer than it did before,
but I'm feeling cautious and will run it for a few more days
before considering moving away from mutt full-time.

--Mark

---

From: [email-masked] (William Morgan)
Date: Wed, 18 Mar 2009 06:49:15 -0700

Reformatted excerpts from Mark Alexander's message of 2009-03-17:
> I wasn't willing to give up completely, though, and started to
> investigate the idea of replacing ferret with Sphinx (yes, I know
> about the Sup-As-Service project, but I was impatient).  I didn't get
> too far with that, though (lack of time, mainly).  Sphinx is a pain,
> frankly, and I was racking my brains for ways of dealing sanely with
> the incremental indexing, the matching of message IDs with Sphinx IDs,
> and the XML interface.

I played around with Sphinx a little bit since that's apparently the hot
shit nowadays, but had a similar experience---the combination of having
to feed everything to it in XML, manually manage the incremental
indexes, and deal with a remote server that would take 1-2 seconds to
pick up changes was hellish. I was not motivated to continue.

So I still think Ferret is the best search engine for Ruby apps, despite
being unmaintained and highly thread-unsafe.

> So I'm really glad to see the new release and hope the ferret-related
> bugs are gone.  So far things look good.  It's lasted lots longer than
> it did before, but I'm feeling cautious and will run it for a few more
> days before considering moving away from mutt full-time.

Glad to hear it. I'm definitely interested in whether it holds up.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Lee Hinman)
Date: Wed, 18 Mar 2009 09:39:34 -0600

> being unmaintained and highly thread-unsafe.

Has anyone looked into using Hyper Estraier
(http://hyperestraier.sourceforge.net/) as a backend instead of Ferret (if
the problems with Ferret are bad enough to want to switch)?

I've had really good luck with it, it indexes fast and the searches are
super-quick. The ruby bindings for it are great!

Just curious.

- Lee

---

From: [email-masked] (Mark Alexander)
Date: Wed, 18 Mar 2009 14:11:05 -0700

On Wed, Mar 18, 2009 at 6:49 AM, William Morgan
<wmorgan-sup at masanjin.net> wrote:
> I'm definitely interested in whether it holds up.

It's holding up very well; no crashes after two days.  This is wonderful!

I have encountered a few odd minor issues.  One is that
it seems not to recognize Message-ID headers that are
split over two lines.  For example, here's one that caused
it to generate a faked-up ID:

Message-ID:
 <72E99544C4530E4F92DCC03303F8CF820145EDC05D at PA-EXMBX01.widget.com>

There's a newline after the colon, and the message ID follows
on the next line, indented with one space.  I believe this one
was generated by Outlook, but I've also seen Alpine do this.

This is messing up threading.  Without delving into
the source, I am guessing this an rmail bug, not a sup bug.

---

From: [email-masked] (William Morgan)
Date: Thu, 19 Mar 2009 10:02:11 -0700

Reformatted excerpts from Lee Hinman's message of 2009-03-18:
> (http://hyperestraier.sourceforge.net/) as a backend instead of Ferret
> (if the problems with Ferret are bad enough to want to switch)?

I looked at it briefly but judging from the mailing list traffic it's
not really maintained, or at least people's questions aren't being
answered. I already have that with Ferret. :)
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Mark Alexander)
Date: Thu, 19 Mar 2009 12:18:07 -0700

I left the new sup running all night, and when
I looked at it this morning, it had dropped
at least one new message.  I didn't figure that
out until I saw replies to the lost message.
Poking about with mutt and searching for
the message using devel/console.sh showed
that it really was missing from the ferret index.
Running sup-sync --all on the maildir source in question
fixed it, but it was still worrisome.

This is on a work machine that gets messages
from dozens of mailing lists, cron jobs, etc.
I have fetchmail running in the background
and polling every two minutes.  Then procmail
is used to stuff things into a dozen folders
in ~/Maildir (for use with mutt).

So I'm wondering if there is some kind of race
condition between sup and fetchmail/procmail.  I'm not
even sure how to debug this since it's apparently
an infrequent event (and one that isn't noticeable
right away).

---

From: [email-masked] (William Morgan)
Date: Thu, 19 Mar 2009 12:25:23 -0700

[resend to list]

Reformatted excerpts from Mark Alexander's message of 2009-03-19:
> So I'm wondering if there is some kind of race condition between sup
> and fetchmail/procmail.  I'm not even sure how to debug this since
> it's apparently an infrequent event (and one that isn't noticeable
> right away).

Hm, that would be bad. The right way to debug this is to wait for it to
happen again (!) and examine the contents of the poll-mode and log-mode
buffers, which should describe what Sup thinks it was doing at the time.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Mark Alexander)
Date: Thu, 19 Mar 2009 13:41:06 -0700

On Thu, Mar 19, 2009 at 12:25 PM, William Morgan
> buffers, which should describe what Sup thinks it was doing at the time.

That's my plan.  I did look at the two log buffers when I noticed
the problem, but by then it was too late.  I'll keep an eye on it.

---

From: [email-masked] (Iain)
Date: Wed, 15 Apr 2009 18:29:15 +0100

> buffers, which should describe what Sup thinks it was doing at the time.

I've noticed what seems to be the same problem.

I've been using getmail (no procmail or fetchmail), to grab messages 
into Maildir folders. Occasionally, messages don't show up in Sup. They 
appear only when I do a manual sup-sync --changed.

I discovered by accident (by impatiently trying to read an email that I 
knew was being delivered by getmail into Sup) that I can reliably 
replicate the problem, or at least a similar problem. It seems to occur 
when Sup polls the Maildir at the same time as getmail is retrieving the 
message.

I can replicate the problem by pressing P repeatedly in Sup to manually 
poll for messages, while getmail is in the process of retrieving. (It 
takes about 4 seconds for getmail to go from initialising a retrieval to 
dropping the mail in the Maildir, so there is a big window for when I 
press P, and I can replicate the problem reliably.) Every time, Sup 
doesn't retrieve the new messages.

The log-mode buffer doesn't show anything relevant in my 
artifically-produced replication of the problem, because it doesn't log 
when you manually poll.

The poll-mode buffer for the source in question displays the exact same 
message on every poll of the source (before getmail runs, during the 
getmail run, after the getmail run) when repeatedly polling before and 
during the getmail run:

Loading from maildir:///home/user/Mail/address at example.com/... 

Found message at 12398138490002177 with labels {mylabel}

-- the timestamp never changes.

If I had to guess, I'd say that it looks like the shortcut taken 
(looking at the mtime of the directory to see if scanning for a new 
Maildir message is required) has a race condition. A weird race 
condition, because I've verified (ls -l --time-style=+%s) that the "new" 
and "tmp" folders are indeed having their timestamps updated when the 
mail drops in (they are).

This might explain the infrequent lost messages: when not hitting P 
repeatedly, this race condition is unlikely to occur, only happening 
when the Sup poll occurs within the few seconds window during which 
getmail is working and working with a new message to deliver.

~Iain