* [sup-talk] preparing for 0.7
@ 2009-03-16 17:27 William Morgan
2009-03-17 18:29 ` Mark Alexander
0 siblings, 1 reply; 10+ messages in thread
From: William Morgan @ 2009-03-16 17:27 UTC (permalink / raw)
Hi all,
I've prepared master for an 0.7 release. If you get a chance, please try
it out and let me know if there's anything obviously broken. The changes
I'm most excited about are the Ferret locking (so hopefully no more
index corruption) and being able to type long answers to Sup's
questions.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7
2009-03-16 17:27 [sup-talk] preparing for 0.7 William Morgan
@ 2009-03-17 18:29 ` Mark Alexander
2009-03-18 13:49 ` William Morgan
0 siblings, 1 reply; 10+ messages in thread
From: Mark Alexander @ 2009-03-17 18:29 UTC (permalink / raw)
On Mon, Mar 16, 2009 at 10:27 AM, William Morgan
<wmorgan-sup at masanjin.net> wrote:
> I've prepared master for an 0.7 release. If you get a chance, please try
> it out and let me know if there's anything obviously broken. The changes
> I'm most excited about are the Ferret locking (so hopefully no more
> index corruption) and being able to type long answers to Sup's
> questions.
Thanks! I'm trying it now.
Last month I discovered sup and attempted to use it
in a serious way as a replacement for mutt. I fed it
around 10000 messages that I had in my ~/Maildir
and tried to use it for several days. But I kept running
into ferret-related crashes every few minutes, so I
abandoned the effort and went back to mutt.
I wasn't willing to give up completely, though, and
started to investigate the idea of replacing ferret
with Sphinx (yes, I know about the Sup-As-Service
project, but I was impatient). I didn't get too far
with that, though (lack of time, mainly). Sphinx
is a pain, frankly, and I was racking my brains for
ways of dealing sanely with the incremental indexing,
the matching of message IDs with Sphinx IDs, and
the XML interface.
So I'm really glad to see the new release and hope
the ferret-related bugs are gone. So far things
look good. It's lasted lots longer than it did before,
but I'm feeling cautious and will run it for a few more days
before considering moving away from mutt full-time.
--Mark
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7
2009-03-17 18:29 ` Mark Alexander
@ 2009-03-18 13:49 ` William Morgan
2009-03-18 15:39 ` Lee Hinman
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: William Morgan @ 2009-03-18 13:49 UTC (permalink / raw)
Reformatted excerpts from Mark Alexander's message of 2009-03-17:
> I wasn't willing to give up completely, though, and started to
> investigate the idea of replacing ferret with Sphinx (yes, I know
> about the Sup-As-Service project, but I was impatient). I didn't get
> too far with that, though (lack of time, mainly). Sphinx is a pain,
> frankly, and I was racking my brains for ways of dealing sanely with
> the incremental indexing, the matching of message IDs with Sphinx IDs,
> and the XML interface.
I played around with Sphinx a little bit since that's apparently the hot
shit nowadays, but had a similar experience---the combination of having
to feed everything to it in XML, manually manage the incremental
indexes, and deal with a remote server that would take 1-2 seconds to
pick up changes was hellish. I was not motivated to continue.
So I still think Ferret is the best search engine for Ruby apps, despite
being unmaintained and highly thread-unsafe.
> So I'm really glad to see the new release and hope the ferret-related
> bugs are gone. So far things look good. It's lasted lots longer than
> it did before, but I'm feeling cautious and will run it for a few more
> days before considering moving away from mutt full-time.
Glad to hear it. I'm definitely interested in whether it holds up.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7
2009-03-18 13:49 ` William Morgan
@ 2009-03-18 15:39 ` Lee Hinman
2009-03-19 17:02 ` William Morgan
2009-03-18 21:11 ` Mark Alexander
2009-03-19 19:18 ` Mark Alexander
2 siblings, 1 reply; 10+ messages in thread
From: Lee Hinman @ 2009-03-18 15:39 UTC (permalink / raw)
Excerpts from William Morgan's message of Wed Mar 18 07:49:15 -0600 2009:
> So I still think Ferret is the best search engine for Ruby apps, despite
> being unmaintained and highly thread-unsafe.
Has anyone looked into using Hyper Estraier
(http://hyperestraier.sourceforge.net/) as a backend instead of Ferret (if
the problems with Ferret are bad enough to want to switch)?
I've had really good luck with it, it indexes fast and the searches are
super-quick. The ruby bindings for it are great!
Just curious.
- Lee
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7
2009-03-18 13:49 ` William Morgan
2009-03-18 15:39 ` Lee Hinman
@ 2009-03-18 21:11 ` Mark Alexander
2009-03-19 19:18 ` Mark Alexander
2 siblings, 0 replies; 10+ messages in thread
From: Mark Alexander @ 2009-03-18 21:11 UTC (permalink / raw)
On Wed, Mar 18, 2009 at 6:49 AM, William Morgan
<wmorgan-sup at masanjin.net> wrote:
> I'm definitely interested in whether it holds up.
It's holding up very well; no crashes after two days. This is wonderful!
I have encountered a few odd minor issues. One is that
it seems not to recognize Message-ID headers that are
split over two lines. For example, here's one that caused
it to generate a faked-up ID:
Message-ID:
<72E99544C4530E4F92DCC03303F8CF820145EDC05D at PA-EXMBX01.widget.com>
There's a newline after the colon, and the message ID follows
on the next line, indented with one space. I believe this one
was generated by Outlook, but I've also seen Alpine do this.
This is messing up threading. Without delving into
the source, I am guessing this an rmail bug, not a sup bug.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7
2009-03-18 15:39 ` Lee Hinman
@ 2009-03-19 17:02 ` William Morgan
0 siblings, 0 replies; 10+ messages in thread
From: William Morgan @ 2009-03-19 17:02 UTC (permalink / raw)
Reformatted excerpts from Lee Hinman's message of 2009-03-18:
> Has anyone looked into using Hyper Estraier
> (http://hyperestraier.sourceforge.net/) as a backend instead of Ferret
> (if the problems with Ferret are bad enough to want to switch)?
I looked at it briefly but judging from the mailing list traffic it's
not really maintained, or at least people's questions aren't being
answered. I already have that with Ferret. :)
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7
2009-03-18 13:49 ` William Morgan
2009-03-18 15:39 ` Lee Hinman
2009-03-18 21:11 ` Mark Alexander
@ 2009-03-19 19:18 ` Mark Alexander
2009-03-19 19:25 ` William Morgan
2 siblings, 1 reply; 10+ messages in thread
From: Mark Alexander @ 2009-03-19 19:18 UTC (permalink / raw)
I left the new sup running all night, and when
I looked at it this morning, it had dropped
at least one new message. I didn't figure that
out until I saw replies to the lost message.
Poking about with mutt and searching for
the message using devel/console.sh showed
that it really was missing from the ferret index.
Running sup-sync --all on the maildir source in question
fixed it, but it was still worrisome.
This is on a work machine that gets messages
from dozens of mailing lists, cron jobs, etc.
I have fetchmail running in the background
and polling every two minutes. Then procmail
is used to stuff things into a dozen folders
in ~/Maildir (for use with mutt).
So I'm wondering if there is some kind of race
condition between sup and fetchmail/procmail. I'm not
even sure how to debug this since it's apparently
an infrequent event (and one that isn't noticeable
right away).
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7
2009-03-19 19:18 ` Mark Alexander
@ 2009-03-19 19:25 ` William Morgan
2009-03-19 20:41 ` Mark Alexander
0 siblings, 1 reply; 10+ messages in thread
From: William Morgan @ 2009-03-19 19:25 UTC (permalink / raw)
[resend to list]
Reformatted excerpts from Mark Alexander's message of 2009-03-19:
> So I'm wondering if there is some kind of race condition between sup
> and fetchmail/procmail. I'm not even sure how to debug this since
> it's apparently an infrequent event (and one that isn't noticeable
> right away).
Hm, that would be bad. The right way to debug this is to wait for it to
happen again (!) and examine the contents of the poll-mode and log-mode
buffers, which should describe what Sup thinks it was doing at the time.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7
2009-03-19 19:25 ` William Morgan
@ 2009-03-19 20:41 ` Mark Alexander
2009-04-15 17:29 ` [sup-talk] Lost Maildir Messages Iain
0 siblings, 1 reply; 10+ messages in thread
From: Mark Alexander @ 2009-03-19 20:41 UTC (permalink / raw)
On Thu, Mar 19, 2009 at 12:25 PM, William Morgan
<wmorgan-sup at masanjin.net> wrote:
> Hm, that would be bad. The right way to debug this is to wait for it to
> happen again (!) and examine the contents of the poll-mode and log-mode
> buffers, which should describe what Sup thinks it was doing at the time.
That's my plan. I did look at the two log buffers when I noticed
the problem, but by then it was too late. I'll keep an eye on it.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Lost Maildir Messages
2009-03-19 20:41 ` Mark Alexander
@ 2009-04-15 17:29 ` Iain
0 siblings, 0 replies; 10+ messages in thread
From: Iain @ 2009-04-15 17:29 UTC (permalink / raw)
> Hm, that would be bad. The right way to debug this is to wait for it to
> happen again (!) and examine the contents of the poll-mode and log-mode
> buffers, which should describe what Sup thinks it was doing at the time.
I've noticed what seems to be the same problem.
I've been using getmail (no procmail or fetchmail), to grab messages
into Maildir folders. Occasionally, messages don't show up in Sup. They
appear only when I do a manual sup-sync --changed.
I discovered by accident (by impatiently trying to read an email that I
knew was being delivered by getmail into Sup) that I can reliably
replicate the problem, or at least a similar problem. It seems to occur
when Sup polls the Maildir at the same time as getmail is retrieving the
message.
I can replicate the problem by pressing P repeatedly in Sup to manually
poll for messages, while getmail is in the process of retrieving. (It
takes about 4 seconds for getmail to go from initialising a retrieval to
dropping the mail in the Maildir, so there is a big window for when I
press P, and I can replicate the problem reliably.) Every time, Sup
doesn't retrieve the new messages.
The log-mode buffer doesn't show anything relevant in my
artifically-produced replication of the problem, because it doesn't log
when you manually poll.
The poll-mode buffer for the source in question displays the exact same
message on every poll of the source (before getmail runs, during the
getmail run, after the getmail run) when repeatedly polling before and
during the getmail run:
Loading from maildir:///home/user/Mail/address at example.com/...
Found message at 12398138490002177 with labels {mylabel}
-- the timestamp never changes.
If I had to guess, I'd say that it looks like the shortcut taken
(looking at the mtime of the directory to see if scanning for a new
Maildir message is required) has a race condition. A weird race
condition, because I've verified (ls -l --time-style=+%s) that the "new"
and "tmp" folders are indeed having their timestamps updated when the
mail drops in (they are).
This might explain the infrequent lost messages: when not hitting P
repeatedly, this race condition is unlikely to occur, only happening
when the Sup poll occurs within the few seconds window during which
getmail is working and working with a new message to deliver.
~Iain
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2009-04-15 17:29 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-03-16 17:27 [sup-talk] preparing for 0.7 William Morgan
2009-03-17 18:29 ` Mark Alexander
2009-03-18 13:49 ` William Morgan
2009-03-18 15:39 ` Lee Hinman
2009-03-19 17:02 ` William Morgan
2009-03-18 21:11 ` Mark Alexander
2009-03-19 19:18 ` Mark Alexander
2009-03-19 19:25 ` William Morgan
2009-03-19 20:41 ` Mark Alexander
2009-04-15 17:29 ` [sup-talk] Lost Maildir Messages Iain
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox