* [sup-talk] preparing for 0.7 @ 2009-03-16 17:27 William Morgan 2009-03-17 18:29 ` Mark Alexander 0 siblings, 1 reply; 10+ messages in thread From: William Morgan @ 2009-03-16 17:27 UTC (permalink / raw) Hi all, I've prepared master for an 0.7 release. If you get a chance, please try it out and let me know if there's anything obviously broken. The changes I'm most excited about are the Ferret locking (so hopefully no more index corruption) and being able to type long answers to Sup's questions. -- William <wmorgan-sup at masanjin.net> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7 2009-03-16 17:27 [sup-talk] preparing for 0.7 William Morgan @ 2009-03-17 18:29 ` Mark Alexander 2009-03-18 13:49 ` William Morgan 0 siblings, 1 reply; 10+ messages in thread From: Mark Alexander @ 2009-03-17 18:29 UTC (permalink / raw) On Mon, Mar 16, 2009 at 10:27 AM, William Morgan <wmorgan-sup at masanjin.net> wrote: > I've prepared master for an 0.7 release. If you get a chance, please try > it out and let me know if there's anything obviously broken. The changes > I'm most excited about are the Ferret locking (so hopefully no more > index corruption) and being able to type long answers to Sup's > questions. Thanks! I'm trying it now. Last month I discovered sup and attempted to use it in a serious way as a replacement for mutt. I fed it around 10000 messages that I had in my ~/Maildir and tried to use it for several days. But I kept running into ferret-related crashes every few minutes, so I abandoned the effort and went back to mutt. I wasn't willing to give up completely, though, and started to investigate the idea of replacing ferret with Sphinx (yes, I know about the Sup-As-Service project, but I was impatient). I didn't get too far with that, though (lack of time, mainly). Sphinx is a pain, frankly, and I was racking my brains for ways of dealing sanely with the incremental indexing, the matching of message IDs with Sphinx IDs, and the XML interface. So I'm really glad to see the new release and hope the ferret-related bugs are gone. So far things look good. It's lasted lots longer than it did before, but I'm feeling cautious and will run it for a few more days before considering moving away from mutt full-time. --Mark ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7 2009-03-17 18:29 ` Mark Alexander @ 2009-03-18 13:49 ` William Morgan 2009-03-18 15:39 ` Lee Hinman ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: William Morgan @ 2009-03-18 13:49 UTC (permalink / raw) Reformatted excerpts from Mark Alexander's message of 2009-03-17: > I wasn't willing to give up completely, though, and started to > investigate the idea of replacing ferret with Sphinx (yes, I know > about the Sup-As-Service project, but I was impatient). I didn't get > too far with that, though (lack of time, mainly). Sphinx is a pain, > frankly, and I was racking my brains for ways of dealing sanely with > the incremental indexing, the matching of message IDs with Sphinx IDs, > and the XML interface. I played around with Sphinx a little bit since that's apparently the hot shit nowadays, but had a similar experience---the combination of having to feed everything to it in XML, manually manage the incremental indexes, and deal with a remote server that would take 1-2 seconds to pick up changes was hellish. I was not motivated to continue. So I still think Ferret is the best search engine for Ruby apps, despite being unmaintained and highly thread-unsafe. > So I'm really glad to see the new release and hope the ferret-related > bugs are gone. So far things look good. It's lasted lots longer than > it did before, but I'm feeling cautious and will run it for a few more > days before considering moving away from mutt full-time. Glad to hear it. I'm definitely interested in whether it holds up. -- William <wmorgan-sup at masanjin.net> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7 2009-03-18 13:49 ` William Morgan @ 2009-03-18 15:39 ` Lee Hinman 2009-03-19 17:02 ` William Morgan 2009-03-18 21:11 ` Mark Alexander 2009-03-19 19:18 ` Mark Alexander 2 siblings, 1 reply; 10+ messages in thread From: Lee Hinman @ 2009-03-18 15:39 UTC (permalink / raw) Excerpts from William Morgan's message of Wed Mar 18 07:49:15 -0600 2009: > So I still think Ferret is the best search engine for Ruby apps, despite > being unmaintained and highly thread-unsafe. Has anyone looked into using Hyper Estraier (http://hyperestraier.sourceforge.net/) as a backend instead of Ferret (if the problems with Ferret are bad enough to want to switch)? I've had really good luck with it, it indexes fast and the searches are super-quick. The ruby bindings for it are great! Just curious. - Lee ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7 2009-03-18 15:39 ` Lee Hinman @ 2009-03-19 17:02 ` William Morgan 0 siblings, 0 replies; 10+ messages in thread From: William Morgan @ 2009-03-19 17:02 UTC (permalink / raw) Reformatted excerpts from Lee Hinman's message of 2009-03-18: > Has anyone looked into using Hyper Estraier > (http://hyperestraier.sourceforge.net/) as a backend instead of Ferret > (if the problems with Ferret are bad enough to want to switch)? I looked at it briefly but judging from the mailing list traffic it's not really maintained, or at least people's questions aren't being answered. I already have that with Ferret. :) -- William <wmorgan-sup at masanjin.net> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7 2009-03-18 13:49 ` William Morgan 2009-03-18 15:39 ` Lee Hinman @ 2009-03-18 21:11 ` Mark Alexander 2009-03-19 19:18 ` Mark Alexander 2 siblings, 0 replies; 10+ messages in thread From: Mark Alexander @ 2009-03-18 21:11 UTC (permalink / raw) On Wed, Mar 18, 2009 at 6:49 AM, William Morgan <wmorgan-sup at masanjin.net> wrote: > I'm definitely interested in whether it holds up. It's holding up very well; no crashes after two days. This is wonderful! I have encountered a few odd minor issues. One is that it seems not to recognize Message-ID headers that are split over two lines. For example, here's one that caused it to generate a faked-up ID: Message-ID: <72E99544C4530E4F92DCC03303F8CF820145EDC05D at PA-EXMBX01.widget.com> There's a newline after the colon, and the message ID follows on the next line, indented with one space. I believe this one was generated by Outlook, but I've also seen Alpine do this. This is messing up threading. Without delving into the source, I am guessing this an rmail bug, not a sup bug. ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7 2009-03-18 13:49 ` William Morgan 2009-03-18 15:39 ` Lee Hinman 2009-03-18 21:11 ` Mark Alexander @ 2009-03-19 19:18 ` Mark Alexander 2009-03-19 19:25 ` William Morgan 2 siblings, 1 reply; 10+ messages in thread From: Mark Alexander @ 2009-03-19 19:18 UTC (permalink / raw) I left the new sup running all night, and when I looked at it this morning, it had dropped at least one new message. I didn't figure that out until I saw replies to the lost message. Poking about with mutt and searching for the message using devel/console.sh showed that it really was missing from the ferret index. Running sup-sync --all on the maildir source in question fixed it, but it was still worrisome. This is on a work machine that gets messages from dozens of mailing lists, cron jobs, etc. I have fetchmail running in the background and polling every two minutes. Then procmail is used to stuff things into a dozen folders in ~/Maildir (for use with mutt). So I'm wondering if there is some kind of race condition between sup and fetchmail/procmail. I'm not even sure how to debug this since it's apparently an infrequent event (and one that isn't noticeable right away). ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7 2009-03-19 19:18 ` Mark Alexander @ 2009-03-19 19:25 ` William Morgan 2009-03-19 20:41 ` Mark Alexander 0 siblings, 1 reply; 10+ messages in thread From: William Morgan @ 2009-03-19 19:25 UTC (permalink / raw) [resend to list] Reformatted excerpts from Mark Alexander's message of 2009-03-19: > So I'm wondering if there is some kind of race condition between sup > and fetchmail/procmail. I'm not even sure how to debug this since > it's apparently an infrequent event (and one that isn't noticeable > right away). Hm, that would be bad. The right way to debug this is to wait for it to happen again (!) and examine the contents of the poll-mode and log-mode buffers, which should describe what Sup thinks it was doing at the time. -- William <wmorgan-sup at masanjin.net> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] preparing for 0.7 2009-03-19 19:25 ` William Morgan @ 2009-03-19 20:41 ` Mark Alexander 2009-04-15 17:29 ` [sup-talk] Lost Maildir Messages Iain 0 siblings, 1 reply; 10+ messages in thread From: Mark Alexander @ 2009-03-19 20:41 UTC (permalink / raw) On Thu, Mar 19, 2009 at 12:25 PM, William Morgan <wmorgan-sup at masanjin.net> wrote: > Hm, that would be bad. The right way to debug this is to wait for it to > happen again (!) and examine the contents of the poll-mode and log-mode > buffers, which should describe what Sup thinks it was doing at the time. That's my plan. I did look at the two log buffers when I noticed the problem, but by then it was too late. I'll keep an eye on it. ^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] Lost Maildir Messages 2009-03-19 20:41 ` Mark Alexander @ 2009-04-15 17:29 ` Iain 0 siblings, 0 replies; 10+ messages in thread From: Iain @ 2009-04-15 17:29 UTC (permalink / raw) > Hm, that would be bad. The right way to debug this is to wait for it to > happen again (!) and examine the contents of the poll-mode and log-mode > buffers, which should describe what Sup thinks it was doing at the time. I've noticed what seems to be the same problem. I've been using getmail (no procmail or fetchmail), to grab messages into Maildir folders. Occasionally, messages don't show up in Sup. They appear only when I do a manual sup-sync --changed. I discovered by accident (by impatiently trying to read an email that I knew was being delivered by getmail into Sup) that I can reliably replicate the problem, or at least a similar problem. It seems to occur when Sup polls the Maildir at the same time as getmail is retrieving the message. I can replicate the problem by pressing P repeatedly in Sup to manually poll for messages, while getmail is in the process of retrieving. (It takes about 4 seconds for getmail to go from initialising a retrieval to dropping the mail in the Maildir, so there is a big window for when I press P, and I can replicate the problem reliably.) Every time, Sup doesn't retrieve the new messages. The log-mode buffer doesn't show anything relevant in my artifically-produced replication of the problem, because it doesn't log when you manually poll. The poll-mode buffer for the source in question displays the exact same message on every poll of the source (before getmail runs, during the getmail run, after the getmail run) when repeatedly polling before and during the getmail run: Loading from maildir:///home/user/Mail/address at example.com/... Found message at 12398138490002177 with labels {mylabel} -- the timestamp never changes. If I had to guess, I'd say that it looks like the shortcut taken (looking at the mtime of the directory to see if scanning for a new Maildir message is required) has a race condition. A weird race condition, because I've verified (ls -l --time-style=+%s) that the "new" and "tmp" folders are indeed having their timestamps updated when the mail drops in (they are). This might explain the infrequent lost messages: when not hitting P repeatedly, this race condition is unlikely to occur, only happening when the Sup poll occurs within the few seconds window during which getmail is working and working with a new message to deliver. ~Iain ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2009-04-15 17:29 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-03-16 17:27 [sup-talk] preparing for 0.7 William Morgan 2009-03-17 18:29 ` Mark Alexander 2009-03-18 13:49 ` William Morgan 2009-03-18 15:39 ` Lee Hinman 2009-03-19 17:02 ` William Morgan 2009-03-18 21:11 ` Mark Alexander 2009-03-19 19:18 ` Mark Alexander 2009-03-19 19:25 ` William Morgan 2009-03-19 20:41 ` Mark Alexander 2009-04-15 17:29 ` [sup-talk] Lost Maildir Messages Iain
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox