Subject: [sup-talk] More about those lost messages

From: [email-masked] (Mark Alexander)
Date: Wed, 8 Apr 2009 10:32:29 -0700

I've been using sup (next branch) for a week or so, and no crashes so
far.  But I'm still having problems with occasional lost messages.
Messages that were definitely in the index in the evening are gone
when I start sup again the next morning.  I've recovered them by doing
a sup-sync --all, but I'm still puzzled by what's going on.

This has happened twice that I've noticed, and in both cases the
message that got lost was part of a thread in which I was
participating, and was in the inbox.  It could be that other messages
got lost in other threads that I was ignoring, but I didn't notice it
because I was ignoring those threads :-) .

When I first noticed this a couple of weeks ago, I thought it might
have something to do with fetchmail running in the background.  I
changed things around so that fetchmail is only running in the
background when sup is not running.  When sup is running, it invokes
fetchmail via the poll hook.  But this doesn't seem to have had any
effect on the lost message problem.

This is just idle, uninformed speculation, but maybe some records that
were in the in-memory index didn't get written to disk when sup
exited?  Would neglecting to use the '$' command do this?

---

From: [email-masked] (William Morgan)
Date: Wed, 08 Apr 2009 15:34:06 -0700

Reformatted excerpts from Mark Alexander's message of 2009-04-08:
> participating, and was in the inbox.

Just to confirm: you saw the message in Sup, you quit Sup, and when you
started Sup again, the message wasn't there?

The next time this happens, would you mind trying this? Before running
sup-sync --all, find the message id of the missing message (i.e. from
the mbox file), and run:

  devel/console.sh                                # start the debug console
  Index.index.search("message_id:XXX").total_hits # from within the console

where XXX is the message id, without the angle brackets. E.g.

  Index.index.search("message_id:a412e2a70904081032x3438abedw3a23d87973f56d35 at mail.gmail.com")

This will tell us whether it's in the index or not, without anything
of Sup's curses interface getting in the way.

If it is in the index (the total_hits is > 0), then try finding the
labels:

  docid = Index.index.search("message_id:XXX").hits.first.doc
  Index.build_message(docid).labels

> exited?  Would neglecting to use the '$' command do this?

Sup saves all state when you quit. That's why I'm wondering whether the
messages actually are in the index, or somehow are just hidden from the
curses interface.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Mark Alexander)
Date: Thu, 9 Apr 2009 08:36:54 -0700

I'm starting to think it may be a user error on my part.  It looks as if
I might have used the 'd' (delete) command on some messages,
perhaps accidentally or carelessly in some cases, and that
prevented them from showing up in threads later.  I'll continue
studying this.