Subject: [sup-talk] new in next: faster saving and bigger indexes

From: [email-masked] (William Morgan)
Date: Sun, 24 Feb 2008 09:15:32 -0800

I've just merged in a changeset that makes Sup store message body
content in the Ferret index. (They've always been indexed, but now
they're stored as well.) This means that changing the labels on a
message can be a copy operation of the previous Ferret document, rather
than requiring downloading and parsing the original message to create a
new Ferret document.

So, this should have two effects:

1. The Ferret index size will expand by about 50%. Sorry.
2. Tweaking message labels should be much, much faster, since the
   message no longer has to be downloaded from the source in order to
   change the labels. If you've ever tried to label a large IMAP thread,
   you no longer have to wait 5 minutes just to save. :)

The index size increase is unfortunate, but it's something that has to
happen anyways if we want search-results-mode to have matching text in
the snippets, which is in the future TODO.

The change was made in such a way that it's incrementally applied
whenever a message is saved or changed in the Ferret index. So, if you
want the above behavior on all messages immediately, you must do
sup-sync --all on a source (which will require downloading each
message). Otherwise, you will get the slow behavior (message body needs
to be downloaded from the source) the first time you save a message
after merging this change, and the fast behavior (no downloading
required) on all subsequent times.

I'm planning on committing another change in the near future that will
also require a sup-sync --all, so you might want to hold out for that if
you have a lot of messages. (I'm planning to have message state be
duplicated outside the Ferret index, so that Ferret index corruption is
a nuisance rather than a nightmare.)

There's also a potential next step here where no interaction with
external sources is *ever* required except at poll time, which would
mean IMAP would become actually useable. (The other option being just
figure out why the Ruby IMAP library is so incredibly slow.)

-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Marcus Williams)
Date: Wed, 27 Feb 2008 15:19:03 +0000

On 24.2.2008, William Morgan wrote:
> whenever a message is saved or changed in the Ferret index.

I'm not sure if this is down to this change in next or not but I'm
seeing an oddity with mail lists where my outgoing message (in sent)
doesnt get updated by the incoming message (from the mail list).

An example might be clearer - I send a message to sup-talk and I get a
copy of my sent message in my inbox. I then get the same message sent
back to me from sup-talk and what used to happen is the headers would
update to the new incoming message (and all my hooks would kick in).
My recent messages to sup-talk dont seem to be doing this.

So... either a recent change has caused this or sup-talk is now not
delivering copies of my messages to me (which is certainly a mailman
option).

Any ideas?

Marcus

---

From: [email-masked] (William Morgan)
Date: Wed, 27 Feb 2008 09:16:42 -0800

> doesnt get updated by the incoming message (from the mail list).

Good catch. This is an unintented consequence of that change. I'm seeing
it too. I'll work on a fix---I have an idea and I don't think it will
be too hard.

In the mean time, feel free to git reset to back to before that commit. :)

---

From: [email-masked] (Christopher Warrington)
Date: Wed, 27 Feb 2008 12:55:12 -0600

Excerpts from William Morgan's message of Sun Feb 24 11:15:32 -0600 2008:
> mean IMAP would become actually useable. (The other option being
> just figure out why the Ruby IMAP library is so incredibly slow.)

Now, we have offline(-ish) mode for IMAP. Horray! I can do e-mail on
the plane.

-- 
Christopher Warrington <chrisw at rice.edu>

---

From: [email-masked] (William Morgan)
Date: Thu, 28 Feb 2008 09:23:29 -0800

> Now, we have offline(-ish) mode for IMAP. Horray! I can do e-mail on

You could always have used offlineimap. :)

---

From: [email-masked] (William Morgan)
Date: Thu, 28 Feb 2008 10:25:45 -0800

Reformatted excerpts from William Morgan's message of 2008-02-27:
> Good catch. This is an unintented consequence of that change. I'm
> seeing it too. I'll work on a fix---I have an idea and I don't think
> it will be too hard.

I think I've solved this in next. Later versions of the same message
can overwrite previous versions, except for labels, which are merged in.
This should address a previous bug as well.

Try it out and lmk. A sup-sync --changed should fix everything.
(Make a backup first, of course!)

---

From: [email-masked] (Marcus Williams)
Date: Thu, 28 Feb 2008 23:05:20 +0000

On 28.2.2008, William Morgan wrote:
> I think I've solved this in next. 

I just did an update but didnt get anything new other than the pgp
stuff. You sure its in next? :)

Marcus

---

From: [email-masked] (William Morgan)
Date: Thu, 28 Feb 2008 16:28:22 -0800

> stuff. You sure its in next? :)

Should be: commit 28f7a766f32040f993af88dae9fe2d994a80f16c.

---

From: [email-masked] (William Morgan)
Date: Thu, 28 Feb 2008 17:27:27 -0800

Hah, I hadn't actually pushed it out yet. In the mean time I found
another bug, and fixed it, so... try now.

---

From: [email-masked] (Christopher Warrington)
Date: Mon, 03 Mar 2008 01:42:53 -0600

Excerpts from William Morgan's message of Thu Feb 28 11:23:29 -0600 2008:
>> Now, we have offline(-ish) mode for IMAP. Horray! I can do e-mail
>> on the plane.

Wow! That's an amazing program! I'm using it now to read and write
e-mail using sup at 30,000 feet (and much quicker than IMAP, too!).

(Message sent at a paltry 200 feet...)