* [sup-talk] new in next: faster saving and bigger indexes
@ 2008-02-24 17:15 William Morgan
2008-02-27 15:19 ` Marcus Williams
2008-02-27 18:55 ` Christopher Warrington
0 siblings, 2 replies; 10+ messages in thread
From: William Morgan @ 2008-02-24 17:15 UTC (permalink / raw)
I've just merged in a changeset that makes Sup store message body
content in the Ferret index. (They've always been indexed, but now
they're stored as well.) This means that changing the labels on a
message can be a copy operation of the previous Ferret document, rather
than requiring downloading and parsing the original message to create a
new Ferret document.
So, this should have two effects:
1. The Ferret index size will expand by about 50%. Sorry.
2. Tweaking message labels should be much, much faster, since the
message no longer has to be downloaded from the source in order to
change the labels. If you've ever tried to label a large IMAP thread,
you no longer have to wait 5 minutes just to save. :)
The index size increase is unfortunate, but it's something that has to
happen anyways if we want search-results-mode to have matching text in
the snippets, which is in the future TODO.
The change was made in such a way that it's incrementally applied
whenever a message is saved or changed in the Ferret index. So, if you
want the above behavior on all messages immediately, you must do
sup-sync --all on a source (which will require downloading each
message). Otherwise, you will get the slow behavior (message body needs
to be downloaded from the source) the first time you save a message
after merging this change, and the fast behavior (no downloading
required) on all subsequent times.
I'm planning on committing another change in the near future that will
also require a sup-sync --all, so you might want to hold out for that if
you have a lot of messages. (I'm planning to have message state be
duplicated outside the Ferret index, so that Ferret index corruption is
a nuisance rather than a nightmare.)
There's also a potential next step here where no interaction with
external sources is *ever* required except at poll time, which would
mean IMAP would become actually useable. (The other option being just
figure out why the Ruby IMAP library is so incredibly slow.)
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-24 17:15 [sup-talk] new in next: faster saving and bigger indexes William Morgan
@ 2008-02-27 15:19 ` Marcus Williams
2008-02-27 17:16 ` William Morgan
2008-02-27 18:55 ` Christopher Warrington
1 sibling, 1 reply; 10+ messages in thread
From: Marcus Williams @ 2008-02-27 15:19 UTC (permalink / raw)
On 24.2.2008, William Morgan wrote:
> The change was made in such a way that it's incrementally applied
> whenever a message is saved or changed in the Ferret index.
I'm not sure if this is down to this change in next or not but I'm
seeing an oddity with mail lists where my outgoing message (in sent)
doesnt get updated by the incoming message (from the mail list).
An example might be clearer - I send a message to sup-talk and I get a
copy of my sent message in my inbox. I then get the same message sent
back to me from sup-talk and what used to happen is the headers would
update to the new incoming message (and all my hooks would kick in).
My recent messages to sup-talk dont seem to be doing this.
So... either a recent change has caused this or sup-talk is now not
delivering copies of my messages to me (which is certainly a mailman
option).
Any ideas?
Marcus
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-27 15:19 ` Marcus Williams
@ 2008-02-27 17:16 ` William Morgan
2008-02-28 18:25 ` William Morgan
0 siblings, 1 reply; 10+ messages in thread
From: William Morgan @ 2008-02-27 17:16 UTC (permalink / raw)
Reformatted excerpts from Marcus Williams's message of 2008-02-27:
> I'm not sure if this is down to this change in next or not but I'm
> seeing an oddity with mail lists where my outgoing message (in sent)
> doesnt get updated by the incoming message (from the mail list).
Good catch. This is an unintented consequence of that change. I'm seeing
it too. I'll work on a fix---I have an idea and I don't think it will
be too hard.
In the mean time, feel free to git reset to back to before that commit. :)
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-24 17:15 [sup-talk] new in next: faster saving and bigger indexes William Morgan
2008-02-27 15:19 ` Marcus Williams
@ 2008-02-27 18:55 ` Christopher Warrington
2008-02-28 17:23 ` William Morgan
1 sibling, 1 reply; 10+ messages in thread
From: Christopher Warrington @ 2008-02-27 18:55 UTC (permalink / raw)
Excerpts from William Morgan's message of Sun Feb 24 11:15:32 -0600 2008:
> There's also a potential next step here where no interaction with
> external sources is *ever* required except at poll time, which would
> mean IMAP would become actually useable. (The other option being
> just figure out why the Ruby IMAP library is so incredibly slow.)
Now, we have offline(-ish) mode for IMAP. Horray! I can do e-mail on
the plane.
--
Christopher Warrington <chrisw at rice.edu>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-27 18:55 ` Christopher Warrington
@ 2008-02-28 17:23 ` William Morgan
2008-03-03 7:42 ` Christopher Warrington
0 siblings, 1 reply; 10+ messages in thread
From: William Morgan @ 2008-02-28 17:23 UTC (permalink / raw)
Reformatted excerpts from Christopher Warrington's message of 2008-02-27:
> Now, we have offline(-ish) mode for IMAP. Horray! I can do e-mail on
> the plane.
You could always have used offlineimap. :)
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-27 17:16 ` William Morgan
@ 2008-02-28 18:25 ` William Morgan
2008-02-28 23:05 ` Marcus Williams
0 siblings, 1 reply; 10+ messages in thread
From: William Morgan @ 2008-02-28 18:25 UTC (permalink / raw)
Reformatted excerpts from William Morgan's message of 2008-02-27:
> Good catch. This is an unintented consequence of that change. I'm
> seeing it too. I'll work on a fix---I have an idea and I don't think
> it will be too hard.
I think I've solved this in next. Later versions of the same message
can overwrite previous versions, except for labels, which are merged in.
This should address a previous bug as well.
Try it out and lmk. A sup-sync --changed should fix everything.
(Make a backup first, of course!)
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-28 18:25 ` William Morgan
@ 2008-02-28 23:05 ` Marcus Williams
2008-02-29 0:28 ` William Morgan
0 siblings, 1 reply; 10+ messages in thread
From: Marcus Williams @ 2008-02-28 23:05 UTC (permalink / raw)
On 28.2.2008, William Morgan wrote:
> I think I've solved this in next.
I just did an update but didnt get anything new other than the pgp
stuff. You sure its in next? :)
Marcus
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-28 23:05 ` Marcus Williams
@ 2008-02-29 0:28 ` William Morgan
2008-02-29 1:27 ` William Morgan
0 siblings, 1 reply; 10+ messages in thread
From: William Morgan @ 2008-02-29 0:28 UTC (permalink / raw)
Reformatted excerpts from Marcus Williams's message of 2008-02-28:
> I just did an update but didnt get anything new other than the pgp
> stuff. You sure its in next? :)
Should be: commit 28f7a766f32040f993af88dae9fe2d994a80f16c.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-29 0:28 ` William Morgan
@ 2008-02-29 1:27 ` William Morgan
0 siblings, 0 replies; 10+ messages in thread
From: William Morgan @ 2008-02-29 1:27 UTC (permalink / raw)
Hah, I hadn't actually pushed it out yet. In the mean time I found
another bug, and fixed it, so... try now.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [sup-talk] new in next: faster saving and bigger indexes
2008-02-28 17:23 ` William Morgan
@ 2008-03-03 7:42 ` Christopher Warrington
0 siblings, 0 replies; 10+ messages in thread
From: Christopher Warrington @ 2008-03-03 7:42 UTC (permalink / raw)
Excerpts from William Morgan's message of Thu Feb 28 11:23:29 -0600 2008:
>> Now, we have offline(-ish) mode for IMAP. Horray! I can do e-mail
>> on the plane.
> You could always have used offlineimap. :)
Wow! That's an amazing program! I'm using it now to read and write
e-mail using sup at 30,000 feet (and much quicker than IMAP, too!).
(Message sent at a paltry 200 feet...)
--
Christopher Warrington <chrisw at rice.edu>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2008-03-03 7:42 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-02-24 17:15 [sup-talk] new in next: faster saving and bigger indexes William Morgan
2008-02-27 15:19 ` Marcus Williams
2008-02-27 17:16 ` William Morgan
2008-02-28 18:25 ` William Morgan
2008-02-28 23:05 ` Marcus Williams
2008-02-29 0:28 ` William Morgan
2008-02-29 1:27 ` William Morgan
2008-02-27 18:55 ` Christopher Warrington
2008-02-28 17:23 ` William Morgan
2008-03-03 7:42 ` Christopher Warrington
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox