Subject: [sup-talk] xapian merged into next

From: [email-masked] (William Morgan)
Date: Mon, 27 Jul 2009 09:13:16 -0700

I've merged the xapian branch into next. Users of next need not worry;
xapian only enabled when you set the environment variable
SUP_INDEX=xapian. (And you'll have to regenerate your index; see my
message of a few weeks ago for how.)
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guillaume Quintard)
Date: Mon, 27 Jul 2009 18:16:50 +0200

> message of a few weeks ago for how.)

The big question is: is it interesting, as a user, to switch? :-)

-- 
Guillaume

---

From: [email-masked] (William Morgan)
Date: Mon, 27 Jul 2009 09:27:42 -0700

> The big question is: is it interesting, as a user, to switch? :-)

Yes. It's noticeably faster than Ferret, especially for loading large
threads in thread-index-mode. (Which isn't Xapian per se, but other
improvements Rich has made).

It's also much larger on disk, though there might be a way to trim that
down.

At some point I want to deprecate Ferret, since it's unmaintained, so
you'll be forced to switch. No timeline on that though.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guillaume Quintard)
Date: Mon, 27 Jul 2009 18:31:46 +0200

> improvements Rich has made).
Yummy!
> It's also much larger on disk, though there might be a way to trim that
Less yummy!

So, we just export SUP_INDEX=xapian and that's it? We start with an
empty sup and we just have to reimport the mbox/maildir/whatever?
Means losing the red states and tags, I guess.

-- 
Guillaume

---

From: [email-masked] (William Morgan)
Date: Mon, 27 Jul 2009 09:44:54 -0700

> Means losing the red states and tags, I guess.

Current instructions:

1. install the ruby xapian library and the ruby gdbm library, if you
   don't have them. These are packaged by your distro, and are not gems.
2. git checkout next
3. git pull
4. cp ~/.sup/sources.yaml /tmp # just in case
5. ruby -Ilib bin/sup-dump > dumpfile
6. SUP_INDEX=xapian ruby -Ilib bin/sup-sync --all --all-sources --restore dumpfile
7. SUP_INDEX=xapian ruby -Ilib bin/sup -o
8. Oooh, fast.

This should not disturb your Ferret index, so you can switch back and forth
between the two. (Message state, of course, is not shared.) However, adding new
messages to one index will prevent it from being automatically added to the
other, so I recommend running in Xapian mode with -o and not pressing 'P'.
Unless, of of course, you're ready to switch permanently, in which case rm -rf
~/.sup/ferret. :)
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guillaume Quintard)
Date: Mon, 27 Jul 2009 18:47:13 +0200

> Unless, of of course, you're ready to switch permanently, in which case rm -rf

There's no reason not to, right?
/me is following blinding the given instructions :-)

-- 
Guillaume

---

From: [email-masked] (William Morgan)
Date: Mon, 27 Jul 2009 09:50:12 -0700

> There's no reason not to, right?

Well, the code isn't quite as well tested...
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guillaume Quintard)
Date: Mon, 27 Jul 2009 19:09:19 +0200

> Well, the code isn't quite as well tested...

Someone has to do it, plus I still have my mbox archive...
...and I was getting tired of this mostly bugless mail experience.

-- 
Guillaume

---

From: [email-masked] (William Morgan)
Date: Mon, 27 Jul 2009 10:34:59 -0700

> ...and I was getting tired of this mostly bugless mail experience.

Ok, you're the official guinea pig then. :)
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guillaume Quintard)
Date: Mon, 27 Jul 2009 22:04:58 +0200

> 5. ruby -Ilib bin/sup-dump > dumpfile
-> produces a empty file (not sure it's normal)

> 6. SUP_INDEX=xapian ruby -Ilib bin/sup-sync --all --all-sources --restore dumpfile
-> error
[Mon Jul 27 22:01:35 +0200 2009] using character set encoding "UTF-8"
./lib/sup/xapian_index.rb:17:in `at': bignum too big to convert into
`long' (RangeError)
	from ./lib/sup/xapian_index.rb:17
	from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
	from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require'
	from ./lib/sup/index.rb:217
	from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
	from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require'
	from ./lib/sup.rb:269
	from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
	from /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require'
	from bin/sup-sync:6

l17 of xapian_index.rb is MAX_DATE = Time.at(2**31)

-- 
Guillaume

---

From: [email-masked] (Richard Heycock)
Date: Tue, 28 Jul 2009 10:33:16 +1000

> you'll be forced to switch. No timeline on that though.

Just to add to Williams answer not only is it faster it's also
*significantly* more robust. I'm running Debian unstable which, at the
moment is really living up to it's name, consequently my machine is
dying a lot more times than it should and I haven't had to rebuild the
index once. Compare that to ferret where I pretty much has to rebuild
the index every time; I even wrote myself a one line script to do it.

William there is work being done on the next xapian "engine" which aims
to reduce the disc size.

rgh

---

From: [email-masked] (Rich Lane)
Date: Mon, 27 Jul 2009 23:33:16 -0400

> ./lib/sup/xapian_index.rb:17:in `at': bignum too big to convert into

Oops, breaking sup-dump would make switching to Xapian a little
difficult. I've posted patches for both these issues. We really should
have more tests to catch this sort of thing...

---

From: [email-masked] (William Morgan)
Date: Tue, 28 Jul 2009 08:13:52 -0700

Reformatted excerpts from Rich Lane's message of 2009-07-27:
> We really should have more tests to catch this sort of thing...

Agreed. Mostly my fault, I'm afraid.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Guillaume Quintard)
Date: Sat, 1 Aug 2009 19:44:50 +0200

I get this when I run "SUP_INDEX=xapian ruby -Ilib bin/sup-sync --all
--all-sources --restore dumpfile" (only had the time to do it today):

./lib/sup/xapian_index.rb:435:in `replace_document':
InvalidArgumentError: Term too long (> 245):
E"name.surname at enst-bretagne.fr,
my_name.my_surname at enst-bretagne.fr, name.surname at enst-bretagne.fr,
name.surname at enst-bretagne.fr,
name.surname at telecom-bretagne.eu,
name.surname at telecom-bretagne.eu,
name.surname at telecom-bretagne.eu, name.surname"@telecom-bretagne.eu
(ArgumentError)
? ? ? ?from ./lib/sup/xapian_index.rb:435:in `index_message'
? ? ? ?from ./lib/sup/xapian_index.rb:117:in `sync_message'
? ? ? ?from /usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
? ? ? ?from ./lib/sup/xapian_index.rb:330:in `synchronize'
? ? ? ?from ./lib/sup/xapian_index.rb:116:in `sync_message'
? ? ? ?from ./lib/sup/util.rb:519:in `send'
? ? ? ?from ./lib/sup/util.rb:519:in `method_missing'
? ? ? ?from ./lib/sup/poll.rb:157:in `add_messages_from'
? ? ? ?from ./lib/sup/source.rb:100:in `each'
? ? ? ?from ./lib/sup/util.rb:558:in `send'
? ? ? ?from ./lib/sup/util.rb:558:in `__pass'
? ? ? ?from ./lib/sup/util.rb:545:in `method_missing'
? ? ? ?from ./lib/sup/poll.rb:141:in `add_messages_from'
? ? ? ?from ./lib/sup/util.rb:519:in `send'
? ? ? ?from ./lib/sup/util.rb:519:in `method_missing'
? ? ? ?from bin/sup-sync:140
? ? ? ?from bin/sup-sync:135:in `each'
? ? ? ?from bin/sup-sync:135

Cheers!

-- 
Guillaume

---

From: [email-masked] (Rich Lane)
Date: Sat, 01 Aug 2009 14:14:34 -0400

> name.surname at telecom-bretagne.eu, name.surname"@telecom-bretagne.eu

That's a very strange email address :). Could you track down the message
causing this and post the headers? It looks like it's getting parsed
incorrectly.

I think we'd be safe not adding terms for email addresses longer than
244 characters on the assumption that the user isn't going to want to
search for them.

---

From: [email-masked] (Guillaume Quintard)
Date: Sat, 01 Aug 2009 20:31:55 +0200

> 244 characters on the assumption that the user isn't going to want to

http://files.getdropbox.com/u/155904/grepped

I did a simple grep, tell me if it's not enough (I'd rather not dive
into the humongous mbox file).
The mails come from the mailing-list admin tool (sympa), encoding
problem it looks, the mangled part is "Propri?taires de liste" (list
owners in french)

-- 
Guillaume

---

From: [email-masked] (Rich Lane)
Date: Sat, 01 Aug 2009 19:03:06 -0400

> problem it looks, the mangled part is "Propri?taires de liste" (list

I posted a patch that should keep Xapian from throwing an exception
with long email addresses. I'm not any sort of encoding expert, but I'd
guess that sup isn't responsible for the mangling.

---

From: [email-masked] (Guillaume Quintard)
Date: Sun, 02 Aug 2009 01:35:25 +0200

> guess that sup isn't responsible for the mangling.

it isn't, the mangling comes from the mbox file, yet sup was able to
import it at some point. I guesse the quotes don't really help.

-- 
Guillaume