Subject: [sup-talk] [PATCH] xapian: drop excessively long terms

From: [email-masked] (Rich Lane)
Date: Sat, 1 Aug 2009 15:56:09 -0700

---
 lib/sup/xapian_index.rb |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/lib/sup/xapian_index.rb b/lib/sup/xapian_index.rb
index 6358a20..5a5dfc1 100644
--- a/lib/sup/xapian_index.rb
+++ b/lib/sup/xapian_index.rb
@@ -304,6 +304,8 @@ class XapianIndex < BaseIndex
 
   DATE_VALUENO = 0
 
+  MAX_TERM_LENGTH = 245
+
   # Xapian can very efficiently sort in ascending docid order. Sup always wants
   # to sort by descending date, so this method maps between them. In order to
   # handle multiple messages per second, we use a logistic curve centered
@@ -428,7 +430,7 @@ class XapianIndex < BaseIndex
 
     @term_generator.document = doc
     text.each { |text,prefix| @term_generator.index_text text, 1, prefix }
-    terms.each { |term| doc.add_term term }
+    terms.each { |term| doc.add_term term if term.length <= MAX_TERM_LENGTH }
     doc.add_value DATE_VALUENO, date_value
     doc.data = m.id
 
-- 
1.6.0.4

---

From: [email-masked] (William Morgan)
Date: Wed, 05 Aug 2009 06:55:47 -0700

Applied, thanks!

BTW I've merged the xapian stuff into master.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Edward Z. Yang)
Date: Wed, 05 Aug 2009 10:54:31 -0400

> BTW I've merged the xapian stuff into master.

Exciting days we live in. :-) Maybe I'll wait another week and update my local
checkout.

Cheers,
Edward

---

From: [email-masked] (Rich Lane)
Date: Wed, 05 Aug 2009 11:36:19 -0400

> BTW I've merged the xapian stuff into master.

Let's not advertise this as stable just yet - I've got a patchset that
removes the GDBM databases which will break compatibility. Using just
Xapian means better consistency if your kernel panics and potentially
better performance. However, I have some Xapian bugs to squash before
it's usable.

---

From: [email-masked] (William Morgan)
Date: Wed, 05 Aug 2009 08:48:35 -0700

Reformatted excerpts from Rich Lane's message of 2009-08-05:
> removes the GDBM databases which will break compatibility.

Since the xapian stuff is only enabled via an obscure environment
variable, I'm not too concerned about it being uber-stable. I've merged
this more for the internal API changes, which seem stable and which I'm
now building off of.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Nicolas Pouillard)
Date: Wed, 05 Aug 2009 19:29:48 +0200

Excerpts from Rich Lane's message of Wed Aug 05 17:36:19 +0200 2009:
> Let's not advertise this as stable just yet -

> I've got a patchset that

I was planning to try a migration today, should I better wait a little?

-- 
Nicolas Pouillard
http://nicolaspouillard.fr

---

From: [email-masked] (Rich Lane)
Date: Wed, 05 Aug 2009 13:48:11 -0400

> I was planning to try a migration today, should I better wait a little?

I'd say go for it, just be aware you'll need to reindex soon. For the
full experience compile svn xapian and export XAPIAN_PREFER_CHERT=1.

---

From: [email-masked] (Nicolas Pouillard)
Date: Wed, 05 Aug 2009 19:58:43 +0200

> full experience compile svn xapian and export XAPIAN_PREFER_CHERT=1.

When you say the svn version you mean the trunk or the 1.0-branch ?

---

From: [email-masked] (Rich Lane)
Date: Wed, 05 Aug 2009 14:06:39 -0400

> When you say the svn version you mean the trunk or the 1.0-branch ?

Trunk.