Subject: [sup-talk] [PATCH] xapian: replace DocumentMethods module with plain monkeypatching

From: Rich Lane <[email-masked]>
Date: Mon, 19 Oct 2009 23:14:07 -0700

---
 lib/sup/xapian_index.rb |   47 ++++++++++++++++++++++-------------------------
 1 files changed, 22 insertions(+), 25 deletions(-)

diff --git a/lib/sup/xapian_index.rb b/lib/sup/xapian_index.rb
index ad45b0e..34d67d5 100644
--- a/lib/sup/xapian_index.rb
+++ b/lib/sup/xapian_index.rb
@@ -565,35 +565,32 @@ EOS
       raise "Invalid term type #{type}"
     end
   end
+end
 
-  module DocumentMethods
-    def entry
-      Marshal.load data
-    end
-
-    def entry=(x)
-      self.data = Marshal.dump x
-    end
+end
 
-    def index_text text, prefix, weight=1
-      term_generator = Xapian::TermGenerator.new
-      term_generator.stemmer = Xapian::Stem.new(STEM_LANGUAGE)
-      term_generator.document = self
-      term_generator.index_text text, weight, prefix
-    end
+class Xapian::Document
+  def entry
+    Marshal.load data
+  end
 
-    def add_term term
-      if term.length <= MAX_TERM_LENGTH
-        super term
-      else
-        warn "dropping excessively long term #{term}"
-      end
-    end
+  def entry=(x)
+    self.data = Marshal.dump x
   end
-end
 
-end
+  def index_text text, prefix, weight=1
+    term_generator = Xapian::TermGenerator.new
+    term_generator.stemmer = Xapian::Stem.new(Redwood::XapianIndex::STEM_LANGUAGE)
+    term_generator.document = self
+    term_generator.index_text text, weight, prefix
+  end
 
-class Xapian::Document
-  include Redwood::XapianIndex::DocumentMethods
+  alias old_add_term add_term
+  def add_term term
+    if term.length <= Redwood::XapianIndex::MAX_TERM_LENGTH
+      old_add_term term
+    else
+      warn "dropping excessively long term #{term}"
+    end
+  end
 end
-- 
1.6.4.2

_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Mon, 02 Nov 2009 11:28:03 -0800

Branch xapian-bugfix, merged into next. Thanks!
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Thu, 15 Oct 2009 05:59:53 -0700

Reformatted excerpts from Tero Tilus's message of 2009-10-12:
> Looks like lib/sup/xapian_index.rb tries to override
> Xapian::Document#add_term with a version which is wired to ditch too
> long terms.  Only that you can't override methods just by including a
> module.  Methods of the including class override methods in included
> module.

Very good point. Thanks!

---

From: Rich Lane <[email-masked]>
Date: Mon, 19 Oct 2009 22:34:37 -0700

---
 lib/sup/xapian_index.rb |   25 +++++++++++++++++++++++++
 1 files changed, 25 insertions(+), 0 deletions(-)

index e1cfe65..c373c17 100644
@@ -560,7 +560,32 @@ EOS
 
 end
 
+class Xapian::Document
+  def entry
+    Marshal.load data
+  end
+
+  def entry=(x)
+
+

---

From: Rich Lane <[email-masked]>
Date: Tue, 20 Oct 2009 02:13:58 -0400

Disregard this one. (I thought master had already gotten my
update-message-state patch)

Excerpts from Rich Lane's message of Tue Oct 20 01:34:37 -0400 2009:
> +    term_generator.stemmer =
> Xapian::Stem.new(Redwood::XapianIndex::STEM_LANGUAGE)

---

From: Tero Tilus <[email-masked]>
Date: Tue, 13 Oct 2009 01:34:49 +0300

sup-sync blows up like this

/home/terotil/src/sup/lib/sup/xapian_index.rb:446:in `replace_document': InvalidArgumentError: Term too long (> 245): Lfwd: =?iso-8859-1?q?tekij=e4n_oikeudet=5d?= (ArgumentError)

Relevant part of the problematic mail looks like this

To: [email-masked]
Subject: [Fwd: tekijän oikeudet]
Content-Length: 11892

This is how I solved it for me, for now

diff --git a/lib/sup/xapian_index.rb b/lib/sup/xapian_index.rb
index ad45b0e..d3b3e25 100644
--- a/lib/sup/xapian_index.rb
+++ b/lib/sup/xapian_index.rb
@@ -443,7 +443,11 @@ EOS
         warn "docid underflow, dropping #{m.id.inspect}"
         return
       end
-      @xapian.replace_document docid, doc
+      begin
+        @xapian.replace_document docid, doc
+      rescue StandardError => err
+        warn "Failed to add message #{m.id.inspect} to Xapian index: #{err}"
+      end
     end
 
     m.labels.each { |l| LabelManager << l }

terotil@sotka:~$ irb
> class Foo; def bar; :bar; end; end
=> nil
> module Baz; def bar; :baz; end; end
=> nil
> class Foo; include Baz; end
=> Foo
> Foo.new.bar
=> :bar
> Foo.ancestors
=> [Foo, Baz, Object, Kernel]  # Foo before Baz, methods in Foo take priority

It is still Foo#bar being called, not Baz#bar.  You need to open up
Xapian::Document and then do alias method chaining to override
methods.  Or you could do tricks like
http://coderrr.wordpress.com/2008/10/29/secure-alias-method-chaining/

-- 
Tero Tilus ## 050 3635 235 ## http://tero.tilus.net/
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk