Subject: [sup-talk] [PATCH] add Message#load_from_index! shortcut

From: [email-masked] (Rich Lane)
Date: Sat, 22 Aug 2009 11:28:15 -0700

Message#parse_header is slow, taking up to 2/3 of the time spent loading
threads in thread-index-mode. This patch adds a shortcut method that the index
can use to efficiently initialize a message.
---
 lib/sup/message.rb      |   25 +++++++++++++++++++++++++
 lib/sup/xapian_index.rb |   30 +++++++++++-------------------
 2 files changed, 36 insertions(+), 19 deletions(-)

diff --git a/lib/sup/message.rb b/lib/sup/message.rb
index 965c10e..56e66de 100644
--- a/lib/sup/message.rb
+++ b/lib/sup/message.rb
@@ -127,6 +127,31 @@ class Message
     @list_unsubscribe = header["list-unsubscribe"]
   end
 
+  ## Expected index entry format:
+  ## :message_id, :subject => String
+  ## :date => Time
+  ## :refs, :replytos => Array of String
+  ## :from => Person
+  ## :to, :cc, :bcc => Array of Person
+  def load_from_index! entry
+    @id = entry[:message_id]
+    @from = entry[:from]
+    @date = entry[:date]
+    @subj = entry[:subject]
+    @to = entry[:to]
+    @cc = entry[:cc]
+    @bcc = entry[:bcc]
+    @refs = (@refs + entry[:refs]).uniq
+    @replytos = entry[:replytos]
+
+    @replyto = nil
+    @list_address = nil
+    @recipient_email = nil
+    @source_marked_read = false
+    @list_subscribe = nil
+    @list_unsubscribe = nil
+  end
+
   def add_ref ref
     @refs << ref
     @dirty = true
diff --git a/lib/sup/xapian_index.rb b/lib/sup/xapian_index.rb
index 85f6ef0..c260728 100644
--- a/lib/sup/xapian_index.rb
+++ b/lib/sup/xapian_index.rb
@@ -71,25 +71,17 @@ class XapianIndex < BaseIndex
     source = SourceManager[entry[:source_id]]
     raise "invalid source #{entry[:source_id]}" unless source
 
-    mk_addrs = lambda { |l| l.map { |e,n| "#{n} <#{e}>" } * ', ' }
-    mk_refs = lambda { |l| l.map { |r| "<#{r}>" } * ' ' }
-    fake_header = {
-      'message-id' => entry[:message_id],
-      'date' => Time.at(entry[:date]),
-      'subject' => entry[:subject],
-      'from' => mk_addrs[[entry[:from]]],
-      'to' => mk_addrs[entry[:to]],
-      'cc' => mk_addrs[entry[:cc]],
-      'bcc' => mk_addrs[entry[:bcc]],
-      'reply-tos' => mk_refs[entry[:replytos]],
-      'references' => mk_refs[entry[:refs]],
-     }
-
-      m = Message.new :source => source, :source_info => entry[:source_info],
-                  :labels => entry[:labels],
-                  :snippet => entry[:snippet]
-      m.parse_header fake_header
-      m
+    m = Message.new :source => source, :source_info => entry[:source_info],
+                    :labels => entry[:labels], :snippet => entry[:snippet]
+
+    mk_person = lambda { |x| Person.new(*x.reverse!) }
+    entry[:from] = mk_person[entry[:from]]
+    entry[:to].map!(&mk_person)
+    entry[:cc].map!(&mk_person)
+    entry[:bcc].map!(&mk_person)
+
+    m.load_from_index! entry
+    m
   end
 
   def add_message m; sync_message m end
-- 
1.6.4

---

From: [email-masked] (Rich Lane)
Date: Sat, 22 Aug 2009 11:54:23 -0700

Another 10% query performance boost.
---
 lib/sup/xapian_index.rb |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/lib/sup/xapian_index.rb b/lib/sup/xapian_index.rb
index c260728..3a23951 100644
--- a/lib/sup/xapian_index.rb
+++ b/lib/sup/xapian_index.rb
@@ -74,7 +74,7 @@ class XapianIndex < BaseIndex
     m = Message.new :source => source, :source_info => entry[:source_info],
                     :labels => entry[:labels], :snippet => entry[:snippet]
 
-    mk_person = lambda { |x| Person.new(*x.reverse!) }
+    mk_person = lambda { |x| QuickPerson.new(*x) }
     entry[:from] = mk_person[entry[:from]]
     entry[:to].map!(&mk_person)
     entry[:cc].map!(&mk_person)
@@ -84,6 +84,14 @@ class XapianIndex < BaseIndex
     m
   end
 
+  class QuickPerson < Person
+    def initialize email, name
+      raise ArgumentError, "email can't be nil" unless email
+      @email = email
+      @name = name
+    end
+  end
+
   def add_message m; sync_message m end
   def update_message m; sync_message m end
   def update_message_state m; sync_message m end
-- 
1.6.4

---

From: [email-masked] (William Morgan)
Date: Mon, 24 Aug 2009 16:23:32 -0700

Reformatted excerpts from Rich Lane's message of 2009-08-22:
> Message#parse_header is slow, taking up to 2/3 of the time spent
> loading threads in thread-index-mode. This patch adds a shortcut
> method that the index can use to efficiently initialize a message.

Yes, great catch. I've applied this to xapian-updates branch and
remerged into next.

BTW can you base your patches off master instead of next? It will save
me a couple keystrokes.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (William Morgan)
Date: Mon, 24 Aug 2009 16:24:19 -0700

> +    mk_person = lambda { |x| QuickPerson.new(*x) }

What about lambda { |x| Person.new x[1], x[0] }. Surely that must be
even faster?
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Rich Lane)
Date: Tue, 25 Aug 2009 21:50:48 -0400

> What about lambda { |x| Person.new x[1], x[0] }. Surely that must be

The slow part is the processing in Person#initialize, which QuickPerson
overrides. You might also be able to avoid that by moving the
initialize() code into Person.from_address.

---

From: [email-masked] (William Morgan)
Date: Tue, 01 Sep 2009 13:59:56 -0700

Reformatted excerpts from Rich Lane's message of 2009-08-25:
> The slow part is the processing in Person#initialize, which
> QuickPerson overrides. You might also be able to avoid that by moving
> the initialize() code into Person.from_address.

Yeah, I think that's a better approach. There's no need for the
constructor to do all that work. I'll work on this when I get a chance,
or feel free to beat me to it.
-- 
William <wmorgan-sup at masanjin.net>