Subject: Re: [sup-devel] [sup-talk] Ferret to Xapian conversion

From: Rich Lane <[email-masked]>
Date: Sun, 03 Jan 2010 15:18:35 -0500

Excerpts from Anthony Martinez's message of 2010-01-03 14:38:52 -0500:
> Speaking of not having the Xapian gem, when I tried "gem install xapian-full",
> it looked like it worked, but didn't actually install a gem on my system. Here's
> the build log: http://pastebin.com/f2d25a13c
> 
> I tried the regular 'xapian' gem, but the version available in Debian unstable
> is newer than the gem bindings, so that one outright refuses to compile.

I noticed that gem thinks the xapian compile failed, but it actually
succeeded and since it was in-place the installed gem worked fine. I've
pushed the ncursesw source to my github so please tinker and send me a
patch. I'm traveling for the next day so I won't be able to fix it soon
myself.
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Sat, 02 Jan 2010 14:34:46 -0800

Reformatted excerpts from William Morgan's message of 2010-01-02:
> I'm just about to start work on a coversion script, so if you want to
> wait a few hours, it should be easy to have an automated version of
> this.

Or a few days.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Kevin Riggle <[email-masked]>
Date: Sun, 03 Jan 2010 18:52:20 -0500

Excerpts from William Morgan's message of Sun Jan 03 10:18:53 -0500 2010:
> If you run this script, please report your experience, since I'd like to
> include it in the 0.10 release coming soon.
> 
When I run the script, I get a number of lines complaining about being unable
to convert various encodings -- is this expected behavior?

eg.:

## read 15375m (about 19%) @ 8.1m/s. 0:31:44 elapsed, about 2:17:28 remaining
[Sun Jan 03 14:53:33 -0500 2010] WARNING: couldn't transcode text from UTF-8 (utf-8) to UTF-8) ("Summer time is popul"...) (got "\223assisting\224 with"... (Iconv::IllegalSequence))
...
[Sun Jan 03 15:31:45 -0500 2010] WARNING: couldn't transcode text from ANSI (ANSI) to UTF-8) ("Dear Librarian, \n\nBr"...) (got invalid encoding ("UTF-8", "ANSI") (Iconv::InvalidEncoding))
## read 33992m (about 41%) @ 8.1m/s. 1:10:13 elapsed, about 1:39:07 remaining
[Sun Jan 03 15:31:55 -0500 2010] WARNING: couldn't transcode text from X-UNKNOWN (ASCII) to UTF-8) ("I agree with Jacky's"...) (got "\240 \240 \240I actually "... (Iconv::IllegalSequence))

Also, the conversion appeared to terminate prematurely, so I didn't bother 
running Sup against the resulting database.  The last few lines were:

## read 34123m (about 42%) @ 8.1m/s. 1:10:28 elapsed, about 1:38:49 remaining
## read 34259m (about 42%) @ 8.1m/s. 1:10:43 elapsed, about 1:38:30 remaining
## read 34388m (about 42%) @ 8.1m/s. 1:10:58 elapsed, about 1:38:12 remaining
kevinr@black-opal:~/src/sup$ 

- Kevin
-- 
Kevin Riggle ([email-masked]) 
MIT Class of 2010, Course VI-3 (Computer Science)
http://free-dissociation.com
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Sat, 02 Jan 2010 13:24:04 -0800

Reformatted excerpts from Anthony Martinez's message of 2010-01-02:
> A quick glance over the recent patches doesn't suggest anything that would break
> the procedure on the Wiki, and I assume that Rich Lane and others are running on
> Xapian without excessive trouble, so the only difference from the Wiki is that
> we can run sup without -o and rm -rf ~/.sup/ferret nowadays.

> Is that an accurate assessment?

Yes.

---

From: Anthony Martinez <[email-masked]>
Date: Sun, 03 Jan 2010 15:47:39 -0700

Excerpts from Rich Lane's message of Sun Jan 03 13:18:35 -0700 2010:
> succeeded and since it was in-place the installed gem worked fine.

I don't know enough about packaging Gems to help out, unfortunately. The way I'm
proceeding for now is to add "-I /var/lib/gems/1.8/gems/xapian-full-1.1.3/lib/"
to my Ruby command line (and in the case of the conversion script, to the
RUBY_INVOCATION environment). Life is ridiculous. :)

It appears to have worked, though, as I'm now writing this from a
Xapian-indexed sup!
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Anthony Martinez <[email-masked]>
Date: Sun, 03 Jan 2010 12:38:52 -0700

> include it in the 0.10 release coming soon.

Here is a patch to make it complain about not having Xapian *before* starting
the state dump, so it doesn't get to step 3 and suddenly find out that it can't
load Xapian.

Help!

diff --git a/bin/sup-convert-ferret-index b/bin/sup-convert-ferret-index
index 96a97aa..e78ac4b 100755
--- a/bin/sup-convert-ferret-index
+++ b/bin/sup-convert-ferret-index
@@ -42,6 +42,12 @@ def run cmd
   puts
 end
 
+begin
+    require 'xapian'
+rescue LoadError
+    Trollop::die "you don't have the xapian gem installed, so this script won't do much for you. `gem install xapian' first."
+end
+
 Redwood::start
 index = Redwood::Index.init
 Trollop::die "you appear to already have a Xapian index--delete #{File.join(Redwood::BASE_DIR, "xapian")} if you really want to do this" unless Redwood::Index.is_a_deprecated_ferret_index?
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Anthony Martinez <[email-masked]>
Date: Sat, 02 Jan 2010 13:06:16 -0700

Ferret is starting to give me trouble (StateErrors when I'm marking messages in
the Unread mode as read), so I figure that there is no better time to convert to
Xapian. However, the documentation on the Wiki hasn't been updated since
(September of) last year.

-- Anthony

---

From: William Morgan <[email-masked]>
Date: Sun, 03 Jan 2010 07:18:53 -0800

If you pull the latest git next, there should be a
sup-convert-ferret-index script which should automate this process. (You
will also start getting yelled at for using Ferret.)

It doesn't remove your old .ferret directory, so if anything goes wrong
(unlikely, but hey), you should be able to recover.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Tue, 05 Jan 2010 13:15:38 -0800

> to convert various encodings -- is this expected behavior?

Not a problem at all. (Typically spam with a wrong declared charset.) I've
turned this down to a debug message so you won't even see it any more.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk