* [sup-talk] Ferret to Xapian conversion
@ 2010-01-02 20:06 Anthony Martinez
2010-01-02 21:24 ` William Morgan
0 siblings, 1 reply; 9+ messages in thread
From: Anthony Martinez @ 2010-01-02 20:06 UTC (permalink / raw)
To: sup-devel
Ferret is starting to give me trouble (StateErrors when I'm marking messages in
the Unread mode as read), so I figure that there is no better time to convert to
Xapian. However, the documentation on the Wiki hasn't been updated since
(September of) last year.
A quick glance over the recent patches doesn't suggest anything that would break
the procedure on the Wiki, and I assume that Rich Lane and others are running on
Xapian without excessive trouble, so the only difference from the Wiki is that
we can run sup without -o and rm -rf ~/.sup/ferret nowadays.
Is that an accurate assessment?
-- Anthony
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [sup-talk] Ferret to Xapian conversion
2010-01-02 20:06 [sup-talk] Ferret to Xapian conversion Anthony Martinez
@ 2010-01-02 21:24 ` William Morgan
2010-01-02 22:34 ` William Morgan
0 siblings, 1 reply; 9+ messages in thread
From: William Morgan @ 2010-01-02 21:24 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from Anthony Martinez's message of 2010-01-02:
> A quick glance over the recent patches doesn't suggest anything that would break
> the procedure on the Wiki, and I assume that Rich Lane and others are running on
> Xapian without excessive trouble, so the only difference from the Wiki is that
> we can run sup without -o and rm -rf ~/.sup/ferret nowadays.
I'm just about to start work on a coversion script, so if you want to
wait a few hours, it should be easy to have an automated version of
this.
> Is that an accurate assessment?
Yes.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [sup-talk] Ferret to Xapian conversion
2010-01-02 21:24 ` William Morgan
@ 2010-01-02 22:34 ` William Morgan
2010-01-03 15:18 ` William Morgan
0 siblings, 1 reply; 9+ messages in thread
From: William Morgan @ 2010-01-02 22:34 UTC (permalink / raw)
To: sup-talk
Reformatted excerpts from William Morgan's message of 2010-01-02:
> I'm just about to start work on a coversion script, so if you want to
> wait a few hours, it should be easy to have an automated version of
> this.
Or a few days.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [sup-talk] Ferret to Xapian conversion
2010-01-02 22:34 ` William Morgan
@ 2010-01-03 15:18 ` William Morgan
2010-01-03 19:38 ` [sup-devel] " Anthony Martinez
2010-01-03 23:52 ` Kevin Riggle
0 siblings, 2 replies; 9+ messages in thread
From: William Morgan @ 2010-01-03 15:18 UTC (permalink / raw)
To: sup-talk
If you pull the latest git next, there should be a
sup-convert-ferret-index script which should automate this process. (You
will also start getting yelled at for using Ferret.)
If you run this script, please report your experience, since I'd like to
include it in the 0.10 release coming soon.
It doesn't remove your old .ferret directory, so if anything goes wrong
(unlikely, but hey), you should be able to recover.
--
William <wmorgan-sup@masanjin.net>
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [sup-devel] [sup-talk] Ferret to Xapian conversion
2010-01-03 15:18 ` William Morgan
@ 2010-01-03 19:38 ` Anthony Martinez
2010-01-03 20:18 ` Rich Lane
2010-01-03 23:52 ` Kevin Riggle
1 sibling, 1 reply; 9+ messages in thread
From: Anthony Martinez @ 2010-01-03 19:38 UTC (permalink / raw)
To: sup-talk; +Cc: sup-devel
Excerpts from William Morgan's message of Sun Jan 03 08:18:53 -0700 2010:
> If you run this script, please report your experience, since I'd like to
> include it in the 0.10 release coming soon.
Here is a patch to make it complain about not having Xapian *before* starting
the state dump, so it doesn't get to step 3 and suddenly find out that it can't
load Xapian.
Speaking of not having the Xapian gem, when I tried "gem install xapian-full",
it looked like it worked, but didn't actually install a gem on my system. Here's
the build log: http://pastebin.com/f2d25a13c
I tried the regular 'xapian' gem, but the version available in Debian unstable
is newer than the gem bindings, so that one outright refuses to compile.
Help!
diff --git a/bin/sup-convert-ferret-index b/bin/sup-convert-ferret-index
index 96a97aa..e78ac4b 100755
--- a/bin/sup-convert-ferret-index
+++ b/bin/sup-convert-ferret-index
@@ -42,6 +42,12 @@ def run cmd
puts
end
+begin
+ require 'xapian'
+rescue LoadError
+ Trollop::die "you don't have the xapian gem installed, so this script won't do much for you. `gem install xapian' first."
+end
+
Redwood::start
index = Redwood::Index.init
Trollop::die "you appear to already have a Xapian index--delete #{File.join(Redwood::BASE_DIR, "xapian")} if you really want to do this" unless Redwood::Index.is_a_deprecated_ferret_index?
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [sup-devel] [sup-talk] Ferret to Xapian conversion
2010-01-03 19:38 ` [sup-devel] " Anthony Martinez
@ 2010-01-03 20:18 ` Rich Lane
2010-01-03 22:47 ` Anthony Martinez
0 siblings, 1 reply; 9+ messages in thread
From: Rich Lane @ 2010-01-03 20:18 UTC (permalink / raw)
To: Anthony Martinez; +Cc: sup-talk, sup-devel
Excerpts from Anthony Martinez's message of 2010-01-03 14:38:52 -0500:
> Speaking of not having the Xapian gem, when I tried "gem install xapian-full",
> it looked like it worked, but didn't actually install a gem on my system. Here's
> the build log: http://pastebin.com/f2d25a13c
>
> I tried the regular 'xapian' gem, but the version available in Debian unstable
> is newer than the gem bindings, so that one outright refuses to compile.
I noticed that gem thinks the xapian compile failed, but it actually
succeeded and since it was in-place the installed gem worked fine. I've
pushed the ncursesw source to my github so please tinker and send me a
patch. I'm traveling for the next day so I won't be able to fix it soon
myself.
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [sup-devel] [sup-talk] Ferret to Xapian conversion
2010-01-03 20:18 ` Rich Lane
@ 2010-01-03 22:47 ` Anthony Martinez
0 siblings, 0 replies; 9+ messages in thread
From: Anthony Martinez @ 2010-01-03 22:47 UTC (permalink / raw)
To: Rich Lane; +Cc: sup-talk, sup-devel
Excerpts from Rich Lane's message of Sun Jan 03 13:18:35 -0700 2010:
> Excerpts from Anthony Martinez's message of 2010-01-03 14:38:52 -0500:
> > Speaking of not having the Xapian gem, when I tried "gem install xapian-full",
> > it looked like it worked, but didn't actually install a gem on my system. Here's
> > the build log: http://pastebin.com/f2d25a13c
>
> I noticed that gem thinks the xapian compile failed, but it actually
> succeeded and since it was in-place the installed gem worked fine.
I don't know enough about packaging Gems to help out, unfortunately. The way I'm
proceeding for now is to add "-I /var/lib/gems/1.8/gems/xapian-full-1.1.3/lib/"
to my Ruby command line (and in the case of the conversion script, to the
RUBY_INVOCATION environment). Life is ridiculous. :)
It appears to have worked, though, as I'm now writing this from a
Xapian-indexed sup!
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [sup-talk] Ferret to Xapian conversion
2010-01-03 15:18 ` William Morgan
2010-01-03 19:38 ` [sup-devel] " Anthony Martinez
@ 2010-01-03 23:52 ` Kevin Riggle
2010-01-05 21:15 ` William Morgan
1 sibling, 1 reply; 9+ messages in thread
From: Kevin Riggle @ 2010-01-03 23:52 UTC (permalink / raw)
To: sup-talk
Excerpts from William Morgan's message of Sun Jan 03 10:18:53 -0500 2010:
> If you run this script, please report your experience, since I'd like to
> include it in the 0.10 release coming soon.
>
When I run the script, I get a number of lines complaining about being unable
to convert various encodings -- is this expected behavior?
eg.:
## read 15375m (about 19%) @ 8.1m/s. 0:31:44 elapsed, about 2:17:28 remaining
[Sun Jan 03 14:53:33 -0500 2010] WARNING: couldn't transcode text from UTF-8 (utf-8) to UTF-8) ("Summer time is popul"...) (got "\223assisting\224 with"... (Iconv::IllegalSequence))
...
[Sun Jan 03 15:31:45 -0500 2010] WARNING: couldn't transcode text from ANSI (ANSI) to UTF-8) ("Dear Librarian, \n\nBr"...) (got invalid encoding ("UTF-8", "ANSI") (Iconv::InvalidEncoding))
## read 33992m (about 41%) @ 8.1m/s. 1:10:13 elapsed, about 1:39:07 remaining
[Sun Jan 03 15:31:55 -0500 2010] WARNING: couldn't transcode text from X-UNKNOWN (ASCII) to UTF-8) ("I agree with Jacky's"...) (got "\240 \240 \240I actually "... (Iconv::IllegalSequence))
Also, the conversion appeared to terminate prematurely, so I didn't bother
running Sup against the resulting database. The last few lines were:
## read 34123m (about 42%) @ 8.1m/s. 1:10:28 elapsed, about 1:38:49 remaining
## read 34259m (about 42%) @ 8.1m/s. 1:10:43 elapsed, about 1:38:30 remaining
## read 34388m (about 42%) @ 8.1m/s. 1:10:58 elapsed, about 1:38:12 remaining
kevinr@black-opal:~/src/sup$
- Kevin
--
Kevin Riggle (kevinr@free-dissociation.com)
MIT Class of 2010, Course VI-3 (Computer Science)
http://free-dissociation.com
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2010-01-05 21:16 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-01-02 20:06 [sup-talk] Ferret to Xapian conversion Anthony Martinez
2010-01-02 21:24 ` William Morgan
2010-01-02 22:34 ` William Morgan
2010-01-03 15:18 ` William Morgan
2010-01-03 19:38 ` [sup-devel] " Anthony Martinez
2010-01-03 20:18 ` Rich Lane
2010-01-03 22:47 ` Anthony Martinez
2010-01-03 23:52 ` Kevin Riggle
2010-01-05 21:15 ` William Morgan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox