Re: [sup-devel] Cannot query Japanese characters

From: William Morgan <wmorgan-sup@masanjin.net>
To: sup-devel <sup-devel@rubyforge.org>
Subject: Re: [sup-devel] Cannot query Japanese characters
Date: Tue, 03 May 2011 22:26:04 +0000	[thread overview]
Message-ID: <1304460745-sup-6241@masanjin.net> (raw)
In-Reply-To: <BANLkTi=tSnbEijoEHG76Z5Fy9-3G4TPxVw@mail.gmail.com>

Reformatted excerpts from Horacio Sanson's message of 2011-05-03:
> index = Index.new "index" => #<Whistlepig::Index:0x00000002093f60>
> entry1 = Entry.new => #<Whistlepig::Entry:0x0000000207d328>
> entry1.add_string "body", "研究会" => #<Whistlepig::Entry:0x0000000207d328>
> docid1 = index.add_entry entry1 => 1
> q1 = Query.new "body", "研究" => body:"研究"
> results1 = index.search q1 => []

The problem here is tokenization. Whistlepig only provides a very simple
tokenizer, namely, it looks for space-separated things [1]. So you have to
space-separate your tokens in both the indexing and querying stages, e.g.:

  entry1.add_string "body", "研 究 会" => #<Whistlepig::Entry:0x90b873c>
  docid1 = index.add_entry entry1      => 1
  q1 = Query.new "body", "研 究"       => AND body:"研" body:"究"
  q1 = Query.new "body", "\"研 究\""   => PHRASE body:"研" body:"究"
  results1 = index.search q1           => [1]

For Japanese, proper tokenization is tricky. You could simply space-separate
every character and deal with the spurious matches across word boundaries.
Or you could do it right by plugging in a proper tokenizer, e.g. something
like http://www.chasen.org/~taku/software/TinySegmenter/.

[1] It also strips any prefix or suffix characters that match [:punct:]. This
is all pretty ad-hoc and undocumented. Providing simpler whitespace-only
tokenizer as an alternative is in the works.
-- 
William <wmorgan-sup@masanjin.net>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel