Re: [sup-devel] Cannot query Japanese characters

From: Horacio Sanson <hsanson@gmail.com>
To: Sup developer discussion <sup-devel@rubyforge.org>
Subject: Re: [sup-devel] Cannot query Japanese characters
Date: Wed, 4 May 2011 11:03:14 +0900	[thread overview]
Message-ID: <BANLkTimWHyrh6ZtCEgQ84kNkqFr-ZvLdCw@mail.gmail.com> (raw)
In-Reply-To: <BANLkTikbENFqT2GsE5uWjqN_DTMq43FFkw@mail.gmail.com>

Forgot to mention you need the mecab ruby gem. In Ubuntu 10.04 this
gem is part of the distribution and can be installed with the command:

sudo apt-get install libmecab-ruby1.8 libmecab-ruby1.9.1 mecab-ipadic-utf8

regards
Horacio

On Wed, May 4, 2011 at 10:42 AM, Horacio Sanson <hsanson@gmail.com> wrote:
> Chasen is the worst tokenizer, is pretty old. The best one is MeCab
> that is the faster and from the same author of Chasen.
> You can see all major Japanese tokenizer in action at this URL:
> http://nomadscafe.jp/test/keitaiso/index.cgi. Just put some
> text in the box and press the button.
>
> After some hacking I got a Heliotrope server that works perfectly with
> Japanese text. All I did was follow your comments
> and applied the MeCab tokenizer to the message body and query strings
> before passing them to Whistelpig or more specific
> to Heliotrope::Index.
>
> There is one problem I don't see how to handle... I do receive email
> in Japanese but also Chinese and Korean. I need a different
> tokenizer for each one and I have no idea how to handle this. Do email
> messages contain a language header that would allow me
> to identify the language and pass it to the corresponding tokenizer??
>
>
> regards,
> Horacio
>
> On Wed, May 4, 2011 at 7:26 AM, William Morgan <wmorgan-sup@masanjin.net> wrote:
>> Reformatted excerpts from Horacio Sanson's message of 2011-05-03:
>>> index = Index.new "index" => #<Whistlepig::Index:0x00000002093f60>
>>> entry1 = Entry.new => #<Whistlepig::Entry:0x0000000207d328>
>>> entry1.add_string "body", "研究会" => #<Whistlepig::Entry:0x0000000207d328>
>>> docid1 = index.add_entry entry1 => 1
>>> q1 = Query.new "body", "研究" => body:"研究"
>>> results1 = index.search q1 => []
>>
>> The problem here is tokenization. Whistlepig only provides a very simple
>> tokenizer, namely, it looks for space-separated things [1]. So you have to
>> space-separate your tokens in both the indexing and querying stages, e.g.:
>>
>>  entry1.add_string "body", "研 究 会" => #<Whistlepig::Entry:0x90b873c>
>>  docid1 = index.add_entry entry1      => 1
>>  q1 = Query.new "body", "研 究"       => AND body:"研" body:"究"
>>  q1 = Query.new "body", "\"研 究\""   => PHRASE body:"研" body:"究"
>>  results1 = index.search q1           => [1]
>>
>> For Japanese, proper tokenization is tricky. You could simply space-separate
>> every character and deal with the spurious matches across word boundaries.
>> Or you could do it right by plugging in a proper tokenizer, e.g. something
>> like http://www.chasen.org/~taku/software/TinySegmenter/.
>>
>> [1] It also strips any prefix or suffix characters that match [:punct:]. This
>> is all pretty ad-hoc and undocumented. Providing simpler whitespace-only
>> tokenizer as an alternative is in the works.
>> --
>> William <wmorgan-sup@masanjin.net>
>> _______________________________________________
>> Sup-devel mailing list
>> Sup-devel@rubyforge.org
>> http://rubyforge.org/mailman/listinfo/sup-devel
>>
>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel