From: Gaute Hope <eg@gaute.vetsj.com>
To: Horacio Sanson <hsanson@gmail.com>
Cc: Sup developer discussion <sup-devel@rubyforge.org>
Subject: Re: [sup-devel] sup 0.13
Date: Mon, 20 May 2013 09:24:23 +0200 [thread overview]
Message-ID: <5199CFA7.9020900@gaute.vetsj.com> (raw)
In-Reply-To: <CAHWBo_ZC176WjeMpPs_rJE9=41exnxxJ0ENfLpHRj1ZtOpg2Ng@mail.gmail.com>
Hi,
There has recently been opened an issue regarding this:
https://github.com/sup-heliotrope/sup/issues/60
Regards, Gaute
On 09. mai 2013 03:39, Horacio Sanson wrote:
> UTF-8 handles most cases but I still have to deal with emails in
> ISO2022-JP, Shift-JIS and EUC-JP. After some research it seems Xapian has
> no support for Asian languages. I will try to make some tests and open an
> issue if I cannot make it work.
>
> I can see in the sup configuration file that the stem language can be
> configured but there are no CJK stemmers for Xapian that I can find.
>
>
> On Thu, May 2, 2013 at 5:17 PM, Gaute Hope <eg@gaute.vetsj.com> wrote:
>
>>
>>
>> On 30. april 2013 11:44, Horacio Sanson wrote:
>>> Great to see Sup getting back on track again..
>>>
>>> I submitted some patches for the Gmail dumper of Heliotrope some time ago
>>> but the lack of non alphabet languages (Japanese, Chinese) made it
>>> impossible for me to keep using heliotrope/turnesole.
>>>
>>> The main issue to support Japanese/Chinese with heliotrope was that
>>> whistlepig (indexer) lacked the ability to tokenize these languages. Also
>>> the half baked UTF-8 support caused several issues with these languages.
>>>
>>> I would like to help in testing/implementing support for these languages,
>>> starting with Japanese, but I would require some guidance. First I would
>>> like to know is there is a way to configure the Xapian tokenizer
>>> (segmenter) within sup? Please consider that I am new to both sup and to
>>> Xapian.
>>
>> Hi Horacio,
>>
>> consider opening an issue at
>> https://github.com/sup-heliotrope/sup/issues to make sure this doesn't
>> disappear. Some changes will probably be made to the indexer when going
>> to Mail (from RMail), but I hope to be able to migrate the existing
>> index. Perhaps its time to get it right for arbitrary languages as well.
>> I am unfamiliar with Japanes/Chinese - does UTF-8 cover the needs?
>>
>> Mail is better at handling UTF-8 and I think there was some fork that
>> had some extra support for Japanese.
>>
>> Regards, Gaute
>>
>
next prev parent reply other threads:[~2013-05-20 7:25 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-04-28 20:32 Hamish D
2013-04-28 20:54 ` Gaute Hope
2013-04-29 11:02 ` Israel Herraiz
2013-04-29 11:39 ` Gaute Hope
2013-04-30 9:44 ` Horacio Sanson
2013-05-02 8:17 ` Gaute Hope
2013-05-09 1:39 ` Horacio Sanson
2013-05-20 7:24 ` Gaute Hope [this message]
2013-05-07 11:54 ` [sup-devel] sup 0.13 - please test Gaute Hope
2013-05-13 7:46 ` Gaute Hope
2013-05-13 23:15 ` Matthieu Rakotojaona
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5199CFA7.9020900@gaute.vetsj.com \
--to=eg@gaute.vetsj.com \
--cc=hsanson@gmail.com \
--cc=sup-devel@rubyforge.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox