From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <eg@gaute.vetsj.com>
Received: from [129.177.117.174] ([129.177.117.174])
        by mx.google.com with ESMTPSA id f9sm8019933eeu.11.2013.05.02.01.18.28
        for <multiple recipients>
        (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
        Thu, 02 May 2013 01:18:28 -0700 (PDT)
Message-ID: <5182210E.4040100@gaute.vetsj.com>
Date: Thu, 02 May 2013 10:17:18 +0200
From: Gaute Hope <eg@gaute.vetsj.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130403 Thunderbird/17.0.5
MIME-Version: 1.0
To: Sup developer discussion <sup-devel@rubyforge.org>
CC: Horacio Sanson <hsanson@gmail.com>
Subject: Re: [sup-devel] sup 0.13
References: <CAOxvSbeLRoCwy2THkKa4TKsNqdsx9n-M5BxrZHGfPEUjfEWjSg@mail.gmail.com> <1367233230-sup-3593@tesla> <517E5BD8.6030803@gaute.vetsj.com> <CAHWBo_aKjp__dkjuJ-tYHpKJuv2G=k_jxTWMo0QyyUtvxZbGXA@mail.gmail.com>
In-Reply-To: <CAHWBo_aKjp__dkjuJ-tYHpKJuv2G=k_jxTWMo0QyyUtvxZbGXA@mail.gmail.com>
X-Enigmail-Version: 1.5.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit



On 30. april 2013 11:44, Horacio Sanson wrote:
> Great to see Sup getting back on track again..
> 
> I submitted some patches for the Gmail dumper of Heliotrope some time ago
> but the lack of non alphabet languages (Japanese, Chinese) made it
> impossible for me to keep using heliotrope/turnesole.
> 
> The main issue to support Japanese/Chinese with heliotrope was that
> whistlepig (indexer) lacked the ability to tokenize these languages. Also
> the half baked UTF-8 support caused several issues with these languages.
> 
> I would like to help in testing/implementing support for these languages,
> starting with Japanese, but I would require some guidance. First I would
> like to know is there is a way to configure the Xapian tokenizer
> (segmenter) within sup? Please consider that I am new to both sup and to
> Xapian.

Hi Horacio,

consider opening an issue at
https://github.com/sup-heliotrope/sup/issues to make sure this doesn't
disappear. Some changes will probably be made to the indexer when going
to Mail (from RMail), but I hope to be able to migrate the existing
index. Perhaps its time to get it right for arbitrary languages as well.
I am unfamiliar with Japanes/Chinese - does UTF-8 cover the needs?

Mail is better at handling UTF-8 and I think there was some fork that
had some extra support for Japanese.

Regards, Gaute