From: Horacio Sanson <hsanson@gmail.com>
To: Gaute Hope <eg@gaute.vetsj.com>
Cc: Sup developer discussion <sup-devel@rubyforge.org>
Subject: Re: [sup-devel] sup 0.13
Date: Thu, 9 May 2013 10:39:08 +0900 [thread overview]
Message-ID: <CAHWBo_ZC176WjeMpPs_rJE9=41exnxxJ0ENfLpHRj1ZtOpg2Ng@mail.gmail.com> (raw)
In-Reply-To: <5182210E.4040100@gaute.vetsj.com>
[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]
UTF-8 handles most cases but I still have to deal with emails in
ISO2022-JP, Shift-JIS and EUC-JP. After some research it seems Xapian has
no support for Asian languages. I will try to make some tests and open an
issue if I cannot make it work.
I can see in the sup configuration file that the stem language can be
configured but there are no CJK stemmers for Xapian that I can find.
On Thu, May 2, 2013 at 5:17 PM, Gaute Hope <eg@gaute.vetsj.com> wrote:
>
>
> On 30. april 2013 11:44, Horacio Sanson wrote:
> > Great to see Sup getting back on track again..
> >
> > I submitted some patches for the Gmail dumper of Heliotrope some time ago
> > but the lack of non alphabet languages (Japanese, Chinese) made it
> > impossible for me to keep using heliotrope/turnesole.
> >
> > The main issue to support Japanese/Chinese with heliotrope was that
> > whistlepig (indexer) lacked the ability to tokenize these languages. Also
> > the half baked UTF-8 support caused several issues with these languages.
> >
> > I would like to help in testing/implementing support for these languages,
> > starting with Japanese, but I would require some guidance. First I would
> > like to know is there is a way to configure the Xapian tokenizer
> > (segmenter) within sup? Please consider that I am new to both sup and to
> > Xapian.
>
> Hi Horacio,
>
> consider opening an issue at
> https://github.com/sup-heliotrope/sup/issues to make sure this doesn't
> disappear. Some changes will probably be made to the indexer when going
> to Mail (from RMail), but I hope to be able to migrate the existing
> index. Perhaps its time to get it right for arbitrary languages as well.
> I am unfamiliar with Japanes/Chinese - does UTF-8 cover the needs?
>
> Mail is better at handling UTF-8 and I think there was some fork that
> had some extra support for Japanese.
>
> Regards, Gaute
>
[-- Attachment #2: Type: text/html, Size: 2444 bytes --]
next prev parent reply other threads:[~2013-05-09 1:39 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-04-28 20:32 Hamish D
2013-04-28 20:54 ` Gaute Hope
2013-04-29 11:02 ` Israel Herraiz
2013-04-29 11:39 ` Gaute Hope
2013-04-30 9:44 ` Horacio Sanson
2013-05-02 8:17 ` Gaute Hope
2013-05-09 1:39 ` Horacio Sanson [this message]
2013-05-20 7:24 ` Gaute Hope
2013-05-07 11:54 ` [sup-devel] sup 0.13 - please test Gaute Hope
2013-05-13 7:46 ` Gaute Hope
2013-05-13 23:15 ` Matthieu Rakotojaona
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAHWBo_ZC176WjeMpPs_rJE9=41exnxxJ0ENfLpHRj1ZtOpg2Ng@mail.gmail.com' \
--to=hsanson@gmail.com \
--cc=eg@gaute.vetsj.com \
--cc=sup-devel@rubyforge.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox