From: Gaute Hope <eg@gaute.vetsj.com>
To: sup-devel <sup-devel@rubyforge.org>
Subject: Re: [sup-devel] Bug: UTF-8 error when sending messages
Date: Tue, 01 Feb 2011 14:34:10 +0100 [thread overview]
Message-ID: <1296566276-sup-4340@qwerzila> (raw)
In-Reply-To: <1296497911-sup-5024@poincare>
Excerpts from Adeel Ahmad Khan's message of 2011-01-31 19:24:42 +0100:
>
> Tero Tilus <tero@tilus.net>:
> > Adeel Ahmad Khan, 2011-01-29 05:11:
> > > invalid byte sequence in UTF-8
> > ...
> > > I am using a nearly fresh installation of Sup 0.12.1 with Ruby
> > > 1.9.2p136. I have LOCALE="en_US.UTF-8".
What editor are you using ? Could you attach some buggy text (make sure
it is transferred in binary - ie upload an archive somewhere like
http://paste.xinu.at/)?
> > Both the errors were from regex matches against message body. Somehow
> > your editor doesn't know your locale or is not obeying it. As a
> > result non-utf8 stuff gets saved to disk and sup gets confused.
>
> You were right. It turned out to be an issue with editor.
I'm not so sure. I think it is a problem with Ruby (1.9) degrading the
the text it reads to US-ASCII (8bit). And when you later try to do UTF-8
stuff with it (append, or regexp with UTF-8) it fails. Perhaps even when
trying to access chars in the string (the body) that are not really
US-ASCII [2].
What my patch does is tell Ruby that the string is an UTF-8 string, no
matter what it deduced from reading the file in the first place - or
what might have happened throughout Sup's processing of the text.
Try this patch, it forces encoding on the entire body: http://ix.io/1rO
(these patches are workarounds; not to be applied to source-tree)
The same happened with labels.txt (or with contacts). Whenever a label
string could be degraded to US-ASCII, Ruby did so (how should it know it
was UTF-8 anyway?), then when trying to append, match or work with the
US-ASCII string towards UTF-8 input it failed.
What I think must be done (as an alternative to supporting different
encodings all the way) - is to _always_ read all files in UTF-8 [1] (or
transcode to UTF-8), and perhaps most difficultly _keep_ the strings
UTF-8 throught the entire Sup processes.
[1] http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings
[2] http://www.ruby-forum.com/topic/194493
Best regards,
Gaute
next prev parent reply other threads:[~2011-02-01 13:33 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <mailman.75.1296431981.25874.sup-devel@rubyforge.org>
2011-01-31 18:24 ` Adeel Ahmad Khan
2011-02-01 13:34 ` Gaute Hope [this message]
2011-01-29 3:11 Adeel Ahmad Khan
2011-01-29 18:17 ` Tero Tilus
2011-01-30 15:58 ` Gaute Hope
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1296566276-sup-4340@qwerzila \
--to=eg@gaute.vetsj.com \
--cc=sup-devel@rubyforge.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox