Subject: Re: [sup-devel] Bug: UTF-8 error when sending messages

From: Gaute Hope <[email-masked]>
Date: Tue, 01 Feb 2011 14:34:10 +0100

Excerpts from Adeel Ahmad Khan's message of 2011-01-31 19:24:42 +0100:
> 
> Tero Tilus <[email-masked]>:
> > Adeel Ahmad Khan, 2011-01-29 05:11:
> > >     invalid byte sequence in UTF-8
> > ...
> > > I am using a nearly fresh installation of Sup 0.12.1 with Ruby
> > > 1.9.2p136.  I have LOCALE="en_US.UTF-8".

What editor are you using ? Could you attach some buggy text (make sure
it is transferred in binary - ie upload an archive somewhere like
http://paste.xinu.at/)?

> > Both the errors were from regex matches against message body.  Somehow
> > your editor doesn't know your locale or is not obeying it.  As a
> > result non-utf8 stuff gets saved to disk and sup gets confused.
>
> You were right.  It turned out to be an issue with editor.

I'm not so sure. I think it is a problem with Ruby (1.9) degrading the
the text it reads to US-ASCII (8bit). And when you later try to do UTF-8
stuff with it (append, or regexp with UTF-8) it fails. Perhaps even when
trying to access chars in the string (the body) that are not really
US-ASCII [2].

What my patch does is tell Ruby that the string is an UTF-8 string, no
matter what it deduced from reading the file in the first place - or
what might have happened throughout Sup's processing of the text.

Try this patch, it forces encoding on the entire body: http://ix.io/1rO

(these patches are workarounds; not to be applied to source-tree)

The same happened with labels.txt (or with contacts). Whenever a label
string could be degraded to US-ASCII, Ruby did so (how should it know it
was UTF-8 anyway?), then when trying to append, match or work with the
US-ASCII string towards UTF-8 input it failed.

What I think must be done (as an alternative to supporting different
encodings all the way) - is to _always_ read all files in UTF-8 [1] (or
transcode to UTF-8), and perhaps most difficultly _keep_ the strings
UTF-8 throught the entire Sup processes.

[1] http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings
[2] http://www.ruby-forum.com/topic/194493

Best regards,
Gaute

---

From: Adeel Ahmad Khan <[email-masked]>
Date: Mon, 31 Jan 2011 13:24:42 -0500

Gaute Hope <[email-masked]>:
> Excerpts from Adeel Ahmad Khan's message of 2011-01-29 04:11:40 +0100:
> >     --- ArgumentError from thread: main
> >     /usr/lib/ruby/gems/1.9.1/gems/sup-0.12.1/lib/sup/modes/edit-message-mode.rb:497:in `block in mentions_attachments?'
> 
> Could you try if this patch fixes it (or just edit the lines manually) ?

The patch resolved that error, but I get a different one now:

  --- ArgumentError from thread: main
  invalid byte sequence in UTF-8
  /usr/lib/ruby/gems/1.9.1/gems/sup-0.12.1/lib/sup/modes/edit-message-mode.rb:389:in `build_message'
  /usr/lib/ruby/gems/1.9.1/gems/sup-0.12.1/lib/sup/modes/edit-message-mode.rb:354:in `send_message'
  /usr/lib/ruby/gems/1.9.1/gems/sup-0.12.1/lib/sup/mode.rb:59:in `handle_input'
  /usr/lib/ruby/gems/1.9.1/gems/sup-0.12.1/lib/sup/buffer.rb:277:in `handle_input'
  /usr/lib/ruby/gems/1.9.1/gems/sup-0.12.1/bin/sup:260:in `<module:Redwood>'
  /usr/lib/ruby/gems/1.9.1/gems/sup-0.12.1/bin/sup:69:in `<top (required)>'
  /usr/bin/sup:19:in `load'
  /usr/bin/sup:19:in `<main>'

It looks like another regex match.

    m.body += "\n" unless m.body =~ /\n\Z/

In any case it seems to be my problem, not sup's.

Adeel
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel