Subject: Re: [sup-devel] sup 0.14: Encoding::UndefinedConversionError from thread: load threads for thread-index-mode

From: Gregor Hoffleit <[email-masked]>
Date: Fri, 16 Aug 2013 17:15:17 +0200

* Gaute Hope <[email-masked]> [2013-08-15 18:39:45 +0200]
> did you have a lot of messages in your index added by Sup 0.13 or
> earlier? Do you get the same error if you move the old xapian folder out
> of the way and try to re-index your messages?
> 
> It is possible to restore labels using sup-dump and sup-sync --restore.

Thanks for this explanation.

Yes, I was running with an index created with Sup < 0.13, and it's
pretty clear to me now that this was the cause of this crash.

Well, I moved away the old xapian folder and called sup-sync --restore,
but this crashed after ten minutes and 10.000 messages (of 500.000).

I just filled an issue #131 to github about this crash.

    Gregor

---

From: Gregor Hoffleit <[email-masked]>
Date: Fri, 16 Aug 2013 15:13:10 +0200

* Gaute Hope <[email-masked]> [2013-08-15 18:49:55 +0200]
> Created pull request #128, please test if you have the chance.
> 
> https://github.com/sup-heliotrope/sup/pull/128

Yep, with this workaround, the crash no longer happens.

    Gregor
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Gregor Hoffleit <[email-masked]>
Date: Thu, 15 Aug 2013 18:21:43 +0200

Sup 0.14 fails for me just after the start with the following exception:

--- Encoding::UndefinedConversionError from thread: load threads for thread-index-mode
"\xE2" from ASCII-8BIT to UTF-8
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:259:in `width'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:259:in `display_length'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:226:in `block in draw_line_from_array'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:224:in `each'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:224:in `each_with_index'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:224:in `draw_line_from_array'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:199:in `draw_line'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/line_cursor_mode.rb:52:in `draw_line'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:46:in `block in draw'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:46:in `each'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:46:in `draw'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/line_cursor_mode.rb:37:in `draw'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/buffer.rb:118:in `draw'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/buffer.rb:102:in `redraw'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/buffer.rb:335:in `draw_screen'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/buffer.rb:766:in `clear'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:639:in `method_missing'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/thread_index_mode.rb:667:in `load_n_threads'
(eval):12:in `load_n_threads'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/thread_index_mode.rb:638:in `block in load_n_threads_background'
/var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup.rb:85:in `block in reporting_thread'

I noticed that this exception looks similar to a report from 2011:

* Horacio Sanson's message of Mo Apr 25 04:25:04 +0200 2011:
> Any attempt to add a label with Japanese characters crashes the application. 
> Seems this is a common problem to all Ruby 1.9 applications. I can see Rails 
> had or has a lot of problems with this:
> 
> http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/
> 
> https://rails.lighthouseapp.com/projects/8994/tickets/4683-ascii-8bit-and-utf-8-in-hell
> 
> 
> The backtrace I get follows:
> 
> /var/lib/gems/1.9.1/gems/console-0.3/lib/console/string.rb:27:in 
> `display_width': "\xE3" from ASCII-8BIT to UTF-8 
> (Encoding::UndefinedConversionError)
>         from /var/lib/gems/1.9.1/gems/console-0.3/lib/console/string.rb:27:in 
> `display_width'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/textfield.rb:129:in 
> `handle_input'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:108:in `ask'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:141:in 
> `ask_many_with_completions'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:198:in 
> `ask_for_labels'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/modes/thread-index-
> mode.rb:585:in `block in edit_labels'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:42:in `asking'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/modes/thread-index-
> mode.rb:584:in `edit_labels'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:92:in `handle'
>         from /home/ryujin/Apps/turnsole/lib/turnsole/ui.rb:73:in `step'
>         from bin/turnsole:134:in `<main>'

to which William responded:

> Thanks for the bug report. This is a bit tricky. The problem is
> actually that ncurses is giving me the characters one byte at a time.
> I'll see what I can do to fix this.

Do you need any more information to debug this problem?

I'm using Ruby 1.9.3p194 on Debian 7.1. I installed Sup 0.14 with
"gem1.9.1 install sup", which pulled in the following dependencies:

# gem1.9.1 list --local

*** LOCAL GEMS ***

chronic (0.9.1)
highline (1.6.19)
locale (2.0.8)
lockfile (2.1.0)
mime-types (1.24)
ncursesw-sup (1.3.1.3)
rmail (1.0.0)
sup (0.14.0)
trollop (2.0)
unicode (0.4.4)
xapian-ruby (1.2.15.1)

Regards,
    Gregor Hoffleit

---

From: Gaute Hope <[email-masked]>
Date: Thu, 15 Aug 2013 18:39:45 +0200

> /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:259:in `display_length'

Hi Gregor,

Otherwise, I used to hackishly fix this error when it occured at some
other point by doing the following fix:

diff --git a/lib/sup/util.rb b/lib/sup/util.rb
index 5cff6fa..4579a38 100644
--- a/lib/sup/util.rb
+++ b/lib/sup/util.rb
@@ -256,7 +256,7 @@ end
 
 class String
   def display_length
-    @display_length ||= Unicode.width(self, false)
+    @display_length ||= Unicode.width(self.fix_encoding, false)
   end
 
   def slice_by_display_length len

The problem is that the string which is being formatted (or sought the
width of) is of some sort of encoding (probably already UTF-8), but Ruby
or NCurses or RMail or whatever has lost the encoding, and although the
bytes are encoded they are treated as binary (ASCII-8BIT). There is
generally no way to figure out what the original encoding was, but we
can guess that it is UTF-8 and fix it.

Regards, Gaute

---

From: Gaute Hope <[email-masked]>
Date: Thu, 15 Aug 2013 18:49:55 +0200

Regards, Gaute