Subject: Re: [sup-devel] sup v2 progress report

From: William Morgan <[email-masked]>
Date: Tue, 29 Mar 2011 04:00:36 +0000

Reformatted excerpts from Nicolas Pouillard's message of 2011-03-28:
> .../heliotrope/lib/heliotrope/decoder.rb:109:in `gsub': incompatible
> character encodings: UTF-8 and ASCII-8BIT
> (Encoding::CompatibilityError)

Can you try with the latest master? If it still doesn't work, are you
able to narrow down the string encodings of from and word?

This 1.9 string encoding stuff is tricky business, especially combined
with the messy world of email. I think I'm getting close.
-- 
William <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: Matthieu Rakotojaona <[email-masked]>
Date: Mon, 28 Mar 2011 06:14:24 +0200

Thank you very much.

 I've been using sup for quite a few months now, and I must say that
the labels-centric point of view really is an improvement over the
other mail management systems.
I've been following sup news very closely, and I'm really excited
about heliotrope/turnsole.

I have tried the new system a little bit, and I have a few things to say :
- you must have 'html2text' (not specified) installed on your system,
as long as you have any html mail. I think this is the case for too
many of us, unfortunately.
- I didn't find rubymail with gem, but found it under the name rmail
- I still had some problem with the encoding stuff. I was using ruby
v1.9.2 (or so I thought), but got the "ArgumentError - invalid byte
sequence in UTF-8" error. Strange thing, the log showed me evidence
that I was using ruby v1.9.1 (problems came from files in
"/usr/lib/ruby/gems/1.9.1"). So I switched back to v1.8.7 from AUR
(I'm using archlinux), and the problem just went away.

In the light of what you wrote, I have two questions :
- Is sup development bound to be stopped, at least when
heliotrope/turnsole will be ready ?
- Is there any mailing-list for this new project, or should we keep
using those related with sup ?
- Do you have any address where I can find and test turnsole ?

Again, thank you very much for your work and your help, and I'm
talking to everyone on these lists.

Regards,

-- 
Matthieu RAKOTOJAONA
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Nicolas Pouillard <[email-masked]>
Date: Tue, 29 Mar 2011 14:19:00 -0700 (PDT)

> with the messy world of email. I think I'm getting close.

Nice, it seems to go a lot further. However there seems to be some strange
blocking behavior, like using CPU and producing nothing in hours.

-- 
Nicolas Pouillard
http://nicolaspouillard.fr
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Tue, 29 Mar 2011 03:11:04 +0000

Reformatted excerpts from Nicolas Pouillard's message of 2011-03-28:
> .../heliotrope/lib/heliotrope/decoder.rb:109:in `gsub': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)

Hm. I think I know what the problem is. Stay tuned.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Ico Doornekamp <[email-masked]>
Date: Thu, 14 Apr 2011 09:51:50 +0200

* On Sun Mar 27 22:41:59 +0200 2011, William Morgan wrote:
 
> Heliotrope, the server component, is close to ready for a version 1 release.
> You can find it at https://github.com/wmorgan/heliotrope/.

Here's another crash, but I've not been able yet to find out what exactly is
happening. The problem seems to be caused by a mismatch of multipart mime
boundary strings. 

/home/ico/external/heliotrope/lib/heliotrope/message.rb:151:in `decode_mime_parts': undefined method `multipart?' for nil:NilClass (NoMethodError)
	from /home/ico/external/heliotrope/lib/heliotrope/message.rb:154:in `decode_mime_parts'
	from /home/ico/external/heliotrope/lib/heliotrope/message.rb:130:in `mime_parts'
	from /home/ico/external/heliotrope/lib/heliotrope/message.rb:116:in `has_attachment?'
	from /home/ico/external/heliotrope/lib/heliotrope/index.rb:77:in `add_message'
	from bin/heliotrope-add:113:in `<main>'
You have new mail in /home/ico/Maildir

I've stripped the offending mail down to the following minimal mbox which causes the crash:

>From [email-masked]  Tue May  9 11:22:38 2006
Date: Tue, 09 May 2006 02:22:18 -0800
From: [email-masked]
Mime-Version: 1.0
Content-Type: multipart/alternative; boundary="one"
Message-Id: <1>

--two

Part 

--three

-- 
:wq
^X^Cy^K^X^C^C^C^C

---

From: Nicolas Pouillard <[email-masked]>
Date: Mon, 28 Mar 2011 05:52:52 -0700 (PDT)

On Sun, 27 Mar 2011 20:41:59 +0000, William Morgan <[email-masked]> wrote:
> Hello all,
> 
> I'm happy to report that Sup version 2 is well underway!
> 
> Sup version 2 features:
> - a client/server model that works over HTTP, allowing
>     a) development of other clients, e.g. web-based and phone-based
>     b) simultaneous access from multiple clients
>     c) IMAP emulation, aka no more lock-in!
> - an improved console-based client
> 
> The client, which I'm calling Turnsole, is coming along swimmingly. I hope to
> release a pre-alpha some-stuff-actually-works version within the next few days.
> Much of the UI code has been borrowed from Sup, but the internals are quite
> different:
> - it's event-based, rather than thread-based, which streamlines a lot of the
>   code and avoids a whole big class of bugs.
> - all the index and email threading code is ripped out
> - most of the email parsing code is gone
> - it handles Ruby 1.9 string encoding stuff correctly, rather than having
>   random checks scattered around
> 
> I think you're going to like it, since:
> - the threads are pre-computed on the server side, so it's much, much faster
> - you can finally view attachments locally!
> 
> Stay tuned for more. We're still a ways off before it's a drop-in replacement,
> but I'm excited about how everything is coming together.

While mass importing an mbox I got this:

$ ... heliotrope-add ...
; forced to decode html. running html2text on 868b mime part...
end offset is 237534473
.../heliotrope/lib/heliotrope/decoder.rb:109:in `gsub': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)
  from .../heliotrope/lib/heliotrope/decoder.rb:109:in `decode_rfc2047'
  from .../heliotrope/lib/heliotrope/message.rb:208:in `decode_header'
  from .../heliotrope/lib/heliotrope/message.rb:30:in `parse!'
  from bin/heliotrope-add:105:in `<main>'

Best regards,

---

From: William Morgan <[email-masked]>
Date: Mon, 28 Mar 2011 22:57:26 +0000

> - I didn't find rubymail with gem, but found it under the name rmail

Thanks, I will update the README.

> sequence in UTF-8" error.

Ok, I'm still working on tracking this down. If you have a backtrace
handy, that would be useful.

> heliotrope/turnsole will be ready ?

I'm planning on moving my efforts entirely over to heliotrope +
turnsole. Which isn't that big of a deal, really, since I haven't done
much on Sup for years.

> using those related with sup ?

I'll keep talking about it here until people complain.

> - Do you have any address where I can find and test turnsole ?

Check back in a few days.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: William Morgan <[email-masked]>
Date: Sun, 27 Mar 2011 20:41:59 +0000

Hello all,

Sup version 2 features:

---

From: Ico Doornekamp <[email-masked]>
Date: Thu, 14 Apr 2011 09:39:44 +0200

> You can find it at https://github.com/wmorgan/heliotrope/.

Heliotrope (Actually Time.parse()) crashes on some illegaly formatted dates:

  /usr/lib/ruby/1.9.1/time.rb:137:in `apply_offset': undefined method `<' for nil:NilClass (NoMethodError)
  	from /usr/lib/ruby/1.9.1/time.rb:197:in `make_time'
  	from /usr/lib/ruby/1.9.1/time.rb:261:in `parse'
  	from /home/ico/external/heliotrope/lib/heliotrope/message.rb:27:in `parse!'
  	from bin/heliotrope-add:108:in `<main>'

The date of the message was "Wed, 7  2005 22:55: 1  -0180".

Fixed by adding a NoMethodError catch:

diff --git a/lib/heliotrope/message.rb b/lib/heliotrope/message.rb
index 1682062..d63e411 100644
--- a/lib/heliotrope/message.rb
+++ b/lib/heliotrope/message.rb
@@ -23,7 +23,7 @@ class Message
     @from = Person.from_string decode_header(validate_field(:from, @m.header["from"]))
     @date = begin
       Time.parse(validate_field(:date, @m.header["date"])).to_i
-    rescue ArgumentError
+    rescue ArgumentError, NoMethodError
       #puts "warning: invalid date field [email-masked]['date']}"
       Time.at 0
     end

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 13 Apr 2011 17:18:24 -0300

> You can find it at https://github.com/wmorgan/heliotrope/.

Hi,

I find this to be great.  Thanks for writing it.

I just tried to follow the instructions on a Debian machine and noticed
that you need to install the package ruby1.9.1-dev in order for the
whistlepig and oklahoma_mixer gems.  Probably most Ruby-addicted people
already have it so they don't see that problem, but since I'm not one of
those, it did bit me.  I suggest this should be in the README file.

I ran into a problem importing email that has Latin1 headers.  The crash
I got was:

/home/alvherre/Code/heliotrope/lib/heliotrope/maildir-walker.rb:44:in `block in get_date_in_file': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/alvherre/Code/heliotrope/lib/heliotrope/maildir-walker.rb:42:in `open'
        from /home/alvherre/Code/heliotrope/lib/heliotrope/maildir-walker.rb:42:in `get_date_in_file'
        from /home/alvherre/Code/heliotrope/lib/heliotrope/maildir-walker.rb:35:in `block in get_files'
        from /home/alvherre/Code/heliotrope/lib/heliotrope/maildir-walker.rb:35:in `map'
        from /home/alvherre/Code/heliotrope/lib/heliotrope/maildir-walker.rb:35:in `get_files'
        from /home/alvherre/Code/heliotrope/lib/heliotrope/maildir-walker.rb:21:in `done?'
        from bin/heliotrope-add:98:in `<main>'

I added a debugging rescue to that block and found that the message has
this header:
List-Id: Consultas T<E9>cnicas sobre Linux y Software Libre <l-linux.velug.org.ve>

Note that the <E9> is actually a Latin-1 "é".  (The message has a valid Date:
header).

This is obviously a bogus message; you're not supposed to use non-ASCII
chars in the header.  But these guys did it anyway.  I'm not sure what
would be a good fix for this problem.  I guess it involves discarding
the failing line so that it can index the message using the proper date
instead of "0".

I haven't tried Turnsole yet.  I checked the WEBrick thingy and it seems
pretty neat.

Thanks!

-- 
Álvaro Herrera <[email-masked]>
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: William Morgan <[email-masked]>
Date: Tue, 29 Mar 2011 21:31:06 +0000

> blocking behavior, like using CPU and producing nothing in hours.

Weird. I wonder if it's triggering some regex worst case like we've seen
before in Sup. Are you able to narrow down what message or text is
causing this? I will add a --verbose option, which might make this
easier.
-- 
William <[email-masked]>
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk