Subject: Re: [sup-talk] PGP INLINE - can't make it work

From: [email-masked]
Date: Tue, 2 Mar 2010 15:19:23 +0100

Sounds cool, I can't wait then. Let's hope the patch is included in 0.11. 
Thanks Michael!
Piotr

On Tuesday 02 March 2010 14:59:19 Michael Stapelberg wrote:
> Hi Piotr,
> 
> Excerpts from piotr.kempa's message of Di Mär 02 11:49:59 +0100 2010:
> > So, I guess if this functionality is to be included in sup (and I think it 
> > doesn't have to be since enigmail just makes broken messages and there's 
no 
> > reason why anybody should put up with that:) it should not be in crypto.rb 
but 
> > somewhere else, where the decision is made to even use functions from 
> > crypto.rb (like decrypt) on a message. I'm no programmer but I guess it 
should 
> As mentioned, I wrote a patch to do this. It is currently in review and will
> hopefully be included soon.
> 
> Best regards,
> Michael
> _______________________________________________
> sup-talk mailing list
> [email-masked]
> http://rubyforge.org/mailman/listinfo/sup-talk
> 
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: Piotr Kempa <[email-masked]>
Date: Wed, 10 Mar 2010 15:41:35 +0100

Excerpts from Michael Stapelberg's message of 2010-03-10 12:14:04 +0100:
> Hi Piotr,
> 
> Excerpts from Piotr Kempa's message of Mi Mär 10 11:53:00 +0100 2010:
> > Thank you. I just tried the patch and unfortunately sup crashes on me
> > now. I patched the 0.11 version and checked that the patches were
> > applied successfully (as far as I could at least). 
> Thanks for testing. Can you please see if the updated version works
> for you (attached to this mail)?
> 
> Best regards,
> Michael

Wow, this works!! Thank you so much Michael! This will make my job soooo much
easier, faster and more pleasant :) 

I have just two messages that give errors to sup-sync, but the remaining ones
(a hundred or more) just work beautifully.  The error is like this: 

   [2010-03-10 15:25:28 +0100] Error while running gpg: gpg: invalid armor header: 
	    gpg: malformed CRC
   gpg: encrypted_mdc packet with unknown version 255
   gpg: quoted printable character in armor - probably a buggy MTA has been used
   
   [2010-03-10 15:25:33 +0100] Error while running gpg: gpg: invalid armor
      header: gpg: CRC error; 9C0326 - DC39C0
   gpg: encrypted_mdc packet with unknown version 255
   gpg: quoted printable character in armor - probably a buggy MTA has been used

(two long lines with the timestamp broken by me)

But I guess this is what it says it is - an invalid message with CRC errors in
the GPG payload, so this should be nothing to worry about. 

Anyway, thanks a million Michael!
-- 
Piotr Kempa
_______________________________________________
sup-talk mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-talk

---

From: [email-masked]
Date: Mon, 8 Mar 2010 11:18:55 +0100

> > hopefully be included soon.

Can anyone confirm if this patch made it to 0.11? I just installed the new 
version and it still doesn't work but if it's there then maybe I'm doing 
something wrong. If it didn't, is it posted anywhere? I'll be happy to patch 
0.11 myself instead of waiting for the next version. I really, really want to 
start using sup :)

Thank you, 
Piotr

---

From: Piotr Kempa <[email-masked]>
Date: Wed, 10 Mar 2010 11:53:00 +0100

Excerpts from Michael Stapelberg's message of 2010-03-09 18:06:35 +0100:
> Hi Piotr,
> 
> Excerpts from piotr.kempa's message of Mo Mär 08 12:20:02 +0100 2010:
> > Thank you and Michael for the explanation. Now it all makes sense. I'll just 
> > keep watching the git repo in case a patch makes it there. Or  this list. If 
> > anybody needs help testing such patch - I'm at your service. Coding it myself 
> > could be way over my head though...
> I just sent an updated patch to sup-devel, you can find it at
> http://rubyforge.org/pipermail/sup-devel/2010-March/000463.html
> 
> Testing it would be appreciated.
> 
> Best regards,
> Michael

Hello Michael, 

I'm attaching the crash log. The crash happens like this: I add the
new source maildir with encrypted emails, run sup, it executes, sees
the new source and starts indexing emails and then crashes. 

Hope this helps somehow. 
 
Piotr

---

From: Michael Stapelberg <[email-masked]>
Date: Tue, 09 Mar 2010 18:06:35 +0100

Hi Piotr,

Best regards,
Michael

---

From: Michael Stapelberg <[email-masked]>
Date: Mon, 08 Mar 2010 11:41:18 +0100

Hi Piotr,

Excerpts from piotr.kempa's message of Mo Mär 08 11:29:47 +0100 2010:
> Okay, I found the patch here: http://www.mail-archive.com/sup-
> [email-masked]/msg02770.html
> 
> And it is included in 0.11, I just checked in the code. 
You are mistaken, this is a different patch (not for inline gpg).

> So in order to decide about encrypted content you actually have to look inside 
> the body of the message, not just at the header. At least I couldn't find any 
> indication in the headers :(
This is actually the very intention of inline gpg -- to work with existing
standards/mailclients which do not support different headers.

Unfortunately I did not yet have time to rework the patch, but I hope to do
so in the next few days.

Best regards,
Michael

---

From: [email-masked]
Date: Mon, 8 Mar 2010 12:20:02 +0100

On Monday 08 March 2010 11:50:56 Sven Moritz Hallberg wrote:
> > However, this still doesn't work because the header doesn't contain 
Content 
> > Type = application/pgp, it is text/plain, which is absolutely crazy and I 
have 
> > no idea what the enigmail developers are thinking...
> 
> In fact I think this might be a design decision on the part of PGP, where 
the
> original thinking went something like this: It should be possible to use it
> with ANY mailer, even those that do not know about it. Thus the support for
> "inline PGP", i.e. a PGP message embedded within the text of a regular 
email.
> >From this standpoint it makes sense to keep the Content-Type as text/plain.
> Imagine your recipient's MUA has no idea how to handle application/pgp. With
> text/plain, she's still able to copy/paste into a separate PGP application.
> 
> The merit of this design decision is debatable, but I think that's the way 
it
> came to be.
> 
> So, I'd say it's unlikely Enigmail is going to change the header. I'd also
> guess that everyone else is doing it the same way.
> 
> So, actually inspecting the body of the message is the way to go...
> 
> HTH,
> pesco

Thank you, 
Piotr

---

From: Michael Stapelberg <[email-masked]>
Date: Wed, 10 Mar 2010 22:21:50 +0100

Hi Piotr,

> easier, faster and more pleasant :) 
You’re welcome. Thanks for testing. I will now post the updated patch to
sup-devel and hope that rlane will merge it soon.

Best regards,
Michael

---

From: [email-masked]
Date: Tue, 2 Mar 2010 11:49:59 +0100

I digged a bit further, trying to decode the emails from commandline with "gpg 
--decode". For some of the inline GPG emails the commandline version fails, 
but for some other it works. Even for those that it works, sup just shows 
undecoded version which leads me to believe it doesn't even try to decode, 
which means it doesn't recognize the email as encrypted at all. 

As far as I can deduct, sup only recognizes encrypted emails if they have 
something like this inside:
Content-Type: multipart/encrypted;
  boundary="nextPart15239034.i9UOPnLxZ9";
  protocol="application/pgp-encrypted"

Or similar. Which is a reasonable behaviour.

On the other hand, if the damned enigmail creates an email with "Content-Type: 
text/plain;" and then proceeds to include "-----BEGIN PGP MESSAGE-----" in the 
body, sup treats this as (no surprise) text/plain and doesn't decode. If it 
tried, it would succeed since this message passed through gpg --decode is 
decoded okay.

So, I guess if this functionality is to be included in sup (and I think it 
doesn't have to be since enigmail just makes broken messages and there's no 
reason why anybody should put up with that:) it should not be in crypto.rb but 
somewhere else, where the decision is made to even use functions from 
crypto.rb (like decrypt) on a message. I'm no programmer but I guess it should 
be something like this:
if payload contains "BEGIN PGP MESSAGE" then run the decrypt function from 
crypto.rb on it. 

For now - does anybody know a way to mass decrypt all messages in my mailbox 
so I can keep them in plain text and use sup anyway? That would be ideal 
solution but I just can't find a program to do that. And gpg itself isn't 
helpful since it just prints decoded payload on --output, without the headers, 
so it's no use in batch. I can try to code this in bash but maybe there's 
something out there already?

Thank you
Piotr

On Friday 26 February 2010 18:43:04 Sven Schober wrote:
> Excerpts from kosmici-atakuja's message of Fri Feb 26 18:08:36 +0100 2010:
> > First of wall I want to say I absolutely love sup! I used all the major 
mail 
> > clients like mutt, claws, thunderbird, kmail. All mail clients just fail 
(mutt 
> > was the closest though...). Sup so far has the potential to be the best of 
> > them. If only I can get it to read the inline gpg that enigmail makes...
> > 
> +1
> 
> > Is there anything I can do to get this working? This is a crucial feature 
for 
> > me...
> > 
> +1
> > Moreover, I searched the list archives and it looks like some people have 
got 
> > it working as there have been patches to make it work a couple of months 
ago. 
> > And my version is recent (January) so those patches should be in it. It 
just 
> > doesn't for me for some reason :(
> > 
> I think there was a (preliminary) patch from Michael Stapelberg
> floating around, but somehow that didn't make it into next?
> 
> > I know I should read crypto.rb and figure it out from there. I tried, I 
just 
> > can't understand it, as I'm not experienced in ruby, or email clients code 
for 
> > that matter. 
> > 
> +1
> 
> > Please help!
> +1
> 
> > Thank you
> > Piotr
> 
> Ciao,
>   Sven
> -- 
> Sven Schober, [email-masked]                    |UNI ULM
> http://www-vs.informatik.uni-ulm.de/dept/staff/schober/  |DISTRIBUTED
> Room O27-346, Phone: +49-731-5024146 [+49-179-5060182]   |SYSTEMS LAB
>

---

From: Sven Schober <[email-masked]>
Date: Fri, 26 Feb 2010 18:43:04 +0100

Excerpts from kosmici-atakuja's message of Fri Feb 26 18:08:36 +0100 2010:
> First of wall I want to say I absolutely love sup! I used all the major mail 
> clients like mutt, claws, thunderbird, kmail. All mail clients just fail (mutt 
+1

> Is there anything I can do to get this working? This is a crucial feature for 
> me...
> 
+1
> Moreover, I searched the list archives and it looks like some people have got 
> it working as there have been patches to make it work a couple of months ago. 
> And my version is recent (January) so those patches should be in it. It just 

> I know I should read crypto.rb and figure it out from there. I tried, I just 
> can't understand it, as I'm not experienced in ruby, or email clients code for 
> that matter. 
> 
+1

+1

Ciao,
  Sven

---

From: Sam Hall <[email-masked]>
Date: Tue, 2 Mar 2010 15:12:39 +0100

On Tue, Mar 02, 2010 at 11:49 (+0100), [email-masked] wrote:
> For now - does anybody know a way to mass decrypt all messages in my
> mailbox so I can keep them in plain text and use sup anyway?

Using mutt with tagging and decrypt-save should work (didn't test that,
though). Google for decrypt-save, it's not documented in the manual.

Best wishes,
  /Sam

---

From: [email-masked]
Date: Mon, 8 Mar 2010 11:29:47 +0100

On Monday 08 March 2010 11:18:55 [email-masked] wrote:
> > > > So, I guess if this functionality is to be included in sup (and I 
think 

However, this still doesn't work because the header doesn't contain Content 
Type = application/pgp, it is text/plain, which is absolutely crazy and I have 
no idea what the enigmail developers are thinking... Anyway, here's how it 
looks like (only the important part):

X-Enigmail-Version: 0.95.7
Content-Type: text/plain;
  charset=UTF-8
Content-Transfer-Encoding: quoted-printable

-----BEGIN PGP MESSAGE-----
Charset: UTF-8
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
=20
hQIOA3WKxDKSmLd7EAf9GP5JLA77IcGOAUrOZ8fGCEtK8Pm838Me+pMD94ulHlDS
LcidQKPSVGXaj4lLU8RqxoMKTpxuaZAm1u286C+SL9eQc51N3gH/X57loKhjCRb+
[...]

All in all, I wouldn't be surprised if you decided not to support such broken 
emails so I'll just go about learning enough ruby to patch it myself :)

Piotr

---

From: Michael Stapelberg <[email-masked]>
Date: Wed, 10 Mar 2010 12:14:04 +0100

Hi Piotr,

Best regards,
Michael

---

From: Sven Moritz Hallberg <[email-masked]>
Date: Mon, 08 Mar 2010 11:50:56 +0100

> no idea what the enigmail developers are thinking...

In fact I think this might be a design decision on the part of PGP, where the
original thinking went something like this: It should be possible to use it
with ANY mailer, even those that do not know about it. Thus the support for
"inline PGP", i.e. a PGP message embedded within the text of a regular email.

The merit of this design decision is debatable, but I think that's the way it
came to be.

HTH,
pesco

---

From: Michael Stapelberg <[email-masked]>
Date: Tue, 02 Mar 2010 14:59:19 +0100

Hi Piotr,

Best regards,
Michael

---

From: [email-masked]
Date: Fri, 26 Feb 2010 18:08:36 +0100

Example email looks like this in sup: 

 -----BEGIN PGP MESSAGE-----
  Charset: UTF-8
  Version: GnuPG v1.4.9 (MingW32)
  Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

  hQIOA3WKxDKSmLd7EAf/cKH72GNHWgkY46sjHuYBlV5Q9udlmjB1ucTqtVfSUysH
[cut]
 NYqMmmJ1HHG58B9bPqGFPNe/jEPeiiqj9Ez58kYzS32VeelJFwV+O8K9/TAvG4HN
  fEIgtYMGj0Uc5v0eZ7yVqFfluJrEX5V7mN2InahTCD+uFEfguvld8rZagEtlyxU5
  =nAMm
  -----END PGP MESSAGE-----

And it gets displayed like this, no error messages or anything. I have 
configured my gpg in sup as per this wiki page:
http://sup.rubyforge.org/wiki/wiki.pl?GPG
(with the numeric ID and such). 

And I enabled verbose debug logging, here's the log:
(cutting leading timestamp for better reading)

using character set encoding "UTF-8"
No 'chronic' gem detected. Install it for date/time query restrictions.
No 'ncursesw' gem detected. Install it for wide character support.
dynamically loading setlocale() from libc.so.6
setting locale...
using xapian index
locking /home/piotr/.sup/lock...
crypto: detected gpg binary in /usr/bin/gpg
starting curses
initializing log buffer
Welcome to Sup! Log level is set to debug.
initializing inbox buffer
ready for interaction!
colormap: for color twiddle_color, using id 1 -> 4, -1
colormap: for color status_color, using id 2 -> 7, 4
colormap: for color tagged_color_highlight, using id 3 -> 3, 6
colormap: for color date_color_highlight, using id 4 -> 0, 6
colormap: for color to_me_color_highlight, using id 5 -> 2, 6
colormap: for color tagged_color, using id 6 -> 3, -1
colormap: for color date_color, using id 7 -> 7, -1
colormap: for color to_me_color, using id 8 -> 2, -1
colormap: for color snippet_color, using id 9 -> 6, -1
scanning maildir /home/piotr/Mail...
done scanning maildir
crypto: running: /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --
use-agent --output /tmp/32042-0-redwood.outp
scanning maildir /home/piotr/Mail...
done scanning maildir
crypto: running: /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --
use-agent --output /tmp/32042-0-redwood.outp
crypto: running: /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --
use-agent --output /tmp/32042-2-redwood.outp
crypto: running: /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --
use-agent --output /tmp/32042-0-redwood.outp
colormap: for color alternate_patina_color, using id 10 -> 0, 4
colormap: for color message_patina_color, using id 11 -> 0, 2
colormap: for color system_buf_color_highlight, using id 12 -> 7, 6

(no errors there as far as I can see... And looks like it's running the gpg 
binary succesfully... I just changed the maildir path for privacy in this log)

GPG is configured to use gpg-agent, the gpg-agent is running, pinentry program 
shows up when running sup and I enter the correct password. I checked the 
password many times already. Also, the environment variable $GPG_AGENT_INFO is 
present and visible in sup environment (it's global and visible anywhere). 
I already tried it without "use gpg-agent" and the effect was the same. 

In mutt this worked, I mean the inline gpg messages were decrypted using this 
same setup of gpg-agent and gpg. It also works in kmail (although not 
perfectly, but decodes the messages).

Now, this may be a clue: when normal gpg message is used (mime) it sometimes 
geta decrypted and sometimes doesn't. If it doesn't, sup only says this in the 
message: 
    x This message has been decrypted for display
    x Attachment: sup-attachment-1267203299-4513. (multipart/mixed; 71b)

When I'm trying to display this attachment it says: 
For some bizarre reason, RubyMail was unable to parse this attachment.
and in the status line:
Couldn't execute view command, viewing as text.

(this may be because I have no run-mailcap file, but the message was in plain 
text, no html, so it shouldn't be needed now should it? I'm planning on 
getting some run-mailcap script from some debian system when I have some time, 
but for now this inline GPG is my main priority)

I'm still not sure why they sometimes decrypt correctly and sometimes not, but 
I can work with this, at least it works sometimes. But even when mime gpg 
works, the inline stuff doesn't. 

Verisions:
I'm using sup version 0.10.2 from Archlinux AUR. 
ruby version is 1.9.1
xapian-core is 1.0.18

Is there anything I can do to get this working? This is a crucial feature for 
me as almost all my work colleagues use thunderbird and enigmail to send their 
email and that uses inline pgp crap as default (which I know is not conforming 
to any standards etc.). And even if I can convince verybody to switch to mime, 
I still have a couple hundred megs of archives in this damn format. 

Please help!
Thank you
Piotr