Subject: [sup-talk] Reply calculation

From: [email-masked] (Edward Z. Yang)
Date: Sat, 25 Jul 2009 14:10:54 -0400

I think sup's "To" address calculation algorithm is wrong, and
messes up in some important edge-cases.  The two edge
cases I have in mind are:

* When a mailing list sends an unsubscribe/subscribe notification

* When someone posts to a mailing list with an explicit Reply-to
  header.

In both of these cases, the mail will commonly have an explicit
Reply-to header.  However, Sup will by default disregard this
header in favor for List-id, which is *wrong*.

You can guess, this has bitten me several times.

I think the correct behavior is to only use List-id when Reply-to
is not explicitly set, since List-id is "magical" whereas Reply-to
is commonly explicit.

Cheers,
Edward

---

From: [email-masked] (Edward Z. Yang)
Date: Sat, 25 Jul 2009 14:24:15 -0400

Here is a patch that changes the behavior:

>From 58a8d325286703f9057dd5d07094d0c6a061377c Mon Sep 17 00:00:00 2001
From: Edward Z. Yang <ezyang at mit.edu>
Date: Sat, 25 Jul 2009 14:17:08 -0400
Subject: [PATCH] Use Reply-to if it is explicitly set.

Previously, Sup would ignore an explicit Reply-to
in favor of List-id.  This is the wrong behavior for
the following cases:

* List subscribe and unsubscribe messages will commonly
  have List-id set, but set Reply-to for their own
  nefarious purposes (the "you should be able to reply
  to this email and have it work")

* People will sometimes send mail to a list with an
  explicit Reply-to header to get responses offlist.

This brings Sup's behavior more inline with traditional
mail clients.  Since most lists already set Reply-to
(and most mail clients don't respect List-id), there will
probably be no breakage.

Signed-off-by: Edward Z. Yang <ezyang at mit.edu>
---
 lib/sup/modes/reply-mode.rb |    8 +++-----
 1 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/lib/sup/modes/reply-mode.rb b/lib/sup/modes/reply-mode.rb
index c79c5db..ca423c1 100644
--- a/lib/sup/modes/reply-mode.rb
+++ b/lib/sup/modes/reply-mode.rb
@@ -81,10 +81,8 @@ EOS
       AccountManager.default_account
     end
 
-    ## now, determine to: and cc: addressess. we ignore reply-to for list
-    ## messages because it's typically set to the list address, which we
-    ## explicitly treat with reply type :list
-    to = @m.is_list_message? ? @m.from : (@m.replyto || @m.from)
+    ## if someone overrode reply-to, it was for a good reason
+    to = (@m.replyto || @m.from)
 
     ## next, cc:
     cc = (@m.to + @m.cc - [from, to]).uniq
@@ -115,7 +113,7 @@ EOS
     } unless not_me_ccs.empty?
 
     @headers[:list] = {
-      "To" => [@m.list_address.full_address],
+      "To" => [@m.replyto.full_address || @m.list_address.full_address],
     } if @m.is_list_message?
 
     refs = gen_references
-- 
1.6.3.3

---

From: [email-masked] (Edward Z. Yang)
Date: Sat, 25 Jul 2009 15:03:31 -0400

> Here is a patch that changes the behavior:

Actually, the patch is not quite correct.  Let me think about this
some more.

Cheers,
Edward

---

From: [email-masked] (Edward Z. Yang)
Date: Sat, 25 Jul 2009 15:15:59 -0400

After thinking about this some, I think that the only way
to reasonably handle all corner cases is to explicitly ask
the user for a choice in corner cases.  Having the default
subtly change based on some non-obvious attributes of an
email is a great way to break muscle memory.

Myself, I have :user in my reply-to.rb hook for now.

Cheers,
Edward

---

From: [email-masked] (William Morgan)
Date: Mon, 27 Jul 2009 09:49:11 -0700

Reformatted excerpts from Edward Z. Yang's message of 2009-07-25:
> After thinking about this some, I think that the only way to
> reasonably handle all corner cases is to explicitly ask the user for a
> choice in corner cases.

What was the breakage when favoring reply-to over list-id? I was buying
your arguments...

Is it possible to identify these corner cases? Is it always when there's
a reply-to and a list-id both set?
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Edward Z. Yang)
Date: Mon, 27 Jul 2009 13:09:13 -0400

> What was the breakage when favoring reply-to over list-id? I was buying

First and foremost, s/list-id/list-post/ in my previous posts; I was quoting
the wrong field and double checked message.rb just now.

I received mail from a mailing list as a call for applications that should
not be sent back to the list.  The mail was constructed with headers like:

From: [email-masked]
To: correct at example.com
Reply-To: correct at example.com
List-Post: mailto:incorrect at example.com

Sup detects List-Post, categorizes the message as a list message, and then
sets the default reply mode as list, which results in List-Post being used
as the to address.

> a reply-to and a list-id both set?

Unfortunately, mailing list administrating is notoriously broken.  I'm not
sure at all what the right solution is.  Take for example this other case:

From: [email-masked]
To: list at example.com
Reply-To: person at example.com
List-Post: mailto:list at example.com

Reply-To, in this case, was set by the mailing list server.  This makes
having Reply-To be the end-all be-all kind of spurious.  If we try
to make Sup do the Right Thing (TM), you probably want to send mail to
the list as a whole, which means you do want to either "reply all" semantics
or "list post" magic.  ("reply all" semantics are what you see in traditional
mail clients when you hit the "reply all" button.)

However, consider the next case:

From: [email-masked]
To: list at example.com
Cc: me at example.com

Which is when someone else hit "Reply all" and you got CC'ed.  This means
that the mail never passed through the mailing list agent, the List-Post/Reply-To
headers never got set, and the only way to tell that you should reply to the
whole list is to explicitly ask for "Reply all" semantics (Sup defaults to
"Reply" semantics, which is damn confusing if you're not paying attention).

The core problem is that subtle changes in state should not require the user
to do things differently; it breaks muscle memory and makes mistakes easy.
We could try to make it so that Sup requires /no/ user intervention, but
this is seems to be an AI-hard problem.  Then, the logical other direction,
is to make the interface as consistent as possible.

Cheers,
Edward

PS. Djb wrote an article along these lines:
    http://cr.yp.to/proto/replyto.html
I've skimmed it but I'm kind of skeptical about client support and not sure
if it actually gives useful recommendations.

---

From: [email-masked] (William Morgan)
Date: Tue, 28 Jul 2009 08:48:48 -0700

Reformatted excerpts from Edward Z. Yang's message of 2009-07-27:
> From: correct at example.com

So in this case, following reply-to is correct.

> From: person at example.com
> To: list at example.com
> Reply-To, in this case, was set by the mailing list server.

In this case, I'd argue that this means the list administrator wants the
default reply behavior to be to the individual and not the list. So I'd
again prefer Sup default to the reply-to address rather than the list
address. (With the caveat that this is overrideable by hooks on a
per-list basis, so if it's a matter of an incompetent list
administrator, or simply disagreeing with them, one can override this
behavior for this list.)

> From: persona at example.com
> To: list at example.com
> Cc: me at example.com
> 

There's not much to be done in this case, EXCEPT that if you receive
more than one copy of the message, you should keep the list header
around. Then the only time you're in a funny situation is when you've
received the first but not the other.

This is presumably why Mutt had you register your mailing list addresses
explicitly, which I always found a little irritating.

(Or to have Sup keep around email addresses known to belong to lists,
and match those in the To: field against that, which seems significantly
more complicated.)

> to do things differently; it breaks muscle memory and makes mistakes easy.

I agree with this in principle, but I see addressing a message as a
fundamental part of composing it. You can remove the notion of a smart
default reply-to address from your Sup, if you like, by using the
reply-to hook.

And as for the default, I think I'm of the opinion that setting the
default reply address so as to obey the reply-to is correcter than
anything else (including whatever Sup does currently).

---

From: [email-masked] (Edward Z. Yang)
Date: Tue, 28 Jul 2009 14:34:46 -0400

> administrator, or simply disagreeing with them, one can override this

Nods.

> received the first but not the other.

This situation is not that uncommon; many mailing list software will notice
that someone is on a CC list and will purposely omit them.

> default reply-to address from your Sup, if you like, by using the

This is true. However, when you compose a non-reply message, Sup prompts
you for To, Cc and Subject, because they are such fundamental components
of all messages you would write (well, maybe not so much Cc) and the
user always has to make a judgment call there.

> anything else (including whatever Sup does currently).

Fair enough.  You may get some complaints because this /will/ break muscle
memory, and still presents the possibility for messages to have different
behavior in subtle ways, as detailed in the Cc versus mailing list example
from above.

Cheers,
Edward