Subject: [sup-talk] migrating email to a new computer

From: [email-masked] (Daniel Wagner)
Date: Wed, 19 Nov 2008 17:50:39 -0500

Hey all, I've got a quick question about how sup handles Maildir.  I'm
going to be migrating to a new machine soon, and I've got a couple of
questions about it.

1. My mail is spread out in hundreds of Maildir directories, most of
   them "legacy" directories that I want to merge as I migrate.  Can I
   do this by just copying the mail files from the various "new" and
   "cur" directories to a central "cur" directory?  I know that Maildir
   uses unique names, so this shouldn't clobber any messages, but I
   haven't seen any information on whether this could cause other kinds
   of problems.
2. A more sup-related question: what things do I need to copy to migrate
   my labels, read/archived/starred status information, etc?  I
   definitely want to maintain as much of my mail's metadata as I can.
   I guess this probably interacts with the previous question, too; in
   the end, I would much prefer to keep my metadata than to merge
   Maildir directories if they are mutually exclusive.
3. I'm assuming I won't lose anything if I update sup.

I think this topic deserves a wiki page, so I'll write up a summary of
any advice I get as well as how my experience goes for future posterity.

Thanks!
~d

---

From: [email-masked] (William Morgan)
Date: Wed, 26 Nov 2008 16:05:13 -0800

Hi Daniel,

Sorry for the delay in replying. I've been moving across country and
only have sporadic internet access right now.

Reformatted excerpts from Daniel Wagner's message of 2008-11-19:
> them "legacy" directories that I want to merge as I migrate.  Can I do
> this by just copying the mail files from the various "new" and "cur"
> directories to a central "cur" directory?

I *think* that just moving files from cur/ to cur/ and new/ to new/ will
work, but moving from new/ to cur/ I think typically changes the
filename slightly. I don't know if the rename is required or optional
though. Sup certainly doesn't care, but other tools might.

> 2. A more sup-related question: what things do I need to copy to
> migrate my labels, read/archived/starred status information, etc?  I

Since you're doing lots of moving things around, combining sources, and
upgrading Sup at the same time, I think the best option is going to be
to rebuild your index from scratch. You can use sup-dump to dump out
state information into a huge text file, and sup-sync --restored
--restore should be able to make use of that information to keep your
message state once your new sources have been created.

But definitely, DEFINITELY test it out first. I haven't tried that stuff
for a while and it might have experienced bitrot.

> 3. I'm assuming I won't lose anything if I update sup.

Nope. And I can't believe you're still using Sup 0.4!

> any advice I get as well as how my experience goes for future

That sounds great. Let us know how it goes, and I'll do my best to help
you though the process if things break. Just be sure to keep a backup of
everything. Message state is a precious asset.
-- 
William <wmorgan-sup at masanjin.net>

---

From: [email-masked] (Daniel Wagner)
Date: Thu, 27 Nov 2008 21:25:21 -0500

Much thanks to Nicolas and William.  I'm just entering final projects
season in school, so I may put off this project about a week or two
longer, but will keep these messages +inbox until then. =)

~d

---

From: [email-masked] (Daniel Wagner)
Date: Mon, 08 Dec 2008 00:14:39 -0500

> though. Sup certainly doesn't care, but other tools might.

As you said, sup didn't care a jot.  Lovely!  Since I'm not using any
other tools for my mail, this is just what I will do.

> for a while and it might have experienced bitrot.

Of *course* I'm going to test first.  I don't trust software any more
since I've seen how it's made. =)

> everything. Message state is a precious asset.

I have a project due tomorrow, so naturally I figured I'd spend a few
hours playing with this stuff tonight.  For the most part, I've been
very pleased -- things have gone relatively smoothly compared to my
expectations!

I sup-dumped, copied all the files in oldmachine/Maildir/**/cur and
oldmachine/Maildir/**/new to newmachine/Maildir/cur, started up sup-sync
--restored --restore, then went and made dinner.  Now for some numbers:

The Numbers According To sup-sync
entries in the dump file    26462
entries scanned             24779
entries added               24373
warnings                       27
fakes                          75

The Numbers According To sup
                            old machine     new machine
messages in the index       26459           24416
messages with +inbox           50              45
messages with +Starred         67              57

The warnings took the form
warning: error (Iconv::InvalidEncoding) decoding message body from XXX: invalid encoding ("utf8//IGNORE", "XXX")
where "XXX" was "unicode-1-1utf7" 17 times, "unknown-8bit" 9 times, and
"X-UNKNOWN" 1 time.  It also faked message-ids 68 times, from headers 3
times, and date headers 4 times.

So, I'm not sure I understand how to reconcile these numbers.  Things
are mismatched in odd ways; for example, how could there be more entries
in the dump than there are messages in the index?  Or how come "entries
added" + "warnings" != "messages in the index"?

That's okay, though.  There are people for whom impermanence is a way of
life.  There are a few categories of messages for which I really, really
want the impermanence to start later, though -- specifically +inbox and
+Starred messages.  Since there's only 15 of these lost, is there some
way I can grab/index those manually?

...or a way to help identify and fix the underlying problem?
~d

P.S. Sorry for all the details.  I realize this is probably much more
interesting to me than to any of you, since it affects me more directly,
but I just couldn't help myself. =P

---

From: Mr. (Mr.)
Date: Mon, 08 Dec 2008 00:33:34 -0500

> messages in the index       26459           24416

Ah, I just realized: copying everything to one directory is fine only if
the filenames have unique names.  A quick investigation shows I probably
nuked around 500-600 messages with this assumption, so that's one easily
fixable problem.  But that still leaves ~1500 messages in the aether...

Love!
~d

---

From: [email-masked] (William Morgan)
Date: Thu, 11 Dec 2008 17:31:41 -0800

> entries in the dump file    26462

What is this number from? Is it a wc -l on the dumpfile, or sup-sync's
"Restored state on X messages" message? If the latter, what is wc -l?
(And if the former, what's the other number?)

> fakes                          75

Warnings and fakes we can ignore. (They still count as being added.)
Did sup-sync report a number of updates?

It should be the case that num adds + num updates = num scanned,
and num adds = index size (when starting from an empty index).

> are mismatched in odd ways; for example, how could there be more
> entries in the dump than there are messages in the index?

Yes, that is weird. 26462 vs 26459 is what concerns me. How did three
new messages appear at dump time?

> Or how come "entries added" + "warnings" != "messages in the index"?

Messages with warnings or that needed fake headers should still be
added, so the question is really 24416 vs 24373. That's 43 new messages
that appeared in the index.

> That's okay, though.  There are people for whom impermanence is a way
> of life.

Uh oh. 

> There are a few categories of messages for which I really, really

The most direct way is to query the index through devel/console.sh. You
can find examples on the wiki or in the mailing list archives. Something
like
  Index.ferret.search("label:inbox", :limit => :all).hits.map { |h| Index.ferret[h.doc] }

will give you all the Ferret records with the index label, and you can
then query them for all the fields listed in index.rb. (They'll appear
as empty hashes, {}, when irb prints them out, but in reality they're
just lazily-filled.

That should give you the information you need to compare the two sets of
documents on both indexes, and figure out what the differences are.
-- 
William <wmorgan-sup at masanjin.net>