* [sup-talk] migrating email to a new computer
@ 2008-11-19 22:50 Daniel Wagner
2008-11-27 0:05 ` William Morgan
2008-11-28 2:25 ` Daniel Wagner
0 siblings, 2 replies; 6+ messages in thread
From: Daniel Wagner @ 2008-11-19 22:50 UTC (permalink / raw)
Hey all, I've got a quick question about how sup handles Maildir. I'm
going to be migrating to a new machine soon, and I've got a couple of
questions about it.
1. My mail is spread out in hundreds of Maildir directories, most of
them "legacy" directories that I want to merge as I migrate. Can I
do this by just copying the mail files from the various "new" and
"cur" directories to a central "cur" directory? I know that Maildir
uses unique names, so this shouldn't clobber any messages, but I
haven't seen any information on whether this could cause other kinds
of problems.
2. A more sup-related question: what things do I need to copy to migrate
my labels, read/archived/starred status information, etc? I
definitely want to maintain as much of my mail's metadata as I can.
I guess this probably interacts with the previous question, too; in
the end, I would much prefer to keep my metadata than to merge
Maildir directories if they are mutually exclusive.
3. I'm assuming I won't lose anything if I update sup.
I think this topic deserves a wiki page, so I'll write up a summary of
any advice I get as well as how my experience goes for future posterity.
Thanks!
~d
^ permalink raw reply [flat|nested] 6+ messages in thread
* [sup-talk] migrating email to a new computer
2008-11-19 22:50 [sup-talk] migrating email to a new computer Daniel Wagner
@ 2008-11-27 0:05 ` William Morgan
2008-12-08 5:14 ` Daniel Wagner
2008-11-28 2:25 ` Daniel Wagner
1 sibling, 1 reply; 6+ messages in thread
From: William Morgan @ 2008-11-27 0:05 UTC (permalink / raw)
Hi Daniel,
Sorry for the delay in replying. I've been moving across country and
only have sporadic internet access right now.
Reformatted excerpts from Daniel Wagner's message of 2008-11-19:
> 1. My mail is spread out in hundreds of Maildir directories, most of
> them "legacy" directories that I want to merge as I migrate. Can I do
> this by just copying the mail files from the various "new" and "cur"
> directories to a central "cur" directory?
I *think* that just moving files from cur/ to cur/ and new/ to new/ will
work, but moving from new/ to cur/ I think typically changes the
filename slightly. I don't know if the rename is required or optional
though. Sup certainly doesn't care, but other tools might.
> 2. A more sup-related question: what things do I need to copy to
> migrate my labels, read/archived/starred status information, etc? I
> definitely want to maintain as much of my mail's metadata as I can.
Since you're doing lots of moving things around, combining sources, and
upgrading Sup at the same time, I think the best option is going to be
to rebuild your index from scratch. You can use sup-dump to dump out
state information into a huge text file, and sup-sync --restored
--restore should be able to make use of that information to keep your
message state once your new sources have been created.
But definitely, DEFINITELY test it out first. I haven't tried that stuff
for a while and it might have experienced bitrot.
> 3. I'm assuming I won't lose anything if I update sup.
Nope. And I can't believe you're still using Sup 0.4!
> I think this topic deserves a wiki page, so I'll write up a summary of
> any advice I get as well as how my experience goes for future
> posterity.
That sounds great. Let us know how it goes, and I'll do my best to help
you though the process if things break. Just be sure to keep a backup of
everything. Message state is a precious asset.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [sup-talk] migrating email to a new computer
2008-11-19 22:50 [sup-talk] migrating email to a new computer Daniel Wagner
2008-11-27 0:05 ` William Morgan
@ 2008-11-28 2:25 ` Daniel Wagner
1 sibling, 0 replies; 6+ messages in thread
From: Daniel Wagner @ 2008-11-28 2:25 UTC (permalink / raw)
Much thanks to Nicolas and William. I'm just entering final projects
season in school, so I may put off this project about a week or two
longer, but will keep these messages +inbox until then. =)
~d
^ permalink raw reply [flat|nested] 6+ messages in thread
* [sup-talk] migrating email to a new computer
2008-11-27 0:05 ` William Morgan
@ 2008-12-08 5:14 ` Daniel Wagner
2008-12-08 5:33 ` Mr.
2008-12-12 1:31 ` William Morgan
0 siblings, 2 replies; 6+ messages in thread
From: Daniel Wagner @ 2008-12-08 5:14 UTC (permalink / raw)
Excerpts from William Morgan's message of Wed Nov 26 19:05:13 -0500 2008:
> I *think* that just moving files from cur/ to cur/ and new/ to new/ will
> work, but moving from new/ to cur/ I think typically changes the
> filename slightly. I don't know if the rename is required or optional
> though. Sup certainly doesn't care, but other tools might.
As you said, sup didn't care a jot. Lovely! Since I'm not using any
other tools for my mail, this is just what I will do.
> to rebuild your index from scratch. You can use sup-dump to dump out
> state information into a huge text file, and sup-sync --restored
> --restore should be able to make use of that information to keep your
> message state once your new sources have been created.
>
> But definitely, DEFINITELY test it out first. I haven't tried that stuff
> for a while and it might have experienced bitrot.
Of *course* I'm going to test first. I don't trust software any more
since I've seen how it's made. =)
> That sounds great. Let us know how it goes, and I'll do my best to help
> you though the process if things break. Just be sure to keep a backup of
> everything. Message state is a precious asset.
I have a project due tomorrow, so naturally I figured I'd spend a few
hours playing with this stuff tonight. For the most part, I've been
very pleased -- things have gone relatively smoothly compared to my
expectations!
I sup-dumped, copied all the files in oldmachine/Maildir/**/cur and
oldmachine/Maildir/**/new to newmachine/Maildir/cur, started up sup-sync
--restored --restore, then went and made dinner. Now for some numbers:
The Numbers According To sup-sync
entries in the dump file 26462
entries scanned 24779
entries added 24373
warnings 27
fakes 75
The Numbers According To sup
old machine new machine
messages in the index 26459 24416
messages with +inbox 50 45
messages with +Starred 67 57
The warnings took the form
warning: error (Iconv::InvalidEncoding) decoding message body from XXX: invalid encoding ("utf8//IGNORE", "XXX")
where "XXX" was "unicode-1-1utf7" 17 times, "unknown-8bit" 9 times, and
"X-UNKNOWN" 1 time. It also faked message-ids 68 times, from headers 3
times, and date headers 4 times.
So, I'm not sure I understand how to reconcile these numbers. Things
are mismatched in odd ways; for example, how could there be more entries
in the dump than there are messages in the index? Or how come "entries
added" + "warnings" != "messages in the index"?
That's okay, though. There are people for whom impermanence is a way of
life. There are a few categories of messages for which I really, really
want the impermanence to start later, though -- specifically +inbox and
+Starred messages. Since there's only 15 of these lost, is there some
way I can grab/index those manually?
...or a way to help identify and fix the underlying problem?
~d
P.S. Sorry for all the details. I realize this is probably much more
interesting to me than to any of you, since it affects me more directly,
but I just couldn't help myself. =P
^ permalink raw reply [flat|nested] 6+ messages in thread
* [sup-talk] migrating email to a new computer
2008-12-08 5:14 ` Daniel Wagner
@ 2008-12-08 5:33 ` Mr.
2008-12-12 1:31 ` William Morgan
1 sibling, 0 replies; 6+ messages in thread
From: Mr. @ 2008-12-08 5:33 UTC (permalink / raw)
Excerpts from Daniel Wagner's message of Mon Dec 08 00:14:39 -0500 2008:
> The Numbers According To sup-sync
> entries in the dump file 26462
> entries scanned 24779
> entries added 24373
> warnings 27
> fakes 75
>
> The Numbers According To sup
> old machine new machine
> messages in the index 26459 24416
Ah, I just realized: copying everything to one directory is fine only if
the filenames have unique names. A quick investigation shows I probably
nuked around 500-600 messages with this assumption, so that's one easily
fixable problem. But that still leaves ~1500 messages in the aether...
Love!
~d
^ permalink raw reply [flat|nested] 6+ messages in thread
* [sup-talk] migrating email to a new computer
2008-12-08 5:14 ` Daniel Wagner
2008-12-08 5:33 ` Mr.
@ 2008-12-12 1:31 ` William Morgan
1 sibling, 0 replies; 6+ messages in thread
From: William Morgan @ 2008-12-12 1:31 UTC (permalink / raw)
Reformatted excerpts from Daniel Wagner's message of 2008-12-07:
> entries in the dump file 26462
What is this number from? Is it a wc -l on the dumpfile, or sup-sync's
"Restored state on X messages" message? If the latter, what is wc -l?
(And if the former, what's the other number?)
> entries scanned 24779
> entries added 24373
> warnings 27
> fakes 75
Warnings and fakes we can ignore. (They still count as being added.)
Did sup-sync report a number of updates?
It should be the case that num adds + num updates = num scanned,
and num adds = index size (when starting from an empty index).
> The Numbers According To sup
> old machine new machine
> messages in the index 26459 24416
> messages with +inbox 50 45
> messages with +Starred 67 57
>
> So, I'm not sure I understand how to reconcile these numbers. Things
> are mismatched in odd ways; for example, how could there be more
> entries in the dump than there are messages in the index?
Yes, that is weird. 26462 vs 26459 is what concerns me. How did three
new messages appear at dump time?
> Or how come "entries added" + "warnings" != "messages in the index"?
Messages with warnings or that needed fake headers should still be
added, so the question is really 24416 vs 24373. That's 43 new messages
that appeared in the index.
> That's okay, though. There are people for whom impermanence is a way
> of life.
Uh oh.
> There are a few categories of messages for which I really, really
> want the impermanence to start later, though -- specifically +inbox and
> +Starred messages. Since there's only 15 of these lost, is there some
> way I can grab/index those manually?
The most direct way is to query the index through devel/console.sh. You
can find examples on the wiki or in the mailing list archives. Something
like
Index.ferret.search("label:inbox", :limit => :all).hits.map { |h| Index.ferret[h.doc] }
will give you all the Ferret records with the index label, and you can
then query them for all the fields listed in index.rb. (They'll appear
as empty hashes, {}, when irb prints them out, but in reality they're
just lazily-filled.
That should give you the information you need to compare the two sets of
documents on both indexes, and figure out what the differences are.
--
William <wmorgan-sup at masanjin.net>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2008-12-12 1:31 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-11-19 22:50 [sup-talk] migrating email to a new computer Daniel Wagner
2008-11-27 0:05 ` William Morgan
2008-12-08 5:14 ` Daniel Wagner
2008-12-08 5:33 ` Mr.
2008-12-12 1:31 ` William Morgan
2008-11-28 2:25 ` Daniel Wagner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox