* [sup-devel] alternative for sup-sync option --restored?
@ 2010-08-06 9:13 Sascha Silbe
2010-08-22 13:40 ` Sascha Silbe
2011-01-19 13:52 ` [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format Sascha Silbe
0 siblings, 2 replies; 6+ messages in thread
From: Sascha Silbe @ 2010-08-06 9:13 UTC (permalink / raw)
To: sup-devel
[-- Attachment #1.1: Type: text/plain, Size: 672 bytes --]
Hi!
I'm reading and writing my mails on both my desktop (which has the entire archive, totalling ~ 16GB) and my laptop (recent mails and partial archive, ~ 1-2GB).
Up to now I used sup-dump and sup-sync --restored to synchronize the two (every time I switched between them, since it's a full dump as opposed to a log file or diff).
After merging the latest changes (including a full dump + restore for index upgrade, taking about 33 hours), the --restored option to sup-sync is gone (and without it sup-sync doesn't do anything useful). How do I go about "merging" the changes done on the other system now?
Sascha
--
http://sascha.silbe.org/
http://www.infra-silbe.de/
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]
[-- Attachment #2: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [sup-devel] alternative for sup-sync option --restored?
2010-08-06 9:13 [sup-devel] alternative for sup-sync option --restored? Sascha Silbe
@ 2010-08-22 13:40 ` Sascha Silbe
2011-01-19 13:52 ` [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format Sascha Silbe
1 sibling, 0 replies; 6+ messages in thread
From: Sascha Silbe @ 2010-08-22 13:40 UTC (permalink / raw)
To: sup-devel
[-- Attachment #1.1: Type: text/plain, Size: 850 bytes --]
Excerpts from Sascha Silbe's message of Fri Aug 06 11:13:42 +0200 2010:
> I'm reading and writing my mails on both my desktop (which has the entire archive, totalling ~ 16GB) and my laptop (recent mails and partial archive, ~ 1-2GB).
> Up to now I used sup-dump and sup-sync --restored to synchronize the two (every time I switched between them, since it's a full dump as opposed to a log file or diff).
> After merging the latest changes (including a full dump + restore for index upgrade, taking about 33 hours), the --restored option to sup-sync is gone (and without it sup-sync doesn't do anything useful). How do I go about "merging" the changes done on the other system now?
Any suggestion? This is preventing me from updating to git master and rebasing my outstanding patches.
Sascha
--
http://sascha.silbe.org/
http://www.infra-silbe.de/
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]
[-- Attachment #2: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format
2010-08-06 9:13 [sup-devel] alternative for sup-sync option --restored? Sascha Silbe
2010-08-22 13:40 ` Sascha Silbe
@ 2011-01-19 13:52 ` Sascha Silbe
2011-01-20 4:13 ` Rich Lane
2011-01-20 15:28 ` Nicolas Pouillard
1 sibling, 2 replies; 6+ messages in thread
From: Sascha Silbe @ 2011-01-19 13:52 UTC (permalink / raw)
To: sup-devel
sup-import-dump imports message state as exported by sup-dump into the index.
It is a direct replacement for the sup-sync --restored functionality that got
lost when merging the maildir branch.
Unlike sup-sync it operates on the index only, so it's fast enough for
periodically importing full dumps to keep multiple sup installations
synchronised.
It should also be easy enough to add support for a "diff" style format that
would allow replaying "logs" if sup were enhanced to write those in the
future.
To give some rough numbers:
Dump file contains 78104 lines, index about 600k entries. 410 entries from the
dump file don't match the index and cause index updates. Transaction mode is
used for all runs.
Cold cache, dry run: 138s real time, 53s user+system
Hot cache, dry run: 42s real time, 40s user+system
Hot cache, changes written to disk: 55s real time, 44s user+system
Hot cache, no updates: 43s real time, 41s user+system
Signed-off-by: Sascha Silbe <sascha-pgp@silbe.org>
---
bin/sup-import-dump | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++
lib/sup/index.rb | 15 ++++++++
2 files changed, 114 insertions(+), 0 deletions(-)
diff --git a/bin/sup-import-dump b/bin/sup-import-dump
new file mode 100644
index 0000000..91a1721
--- /dev/null
+++ b/bin/sup-import-dump
@@ -0,0 +1,99 @@
+#!/usr/bin/env ruby
+
+require 'uri'
+require 'rubygems'
+require 'trollop'
+require "sup"; Redwood::check_library_version_against "git"
+
+PROGRESS_UPDATE_INTERVAL = 15 # seconds
+
+class AbortExecution < SystemExit
+end
+
+opts = Trollop::options do
+ version "sup-import-dump (sup #{Redwood::VERSION})"
+ banner <<EOS
+Imports message state previously exported by sup-dump into the index.
+sup-import-dump operates on the index only, so the messages must have already
+been added using sup-sync. If you need to recreate the index, see sup-sync
+--restore <filename> instead.
+
+Messages not mentioned in the dump file will not be modified.
+
+Usage:
+ sup-import-dump [options] <dump file>
+
+Options:
+EOS
+ opt :verbose, "Print message ids as they're processed."
+ opt :ignore_missing, "Silently skip over messages that are not in the index."
+ opt :warn_missing, "Warn about messages that are not in the index, but continue."
+ opt :abort_missing, "Abort on encountering messages that are not in the index. (default)"
+ opt :atomic, "Use transaction to apply all changes atomically."
+ opt :dry_run, "Don't actually modify the index. Probably only useful with --verbose.", :short => "-n"
+ opt :version, "Show version information", :short => :none
+
+ conflicts :ignore_missing, :warn_missing, :abort_missing
+end
+Trollop::die "No dump file given" if ARGV.empty?
+Trollop::die "Extra arguments given" if ARGV.length > 1
+dump_name = ARGV.shift
+missing_action = [:ignore_missing, :warn_missing, :abort_missing].find { |x| opts[x] } || :abort_missing
+
+Redwood::start
+index = Redwood::Index.init
+
+index.lock_interactively or exit
+begin
+ num_read = 0
+ num_changed = 0
+ index.load
+ index.begin_transaction if opts[:atomic]
+
+ IO.foreach dump_name do |l|
+ l =~ /^(\S+) \((.*?)\)$/ or raise "Can't read dump line: #{l.inspect}"
+ mid, labels = $1, $2
+ num_read += 1
+
+ unless index.contains_id? mid
+ if missing_action == :abort_missing
+ $stderr.puts "Message #{mid} not found in index, aborting."
+ raise AbortExecution, 10
+ elsif missing_action == :warn_missing
+ $stderr.puts "Message #{mid} not found in index, skipping."
+ end
+
+ next
+ end
+
+ m = index.build_message mid
+ new_labels = labels.to_set_of_symbols
+
+ if m.labels == new_labels
+ puts "#{mid} unchanged" if opts[:verbose]
+ next
+ end
+
+ puts "Changing flags for #{mid} from '#{m.labels.to_a * ' '}' to '#{new_labels.to_a * ' '}'" if opts[:verbose]
+ num_changed += 1
+
+ next if opts[:dry_run]
+
+ m.labels = new_labels
+ index.update_message_state m
+ end
+
+ index.commit_transaction if opts[:atomic]
+ puts "Updated #{num_changed} of #{num_read} messages."
+rescue AbortExecution
+ index.cancel_transaction if opts[:atomic]
+ raise
+rescue Exception => e
+ index.cancel_transaction if opts[:atomic]
+ File.open("sup-exception-log.txt", "w") { |f| f.puts e.backtrace }
+ raise
+ensure
+ index.save_index unless opts[:atomic]
+ Redwood::finish
+ index.unlock
+end
diff --git a/lib/sup/index.rb b/lib/sup/index.rb
index b90c2b1..bcc449b 100644
--- a/lib/sup/index.rb
+++ b/lib/sup/index.rb
@@ -260,6 +260,21 @@ EOS
end
end
+ # wrap all future changes inside a transaction so they're done atomically
+ def begin_transaction
+ synchronize { @xapian.begin_transaction }
+ end
+
+ # complete the transaction and write all previous changes to disk
+ def commit_transaction
+ synchronize { @xapian.commit_transaction }
+ end
+
+ # abort the transaction and revert all changes made since begin_transaction
+ def cancel_transaction
+ synchronize { @xapian.cancel_transaction }
+ end
+
## xapian-compact takes too long, so this is a no-op
## until we think of something better
def optimize
--
1.7.2.3
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format
2011-01-19 13:52 ` [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format Sascha Silbe
@ 2011-01-20 4:13 ` Rich Lane
2011-01-20 15:28 ` Nicolas Pouillard
1 sibling, 0 replies; 6+ messages in thread
From: Rich Lane @ 2011-01-20 4:13 UTC (permalink / raw)
To: Sascha Silbe; +Cc: sup-devel
Applied to master.
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format
2011-01-19 13:52 ` [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format Sascha Silbe
2011-01-20 4:13 ` Rich Lane
@ 2011-01-20 15:28 ` Nicolas Pouillard
2011-01-21 12:36 ` Sascha Silbe
1 sibling, 1 reply; 6+ messages in thread
From: Nicolas Pouillard @ 2011-01-20 15:28 UTC (permalink / raw)
To: Sascha Silbe, Sup developer discussion, sup-devel
On Wed, 19 Jan 2011 14:52:15 +0100, Sascha Silbe <sascha-pgp@silbe.org> wrote:
> sup-import-dump imports message state as exported by sup-dump into the index.
> It is a direct replacement for the sup-sync --restored functionality that got
> lost when merging the maildir branch.
> Unlike sup-sync it operates on the index only, so it's fast enough for
> periodically importing full dumps to keep multiple sup installations
> synchronised.
So it does not clear the index, right? It only assign the set of labels to the
given message. Just what I need to upgrade my sup index with my notmuch index.
> It should also be easy enough to add support for a "diff" style format that
> would allow replaying "logs" if sup were enhanced to write those in the
> future.
Please do.
I hacked such a logging feature [1] in the past (at the time where sup
used ferret and that message loss happened), then I merged the previous index
dump with the log and compared this to the actual dump (I wrote some tools
to play with this log format).
Cheers,
[1]: http://gitorious.org/~ertai/sup/clone-by-ertai/commits/to-submit-log-labels-mapping
--
Nicolas Pouillard
http://nicolaspouillard.fr
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format
2011-01-20 15:28 ` Nicolas Pouillard
@ 2011-01-21 12:36 ` Sascha Silbe
0 siblings, 0 replies; 6+ messages in thread
From: Sascha Silbe @ 2011-01-21 12:36 UTC (permalink / raw)
To: Nicolas Pouillard; +Cc: Sup developer discussion
[-- Attachment #1.1: Type: text/plain, Size: 1168 bytes --]
Excerpts from Nicolas Pouillard's message of Thu Jan 20 16:28:17 +0100 2011:
> So it does not clear the index, right? It only assign the set of labels to the
> given message. Just what I need to upgrade my sup index with my notmuch index.
Yes, that's exactly the purpose (except that I'm using it to synchronise
two sup installations, not sup and notmuch).
> > It should also be easy enough to add support for a "diff" style format that
> > would allow replaying "logs" if sup were enhanced to write those in the
> > future.
> Please do.
Eventually I will, but I don't know when I'll get around to do it.
> I hacked such a logging feature [1] in the past [...]
Nice! Looks like the only other places (besides sync_message) that write
to the index are delete and delete_message (the latter is part of a
not-yet-upstreamed patch of mine). Looks like the hardest part will
actually be adding the configuration option for the file name (with
support for a date template). ;)
> [1]: http://gitorious.org/~ertai/sup/clone-by-ertai/commits/to-submit-log-labels-mapping
Sascha
--
http://sascha.silbe.org/
http://www.infra-silbe.de/
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]
[-- Attachment #2: Type: text/plain, Size: 143 bytes --]
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-01-21 13:42 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-08-06 9:13 [sup-devel] alternative for sup-sync option --restored? Sascha Silbe
2010-08-22 13:40 ` Sascha Silbe
2011-01-19 13:52 ` [sup-devel] [PATCH] add sup-import-dump: import message state in sup-dump format Sascha Silbe
2011-01-20 4:13 ` Rich Lane
2011-01-20 15:28 ` Nicolas Pouillard
2011-01-21 12:36 ` Sascha Silbe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox