community/pipermail-archives/sup-devel/2011-07.txt (42135B) - raw
1 From sascha-pgp@silbe.org Sat Jul 2 11:06:54 2011
2 From: sascha-pgp@silbe.org (Sascha Silbe)
3 Date: Sat, 2 Jul 2011 17:06:54 +0200
4 Subject: [sup-devel] [PATCH] Catch errors while saving a message to disk for
5 editing
6 Message-ID: <1309619214-2446-1-git-send-email-sascha-pgp@silbe.org>
7
8 Running out of disk space in /tmp caused sup to crash with the following
9 exception:
10
11 --- Errno::ENOSPC from thread: main
12 No space left on device - /tmp/sascha_silbe/sup.reply-mode20110702-31427-rtg4kl-0
13 /usr/lib/ruby/1.8/tempfile.rb:97:in `close'
14 /usr/lib/ruby/1.8/tempfile.rb:97:in `_close'
15 /usr/lib/ruby/1.8/tempfile.rb:112:in `close'
16 ./lib/sup/modes/edit-message-mode.rb:180:in `edit_message'
17 ./lib/sup/mode.rb:59:in `send'
18 ./lib/sup/mode.rb:59:in `handle_input'
19 ./lib/sup/buffer.rb:278:in `handle_input'
20 bin/sup:271
21
22 Signed-off-by: Sascha Silbe <sascha-pgp at silbe.org>
23 ---
24 lib/sup/modes/edit-message-mode.rb | 24 +++++++++++++++++-------
25 1 files changed, 17 insertions(+), 7 deletions(-)
26
27 diff --git a/lib/sup/modes/edit-message-mode.rb b/lib/sup/modes/edit-message-mode.rb
28 index 5ed7833..256e314 100644
29 --- a/lib/sup/modes/edit-message-mode.rb
30 +++ b/lib/sup/modes/edit-message-mode.rb
31 @@ -172,12 +172,21 @@ def edit_to; edit_field "To" end
32 def edit_cc; edit_field "Cc" end
33 def edit_subject; edit_field "Subject" end
34
35 - def edit_message
36 - @file = Tempfile.new "sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}"
37 + def save_message_to_file
38 + @file = Tempfile.new ["sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}", ".eml"]
39 @file.puts format_headers(@header - NON_EDITABLE_HEADERS).first
40 @file.puts
41 @file.puts @body.join("\n")
42 @file.close
43 + end
44 +
45 + def edit_message
46 + begin
47 + write_message_to_file
48 + rescue SystemCallError => e
49 + BufferManager.flash "Can't save message to file: #{e.message}"
50 + return
51 + end
52
53 editor = $config[:editor] || ENV['EDITOR'] || "/usr/bin/vi"
54
55 @@ -197,11 +206,12 @@ def edit_message
56 end
57
58 def edit_message_async
59 - @file = Tempfile.new ["sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}", ".eml"]
60 - @file.puts format_headers(@header - NON_EDITABLE_HEADERS).first
61 - @file.puts
62 - @file.puts @body.join("\n")
63 - @file.close
64 + begin
65 + write_message_to_file
66 + rescue SystemCallError => e
67 + BufferManager.flash "Can't save message to file: #{e.message}"
68 + return
69 + end
70
71 @mtime = File.mtime @file.path
72
73 --
74 1.7.4.1
75
76
77 From alex.shulgin@gmail.com Sat Jul 2 16:28:27 2011
78 From: alex.shulgin@gmail.com (Alexander Shulgin)
79 Date: Sat, 2 Jul 2011 23:28:27 +0300
80 Subject: [sup-devel] [PATCH] Catch errors while saving a message to disk
81 for editing
82 In-Reply-To: <1309619214-2446-1-git-send-email-sascha-pgp@silbe.org>
83 References: <1309619214-2446-1-git-send-email-sascha-pgp@silbe.org>
84 Message-ID: <BANLkTikhM6fvZoSJu=Uo_qPfTsPy3uW5tA@mail.gmail.com>
85
86 On Sat, Jul 2, 2011 at 18:06, Sascha Silbe <sascha-pgp at silbe.org> wrote:
87
88 > ?lib/sup/modes/edit-message-mode.rb | ? 24 +++++++++++++++++-------
89 > ?1 files changed, 17 insertions(+), 7 deletions(-)
90 >
91 > diff --git a/lib/sup/modes/edit-message-mode.rb b/lib/sup/modes/edit-message-mode.rb
92 > index 5ed7833..256e314 100644
93 > --- a/lib/sup/modes/edit-message-mode.rb
94 > +++ b/lib/sup/modes/edit-message-mode.rb
95 > @@ -172,12 +172,21 @@ def edit_to; edit_field "To" end
96 > ? def edit_cc; edit_field "Cc" end
97 > ? def edit_subject; edit_field "Subject" end
98 >
99 > - ?def edit_message
100 > - ? ?@file = Tempfile.new "sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}"
101 > + ?def save_message_to_file
102
103 Didn't you mean 'write_message_to_file' here instead?
104
105 > + ? ?@file = Tempfile.new ["sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}", ".eml"]
106 > ? ? @file.puts format_headers(@header - NON_EDITABLE_HEADERS).first
107 > ? ? @file.puts
108 > ? ? @file.puts @body.join("\n")
109 > ? ? @file.close
110 > + ?end
111 > +
112 > + ?def edit_message
113 > + ? ?begin
114 > + ? ? ?write_message_to_file
115 > + ? ?rescue SystemCallError => e
116 > + ? ? ?BufferManager.flash "Can't save message to file: #{e.message}"
117 > + ? ? ?return
118 > + ? ?end
119 >
120 > ? ? editor = $config[:editor] || ENV['EDITOR'] || "/usr/bin/vi"
121 >
122 [snip]
123
124 From sascha-ml-reply-to-2011-3@silbe.org Sat Jul 2 17:54:04 2011
125 From: sascha-ml-reply-to-2011-3@silbe.org (Sascha Silbe)
126 Date: Sat, 02 Jul 2011 23:54:04 +0200
127 Subject: [sup-devel] [PATCH] Catch errors while saving a message to disk
128 for editing
129 In-Reply-To: <BANLkTikhM6fvZoSJu=Uo_qPfTsPy3uW5tA@mail.gmail.com>
130 References: <1309619214-2446-1-git-send-email-sascha-pgp@silbe.org>
131 <BANLkTikhM6fvZoSJu=Uo_qPfTsPy3uW5tA@mail.gmail.com>
132 Message-ID: <1309643493-sup-6539@twin.sascha.silbe.org>
133
134 Excerpts from Alexander Shulgin's message of Sat Jul 02 22:28:27 +0200 2011:
135
136 > > - ?def edit_message
137 > > - ? ?@file = Tempfile.new "sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}"
138 > > + ?def save_message_to_file
139 >
140 > Didn't you mean 'write_message_to_file' here instead?
141
142 Oops, yes. I forgot to update the index after fixing this (there are
143 other, unrelated changes in the working directory that I didn't want to
144 commit, so I didn't use "git commit -a" as usual). Will post a new
145 version. Thanks for spotting this!
146
147 Sascha
148
149 --
150 http://sascha.silbe.org/
151 http://www.infra-silbe.de/
152 -------------- next part --------------
153 A non-text attachment was scrubbed...
154 Name: signature.asc
155 Type: application/pgp-signature
156 Size: 500 bytes
157 Desc: not available
158 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20110702/f2b87823/attachment.bin>
159
160 From sascha-pgp@silbe.org Sat Jul 2 17:56:16 2011
161 From: sascha-pgp@silbe.org (Sascha Silbe)
162 Date: Sat, 2 Jul 2011 23:56:16 +0200
163 Subject: [sup-devel] [PATCH v2] Catch errors while saving a message to disk
164 for editing
165 In-Reply-To: <BANLkTikhM6fvZoSJu=Uo_qPfTsPy3uW5tA@mail.gmail.com>
166 References: <BANLkTikhM6fvZoSJu=Uo_qPfTsPy3uW5tA@mail.gmail.com>
167 Message-ID: <1309643776-8936-1-git-send-email-sascha-pgp@silbe.org>
168
169 Running out of disk space in /tmp caused sup to crash with the following
170 exception:
171
172 --- Errno::ENOSPC from thread: main
173 No space left on device - /tmp/sascha_silbe/sup.reply-mode20110702-31427-rtg4kl-0
174 /usr/lib/ruby/1.8/tempfile.rb:97:in `close'
175 /usr/lib/ruby/1.8/tempfile.rb:97:in `_close'
176 /usr/lib/ruby/1.8/tempfile.rb:112:in `close'
177 ./lib/sup/modes/edit-message-mode.rb:180:in `edit_message'
178 ./lib/sup/mode.rb:59:in `send'
179 ./lib/sup/mode.rb:59:in `handle_input'
180 ./lib/sup/buffer.rb:278:in `handle_input'
181 bin/sup:271
182
183 Signed-off-by: Sascha Silbe <sascha-pgp at silbe.org>
184 ---
185 v1->v2: fix typo (save_message_to_file vs. write_message_to_file)
186
187 lib/sup/modes/edit-message-mode.rb | 24 +++++++++++++++++-------
188 1 files changed, 17 insertions(+), 7 deletions(-)
189
190 diff --git a/lib/sup/modes/edit-message-mode.rb b/lib/sup/modes/edit-message-mode.rb
191 index 5ed7833..4387f7b 100644
192 --- a/lib/sup/modes/edit-message-mode.rb
193 +++ b/lib/sup/modes/edit-message-mode.rb
194 @@ -172,12 +172,21 @@ def edit_to; edit_field "To" end
195 def edit_cc; edit_field "Cc" end
196 def edit_subject; edit_field "Subject" end
197
198 - def edit_message
199 - @file = Tempfile.new "sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}"
200 + def save_message_to_file
201 + @file = Tempfile.new ["sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}", ".eml"]
202 @file.puts format_headers(@header - NON_EDITABLE_HEADERS).first
203 @file.puts
204 @file.puts @body.join("\n")
205 @file.close
206 + end
207 +
208 + def edit_message
209 + begin
210 + save_message_to_file
211 + rescue SystemCallError => e
212 + BufferManager.flash "Can't save message to file: #{e.message}"
213 + return
214 + end
215
216 editor = $config[:editor] || ENV['EDITOR'] || "/usr/bin/vi"
217
218 @@ -197,11 +206,12 @@ def edit_message
219 end
220
221 def edit_message_async
222 - @file = Tempfile.new ["sup.#{self.class.name.gsub(/.*::/, '').camel_to_hyphy}", ".eml"]
223 - @file.puts format_headers(@header - NON_EDITABLE_HEADERS).first
224 - @file.puts
225 - @file.puts @body.join("\n")
226 - @file.close
227 + begin
228 + save_message_to_file
229 + rescue SystemCallError => e
230 + BufferManager.flash "Can't save message to file: #{e.message}"
231 + return
232 + end
233
234 @mtime = File.mtime @file.path
235
236 --
237 1.7.4.1
238
239
240 From hsanson@gmail.com Mon Jul 4 21:52:53 2011
241 From: hsanson@gmail.com (Horacio Sanson)
242 Date: Tue, 5 Jul 2011 10:52:53 +0900
243 Subject: [sup-devel] Heliotrope improving but still found some issues
244 Message-ID: <201107051052.53166.hsanson@gmail.com>
245
246
247 So I tried the latest heliotrope with the leveldb-ruby 0.6 gem, whistlepig 0.7
248 and MeCab hooks for Japanese text support and it works better than before.
249 Unfortunately got two issues:
250
251 First any attempt to search using japanese text fails with the dreaded
252 incompatible character encodings error:
253
254 #####################################################
255 [2011-07-05 10:22:17] INFO WEBrick 1.3.1
256 [2011-07-05 10:22:17] INFO ruby 1.9.2 (2010-08-18) [x86_64-linux]
257 [2011-07-05 10:22:17] INFO WEBrick::HTTPServer#start: pid=13523 port=8042
258 search(body:"??", 0, 20) took 2.1ms
259 Encoding::CompatibilityError - incompatible character encodings: ASCII-8BIT
260 and UTF-8:
261 bin/heliotrope-server:223:in `block in <class:HeliotropeServer>'
262 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:1152:in `call'
263 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:1152:in `block in
264 compile!'
265 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:724:in
266 `instance_eval'
267 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:724:in
268 `route_eval'
269 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:708:in `block (2
270 levels) in route!'
271 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:758:in `block in
272 process_route'
273 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:755:in `catch'
274 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:755:in
275 `process_route'
276 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:707:in `block in
277 route!'
278 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:706:in `each'
279 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:706:in `route!'
280 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:843:in `dispatch!'
281 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:644:in `block in
282 call!'
283 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:808:in
284 `instance_eval'
285 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:808:in `block in
286 invoke'
287 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:808:in `catch'
288 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:808:in `invoke'
289 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:644:in `call!'
290 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:629:in `call'
291 /var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/head.rb:9:in `call'
292 /var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/showexceptions.rb:21:in
293 `call'
294 /var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/lint.rb:48:in `_call'
295 /var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/lint.rb:36:in `call'
296 /var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/showexceptions.rb:24:in `call'
297 /var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/commonlogger.rb:18:in `call'
298 /var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/content_length.rb:13:in `call'
299 /var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/handler/webrick.rb:52:in
300 `service'
301 /usr/lib/ruby/1.9.1/webrick/httpserver.rb:111:in `service'
302 /usr/lib/ruby/1.9.1/webrick/httpserver.rb:70:in `run'
303 /usr/lib/ruby/1.9.1/webrick/server.rb:183:in `block in start_thread'
304 127.0.0.1 - - [05/Jul/2011 10:22:20] "GET /search?q=%E6%89%8B%E7%B4%99
305 HTTP/1.1" 500 89118 0.0331
306 localhost - - [05/Jul/2011:10:22:20 JST] "GET /search?q=%E6%89%8B%E7%B4%99
307 HTTP/1.0" 500 89118
308 - -> /search?q=%E6%89%8B%E7%B4%99
309 [2011-07-05 10:22:20] ERROR Errno::ECONNRESET: Connection reset by peer
310 /usr/lib/ruby/1.9.1/webrick/httpserver.rb:56:in `eof?'
311 /usr/lib/ruby/1.9.1/webrick/httpserver.rb:56:in `run'
312 #######################################################
313
314 The problem seems to be the header method in the heliotrope-server that uses
315 multiline strings (e.g. <<- EOS). By forcing the resulting text to UTF-8
316 encoding the search works as expected with japanese and non japanese text (see
317 attached patch).
318
319
320 The second problem is actually not heliotrope problem. Is the artificial
321 limitations imposed by Gmail. After running heliotrope-add for some time it
322 would fail when the IMAP fetch returns nil. Just after it failed I tried to
323 use my current email reader (kmail) and got an interesting error saying:
324 "exceeded IMAP bandwidth limits". These indicates the nil is due to Gmail
325 limiting the maximum bandwidth I can consume downloading emails.
326
327 The latest heliotrope now catches this error and ignores it but after a while
328 ignoring it I started getting sys-write errors on the socket. I believe this
329 is also GMail abruptly breaking the socket connection to enforce it's
330 bandwidth limits.
331
332 Maybe limiting the rate of gmail-dumper so it reads mails at a lower pace
333 would eliminate these problems or simply stop reading emails for some time
334 when we get the first nil response.
335
336 Overall heliotrope is now usable for Japanese language users (at least for me
337 ). Now I will start playing with turnsole to see if it can handle japanese.
338
339 --
340 regards,
341 Horacio Sanson
342 -------------- next part --------------
343 A non-text attachment was scrubbed...
344 Name: 0001-Fix-encoding-exception.patch
345 Type: text/x-patch
346 Size: 653 bytes
347 Desc: not available
348 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20110705/b685de98/attachment.bin>
349
350 From dmishd@gmail.com Tue Jul 5 05:32:59 2011
351 From: dmishd@gmail.com (Hamish D)
352 Date: Tue, 5 Jul 2011 10:32:59 +0100
353 Subject: [sup-devel] gmail limits [was: Heliotrope improving but still
354 found some issues]
355 Message-ID: <CAOxvSbd7N9Zj+tMh=rXimPCtUBJmyy+k-UeeK8DG3mqUqcfgPg@mail.gmail.com>
356
357 On 5 Jul 2011 02:53, "Horacio Sanson" <hsanson at gmail.com> wrote:
358 > The second problem is actually not heliotrope problem. Is the artificial
359 > limitations imposed by Gmail. After running heliotrope-add for some time
360 it
361 > would fail when the IMAP fetch returns nil. Just after it failed I tried
362 to
363 > use my current email reader (kmail) and got an interesting error saying:
364 > "exceeded IMAP bandwidth limits". These indicates the nil is due to Gmail
365 > limiting the maximum bandwidth I can consume downloading emails.
366 >
367 > Maybe limiting the rate of gmail-dumper so it reads mails at a lower pace
368 > would eliminate these problems or simply stop reading emails for some time
369 > when we get the first nil response.
370
371 I would prefer if heliotrope could recognise the IMAP limit exceeded
372 situation and give a sensible error message - 'gmail won't give any more
373 messages now, please try again after an hour' or something along those
374 lines. I got through over 12000 messages before hitting trouble, so people
375 with smaller accounts may never hit the problem and would like the speed to
376 be as high as possible.
377
378 Hamish Downer
379 -------------- next part --------------
380 An HTML attachment was scrubbed...
381 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20110705/58dafb8d/attachment.html>
382
383 From hsanson@gmail.com Tue Jul 5 10:01:19 2011
384 From: hsanson@gmail.com (Horacio Sanson)
385 Date: Tue, 5 Jul 2011 23:01:19 +0900
386 Subject: [sup-devel] Heliotrope improving but still found some issues
387 In-Reply-To: <201107051052.53166.hsanson@gmail.com>
388 References: <201107051052.53166.hsanson@gmail.com>
389 Message-ID: <CAHWBo_ZuLKhx-MW4O=1Nwi3A3KkSfyDQUpmnbmvHL62d9P1XuQ@mail.gmail.com>
390
391 Speaked too fast... got a few more issues.
392
393 First the patch I sent was incomplete, please ignore and use the one
394 attached here if you would. The previous patch completely removed the
395 header section of the web page.
396
397 Second the GMail labels work great as long as they are in english.
398 This is because the labels are UTF-7 encoded and look like this in
399 japanese: "+&mlkwrtdrmkiwwzdxmlgw4zdrmpm-" and clicking on any of
400 these labels in the web interface result in systax error. Fixing this
401 is as simple as replacing line 207 of imap-dumper.rb with the
402 following code:
403
404 labels = (data.attr["X-GM-LABELS"] || []).map { |label|
405 Net::IMAP.decode_utf7(label.to_s).downcase }
406
407 I am pretty sure the utf7 decoding is language independent and can be
408 applied safely to all labels in any language but I cannot bet on it
409 since I only have tested Japanese. Not sure if this conversion would
410 require a separate hook or something like that.
411
412
413
414 regards,
415 Horacio
416
417 On Tue, Jul 5, 2011 at 10:52 AM, Horacio Sanson <hsanson at gmail.com> wrote:
418 >
419 > So I tried the latest heliotrope with the leveldb-ruby 0.6 gem, whistlepig 0.7
420 > and MeCab hooks for Japanese text support and it works better than before.
421 > Unfortunately got two issues:
422 >
423 > First any attempt to search using japanese text fails with the dreaded
424 > incompatible character encodings error:
425 >
426 > #####################################################
427 > [2011-07-05 10:22:17] INFO ?WEBrick 1.3.1
428 > [2011-07-05 10:22:17] INFO ?ruby 1.9.2 (2010-08-18) [x86_64-linux]
429 > [2011-07-05 10:22:17] INFO ?WEBrick::HTTPServer#start: pid=13523 port=8042
430 > search(body:"??", 0, 20) took 2.1ms
431 > Encoding::CompatibilityError - incompatible character encodings: ASCII-8BIT
432 > and UTF-8:
433 > ?bin/heliotrope-server:223:in `block in <class:HeliotropeServer>'
434 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:1152:in `call'
435 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:1152:in `block in
436 > compile!'
437 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:724:in
438 > `instance_eval'
439 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:724:in
440 > `route_eval'
441 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:708:in `block (2
442 > levels) in route!'
443 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:758:in `block in
444 > process_route'
445 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:755:in `catch'
446 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:755:in
447 > `process_route'
448 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:707:in `block in
449 > route!'
450 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:706:in `each'
451 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:706:in `route!'
452 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:843:in `dispatch!'
453 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:644:in `block in
454 > call!'
455 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:808:in
456 > `instance_eval'
457 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:808:in `block in
458 > invoke'
459 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:808:in `catch'
460 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:808:in `invoke'
461 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:644:in `call!'
462 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/base.rb:629:in `call'
463 > ?/var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/head.rb:9:in `call'
464 > ?/var/lib/gems/1.9.1/gems/sinatra-1.2.5/lib/sinatra/showexceptions.rb:21:in
465 > `call'
466 > ?/var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/lint.rb:48:in `_call'
467 > ?/var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/lint.rb:36:in `call'
468 > ?/var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/showexceptions.rb:24:in `call'
469 > ?/var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/commonlogger.rb:18:in `call'
470 > ?/var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/content_length.rb:13:in `call'
471 > ?/var/lib/gems/1.9.1/gems/rack-1.2.2/lib/rack/handler/webrick.rb:52:in
472 > `service'
473 > ?/usr/lib/ruby/1.9.1/webrick/httpserver.rb:111:in `service'
474 > ?/usr/lib/ruby/1.9.1/webrick/httpserver.rb:70:in `run'
475 > ?/usr/lib/ruby/1.9.1/webrick/server.rb:183:in `block in start_thread'
476 > 127.0.0.1 - - [05/Jul/2011 10:22:20] "GET /search?q=%E6%89%8B%E7%B4%99
477 > HTTP/1.1" 500 89118 0.0331
478 > localhost - - [05/Jul/2011:10:22:20 JST] "GET /search?q=%E6%89%8B%E7%B4%99
479 > HTTP/1.0" 500 89118
480 > - -> /search?q=%E6%89%8B%E7%B4%99
481 > [2011-07-05 10:22:20] ERROR Errno::ECONNRESET: Connection reset by peer
482 > ? ? ? ?/usr/lib/ruby/1.9.1/webrick/httpserver.rb:56:in `eof?'
483 > ? ? ? ?/usr/lib/ruby/1.9.1/webrick/httpserver.rb:56:in `run'
484 > #######################################################
485 >
486 > The problem seems to be the header method in the heliotrope-server that uses
487 > multiline strings (e.g. <<- EOS). By forcing the resulting text to UTF-8
488 > encoding the search works as expected with japanese and non japanese text (see
489 > attached patch).
490 >
491 >
492 > The second problem is actually not heliotrope problem. Is the artificial
493 > limitations imposed by Gmail. After running heliotrope-add for some time it
494 > would fail when the IMAP fetch returns nil. Just after it failed I tried to
495 > use my current email reader (kmail) and got an interesting error saying:
496 > "exceeded IMAP bandwidth limits". These indicates the nil is due to Gmail
497 > limiting the maximum bandwidth I can consume downloading emails.
498 >
499 > The latest heliotrope now catches this error and ignores it but after a while
500 > ignoring it I started getting sys-write errors on the socket. I believe this
501 > is also GMail abruptly breaking the socket connection to enforce it's
502 > bandwidth limits.
503 >
504 > Maybe limiting the rate of gmail-dumper so it reads mails at a lower pace
505 > would eliminate these problems or simply stop reading emails for some time
506 > when we get the first nil response.
507 >
508 > Overall heliotrope is now usable for Japanese language users (at least for me
509 > ). Now I will start playing with turnsole to see if it can handle japanese.
510 >
511 > --
512 > regards,
513 > Horacio Sanson
514 >
515 -------------- next part --------------
516 A non-text attachment was scrubbed...
517 Name: 0001-Fix-encoding-exception.patch
518 Type: text/x-patch
519 Size: 986 bytes
520 Desc: not available
521 URL: <http://rubyforge.org/pipermail/sup-devel/attachments/20110705/7e704fc8/attachment-0001.bin>
522
523 From hsanson@gmail.com Wed Jul 6 22:24:15 2011
524 From: hsanson@gmail.com (Horacio Sanson)
525 Date: Thu, 7 Jul 2011 11:24:15 +0900
526 Subject: [sup-devel] How are the queries supposed to work?
527 Message-ID: <201107071124.16433.hsanson@gmail.com>
528
529
530 Finally after several attempts the gmail-dumper finished indexing my +450,000
531 emails but searching emails does not work as I expected or I am doing
532 something wrong.
533
534 Here is an example I have been trying to understand:
535
536 I am using the heliotrope-console with this command:
537
538 bash> ruby1.9.1 -Ilib bin/heliotrope-console -d ~/.heliotrope
539
540 Then in that console :
541
542 # Create a query for EVERY
543 index.set_query(Query.new("body", "*"))
544
545 # Get first 5 matches
546 r = index.get_some_results 5
547
548 # Inspect the one result
549
550 puts r[1][:subject]
551 => [Rails] Test fixtures not loading
552
553 puts r[1][:direct_recipients].inspect
554 => #<Set: {"Ruby on Rails: Talk <rubyonrails-talk at googlegroups.com>"}>
555
556 puts r[1][:snippet].inspect
557 => PLEASE HELP. This is driving me insane. I have a simple database table
558
559 puts r[1][:labels]
560 => #<Set: {"unread"}>
561
562
563 # So I have a message indexed with a subject that contains "Rails" a direct
564 recipient with "rubyonrails-talk at googlegroups.com" and a body that contains
565 "PLEASE HELP".
566
567
568 # Now I tried several queries that I thought would return that message but
569 they all returned zero results:
570
571 index.set_query(Query.new("body", "HELP"))
572 index.set_query(Query.new("body", "PLEASE"))
573 index.set_query(Query.new("labels", "unread")
574 index.set_query(Query.new("from", "rubyonrails-talk at googlegroups.com")
575 index.set_query(Query.new("to", "rubyonrails-talk at googlegroups.com")
576 index.set_query(Query.new("body", "rubyonrails-talk at googlegroups.com")
577
578 # And the interesting part is that these queries do return the message I
579 expect:
580
581 index.set_query(Query.new("body", "fixtures"))
582 index.set_query(Query.new("subject", "fixtures"))
583
584 # This made me think that only the subject is searchable since the word
585 "fixtures" only appears in the subject but then these queries with words in the
586 subject return zero results:
587
588 index.set_query(Query.new("subject", "Rails"))
589 index.set_query(Query.new("subject", "[Rails]"))
590 index.set_query(Query.new("subject", "Test fixtures"))
591 index.set_query(Query.new("subject", "test fixtures"))
592
593 On all tests I made sure to run index.reset_query! before setting the new
594 query with index.set_query. Is this the correct way???
595
596 --
597 regards,
598 Horacio Sanson
599
600 From wmorgan-sup@masanjin.net Thu Jul 7 02:07:41 2011
601 From: wmorgan-sup@masanjin.net (William Morgan)
602 Date: Thu, 07 Jul 2011 06:07:41 +0000
603 Subject: [sup-devel] How are the queries supposed to work?
604 In-Reply-To: <201107071124.16433.hsanson@gmail.com>
605 References: <201107071124.16433.hsanson@gmail.com>
606 Message-ID: <1310018610-sup-9096@masanjin.net>
607
608 Hi Horacio,
609
610 Reformatted excerpts from Horacio Sanson's message of 2011-07-07:
611 > # Now I tried several queries that I thought would return that message but
612 > they all returned zero results:
613 >
614 > index.set_query(Query.new("body", "HELP"))
615 > index.set_query(Query.new("body", "PLEASE"))
616
617 These two are due to case folding. If you try "help" and "please", it
618 should work.
619
620 > index.set_query(Query.new("labels", "unread")
621
622 This one should be Query.new("body", "~unread"). The label syntax is
623 different in heliotrope from in Sup; they aren't regular fielded terms.
624
625 > index.set_query(Query.new("from", "rubyonrails-talk at googlegroups.com")
626 > index.set_query(Query.new("to", "rubyonrails-talk at googlegroups.com")
627 > index.set_query(Query.new("body", "rubyonrails-talk at googlegroups.com")
628
629 This I don't quite understand. Similar queries work on my system. Would
630 you be able to send the the message that this corresponds to?
631
632 > index.set_query(Query.new("body", "fixtures"))
633 > index.set_query(Query.new("subject", "fixtures"))
634
635 These ones work due to the lower casing.
636
637 > index.set_query(Query.new("subject", "Rails"))
638 > index.set_query(Query.new("subject", "[Rails]"))
639 > index.set_query(Query.new("subject", "Test fixtures"))
640 > index.set_query(Query.new("subject", "test fixtures"))
641
642 I would expect the last one to work. Did it?
643
644 > On all tests I made sure to run index.reset_query! before setting the
645 > new query with index.set_query. Is this the correct way???
646
647 The reset_query! is unnecessary.
648
649 Thanks for all your testing. Much of this is undocumented, so I ask you
650 to bear with me.
651 --
652 William <wmorgan-sup at masanjin.net>
653
654 From wmorgan-sup@masanjin.net Thu Jul 7 02:31:30 2011
655 From: wmorgan-sup@masanjin.net (William Morgan)
656 Date: Thu, 07 Jul 2011 06:31:30 +0000
657 Subject: [sup-devel] Heliotrope improving but still found some issues
658 In-Reply-To: <201107051052.53166.hsanson@gmail.com>
659 References: <201107051052.53166.hsanson@gmail.com>
660 Message-ID: <1310019783-sup-5419@masanjin.net>
661
662 Hi Horacio,
663
664 Thanks for all your help testing. I am committed to making Heliotrope and
665 Turnsole i18n-friendly, so it's great to have some stress applied to that
666 area.
667
668 Reformatted excerpts from Horacio Sanson's message of 2011-07-05:
669 > Encoding::CompatibilityError - incompatible character encodings: ASCII-8BIT
670 > and UTF-8:
671 > bin/heliotrope-server:223:in `block in <class:HeliotropeServer>'
672
673 Ok, I will check out your patch.
674
675 > The latest heliotrope now catches this error and ignores it but after
676 > a while ignoring it I started getting sys-write errors on the socket.
677 > I believe this is also GMail abruptly breaking the socket connection
678 > to enforce it's bandwidth limits.
679
680 I suspect you're right. I wonder if there is a way to detect the rate
681 limiting kicking on. If not, I can add a delay. You can also manually
682 delay with judicious use of the -n flag.
683
684 > Overall heliotrope is now usable for Japanese language users (at least
685 > for me ). Now I will start playing with turnsole to see if it can
686 > handle japanese.
687
688 Great! I look forward to your bug reports. If you could please file bugs
689 in the corresponding Github projects, that will help me keep track of
690 things.
691 --
692 William <wmorgan-sup at masanjin.net>
693
694 From wmorgan-sup@masanjin.net Thu Jul 7 02:33:05 2011
695 From: wmorgan-sup@masanjin.net (William Morgan)
696 Date: Thu, 07 Jul 2011 06:33:05 +0000
697 Subject: [sup-devel] Heliotrope improving but still found some issues
698 In-Reply-To: <CAHWBo_ZuLKhx-MW4O=1Nwi3A3KkSfyDQUpmnbmvHL62d9P1XuQ@mail.gmail.com>
699 References: <201107051052.53166.hsanson@gmail.com>
700 <CAHWBo_ZuLKhx-MW4O=1Nwi3A3KkSfyDQUpmnbmvHL62d9P1XuQ@mail.gmail.com>
701 Message-ID: <1310020323-sup-7703@masanjin.net>
702
703 Reformatted excerpts from Horacio Sanson's message of 2011-07-05:
704 > I am pretty sure the utf7 decoding is language independent and can be
705 > applied safely to all labels in any language but I cannot bet on it
706
707 I think you are right. It should be safe to utf7-decode all labels.
708 Good catch.
709 --
710 William <wmorgan-sup at masanjin.net>
711
712 From hsanson@gmail.com Thu Jul 7 10:48:23 2011
713 From: hsanson@gmail.com (Horacio Sanson)
714 Date: Thu, 7 Jul 2011 23:48:23 +0900
715 Subject: [sup-devel] How are the queries supposed to work?
716 In-Reply-To: <1310018610-sup-9096@masanjin.net>
717 References: <201107071124.16433.hsanson@gmail.com>
718 <1310018610-sup-9096@masanjin.net>
719 Message-ID: <201107072348.23637.hsanson@gmail.com>
720
721 On Thursday 07 July 2011 15:07:41 William Morgan wrote:
722 > Hi Horacio,
723 >
724 > Reformatted excerpts from Horacio Sanson's message of 2011-07-07:
725 > > # Now I tried several queries that I thought would return that message
726 > > but they all returned zero results:
727 > >
728 > > index.set_query(Query.new("body", "HELP"))
729 > > index.set_query(Query.new("body", "PLEASE"))
730 >
731 > These two are due to case folding. If you try "help" and "please", it
732 > should work.
733 >
734
735 Indeed lowercasing all the queries make them work.
736
737 > > index.set_query(Query.new("labels", "unread")
738 >
739 > This one should be Query.new("body", "~unread"). The label syntax is
740 > different in heliotrope from in Sup; they aren't regular fielded terms.
741 >
742 > > index.set_query(Query.new("from", "rubyonrails-talk at googlegroups.com")
743 > > index.set_query(Query.new("to", "rubyonrails-talk at googlegroups.com")
744 > > index.set_query(Query.new("body", "rubyonrails-talk at googlegroups.com")
745 >
746 > This I don't quite understand. Similar queries work on my system. Would
747 > you be able to send the the message that this corresponds to?
748 >
749
750 Sorry my mistake. The queries I did were with:
751
752 "<rubyonrails-talk at googlegroups.com>"
753
754 in this case the result is zero but if I remove the "<" and ">" then I get the
755 expected results. The same goes for "[rails]" that does not work unless I
756 remove the square brackets.
757
758 > > index.set_query(Query.new("body", "fixtures"))
759 > > index.set_query(Query.new("subject", "fixtures"))
760 >
761 > These ones work due to the lower casing.
762 >
763 > > index.set_query(Query.new("subject", "Rails"))
764 > > index.set_query(Query.new("subject", "[Rails]"))
765 > > index.set_query(Query.new("subject", "Test fixtures"))
766 > > index.set_query(Query.new("subject", "test fixtures"))
767 >
768 > I would expect the last one to work. Did it?
769 >
770
771 You are rigth, the last query works correclty. Maybe I was already tired of so
772 much testing and forgot to actually run the query after setting it.
773
774 > > On all tests I made sure to run index.reset_query! before setting the
775 > > new query with index.set_query. Is this the correct way???
776 >
777 > The reset_query! is unnecessary.
778 >
779 > Thanks for all your testing. Much of this is undocumented, so I ask you
780 > to bear with me.
781
782 Once the UTF-7 encoding issue with the labels get's fixed I will test querying
783 with Japanese labels.
784
785 --
786 regards,
787 Horacio Sanson
788
789 From wmorgan-sup@masanjin.net Thu Jul 7 14:08:47 2011
790 From: wmorgan-sup@masanjin.net (William Morgan)
791 Date: Thu, 07 Jul 2011 18:08:47 +0000
792 Subject: [sup-devel] How are the queries supposed to work?
793 In-Reply-To: <201107072348.23637.hsanson@gmail.com>
794 References: <201107071124.16433.hsanson@gmail.com>
795 <1310018610-sup-9096@masanjin.net>
796 <201107072348.23637.hsanson@gmail.com>
797 Message-ID: <1310062014-sup-3818@masanjin.net>
798
799 Ok great, glad we're in sync. I've added an issue for improving this
800 documentation.
801 --
802 William <wmorgan-sup at masanjin.net>
803
804 From wmorgan-sup@masanjin.net Sat Jul 9 18:22:15 2011
805 From: wmorgan-sup@masanjin.net (William Morgan)
806 Date: Sat, 09 Jul 2011 22:22:15 +0000
807 Subject: [sup-devel] Heliotrope improving but still found some issues
808 In-Reply-To: <201107051052.53166.hsanson@gmail.com>
809 References: <201107051052.53166.hsanson@gmail.com>
810 Message-ID: <1310250042-sup-5348@masanjin.net>
811
812 Hi Horacio,
813
814 Reformatted excerpts from Horacio Sanson's message of 2011-07-05:
815 > First any attempt to search using japanese text fails with the dreaded
816 > incompatible character encodings error:
817
818 I'm having trouble reproducing this, or even understanding why your fix would
819 help, since all string literals in the code should be UTF-8-encoded.
820
821 Could you please apply this patch and tell me what the output is when
822 you feed it a crashing search term? Thanks!
823
824 --- cut here ---
825 diff --git a/bin/heliotrope-server b/bin/heliotrope-server
826 index c9754d4..ca764c0 100644
827 --- a/bin/heliotrope-server
828 +++ b/bin/heliotrope-server
829 @@ -219,6 +219,19 @@ class HeliotropeServer < Sinatra::Base
830 end
831 nav += "</div>"
832
833 + puts "start"
834 + p query.original_query_s.encoding
835 + p query.parsed_query_s.encoding
836 + p header("Search: #{query.original_query_s}", query.original_query_s).enc
837 + p "<div>Parsed query: #{escape_html query.parsed_query_s}</div>".encoding
838 + p "<div>Search took #{sprintf '%.2f', info[:elapsed]}s and #{info[:contin
839 + p "#{nav}<table>".encoding
840 + p results.size
841 + p results.map { |r| threadinfo_to_html r }.join.encoding
842 + p "</table>#{nav}".encoding
843 + p footer.encoding
844 + puts "end"
845 +
846 header("Search: #{query.original_query_s}", query.original_query_s) +
847 "<div>Parsed query: #{escape_html query.parsed_query_s}</div>" +
848 "<div>Search took #{sprintf '%.2f', info[:elapsed]}s and #{info[:contin
849 --- cut here ---
850 --
851 William <wmorgan-sup at masanjin.net>
852
853 From hsanson@gmail.com Sat Jul 9 19:50:34 2011
854 From: hsanson@gmail.com (Horacio Sanson)
855 Date: Sun, 10 Jul 2011 08:50:34 +0900
856 Subject: [sup-devel] Heliotrope improving but still found some issues
857 In-Reply-To: <1310250042-sup-5348@masanjin.net>
858 References: <201107051052.53166.hsanson@gmail.com>
859 <1310250042-sup-5348@masanjin.net>
860 Message-ID: <201107100850.34478.hsanson@gmail.com>
861
862 Hello,
863
864 On Sunday 10 July 2011 07:22:15 William Morgan wrote:
865 > Hi Horacio,
866 >
867 > Reformatted excerpts from Horacio Sanson's message of 2011-07-05:
868 > > First any attempt to search using japanese text fails with the dreaded
869 >
870 > > incompatible character encodings error:
871 > I'm having trouble reproducing this, or even understanding why your fix
872 > would help, since all string literals in the code should be UTF-8-encoded.
873 >
874 > Could you please apply this patch and tell me what the output is when
875 > you feed it a crashing search term? Thanks!
876 >
877 > --- cut here ---
878 > diff --git a/bin/heliotrope-server b/bin/heliotrope-server
879 > index c9754d4..ca764c0 100644
880 > --- a/bin/heliotrope-server
881 > +++ b/bin/heliotrope-server
882 > @@ -219,6 +219,19 @@ class HeliotropeServer < Sinatra::Base
883 > end
884 > nav += "</div>"
885 >
886 > + puts "start"
887 > + p query.original_query_s.encoding
888 > + p query.parsed_query_s.encoding
889 > + p header("Search: #{query.original_query_s}",
890 > query.original_query_s).enc + p "<div>Parsed query: #{escape_html
891 > query.parsed_query_s}</div>".encoding + p "<div>Search took #{sprintf
892 > '%.2f', info[:elapsed]}s and #{info[:contin + p
893 > "#{nav}<table>".encoding
894 > + p results.size
895 > + p results.map { |r| threadinfo_to_html r }.join.encoding
896 > + p "</table>#{nav}".encoding
897 > + p footer.encoding
898 > + puts "end"
899 > +
900 > header("Search: #{query.original_query_s}", query.original_query_s)
901 > + "<div>Parsed query: #{escape_html query.parsed_query_s}</div>" +
902 > "<div>Search took #{sprintf '%.2f', info[:elapsed]}s and #{info[:contin
903 > --- cut here ---
904
905 Seems the problem is not heliotrope. The problem are my hooks that use MeCab
906 to split Japanese words.
907
908 If I run a search for japanese using my query hook this is the output:
909
910 search(body:"???", 0, 20) took 0.1ms
911 start
912 #<Encoding:ASCII-8BIT>
913 #<Encoding:UTF-8>
914 #<Encoding:ASCII-8BIT>
915 #<Encoding:UTF-8>
916 "<div>Search took 0.00s and was NOT continued</div>"
917 #<Encoding:UTF-8>
918 0
919 #<Encoding:ASCII-8BIT>
920 #<Encoding:UTF-8>
921 #<Encoding:UTF-8>
922 end
923
924
925 If I put a force_encoding at the end of the hook I get:
926
927 start
928 #<Encoding:UTF-8>
929 #<Encoding:UTF-8>
930 #<Encoding:UTF-8>
931 #<Encoding:UTF-8>
932 "<div>Search took 0.00s and was NOT continued</div>"
933 #<Encoding:UTF-8>
934 20
935 #<Encoding:UTF-8>
936 #<Encoding:UTF-8>
937 #<Encoding:UTF-8>
938 end
939
940 I need to re-index my emails with the new UTF-8 hooks and test the search
941 again.
942
943
944 --
945 regards,
946 Horacio Sanson
947
948 From wmorgan-sup@masanjin.net Fri Jul 15 00:07:28 2011
949 From: wmorgan-sup@masanjin.net (William Morgan)
950 Date: Fri, 15 Jul 2011 04:07:28 +0000
951 Subject: [sup-devel] Heliotrope improving but still found some issues
952 In-Reply-To: <201107100850.34478.hsanson@gmail.com>
953 References: <201107051052.53166.hsanson@gmail.com>
954 <1310250042-sup-5348@masanjin.net>
955 <201107100850.34478.hsanson@gmail.com>
956 Message-ID: <1310702820-sup-727@masanjin.net>
957
958 Reformatted excerpts from Horacio Sanson's message of 2011-07-09:
959 > I need to re-index my emails with the new UTF-8 hooks and test the
960 > search again.
961
962 Can you try with the latest master? You may not need to reindex.
963 --
964 William <wmorgan-sup at masanjin.net>
965
966 From dmishd@gmail.com Sun Jul 17 13:42:11 2011
967 From: dmishd@gmail.com (Hamish D)
968 Date: Sun, 17 Jul 2011 18:42:11 +0100
969 Subject: [sup-devel] error after leveldb-ruby upgrade
970 Message-ID: <CAOxvSbddWV33kzmS0OPCg-jvcKjzwu1JyGFjqL4Puaxiv-a=tQ@mail.gmail.com>
971
972 Hello
973
974 I've just done a git pull, and noticed there was a bump in version of
975 leveldb-ruby, so did a gem update to 0.7.
976
977 Then when trying to run heliotrope-add or heliotrope-server, it
978 immediately dies with:
979
980
981 $ heliotrope-server -d ~/.heliotrope
982 /var/lib/gems/1.9.1/gems/leveldb-ruby-0.7/lib/leveldb.rb:11:in `make':
983 Corruption: checksum mismatch (LevelDB::Error)
984 from /var/lib/gems/1.9.1/gems/leveldb-ruby-0.7/lib/leveldb.rb:11:in `new'
985 from /home/mish/dev/sup/heliotrope/lib/heliotrope/index.rb:45:in `initialize'
986 from /home/mish/dev/sup/heliotrope/bin/heliotrope-server:583:in `new'
987 from /home/mish/dev/sup/heliotrope/bin/heliotrope-server:583:in `<main>'
988
989
990 Any ideas?
991 Hamish
992
993 From wmorgan-sup@masanjin.net Sun Jul 17 16:05:27 2011
994 From: wmorgan-sup@masanjin.net (William Morgan)
995 Date: Sun, 17 Jul 2011 20:05:27 +0000
996 Subject: [sup-devel] error after leveldb-ruby upgrade
997 In-Reply-To: <CAOxvSbddWV33kzmS0OPCg-jvcKjzwu1JyGFjqL4Puaxiv-a=tQ@mail.gmail.com>
998 References: <CAOxvSbddWV33kzmS0OPCg-jvcKjzwu1JyGFjqL4Puaxiv-a=tQ@mail.gmail.com>
999 Message-ID: <1310933008-sup-9080@masanjin.net>
1000
1001 Reformatted excerpts from Hamish D's message of 2011-07-17:
1002 > $ heliotrope-server -d ~/.heliotrope
1003 > /var/lib/gems/1.9.1/gems/leveldb-ruby-0.7/lib/leveldb.rb:11:in `make':
1004 > Corruption: checksum mismatch (LevelDB::Error)
1005
1006 That error gets thrown when LevelDB detects things are corrupted, but
1007 I'd be surprised if the upgrade to 0.7 caused this--the only change was
1008 one in how memory in the Ruby world is free'd. None of the actual
1009 LevelDB code changed.
1010
1011 If you downgrade to 0.6, do you still get that error? Are you running
1012 out of disk space, or anything like that?
1013 --
1014 William <wmorgan-sup at masanjin.net>
1015
1016 From dmishd@gmail.com Sun Jul 17 18:51:55 2011
1017 From: dmishd@gmail.com (Hamish D)
1018 Date: Sun, 17 Jul 2011 23:51:55 +0100
1019 Subject: [sup-devel] error after leveldb-ruby upgrade
1020 In-Reply-To: <1310933008-sup-9080@masanjin.net>
1021 References: <CAOxvSbddWV33kzmS0OPCg-jvcKjzwu1JyGFjqL4Puaxiv-a=tQ@mail.gmail.com>
1022 <1310933008-sup-9080@masanjin.net>
1023 Message-ID: <CAOxvSbd+BmyYST+fvNiY9CHPc8_sYEkb_6ugDRWv875imhCfgw@mail.gmail.com>
1024
1025 On 17 July 2011 21:05, William Morgan <wmorgan-sup at masanjin.net> wrote:
1026 > Reformatted excerpts from Hamish D's message of 2011-07-17:
1027 >> $ heliotrope-server -d ~/.heliotrope
1028 >> /var/lib/gems/1.9.1/gems/leveldb-ruby-0.7/lib/leveldb.rb:11:in `make':
1029 >> Corruption: checksum mismatch (LevelDB::Error)
1030 >
1031 > If you downgrade to 0.6, do you still get that error? Are you running
1032 > out of disk space, or anything like that?
1033
1034 I still get that error with 0.6 (and 0.5) so I guess there must be
1035 corruption :/ I did upgrade straight from 0.5 to 0.7 o the off chance
1036 that made any difference, but I imagine not. On disk space I have
1037 multiple GB to spare, so I don't think that would be it.
1038
1039 Is there any way to recover from the error? Or do I just have to
1040 delete it all and start again?
1041
1042 Hamish
1043
1044 From wmorgan-sup@masanjin.net Mon Jul 18 00:21:18 2011
1045 From: wmorgan-sup@masanjin.net (William Morgan)
1046 Date: Mon, 18 Jul 2011 04:21:18 +0000
1047 Subject: [sup-devel] error after leveldb-ruby upgrade
1048 In-Reply-To: <CAOxvSbd+BmyYST+fvNiY9CHPc8_sYEkb_6ugDRWv875imhCfgw@mail.gmail.com>
1049 References: <CAOxvSbddWV33kzmS0OPCg-jvcKjzwu1JyGFjqL4Puaxiv-a=tQ@mail.gmail.com>
1050 <1310933008-sup-9080@masanjin.net>
1051 <CAOxvSbd+BmyYST+fvNiY9CHPc8_sYEkb_6ugDRWv875imhCfgw@mail.gmail.com>
1052 Message-ID: <1310962828-sup-2572@masanjin.net>
1053
1054 Reformatted excerpts from Hamish D's message of 2011-07-17:
1055 > Is there any way to recover from the error? Or do I just have to
1056 > delete it all and start again?
1057
1058 LevelDB provides a recovery function. I'm not sure how well it works.
1059 Give me a day or two to export it with the ruby bindings, and you can
1060 give it a try.
1061 --
1062 William <wmorgan-sup at masanjin.net>
1063