Subject: Re: [sup-devel] [issue117] Problem with "special" characters (space etc.) in source uri (filesystem path).

From: Marc Hartstein <[email-masked]>
Date: Thu, 19 Aug 2010 13:44:49 -0400

Excerpts from anonymous's message of Thu Aug 19 03:58:42 -0400 2010:
> 
>     uri=URI.encode(uri) ### <--- Added this line in order to encode "special"
> characters within the uri.
> 
> This makes sup-add work, it encodes the non-standard characters (like
> "%20" for " "). They appear as such in sources.yaml. But this breaks
> another part: sup-sync now looks for files containing "%20" instead of
> " " on the disk, which obviously don't exist:

If you're going to URI encode the source URIs (makes sense if they're
truly URIs), you're going to have to URI decode them before use.

I'm not sure if this is the best approach, but you should be able to
turn what you've done into a working patch by finding the places where
sup uses the URIs and using URI.decode(uri) to turn them back into
normal strings at the right moment.
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel

---

From: anonymous <[email-masked]>
Date: Thu, 19 Aug 2010 07:58:42 +0000

New submission from anonymous:

Hey there,

sup has problems with "non-standard", but allowed, characters in the source uri.
My example: I have maildir storages I want to add, and they contain spaces
(which are allowed characters on the filesystem). Here's the error message:

===>
$ sup-add 'maildir:~/maildir-Mail/Inbox/Dresden-Initiativen (Tauschnetz,
Evoluzzer, Vernetzungstreffen ...)'
[Thu Aug 19 09:47:57 +0200 2010] Flushing Xapian updates to disk. This may take
a while...
/usr/lib/ruby/1.8/uri/common.rb:436:in `split': bad URI(is not URI?):
maildir:~/maildir-Mail/Inbox/Dresden-Initiativen (Tauschnetz, Evoluzzer,
Vernetzungstreffen ...) (URI::InvalidURIError)
        from /usr/lib/ruby/1.8/uri/common.rb:485:in `parse'
        from /usr/lib/ruby/1.8/uri/common.rb:608:in `URI'
        from /usr/bin/sup-add:94
        from /usr/bin/sup-add:86:in `each'
        from /usr/bin/sup-add:86
<===

I can circumvent that problem by adding the line marked in the following code
with a comment into /usr/bin/sup-add (it was inserted after line 86 in my
version of the code):

===>
begin
  Redwood::SourceManager.load_sources

  ARGV.each do |uri|
    uri=URI.encode(uri) ### <--- Added this line in order to encode "special"
characters within the uri.
    labels = $opts[:labels] ? $opts[:labels].split(/\s*,\s*/).uniq : []

    if !$opts[:force_new] && Redwood::SourceManager.source_for(uri)
      say "Already know about #{uri}; skipping."
      next
    end
<===
(I don't know anything about ruby; I got this idea from a similar problem with
another program by searching the web.)

This makes sup-add work, it encodes the non-standard characters (like "%20" for
" "). They appear as such in sources.yaml. But this breaks another part:
sup-sync now looks for files containing "%20" instead of " " on the disk, which
obviously don't exist:

===>
$ sup-add 'maildir:~/maildir-Mail/Inbox/Dresden-Initiativen (Tauschnetz,
Evoluzzer, Vernetzungstreffen ...)'
Adding
maildir:~/maildir-Mail/Inbox/Dresden-Initiativen%20(Tauschnetz,%20Evoluzzer,%20Vernetzungstreffen%20...)...
[Thu Aug 19 09:45:29 +0200 2010] Flushing Xapian updates to disk. This may take
a while...

# Works.

$ cat ~/.sup/sources.yaml
---
- !masanjin.net,2006-10-01/Redwood/Maildir
  uri:
maildir:~/maildir-Mail/Inbox/Dresden-Initiativen%20(Tauschnetz,%20Evoluzzer,%20Vernetzungstreffen%20...)
  cur_offset:
  usual: true
  archived: false
  id: 1
  labels: []

  mtimes:
    cur: 1970-01-01 01:00:00 +01:00
    new: 1970-01-01 01:00:00 +01:00

# "Special" characters are Encoded.

$ sup-sync
Scanning
maildir:~/maildir-Mail/Inbox/Dresden-Initiativen%20(Tauschnetz,%20Evoluzzer,%20Vernetzungstreffen%20...)...
[Thu Aug 19 09:45:59 +0200 2010] WARNING: problem getting messages from
maildir:~/maildir-Mail/Inbox/Dresden-Initiativen%20(Tauschnetz,%20Evoluzzer,%20Vernetzungstreffen%20...):
/home/felics/maildir-Mail/Inbox/Dresden-Initiativen%20(Tauschnetz,%20Evoluzzer,%20Vernetzungstreffen%20...)/cur
not a directory
Scanned 0, added 0, updated 0 messages from
maildir:~/maildir-Mail/Inbox/Dresden-Initiativen%20(Tauschnetz,%20Evoluzzer,%20Vernetzungstreffen%20...).
[Thu Aug 19 09:45:59 +0200 2010] Flushing Xapian updates to disk. This may take
a while...

# That directory does not exist. How should it, it has " " within it's name, not
"%20".
<===

Next try: Exchanging back in sources.yaml all "%20" to " ". But that breaks
again sup-sync (the "URI::InvalidURIError" as above). This I was unable to fix,
since I did not find a point where a simple insertion of a URI.encode statement
could help.

This problems are also there for paths containing only "normal" characters and
spaces, i.e. no parentheses. And there are also other special characters that
make problems: '<', '>', '"'. Probably umlauts.

Clean solution would be to make sup safe for any character (in any encoding)
that is allowed to appear within the sources.

For me, that's an KO-argument not to use sup, since I still need that directory
structure to represent some email organisation on filesystem level, and I will
not rename every single directory just for sup (it's part of a bigger email setup).

Bye,
Felix.

----------
keyword: encoding, maildir
messages: 271
nosy: anonymous
priority: bug
ruby_version: 1.8.7 (2008-08-11 patchlevel 72) [i486-linux] (Debian)
status: unread
sup_version: 0.11-2 (Debian)
title: Problem with "special" characters (space etc.) in source uri (filesystem path).

_________________________________________
Sup issue tracker <[email-masked]>
<http://masanjin.net/sup-bugs/issue117>
_________________________________________
_______________________________________________
Sup-devel mailing list
[email-masked]
http://rubyforge.org/mailman/listinfo/sup-devel