Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates
 209.85.217.13 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding:sender;
        b=wQR/dWYYwTEMmtcy6biI2r1PcMGdUCQWXvyjnbdQPN6+UNVHOFkarzkobRSmjAUHyH
         0uJF1eb6vpHFfTp3nrOznUF4iGhLfNTndd6QBPQpyworTE4UPEEhS6gGSPBPCgfuplKF
         5AsP/H4e8DHWzjIq7ZKBO8py9uNd3WZ0MKL80=
Message-ID: <48EF9091.7000603@apache.org>
Date: Fri, 10 Oct 2008 10:27:45 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.17 (X11/20080925)
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Re: Hadoop chokes on file names with ":" in them
References: <5F3F3CEE-2BA7-46A6-B686-95531D45E467@cse.unl.edu>
In-Reply-To: <5F3F3CEE-2BA7-46A6-B686-95531D45E467@cse.unl.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Doug Cutting <cutting@gmail.com>

The safest thing is to restrict your Hadoop file names to a 
common-denominator set of characters that are well supported by Unix, 
Windows, and URIs.  Colon is a special character on both Windows and in 
URIs.  Quoting is in theory possible, but it's hard to get it right 
everywhere in practice.  One can devise heuristics that determine 
whether a colon is intende to be part of a name in a relative path 
rather than indicating a URI scheme or a Windows device, but making sure 
that all components observe that heuristic (Java's URI handler, Windows 
FS, etc.) is impossible and this leads to inconsistent behavior.  HDFS 
prohibits colons in filenames for this reason.

Doug

Brian Bockelman wrote:
> Hey all,
> 
> Hadoop tries to parse file names with ":" in them as a relative URL:
> 
> [brian@red ~]$ hadoop fs -put /tmp/test 
> /user/brian/StageOutTest-24328-Fri-Oct-10-07:58:44-2008
> put: Pathname /user/brian/StageOutTest-24328-Fri-Oct-10-07:58:44-2008 
> from /user/brian/StageOutTest-24328-Fri-Oct-10-07:58:44-2008 is not a 
> valid DFS filename.
> Usage: java FsShell [-put <localsrc> ... <dst>]
> 
> Our users do timestamps like that *a lot*.  It appears that Hadoop tries 
> to interpret the ":" as a sign that you are trying to use a relative URL.
> 
> Is there any reason to not support the ":" character in file names?
> 
> Brian