hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject symlink support in Hadoop 2 GA
Date Tue, 17 Sep 2013 01:49:47 GMT
Hi all,

I wanted to broadcast plans for putting the FileSystem symlinks work
(HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I think
it's pretty important we get it in since it's not a compatible change; if
it misses the GA train, we're not going to have symlinks until the next
major release.

However, we're still dealing with ongoing issues revealed via testing.
There's user-code out there that only handles files and directories and
will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912
for a nice example where globStatus returning symlinks broke Pig; some of
us had a conference call to talk it through, and one definite conclusion
was that this wasn't solvable in a generally compatible manner.

There are also still some gaps in symlink support right now. For example,
the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink
resolution, and tooling like the FsShell and Distcp still need to be
updated as well.

So, there's definitely work to be done, but there are a lot of users
interested in the feature, and symlinks really should be in GA. Would
appreciate any thoughts/input on the matter.

Thanks,
Andrew

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message