hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4044) Create symbolic links in HDFS
Date Fri, 10 Oct 2008 17:53:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638608#action_12638608

Doug Cutting commented on HADOOP-4044:

Perhaps we're prematurely optimizing here.  Perhaps the extra RPCs to resolve paths won't
hurt things that much.  The namenode benchmarks I've seen show that it handles 50k opens/second.
 How much slower would that be if we always first resolved the path.  In this naive approach
we'd add a method which must always be called before a file is opened (or renamed, etc.) that
would return an 'isResolved' boolean and a path.  If the path is resolved then it can be used,
otherwise we must resolve it further.  The resolution loop could then be independent of the
FileSystem method, so no ThreadLocal or other mechanism would be required to maintain this

public class ResolvedPath extends Path {}

public class LinkResolution {
  public isResolved() { ...}
  public Path getUnresolvedPath();
  public ResolvedPath getResolvedPath();

public class FileSystem {
  public abstract LinkResolution resolveLinks(Path p);
  public abstract FSInputStream open(ResolvedPath p);

  private ResolvedPath getResolvedPath(Path p) throws IOException {
    LinkResolution resolution = resolveLinks(p);
    int count = 0;
    while (!resolution.isResolved()) && count < MAX_LINKS) {
      p = resolution.getUnresolvedPath();
      resolution p.getFileSystem(conf).resolveLinks(p);
    return resolution.getResolvedPath();

  public FSInputStream open(Path p, ...) throws IOException {
    return open(getResolvedPath(p), ...);
No exceptions, no thread locals, no anonymous classes, etc.  Should we benchmark this before
we rule it out?

> Create symbolic links in HDFS
> -----------------------------
>                 Key: HADOOP-4044
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4044
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: symLink1.patch, symLink1.patch, symLink4.patch, symLink5.patch,
symLink6.patch, symLink8.patch, symLink9.patch
> HDFS should support symbolic links. A symbolic link is a special type of file that contains
a reference to another file or directory in the form of an absolute or relative path and that
affects pathname resolution. Programs which read or write to files named by a symbolic link
will behave as if operating directly on the target file. However, archiving utilities can
handle symbolic links specially and manipulate them directly.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message