hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "FKorning (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8274) In pseudo or cluster model under Cygwin, tasktracker can not create a new job because of symlink problem.
Date Sun, 29 Apr 2012 16:03:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264577#comment-13264577
] 

FKorning commented on HADOOP-8274:
----------------------------------

If the symlinks really are necessary, here's a java.io.File wrapper
that understands cygwin symlinks (and windows shortcuts).

{code}
package net.muxbus.io;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.net.URI;


/**
 * 
 * @author      Francis Korning   fkorning@yahoo.ca
 * @date        2012-04-30    
 * 
 * @project     muxbus            muxbus.net
 * @source      sourceforge       https://sourceforge.net/p/muxbus/ 
 *     
 * @copyright   2012              F.Korning
 * @license     LGPL              http://www.gnu.org/copyleft/lesser.html
 * 
 * @credit      Jesse Hager       jessehager@iname.com    (.lnk shortcut file format).
 * 
 * 
 * LinkedFile is a java.io.File wrapper that understands both Cygwin Symbolic Links
 * and Windows Explorer Shorcut Links (.lnk).  If a given File is a Link, it tries
 * to resolve the File by following link paths recursively until a final canonical
 * target is found, and if the file exists will act as a wrapper for the link
 * target. A LinkedFile always tracks its source and provides link reflection.
 *
 * A common frustration for POSIX platform developers is that Java on windows does
 * not understand symbolic Links which aren't handled by the windows io native libs.
 * This means that Cygwin or Interix (aka Winterix or Windows Service for Unix)
 * integrated java applications that depend on POSIX tools may break when it comes
 * to resolution of paths with linked files and directories.
 * 
 * The first workaround strategy, which works where links are predictably created,
 * is to ensure that cygwin paths and windows paths are equivalent and identical.
 * For example, you create a circular "/cygwin" link that maps back to the root "/":
 * 
 * <code>
 * <pre>                
 *     /cygwin      ->  /      ->  \cygwin\      ->  C:\cygwin\
 *     /cygwin/tmp  ->  /tmp/  ->  \cygwin\tmp\  ->  C:\cygwin\tmp\
 * </pre>
 * </code>
 * 
 * But you often have no control over much of the integrated codebase and tools.
 * If some of the components create symlinks dynamically where java expects real
 * files and directories this workaround will fail.
 * 
 * Now the only OS-native IO alternative would be to use Windows NTFS Junctions.
 * These are completely transparent and work in shells, in Explorer, and for java.
 * The best implementation would be the junction provided in the SysInternals suite.
 * You could wrap the cygwin 'ln' command and force it to use a junction instead,
 * but this breaks some POSIX compatibility and requires Administrator superuser
 * rights which you may not want to assign to all your running service daemons.
 * 
 * Next Interix symbolic links and the mklink command are built-in since Windows Vista.
 * These fall in between as they act like native symlinks in the shell and in java,
 * look like cygwin links inside the shell, but again they can only be created by the
 * Administrator, and also break POSIX. Unlike cygwin they do appear as shortcuts in
 * a Windows Explorer, with the caveat that these are broken because of permissions.
 * 
 * Note that both strategies only work if the tools create links via the 'ln' command.
 * But what if the offending integrated tools are in perl or python, or even better yet in
 * compiled c/c++ code making use of libc calls for which we have no source nor license?
 * This means that we will need to adopt a strategy to adapt java.io.File to Cygwin libc.
 * Also, most multi-platform java developers favour Cygwin instead of Microsoft's Interix.
 * All of the POSIX daemons we're looking at expect cygwin shells, and this compatibility
 * is our main focus here; we ideally an a model that works without having to wrap 'ln'.
 * 
 * Now Both Cygwin Symlinks and Explorer Shortcuts are just plain binary files underneath
 * with some information pointing to their link targets.  They have the advantage that
 * we can parse these and thus this allows us to add some link reflection to java.io.
 * (This also creates a strange case where windows would have more functionality than
 * unix, unless one would write a unix java native libc wrapper for link functions).
 * They also require no special Administrator rights for creation and maintenance.
 * 
 * A side-effect is that shortcuts created under Windows Explorer have the feature that 
 * a (.lnk) extension is appended, which is obscured from the explorer.  As the names
 * differ, both a Cygwin Symlink and a Windows Shortcut can co-exist in the same place
 * and make for a consistent interface both in the cygwin shell and Windows Explorer.
 *
 * This means that one could well mirror every cygwin symlink with a windows shortcut,
 * and thus have symbolic links that work even under non-Administrator user shells and
 * are mirrored by corresponding Windows Shortcuts that work in the Windows Explorer.
 * Cygwin also comes with its own programmatic 'mkshortcut' command to create Shorcuts
 * from a shell (which you can't under windows by default - doesn't Windows suck?).
 * 
 * <code>
 * <pre>
 *     dir       ->         /dir/            symlink
 *     dir.lnk   ->         /dir/            shortcut
 *     file      ->         /dir/file        symlink
 *     file.lnk  ->         /dir/file        shortcut
 * </pre>
 * </code>
 * 
 * Note that synchronizing of Cygwin symlinks and Windows Shortcuts is left out-of-scope.
 * Our primary concern is to get our POSIX daemons running, not to add sugar to Explorer.
 * However LinkedFile will support and resolve links for both link formats, with first
 * preferrence given to Cygwin Symlinks if found, and then to Windows Shortcuts of the
 * same name (without the .lnk extension), which you may extend for such a facility.
 * 
 * For those wishing to extend the functionality to support transparent mirroring between
 * Explorer and the shell, note that you will have to map links to their (.lnk) shortcuts.
 * Also note that Windows Shortcuts are never interpreted as links by the Cygwin shell.
 * Most critically note that Windows shortcuts are unpredictable in the shell if moved,
 * as they seem to only recalculate relative paths after a link is accessed in Explorer. 
 *
 * 
 * @see cygwin:             http://www.cygwin.com
 * @see cygwin  symlinks:   http://cygwin.com/ml/cygwin/2010-11/msg00394.html 
 * @see interix links:      http://technet.microsoft.com/en-us/library/bb463204.aspx
 * @see windows shortcuts:  http://ithreats.files.wordpress.com/2009/05/lnk_the_windows_shortcut_file_format.pdf
 * @see windows junctions:  http://technet.microsoft.com/en-us/sysinternals
 * 
 */
public class LinkedFile
    extends java.io.File
{

    
    public static final String TOK_SYMLINK = "!<symlink>\u00FF\u00FE";
    
    public static final String TOK_SHORTCUT = "\u004C\u0000\u0000\u0000\u0001\u0014\u0002";
    public static final String TOK_SHORTCUT_SEP = "\u0000";
    
    public static final int    SHORTCUT_MAX = 1024;    // shortcuts should probably never
be that big
    public static final int    SHORTCUT_OFFSET = 76;   // first shortcut section length offset
at 0x4C    


    
    // utilities
    
    
    /**
     * read a link header.
     * 
     * both cygwin symlinks and windows shortcuts are plain binary files.
     * cygwin is faily simple, consisting of magic header and the target.
     * shortcuts have variable length structures but will not exceed 1kb.
     */
    public static String header (File file)
        throws IOException
    {
        if (file == null)
            return null;
        
        String header = null;
        
        if (file.exists() && file.canRead())
        {
            if (file.isFile() || (!file.isDirectory()))
            {                
                char[] buf = new char [SHORTCUT_MAX];
                int length = new BufferedReader(new FileReader(file)).read(buf, 0, SHORTCUT_MAX);
                header = new String(buf);

                //System.err.println ("length: " + length);
                //System.err.println ("header: " + header);
                
                if (isSymlinkHeader(header))
                {
                    //System.err.println("isSymlink: true");                    
                }
                else if (isShortcutHeader(header))
                {
                    //System.err.println("isShortcut: true");
                }
                else
                {
                    header = null;
                }
                
            }
        }
        
        return header;
    }

    
    public static boolean isLinkHeader (String header)
        throws IOException
    {
        return ( isSymlinkHeader(header) || isShortcutHeader(header) );
    }
    
    
    public static boolean isSymlinkHeader (String header)
        throws IOException
    {
        if (header == null)
            return false;
        
        if (header.startsWith(TOK_SYMLINK))
        {
            return true;
        }
        
        return false;
    }

    
    public static boolean isShortcutHeader (String header)
        throws IOException
    {
        if (header == null)
            return false;
        
        if (header.startsWith(TOK_SHORTCUT))
        {
            return true;
        }
        
        return false;
    }


    /**
     * parse a 2-char short length offset from a position in a windows shortcut file.
     */
    public static int shortcutHeaderOffset (String header, int pos)
    {
        if (header == null)
            return 0;

        char small = header.charAt(pos);
        char large = header.charAt(pos +1);
        int offset = small + (large * 16 * 16);
        
        //System.err.println ("offset [" + pos + "] '" + (short)small + " " + (short)large
+ "' : " + offset);        
        return offset;
    }
    
    
    /**
     * parse the final link target from a cygwin symlink or windows shortcut.
     * 
     * cygwin symlink targets are immediately after the header "<!symlink> 0xFF xFE".
     * 
     * windows shortcuts start with the header "L 0x00 0x00 0x00 0x01 0x14 0x02".
     * the windows shortcut files are organized into sections of variable lengths,
     * starting after the header at 0x4c (76). the lengths are 2-char short offsets.
     * windows shortcut final name is at the end of the second "file location" section.
     */
    public static String headerTarget (String header)
    {
        if (header == null)
            return null;

        String target = null;

        try
        {
            if (isSymlinkHeader(header))
            {
                StringBuffer buf = new StringBuffer("");
                for (int i = TOK_SYMLINK.length();  i < header.length(); i+=2)
                {
                    buf.append (header.charAt(i));
                }
                
                target = buf.toString();                
            }
            else if (isShortcutHeader(header))
            {
                // first get the length of the next "item id" section 
                int a = shortcutHeaderOffset(header, SHORTCUT_OFFSET);
                // now get the length of the next "file location" section
                int b = shortcutHeaderOffset(header, SHORTCUT_OFFSET + a  + 2);
                
                // the final name is the last field of the "file location" section 
                int j = SHORTCUT_OFFSET + a + b;                
                int i = header.lastIndexOf(TOK_SHORTCUT_SEP, j - 2) + 1;
                
                target = header.substring (i,j);                
            }
        }
        catch (IOException e) {}

        //System.err.println ("target: " + target);
        return target;
    }
    
    
    
    public static boolean isLink (File file)
    {
        if (file == null)
            return false;
        
        try
        {
            if (file.exists() && file.canRead())
            {
                if (file.isFile() || (!file.isDirectory()))
                {
                    String header = header(file);
                    return isLinkHeader(header);
                }
            }
        }
        catch (IOException e) {}
        
        return false;
    }

    
    public static boolean isSymlink (File file)
    {
        if (file == null)
            return false;
        
        try
        {
            if (file.exists() && file.canRead())
            {
                if (file.isFile() || (!file.isDirectory()))
                {
                    String header = header(file);
                    return isSymlinkHeader(header);
                }
            }
        }
        catch (IOException e) {}
        
        return false;
    }

    
    public static boolean isShortcut (File file)
    {
        if (file == null)
            return false;
        
        try {
            if (file.exists() && file.canRead())
            {
                if (file.isFile() || (!file.isDirectory()))
                {
                    String header = header(file);
                    return isShortcutHeader(header);
                }
            }
        }
        catch (IOException e) {}
        
        return false;
    }
    
    
    /**
     * Resolve a target by recursively traversing links until the canonical target.
     * If the file is not a link, return the file's canonical AbsolutePath.
     */
    public static String resolveTarget (File file)
    {
        String target = file.getAbsolutePath();
        //System.err.println("\path: " + target);
        
        try
        {
            if (file.exists() && file.canRead())
            {
                if (file.isFile() || (!file.isDirectory()))
                {
                    String header = header(file);
                    if (isLinkHeader(header))
                    {
                        target = headerTarget(header);
                        boolean absolute = (target.substring(1)).startsWith(":");
                        boolean logical = (target.startsWith("/") || target.startsWith("\\"));
                        boolean parent = (target.startsWith("../") || target.startsWith("..\\"));
                       
                        boolean current = (target.startsWith("./") || target.startsWith(".\\"));
                        
                        File link = null;
                        if (absolute)
                            link = new File (target.substring(0,2), target.substring(2));
                        else if (logical)
                            link = new File(target.substring(0,1), target.substring(1));
                        else if (parent)
                            link = new File(target.substring(0,2), target.substring(2)); 
                      
                        else if (current)
                            link = new File(target.substring(0,1), target.substring(1));
                        else
                            link = new File (target);
                                                
                        if (! file.equals(link))
                        {
                            target = resolveTarget(link);
                        }
                    }
                }
            }
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
        
        //System.err.println("resolve: " + target);
        return target;
    }

    
    /**
     * 
     * Resolve a file by recursively traversing links until the canonical target.
     * If the file is not a link, return the file itself.

     * @param file
     * @return
     */
    public static File resolveFile (File file)
    {
        //System.err.println("file: " + file.getAbsolutePath());
        
        try
        {
            if (file.exists() && file.canRead())
            {
                if (file.isFile() || (!file.isDirectory()))
                {
                    String header = header(file);
                    if (isLinkHeader(header))
                    {
                        String target = headerTarget(header);
                        boolean absolute = (target.substring(1)).startsWith(":");
                        boolean logical = (target.startsWith("/") || target.startsWith("\\"));
                        boolean parent = (target.startsWith("../") || target.startsWith("..\\"));
                       
                        boolean current = (target.startsWith("./") || target.startsWith(".\\"));
                        
                        File link = null;
                        if (absolute)
                            link = new File (target.substring(0,2), target.substring(2));
                        else if (logical)
                            link = new File(target.substring(0,1), target.substring(1));
                        else if (parent)
                            link = new File(target.substring(0,2), target.substring(2)); 
                      
                        else if (current)
                            link = new File(target.substring(0,1), target.substring(1));
                        else
                            link = new File (target);
                        
                        if (! file.equals(link))
                        {    
                            file = resolveFile(link.getAbsoluteFile());
                        }
                    }
                }
            }
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }

        //System.err.println("resolve: " + file.getAbsolutePath());
        return file;
    }
    
    
    // commmands

    
    /**
     * LinkedFile command resolves a link and prints its final link target name.
     * 
     * @param args
     * @throws IOException
     */
    public static void main (String[] args)
        throws IOException
    {
        try
        {
            if (args.length == 0)
            {
                System.err.println ("usage: LinkedFile <filepath>");
                System.exit (1);
            }
            
            LinkedFile file = new LinkedFile(args[0]);
            System.out.println(file);
        }
        catch (Exception e)
        {
            e.printStackTrace();
            System.exit (1);
        }
        
    }


    
    
    // fields
    
    
    protected String source;        // the file source
    protected String target;        // the resolved target path
    
    protected String header;        // the link header, if source is a link
    protected boolean symlink;      // file is a cygwin symlink
    protected boolean shortcut;     // file is a windows shortcut

    protected File link;            // the original source file link

    
    
    

    // instantiators 

    
    // constructors
    
    /**
     * Creates a new File instance by converting the given pathname string into
     * an abstract pathname.
     */
    public LinkedFile(String pathname)
    {
        super((resolveFile(new File(pathname))).getAbsolutePath());
        
        this.target = getAbsolutePath();
        this.source = "" + pathname;
        
        this.link = new File(pathname);
        this.symlink = isSymlink(link);
        this.shortcut = isShortcut(link);
    }

    
    /**
     * Creates a new File instance from a parent abstract pathname and a child
     * pathname string.
     */
    public LinkedFile(File parent, String child)
    {
        super((resolveFile(new File(parent, child))).getAbsolutePath());
        
        this.target = getAbsolutePath();
        this.source = "" + parent.getName() + "/" + child;
        
        this.link = new File(parent, child);
        this.symlink = isSymlink(link);
        this.shortcut = isShortcut(link);
    }

    
    /**
     * Creates a new File instance from a parent pathname string and a child
     * pathname string.
     */
    public LinkedFile(String parent, String child)
    {
        super((resolveFile(new File(parent, child))).getAbsolutePath());
        
        this.target = getAbsolutePath();
        this.source = "" + parent + "/" + child;
        
        this.link = new File(parent, child);
        this.symlink = isSymlink(link);
        this.shortcut = isShortcut(link);
    }

    
    /**
     * Creates a new File instance by converting the given file: URI into an
     * abstract pathname.
     */
    public LinkedFile(URI uri)
    {
        super((resolveFile(new File(uri))).getAbsolutePath());
        
        this.target = getAbsolutePath();
        this.source = "" + uri;
        
        this.link = new File(uri);
        this.symlink = isSymlink(link);
        this.shortcut = isShortcut(link);
    }

    
    // accessors


    public boolean isLink()
    {
        return (symlink || shortcut);
    }

    
    public boolean isSymlink()
    {
        return (symlink);
    }


    public boolean isShortcut()
    {
        return (shortcut);
    }

    
    public String getSource()
    {
        return source;
    }

    public String getTarget()
    {
        return target;
    }
    
    
    
    // overrides
    
    
    public String getName()
    {
        return super.getName().replace('\\', '/');
    }

    public String getPath()
    {
        return super.getPath().replace('\\', '/');
    }
    
    public String getAbsolutePath()
    {
        return super.getAbsolutePath().replace('\\', '/');
    }
    
    public String getCanonicalPath()
        throws IOException
    {
        return super.getCanonicalPath().replace('\\', '/');
    }

    
    public String toString()
    {

//        System.err.println("");
//        System.err.println("file: " + getAbsolutePath());
//        System.err.println("path: " + getPath());        
//        System.err.println("name: " + getName());
//                
//        System.err.println("source: " + getSource());
//        System.err.println("target: " + getTarget());
//
//        System.err.println("exists: " + exists());        
//        System.err.println("canRead: " + canRead());
//        System.err.println("canWrite: " + canWrite());
//        System.err.println("canExecute: " + canExecute());
//
//        System.err.println("isDirectory: " + isDirectory());
//        System.err.println("isSymlink: " + isSymlink());
//        System.err.println("isShortcut: " + isShortcut());
        

        StringBuffer s = new StringBuffer("");

        if (isLink())
            s.append(getSource() + " -> ");

        s.append(getTarget());

        if (isDirectory())
            s.append("/");
        
        return s.toString();
    }

}
{code}
                
> In pseudo or cluster model under Cygwin, tasktracker can not create a new job because
of symlink problem.
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-8274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8274
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.205.0, 1.0.0, 1.0.1, 0.22.0
>         Environment: windows7+cygwin 1.7.11-1+jdk1.6.0_31+hadoop 1.0.0
>            Reporter: tim.wu
>
> The standalone model is ok. But, in pseudo or cluster model, it always throw errors,
even I just run wordcount example.
> The HDFS works fine, but tasktracker can not create threads(jvm) for new job.  It is
empty under /logs/userlogs/job-xxxx/attempt-xxxx/.
> The reason looks like that in windows, Java can not recognize a symlink of folder as
a folder. 
> The detail description is as following,
> ======================================================================================================
> First, the error log of tasktracker is like:
> ======================
> 12/03/28 14:35:13 INFO mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201203280212_0005_m_-1386636958
> 12/03/28 14:35:13 INFO mapred.JvmManager: JVM Runner jvm_201203280212_0005_m_-1386636958
spawned.
> 12/03/28 14:35:17 INFO mapred.JvmManager: JVM Not killed jvm_201203280212_0005_m_-1386636958
but just removed
> 12/03/28 14:35:17 INFO mapred.JvmManager: JVM : jvm_201203280212_0005_m_-1386636958 exited
with exit code -1. Number of tasks it ran: 0
> 12/03/28 14:35:17 WARN mapred.TaskRunner: attempt_201203280212_0005_m_000002_0 : Child
Error
> java.io.IOException: Task process exit with nonzero status of -1.
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> 12/03/28 14:35:21 INFO mapred.TaskTracker: addFreeSlot : current free slots : 2
> 12/03/28 14:35:24 INFO mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201203280212_0005_m_000002_1
task's state:UNASSIGNED
> 12/03/28 14:35:24 INFO mapred.TaskTracker: Trying to launch : attempt_201203280212_0005_m_000002_1
which needs 1 slots
> 12/03/28 14:35:24 INFO mapred.TaskTracker: In TaskLauncher, current free slots : 2 and
trying to launch attempt_201203280212_0005_m_000002_1 which needs 1 slots
> 12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stdout log for task: attempt_201203280212_0005_m_000002_0
> java.io.FileNotFoundException: D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_0\log.index
(The system cannot find the path specified)
>         at java.io.FileInputStream.open(Native Method)
>         at java.io.FileInputStream.<init>(FileInputStream.java:120)
>         at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
>         at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188)
>         at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:423)
>         at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
>         at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>         at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>         at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stderr log for task: attempt_201203280212_0005_m_000002_0
> java.io.FileNotFoundException: D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_0\log.index
(The system cannot find the path specified)
>         at java.io.FileInputStream.open(Native Method)
>         at java.io.FileInputStream.<init>(FileInputStream.java:120)
>         at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
>         at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188)
>         at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:423)
>         at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
>         at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>         at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>         at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>         at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> =======================================
> I've tried to remote debug tasktracker. In 
> org.apache.hadoop.mapredTaskLog.createTaskAttemptLogDir(TaskAttemptID, boolean, String[])
line: 97:
> public static void createTaskAttemptLogDir(TaskAttemptID taskID,
>       boolean isCleanup, String[] localDirs) throws IOException{
>     String cleanupSuffix = isCleanup ? ".cleanup" : "";
>     String strAttemptLogDir = getTaskAttemptLogDir(taskID, 
>         cleanupSuffix, localDirs);
>     File attemptLogDir = new File(strAttemptLogDir);
>     if (!attemptLogDir.mkdirs()) {
>       throw new IOException("Creation of " + attemptLogDir + " failed.");
>     }
>     String strLinkAttemptLogDir = 
>         getJobDir(taskID.getJobID()).getAbsolutePath() + File.separatorChar + 
>         taskID.toString() + cleanupSuffix;
>     if (FileUtil.symLink(strAttemptLogDir, strLinkAttemptLogDir) != 0) {
>       throw new IOException("Creation of symlink from " + 
>                             strLinkAttemptLogDir + " to " + yestrAttemptLogDir +
>                             " failed.");
>     }
>     //Set permissions for target attempt log dir 
>     FsPermission userOnly = new FsPermission((short) 0777); //FsPermission userOnly =
new FsPermission((short) 0700);
>     FileUtil.setPermission(attemptLogDir, userOnly);
>   }
> and  symlink() function
> public static int symLink(String target, String linkname) throws IOException{
>     String cmd = "ln -s " + target + " " + linkname;
>     Process p = Runtime.getRuntime().exec(cmd, null);
>     int returnVal = -1;
>     try{
>       returnVal = p.waitFor();
>     } catch(InterruptedException e){
>       //do nothing as of yet
>     }
>     if (returnVal != 0) {
>       LOG.warn("Command '" + cmd + "' failed " + returnVal + 
>                " with: " + copyStderr(p));
>     }
>     return returnVal;
>   }
> we know hadoop will create a log folder in ${hadoop.tmp.dir},  and then invoke "ln -s
" to create its symlink under /logs/userlog/job-xxx/attermp-xxxx. 
> In my case, 
> strLinkAttemptLogDir = D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1
> strAttemptLogDir=/tmp/hadoop-timwu/mapred/local\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1
> After a subtrack is created by tasktracker, it runs error in the following function:
> in org.apache.hadoop.mapred.java , DefaultTaskController.launchTask(String, String, String,
List<String>, List<String>, File, String, String) line: 107        
>       ...............
>       //mkdir the loglocation
>       String logLocation = TaskLog.getAttemptDir(jobId, attemptId).toString();
>       if (!localFs.mkdirs(new Path(logLocation))) {
>         throw new IOException("Mkdirs failed to create " 
>                    + logLocation);
>       }
>      ..............
> mkdir() return false, because logLocation is a symlink file. In my case, it is ogLocation=D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1.
 If I open it from explorer in windows, it is just a file,  but not a folder or shortcut.
And its content is like,
>          <symlink>/tmp/hadoop-timwu/mapred/local\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1

> Because the mkdir() is
> public boolean mkdirs(Path f) throws IOException {
>     Path parent = f.getParent();
>     File p2f = pathToFile(f);
>     return (parent == null || mkdirs(parent)) &&
>       (p2f.mkdir() || p2f.isDirectory());
>   }
> So, p2f.isDirectory returns false.  And p2f.isFile will return true. So, for java, it
is a file. Hence, IOException("Mkdirs failed to create D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000001_1")
> will be throws in child threads, and return -1.  Then, we will get the above exception
in main thread.
> Is it any way to close this symlink? Or any other way I can follow?
> BTW, in core-site.xml, I set  hadoop.tmp.dir = /tmp/hadoop-${user.name},  and my $User.name
is timwu. So, it should create a tmp folder /tmp/hadoop-timwu  under cygwin's.   However,
in deed , it create  a folder of d:/tmp/hadoop-timwu. If in cygwin, it is /cygdriver/d/tmp/hadoop-timwu.
 Is it correct?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message