hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3203) TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
Date Wed, 09 Apr 2008 18:41:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587296#action_12587296
] 

Chris Douglas commented on HADOOP-3203:
---------------------------------------

bq. Using listStatus(Path) makes only one RPC. Do you think we should go for FileSystem.getFileStatus(Path)
and make two RPCs?

This section of code should be called once per TaskTracker per Job, right? The difference
between one RPC and two RPCs here is insufficient to measure, let alone effect a performance
bottleneck or load we should worry about. Since the 5x expansion of the jar is a pessimistic
estimate, the config should scarcely matter, but given HADOOP-3048 and the possibility of
serialized objects in the config, it's worth checking its size, too.

> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-3203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3203
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.15.3
>            Reporter: Chris Douglas
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.18.0
>
>         Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
>     // Get sizes of JobFile and JarFile
>     // sizes are -1 if they are not present.
>     FileSystem fileSystem = FileSystem.get(fConf);
>     FileStatus status[] = fileSystem.listStatus(new Path(jobFile).getParent());
>     long jarFileSize = -1;
>     long jobFileSize = -1;
>     for(FileStatus stat : status) {
>       if (stat.getPath().toString().contains("job.xml")) {
>         jobFileSize = stat.getLen();
>       } else {
>         jobFileSize = -1;
>       }
>       if (stat.getPath().toString().contains("job.jar")) {
>         jarFileSize = stat.getLen();
>       } else {
>         jarFileSize = -1;
>       }
>     }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message