hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6532) Path objects are heavy
Date Tue, 23 Feb 2010 00:41:30 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837009#action_12837009
] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6532:
------------------------------------------------

I ran wordcount with the v2 patch twice.  Both took 16 mins.  Not sure why it ran slower than
the previous patch.
{noformat}
-bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}3
Mon Feb 22 23:23:03 UTC 2010
10/02/22 23:23:19 INFO input.FileInputFormat: Total input paths to process : 100000
10/02/22 23:33:53 INFO mapred.JobClient: Running job: job_201002042035_75937
10/02/22 23:33:54 INFO mapred.JobClient:  map 0% reduce 0%
...
10/02/22 23:39:08 INFO mapred.JobClient:  map 100% reduce 100%
...
10/02/22 23:39:14 INFO mapred.JobClient:     Reduce input records=17729

real    16m11.171s
user    2m24.217s
sys     0m39.462s
{noformat}
Anyway, below are some improvement suggestions:
- use a variable to store harPath.depth() so that the same value won't be computed again and
again.

- Unfortunately, Path is quite expensive (see HADOOP-6532).  It is better to check child.startsWith(parentString)
before creating thisPath.  i.e. replace
{code}
+        String child = lineFeed.substring(0, lineFeed.indexOf(" "));
+        Path thisPath = new Path(child);
+        if ((child.startsWith(parentString)) && (thisPath.depth() == harPath.depth()
+ 1)) {
+          ...
{code}
with
{code}
+        String child = lineFeed.substring(0, lineFeed.indexOf(" "));
+        if (child.startsWith(parentString)) {
+          Path thisPath = new Path(child);
+          if (thisPath.depth() == harPath.depth() + 1) {
+            ...
{code}

- Also, there are lines more than 80 characters.


> Path objects are heavy
> ----------------------
>
>                 Key: HADOOP-6532
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6532
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Tsz Wo (Nicholas), SZE
>         Attachments: Fun.java
>
>
> Compared with java.lang.String, org.apache.hadoop.fs.Path is much heavier since it contains
URI.  The size of a Path is roughly 3 times of a String.  See some numbers in the comments.
> A major impact of decreasing Path size is allowing ls, archive, etc. on directories with
many files.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message