hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HADOOP-5244) Distributed cache spends a lot of time runing du -s
Date Fri, 13 Feb 2009 18:50:59 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arun C Murthy resolved HADOOP-5244.
-----------------------------------

    Resolution: Duplicate

Duplicate of HADOOP-4780.

> Distributed cache spends a lot of time runing du -s
> ---------------------------------------------------
>
>                 Key: HADOOP-5244
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5244
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Ben Maurer
>
> When running a MapReduce job that has a large jar on the class path (eg, jruby from hbase),
the task tracker takes a large amount of CPU time during startup. Using jstack, I got the
following stack trace:
> "Thread-8941" daemon prio=10 tid=0x00002aab08005c00 nid=0x2807 waiting on condition [0x0000000043eca000..0x0000000043ecbc90]
>    java.lang.Thread.State: RUNNABLE
> 	at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
> 	at java.lang.StringCoding.encode(StringCoding.java:272)
> 	at java.lang.String.getBytes(String.java:947)
> 	at java.io.UnixFileSystem.getLength(Native Method)
> 	at java.io.File.length(File.java:848)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:428)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
> 	at org.apache.hadoop.filecache.DistributedCache.getLocalCache(DistributedCache.java:210)
> 	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:178)
> Running the system "du" returns very quickly on the directory. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message