hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-2011) Reduce number of getFileStatus call made from every task(TaskDistributedCache) setup
Date Sat, 14 Aug 2010 00:10:16 GMT
Reduce number of getFileStatus call made from every task(TaskDistributedCache) setup
------------------------------------------------------------------------------------

                 Key: MAPREDUCE-2011
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2011
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: distributed-cache
            Reporter: Koji Noguchi


On our cluster, we had jobs with 20 dist cache and very short-lived tasks resulting in 500
map tasks launched per second resulting in  10,000 getFileStatus calls to the namenode.  Namenode
can handle this but asking to see if we can reduce this somehow.  


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message