hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonger (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-15087) Write directly without creating temp directory to avoid rename
Date Mon, 04 Dec 2017 03:26:00 GMT
Yonger created HADOOP-15087:
-------------------------------

             Summary: Write directly without creating temp directory to avoid rename 
                 Key: HADOOP-15087
                 URL: https://issues.apache.org/jira/browse/HADOOP-15087
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs/s3
            Reporter: Yonger


Rename in workloads like Teragen/Terasort who use Hadoop default outputcommitters really hurt
performance a lot. 
Stocator announce it doesn't create the temporary directories any all, and still preserves
Hadoop's fault tolerance. I add a switch when creating file via integrating it's code into
s3a, I got 5x performance gain in Teragen and 15% performance improvement in Terasort.

 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


Mime
View raw message