hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yankay (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-8800) Dynamic Compress Stream
Date Thu, 13 Sep 2012 13:24:11 GMT
yankay created HADOOP-8800:

             Summary: Dynamic Compress Stream
                 Key: HADOOP-8800
                 URL: https://issues.apache.org/jira/browse/HADOOP-8800
             Project: Hadoop Common
          Issue Type: New Feature
          Components: io
    Affects Versions: 2.0.1-alpha
            Reporter: yankay

We use compress in MapReduce in some case because It use CPU to improve IO throughput.

But we can only set one compress algorithm in configure file. The hadoop cluster is changing
every time.  So a compress algorithm may not work well in all case. 

Why not provide a algorithm named dynamic. It can change compress level and algorithm dynamic
based on performance. Like tcp, it starts up slowly, and try run faster and faster.

I would write a detail design here, and try to submit a patch.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message