hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Zhang (JIRA)" <j...@apache.org>
Subject [jira] Created: (HIVE-1343) add an interface in RCFile to support concatenation of two files without (de)compression
Date Thu, 13 May 2010 23:01:44 GMT
add an interface in RCFile to support concatenation of two files without (de)compression
----------------------------------------------------------------------------------------

                 Key: HIVE-1343
                 URL: https://issues.apache.org/jira/browse/HIVE-1343
             Project: Hadoop Hive
          Issue Type: New Feature
    Affects Versions: 0.6.0
            Reporter: Ning Zhang
            Assignee: He Yongqiang
             Fix For: 0.6.0


If two files are concatenated, we need to read each record in these files and write them back
to the destination file. The IO cost is mostly unavoidable due to the lack of append functionality
in HDFS. However the CPU cost could be significantly reduced by avoiding compression and decompression
of the files.

The File Format layer should provide API that implement the block-level concatenation. 



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message