hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Yongqiang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1950) Block merge for RCFile
Date Tue, 15 Feb 2011 01:31:57 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994616#comment-12994616
] 

He Yongqiang commented on HIVE-1950:
------------------------------------

QTestUtil.java is not related to this jira. should open a new one for it.

>>jobExecHelper is constructed in both the constructors and initialize(). Is there a
reason?
This is because the existing code may use ExecDriver and may not call initialize() (like ExecDriver's
main()).

>>checkFatalError: why removed some code?
No code is removed, just some code is moved to jobExecHelper.

> Block merge for RCFile
> ----------------------
>
>                 Key: HIVE-1950
>                 URL: https://issues.apache.org/jira/browse/HIVE-1950
>             Project: Hive
>          Issue Type: New Feature
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>         Attachments: HIVE-1950.1.patch, HIVE-1950.2.patch, HIVE-1950.3.patch, HIVE-1950.4.patch
>
>
> In our env, there are a lot of small files inside one partition/table. In order to reduce
the namenode load, we have one dedicated housekeeping job running to merge these file. Right
now the merge is an 'insert overwrite' in hive, and requires decompress the data and compress
it. This jira is to add a command in Hive to do the merge without decompress and recompress
the data.
> Something like "alter table tbl_name [partition ()] merge files". In this jira the new
command will only support RCFile, since there need some new APIs to the fileformat.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message