hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanth Jayachandran <pjayachand...@hortonworks.com>
Subject Re: How to merge the output file when insert data into a table
Date Thu, 28 Aug 2014 16:44:25 GMT
Hi

If its an rcfile you can use "alter table tablename concatenate" to merge them into 1 file.
For text files, you can may have to reload these 2 files into another table with "Order By".
This will force one reducer to generate total ordering and thereby generating 1 output file.
But remember if the data size huge order by will take very long time. 

Thanks
Prasanth

> On Aug 28, 2014, at 7:45 AM, 王洋 <wangyang.joe@gmail.com> wrote:
> 
> Hello,
>      I have two table ,one is stored as textfile for loading data from local data, the
other is stored as rcfile for compressing data。
>      when I execute the following sql two times
>      
>      insert into table tablename1 select * from tablename2
>    
>      I find that there are two files in the warehouse
> 
>       hadoop fs -ls /user/hive/warehouse/tablename2/
>       Found 2 items
>      -rwxrwxrwt   3 root supergroup        423 2014-08-28 21:09 /user/hive/warehouse/tablename2/000000_0
>      -rwxrwxrwt   3 root supergroup        423 2014-08-28 21:11 /user/hive/warehouse/tablename2/000000_0_copy_1
> 
>    I need to merge the two files to reduce the small file, is there have some way to
merge the files?  
>    thank you !

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message