hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王锋 <wfeng1...@163.com>
Subject size of RCFile in hive
Date Wed, 19 Sep 2012 14:25:54 GMT
Hi
   I tried to convert and merge many small text files using RCFiles using hivesql,but hive
 produced some small rcfiles.
set hive.exec.compress.output=true;
set mapred.output.compress=true;
set mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;
set io.compression.codecs=com.hadoop.compression.lzo.LzoCodec;
hive.merge.mapfiles=true
hive.merge.mapredfiles=true
hive.merge.size.per.task=640000000
hive.merge.size.smallfiles.avgsize=80000000
insert  overwrite table rctable select .....


  the settings:
hive.merge.mapfiles=true
hive.merge.mapredfiles=true
hive.merge.size.per.task=640000000
hive.merge.size.smallfiles.avgsize=80000000
didn't work.


who could tell me how to solve it?
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message