hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanth Jayachandran <pjayachand...@hortonworks.com>
Subject Re: hive on Tez - merging orc files
Date Fri, 24 Apr 2015 07:42:36 GMT
Hi

This has been fixed recently https://issues.apache.org/jira/browse/HIVE-9529. Merging is triggered
in two different ways. INSERT/CTAS can trigger merging of small files and CONCATENATE can
trigger merging of small files. The later had a bug which generated MR task instead of TEZ
task which was fixed recently. Earlier one will use TEZ task always.

Thanks
Prasanth

> On Apr 24, 2015, at 12:33 AM, patcharee <Patcharee.Thongtra@uni.no> wrote:
> 
> Hi,
> 
> Is there anyone using hortonworks sandbox 2.2? I am trying to use hive on Tez on the
sandbox. I set the running engine in hive-site.xml to Tez.
> 
>    <property>
>      <name>hive.execution.engine</name>
>      <value>tez</value>
>    </property>
> 
> Then I ran the script that alters a table to merge small orc files (alter table orc_merge5a
partition(st=0.8) concatenate;). The merging feature worked, but Hive does not use Tez, it
used MapReduce, so weird!
> 
> Another point, I tried to run the same script on the production cluster which is on always
Tez, the merging feature sometimes worked, sometimes did not.
> 
> I would appreciate any suggestions.
> 
> BR,
> Patcharee


Mime
View raw message