Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which
 is an MX secondary for user@hive.apache.org)
Message-ID: <553A2A9E.70005@uib.no>
Date: Fri, 24 Apr 2015 13:35:58 +0200
From: patcharee <Patcharee.Thongtra@uni.no>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: user@hive.apache.org, pjayachandran@hortonworks.com
Subject: Re: hive on Tez - merging orc files
References: <5539F1AD.2010807@uib.no>
 <154CF9A2-E582-454D-A5A7-FEDD126D886E@hortonworks.com>
 <5539FB8A.9050304@uib.no>
 <6DE75112-DB39-4094-A787-47DC6778F305@hortonworks.com>
In-Reply-To: <6DE75112-DB39-4094-A787-47DC6778F305@hortonworks.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

I generated the new hive-exec.jar as you suggested. On the sandbox, Hive 
0.14 with the new jar file is now using Tez to alter table concate, and 
It concatenates files correctly on Tez. Thanks!

However I also tested on the production cluster using Hive 0.14 as well, 
merging did not work and generated another exception below

2015-04-24 13:01:52,259 INFO [main] app.DAGAppMaster: Running DAG: alter 
table orc_merge5a partit...concatenate
2015-04-24 13:01:52,355 INFO [IPC Server handler 0 on 46526] ipc.Server: 
IPC Server handler 0 on 46526, call 
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.getDAGStatus 
from 10.2.1.254:39356 Call#361 Retry#0
org.apache.tez.dag.api.TezException: No running dag at present
         at 
org.apache.tez.dag.api.client.DAGClientHandler.getDAG(DAGClientHandler.java:84)
         at 
org.apache.tez.dag.api.client.DAGClientHandler.getACLManager(DAGClientHandler.java:151)
         at 
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:94)
         at 
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
         at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:415)
         at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

What could be the cause of this exception? Any ideas?

BR,
Patcharee

On 24. april 2015 10:27, Prasanth Jayachandran wrote:
> You can download the branch-0.14 source code from https://github.com/apache/hive/tree/branch-0.14, apply HIVE-9529-branch-1.0.0.patch from https://issues.apache.org/jira/browse/HIVE-9529 and compile it using �mvn clean install -DskipTests -Phadoop-2,dist�. This will generate tar file under hive/packaging/target. You can extract the tar file, copy the hive-exec-x.x.x.jar into /usr/hdp/2.2.*.*/hive/lib/ (take backup of hive-exec.jar and replace with the new one). Rerunning hive cli should use the new hive-exec jar with the patch.
>
> Thanks
> Prasanth
>
>> On Apr 24, 2015, at 1:15 AM, patcharee <Patcharee.Thongtra@uni.no> wrote:
>>
>> Hi,
>>
>> The sandbox 2.2 comes with hive 0.14. Does it also have the bug? If so, how can I patch hive on sandbox?
>>
>> BR,
>> Patcharee
>>
>> On 24. april 2015 09:42, Prasanth Jayachandran wrote:
>>> Hi
>>>
>>> This has been fixed recently https://issues.apache.org/jira/browse/HIVE-9529. Merging is triggered in two different ways. INSERT/CTAS can trigger merging of small files and CONCATENATE can trigger merging of small files. The later had a bug which generated MR task instead of TEZ task which was fixed recently. Earlier one will use TEZ task always.
>>>
>>> Thanks
>>> Prasanth
>>>
>>>> On Apr 24, 2015, at 12:33 AM, patcharee <Patcharee.Thongtra@uni.no> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Is there anyone using hortonworks sandbox 2.2? I am trying to use hive on Tez on the sandbox. I set the running engine in hive-site.xml to Tez.
>>>>
>>>>     <property>
>>>>       <name>hive.execution.engine</name>
>>>>       <value>tez</value>
>>>>     </property>
>>>>
>>>> Then I ran the script that alters a table to merge small orc files (alter table orc_merge5a partition(st=0.8) concatenate;). The merging feature worked, but Hive does not use Tez, it used MapReduce, so weird!
>>>>
>>>> Another point, I tried to run the same script on the production cluster which is on always Tez, the merging feature sometimes worked, sometimes did not.
>>>>
>>>> I would appreciate any suggestions.
>>>>
>>>> BR,
>>>> Patcharee