hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lefty Leverenz <leftylever...@gmail.com>
Subject Re: Question about running Hive on Tez
Date Thu, 19 Dec 2013 00:38:31 GMT
Should hive-on-tez-conf.txt be added to the wiki, or is it not soup yet?

-- Lefty


On Mon, Dec 16, 2013 at 10:25 AM, Cheolsoo Park <piaozhexiu@gmail.com>wrote:

> Closing the loop. We identified the issue with help from the Tez team. It
> was mis-configured mapreduce.reduce.cpu.vcores that caused problems.
>
> If anyone who tries Hive on Tez with EMR Hadoop and sees that reducers are
> stuck, this<http://mail-archives.apache.org/mod_mbox/incubator-tez-user/201312.mbox/%3Cc3f953c465a3b2f4b02d62348ce67161%40mail.gmail.com%3E>
might
> be helpful.
>
> Thanks,
> Cheolsoo
>
> On Fri, Dec 13, 2013 at 4:51 PM, Gunther Hagleitner <
> ghagleitner@hortonworks.com> wrote:
>
>> dev on bcc
>>
>> Zhenxiao,
>>
>> Cool you got it set up.
>>
>>  The query runs a full order by before the limit - are you sure it's not
>> just still running? Hive on Tez prints "total tasks/completed tasks", so no
>> update may mean none of the reduce tasks have finished yet.
>>
>> If not, it'd be great to see the yarn logs (yarn logs -applicationId) and
>> get more info about the table you're using (size, file format, etc). If the
>> logs are really big you might want to consider opening/attaching them to a
>> jira (issues.apache.org) (or send them directly to me).
>>
>> There are a bunch of settings that might be of interest to you (in
>> general not just for this query) - I've attached a text doc with some
>> details.
>>
>> Thanks,
>> Gunther.
>>
>>
>>
>>
>> On Fri, Dec 13, 2013 at 1:12 PM, Zhenxiao Luo <zluo@netflix.com> wrote:
>>
>>> Hi,
>>>
>>> Excuse me. May I ask a question about running Hive on Tez?
>>>
>>> I've installed Hive on Tez, and running a simple query from hiveCli,
>>>
>>> hive> set hive.optimize.tez=true;
>>> hive> select * from table order by title_id limit 5;
>>>
>>> While, each time, I could see from the TezJobMonitor that, all the map
>>> jobs are done, but the reducer never get started, and the job is
>>> running forever there.
>>>
>>> I tried a number of times, and each time the same failure(job running
>>> hangs) happens again and again.
>>> Does anyone successfully running queries using Hive on Tez? Are there
>>> any tips or suggestions?
>>>
>>> Here is my job log:
>>>
>>> 13/12/13 20:57:31 INFO client.TezSession: Submitting dag to
>>> TezSession, sessionName=HIVE-365b35bc-2461-4e2f-83f9-8da1fa356a86,
>>> applicationId=application_1386896881353_0027
>>> 13/12/13 20:57:33 INFO client.TezSession: Submitted dag to TezSession,
>>> sessionName=HIVE-365b35bc-2461-4e2f-83f9-8da1fa356a86,
>>> applicationId=application_1386896881353_0027,
>>> dagId=dag_1386896881353_0027_1
>>> 13/12/13 20:57:33 INFO client.RMProxy: Connecting to ResourceManager
>>> at /10.183.195.180:9022
>>> 13/12/13 20:57:33 INFO log.PerfLogger: </PERFLOG method=TezSubmitDag
>>> start=1386968251250 end=1386968253338 duration=2088
>>> from=org.apache.hadoop.hive.ql.exec.tez.TezTask>
>>>
>>>
>>> 13/12/13 20:57:33 INFO tez.TezJobMonitor:
>>>
>>> 13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG method=TezRunDag
>>> from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
>>> 13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG
>>> method=TezSubmitToRunningDag
>>> from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
>>> 13/12/13 20:57:33 INFO log.PerfLogger: </PERFLOG
>>> method=TezSubmitToRunningDag start=1386968253341 end=1386968253402
>>> duration=61 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
>>> Status: Running (application id: application_1386896881353_0027)
>>>
>>> 13/12/13 20:57:33 INFO tez.TezJobMonitor: Status: Running (application
>>> id: application_1386896881353_0027)
>>>
>>> 13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG
>>> method=TezRunVertex.Reducer 2
>>> from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
>>> 13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG
>>> method=TezRunVertex.Map 1
>>> from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
>>> Map 1: -/- Reducer 2: -/-
>>> 13/12/13 20:57:33 INFO tez.TezJobMonitor: Map 1: -/- Reducer 2: -/-
>>> Map 1: -/- Reducer 2: 0/1
>>> 13/12/13 20:57:33 INFO tez.TezJobMonitor: Map 1: -/- Reducer 2: 0/1
>>> Map 1: 0/16 Reducer 2: 0/1
>>> 13/12/13 20:57:34 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
>>> Map 1: 0/16 Reducer 2: 0/1
>>> 13/12/13 20:57:37 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
>>> Map 1: 0/16 Reducer 2: 0/1
>>> 13/12/13 20:57:40 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
>>> Map 1: 0/16 Reducer 2: 0/1
>>> 13/12/13 20:57:43 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
>>> Map 1: 0/16 Reducer 2: 0/1
>>> 13/12/13 20:57:46 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
>>> Map 1: 0/16 Reducer 2: 0/1
>>> 13/12/13 20:57:49 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
>>> Map 1: 0/16 Reducer 2: 0/1
>>> 13/12/13 20:57:52 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
>>> Map 1: 0/16 Reducer 2: 0/1
>>> 13/12/13 20:57:55 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
>>> Map 1: 1/16 Reducer 2: 0/1
>>> 13/12/13 20:57:56 INFO tez.TezJobMonitor: Map 1: 1/16 Reducer 2: 0/1
>>> Map 1: 2/16 Reducer 2: 0/1
>>> 13/12/13 20:57:58 INFO tez.TezJobMonitor: Map 1: 2/16 Reducer 2: 0/1
>>> Map 1: 3/16 Reducer 2: 0/1
>>> 13/12/13 20:57:58 INFO tez.TezJobMonitor: Map 1: 3/16 Reducer 2: 0/1
>>> Map 1: 5/16 Reducer 2: 0/1
>>> 13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 5/16 Reducer 2: 0/1
>>> Map 1: 8/16 Reducer 2: 0/1
>>> 13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 8/16 Reducer 2: 0/1
>>> Map 1: 12/16 Reducer 2: 0/1
>>> 13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 12/16 Reducer 2: 0/1
>>> Map 1: 15/16 Reducer 2: 0/1
>>> 13/12/13 20:58:00 INFO tez.TezJobMonitor: Map 1: 15/16 Reducer 2: 0/1
>>> 13/12/13 20:58:00 INFO log.PerfLogger: </PERFLOG
>>> method=TezRunVertex.Map 1 start=1386968253402 end=1386968280223
>>> duration=26821 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:00 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:03 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:06 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:09 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:12 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:15 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:18 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:21 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:24 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:27 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:30 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:33 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:36 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:39 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:42 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:45 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:48 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:51 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:54 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:58:57 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:00 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:03 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:06 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:09 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:12 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:15 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:18 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:21 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:24 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:27 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:30 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:33 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:36 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:39 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:42 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:45 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:48 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:52 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:55 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 20:59:58 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 21:00:01 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 21:00:04 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 21:00:07 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 21:00:10 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>> Map 1: 16/16 Reducer 2: 0/1
>>> 13/12/13 21:00:13 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
>>>
>>> Thanks,
>>> Zhenxiao
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>

Mime
View raw message