hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Echo Li <echo...@gmail.com>
Subject Re: Tez Vertex failure
Date Sun, 05 Oct 2014 17:42:31 GMT
I try another query and was able to fetch the logs:

query: select ymd,count(1),count(distinct sessionId) from visit where
ymd<=20140831 group by ymd order by ymd ;

in app logs:

2014-10-05 17:32:58,520 INFO [ORC_GET_SPLITS #4]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: ORC pushdown predicate:
leaf-0 = (LESS_THAN_EQUALS ymd 20140831)
expr = leaf-0
2014-10-05 17:32:58,520 INFO [ORC_GET_SPLITS #3]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: ORC pushdown predicate:
leaf-0 = (LESS_THAN_EQUALS ymd 20140831)
expr = leaf-0
2014-10-05 17:32:58,520 INFO [ORC_GET_SPLITS #6]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: ORC pushdown predicate:
leaf-0 = (LESS_THAN_EQUALS ymd 20140831)
expr = leaf-0
2014-10-05 17:32:58,520 INFO [ORC_GET_SPLITS #1]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: ORC pushdown predicate:
leaf-0 = (LESS_THAN_EQUALS ymd 20140831)
expr = leaf-0
2014-10-05 17:32:58,532 ERROR [ORC_GET_SPLITS #3]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: Unexpected Exception
java.lang.ArrayIndexOutOfBoundsException: 4
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:696)
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInputFormat.java:824)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2014-10-05 17:32:58,532 ERROR [ORC_GET_SPLITS #4]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: Unexpected Exception
java.lang.ArrayIndexOutOfBoundsException: 4
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:696)
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInputFormat.java:824)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2014-10-05 17:32:58,532 ERROR [ORC_GET_SPLITS #1]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: Unexpected Exception
java.lang.ArrayIndexOutOfBoundsException: 3
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:696)
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInputFormat.java:824)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2014-10-05 17:32:58,536 ERROR [AsyncDispatcher event handler]
org.apache.tez.dag.app.dag.impl.VertexImpl: Vertex Input: visit initializer
failed
java.lang.RuntimeException: serious problem
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$Context.waitForTasks(OrcInputFormat.java:468)
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:921)
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:946)
    at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:291)
    at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:372)
    at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getSplits(TezGroupedSplitsInputFormat.java:68)
    at
org.apache.tez.mapreduce.hadoop.MRHelpers.generateOldSplits(MRHelpers.java:263)
    at
org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:139)
    at
org.apache.tez.dag.app.dag.RootInputInitializerRunner$InputInitializerCallable$1.run(RootInputInitializerRunner.java:154)
    at
org.apache.tez.dag.app.dag.RootInputInitializerRunner$InputInitializerCallable$1.run(RootInputInitializerRunner.java:146)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
    at
org.apache.tez.dag.app.dag.RootInputInitializerRunner$InputInitializerCallable.call(RootInputInitializerRunner.java:146)
    at
org.apache.tez.dag.app.dag.RootInputInitializerRunner$InputInitializerCallable.call(RootInputInitializerRunner.java:114)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 4
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:696)
    at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInputFormat.java:824)
    ... 3 more

On Sun, Oct 5, 2014 at 10:03 AM, Echo Li <echolql@gmail.com> wrote:

> Thanks guys, these are great info.
>
> - it works fine when use "set hive.execution.engine=mr"
>
> - there is nothing else printed after the exception; and no application
> logs either
>
> is this a bug?
>
> On Fri, Oct 3, 2014 at 6:39 PM, Gopal V <gopalv@apache.org> wrote:
>
>> On 10/3/14, 5:20 PM, Echo Li wrote:
>>
>>> thanks for reply!
>>>
>>> the query is: *select count(customerid) from tableName where
>>> ymd=20140930 ;*
>>>
>>
>> That is simple enough that it should work anyway.
>>
>> There is strong possibility that the rest of that RuntimeException gives
>> a clue to the problem - if anything is printed.
>>
>> Since the mappers are stuck at -/-, I would speculate that ORC split
>> generation is failing on Google Cloud Storage.
>>
>> The data you need to collect can be collected via
>>
>> "yarn logs -applicationId application_1412033199033_6623"
>>
>> which will be more helpful in locating the root-cause of the issue.
>>
>> Cheers,
>> Gopal
>>
>>
>>
>>> when I try to get the application logs only got this:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *dag_1412033199033_6639_1.dot : Total file length is 1562 bytes.stderr :
>>> Total file length is 0 bytes.stderr_dag_1412033199033_6639_1 : Total
>>> file
>>> length is 0 bytes.stderr_dag_1412033199033_6639_1_post : Total file
>>> length
>>> is 0 bytes.stdout : Total file length is 0
>>> bytes.stdout_dag_1412033199033_6639_1 : Total file length is 0
>>> bytes.stdout_dag_1412033199033_6639_1_post : Total file length is 0
>>> bytes.syslog : Total file length is 7436
>>> bytes.syslog_dag_1412033199033_6639_1 : Total file length is 14695
>>> bytes.syslog_dag_1412033199033_6639_1_post : Total file length is 136
>>> bytes.*
>>>
>>> Also when I run this query: select * from visit where ymd=20140930 limit
>>> 10;
>>> got error:
>>> *Failed with exception java.io.IOException:java.lang.RuntimeException:
>>> serious problem*
>>>
>>> The table is ORC format, in google cloud storage.
>>>
>>>
>>> On Fri, Oct 3, 2014 at 4:54 PM, Vikram Dixit <vikram@hortonworks.com>
>>> wrote:
>>>
>>>  Hi,
>>>>
>>>> Can you share the query and the application logs with us? Which
>>>> version of hive is this?
>>>>
>>>> Thanks
>>>> Vikram.
>>>>
>>>> On Fri, Oct 3, 2014 at 4:51 PM, Echo Li <echolql@gmail.com> wrote:
>>>> > Hi guys,
>>>> >
>>>> > I run some simple hive query and got the error below:
>>>> >
>>>> > Status: Running (application id: application_1412033199033_6623)
>>>> >
>>>> > Map 1: -/-    Reducer 2: 0/1
>>>> > Status: Failed
>>>> > Vertex failed, vertexName=Map 1, vertexId=vertex_1412033199033_
>>>> 6623_1_01,
>>>> > diagnostics=[Vertex Input: visit initializer failed., serious problem]
>>>> > Vertex killed, vertexName=Reducer 2,
>>>> > vertexId=vertex_1412033199033_6623_1_00, diagnostics=[Vertex received
>>>> Kill
>>>> > in INITED state.]
>>>> > DAG failed due to vertex failure. failedVertices:1 killedVertices:1
>>>> > FAILED: Execution Error, return code 2 from
>>>> > org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>> >
>>>> >
>>>> > Any idea what's the cause and what's the fix? I suspect its some sort
>>>> data
>>>> > issue as it only happens to certain partitions, anybody had any
>>>> experience
>>>> > on this?
>>>> >
>>>> > Thanks guys.
>>>> >
>>>> >
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>>> immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message