hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zoltan Haindrich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15467) escape1.q hangs in TestMiniLlapLocalCliDriver
Date Wed, 30 Aug 2017 08:24:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146880#comment-16146880
] 

Zoltan Haindrich commented on HIVE-15467:
-----------------------------------------

the same issue happened again for me...I've looked into it a bit more...and it seems like
there is some issues with the nodemanagers...they report that the local dirs are bad

resourcemanager ui shows this info:
{code}
NodeHealthReport 	4/4 local-dirs are bad: /home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-localDir-nm-1_0,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-localDir-nm-1_2,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-localDir-nm-1_1,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-localDir-nm-1_3;
4/4 log-dirs are bad: /home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-logDir-nm-1_1,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-logDir-nm-1_0,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-logDir-nm-1_3,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-logDir-nm-1_2

{code}

nodemanagers are in an unworkable state...and because of this the tez AM stucks in initializing
state
the resourcemanager ui seems to be not available...and I've not found any other usefull info...

I've switched to a dfferent cli  driver which didn't get stuck...

> escape1.q hangs in TestMiniLlapLocalCliDriver
> ---------------------------------------------
>
>                 Key: HIVE-15467
>                 URL: https://issues.apache.org/jira/browse/HIVE-15467
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Pengcheng Xiong
>            Assignee: Prasanth Jayachandran
>
> here is part of the log before it hangs
> {code}
> 2016-12-19T15:21:05,779  INFO [LlapScheduler] tezplugins.LlapTaskSchedulerService: ScheduleResult
for Task: TaskInfo{task=attempt_1482189645956_0001_33_00_000000_1, priority=1, startTime=0,
containerId=null, assignedNode=, uniqueId=54, localityDelayTimeout=0} = DELAYED_RESOURCES
> 2016-12-19T15:21:05,779 DEBUG [LlapScheduler] tezplugins.LlapTaskSchedulerService: Attempting
to preempt on any host for task=attempt_1482189645956_0001_33_00_000000_1, pendingPreemptions=0
> 2016-12-19T15:21:05,779  INFO [LlapScheduler] tezplugins.LlapTaskSchedulerService: Preempting
for task=attempt_1482189645956_0001_33_00_000000_1 on any available host
> 2016-12-19T15:21:05,779 DEBUG [LlapScheduler] tezplugins.LlapTaskSchedulerService: Unable
to schedule all requests at priority=1. Skipping subsequent priority levels
> 2016-12-19T15:21:07,953 DEBUG [AMReporterQueueDrainer] impl.AMReporter: Removing am localhost:61788
with last associated dag QueryIdentifier{appIdentifier='application_1482189645956_0001', dagIdentifier=33}
from heartbeat with taskCount=0, amFailed=false
> 2016-12-19T15:21:08,634  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:11,700  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:14,755  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:17,814  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:20,871  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:23,931  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:26,977  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:30,027  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:33,078  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:36,133  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:39,179  INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main] SessionState:
Map 1: 0(+1,-1)/1
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message