ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudhir Prakash (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-5800) Race condition when starting all services causing Hive service check to fail
Date Tue, 20 May 2014 16:04:38 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003543#comment-14003543
] 

Sudhir Prakash commented on AMBARI-5800:
----------------------------------------

I don't have the log for the 55s startup, but that was during the initial startup immediately
after installation. Here is a log where it took 29 secs:

{code}
2014-05-16 13:51:20,850 INFO  [main]: server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(605))
- STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting HiveServer2
STARTUP_MSG:   host = hadoopvm1-1/39.0.8.1
STARTUP_MSG:   args = [-hiveconf, hive.metastore.uris=", ", -hiveconf, hive.log.file=hiveserver2.log,
-hiveconf, hive.log.dir=/var/opt/teradata/log/hive]
STARTUP_MSG:   version = 0.13.0.2.1.2.0-402
STARTUP_MSG:   classpath = /etc/hadoop/conf:/usr/lib/hadoop/lib....<truncated>
STARTUP_MSG:   build = git://s111-s8/var/opt/teradata/workspace/BIGTOP-HDP_RPM_REPO-baikal-GA-suse11/bigtop/build/hive/rpm/BUILD/hive-0.13.0.2.1.2.0
-r c19175be5345f7d4bc7ca0a8f95a09a4330de2f4; compiled by 'jenkins' on Sun Apr 27 18:52:34
EDT 2014
************************************************************/
2014-05-16 13:51:48,369 INFO  [main]: hive.metastore (HiveMetaStoreClient.java:open(297))
- Trying to connect to metastore with URI thrift://byn001-1:9083
2014-05-16 13:51:48,436 INFO  [main]: hive.metastore (HiveMetaStoreClient.java:open(385))
- Connected to metastore.
2014-05-16 13:51:49,298 INFO  [main]: session.SessionState (SessionState.java:start(361))
- No Tez session required at this point. hive.execution.engine=mr.
2014-05-16 13:51:49,307 INFO  [main]: service.CompositeService (SessionManager.java:init(70))
- HiveServer2: Async execution thread pool size: 100
2014-05-16 13:51:49,307 INFO  [main]: service.CompositeService (SessionManager.java:init(72))
- HiveServer2: Async execution wait queue size: 100
2014-05-16 13:51:49,308 INFO  [main]: service.CompositeService (SessionManager.java:init(74))
- HiveServer2: Async execution thread keepalive time: 10
2014-05-16 13:51:49,309 INFO  [main]: service.AbstractService (AbstractService.java:init(89))
- Service:OperationManager is inited.
2014-05-16 13:51:49,309 INFO  [main]: service.AbstractService (AbstractService.java:init(89))
- Service:SessionManager is inited.
2014-05-16 13:51:49,309 INFO  [main]: service.AbstractService (AbstractService.java:init(89))
- Service:CLIService is inited.
2014-05-16 13:51:49,309 INFO  [main]: service.AbstractService (AbstractService.java:init(89))
- Service:ThriftBinaryCLIService is inited.
2014-05-16 13:51:49,310 INFO  [main]: service.AbstractService (AbstractService.java:init(89))
- Service:HiveServer2 is inited.
2014-05-16 13:51:49,310 INFO  [main]: service.AbstractService (AbstractService.java:start(104))
- Service:OperationManager is started.
2014-05-16 13:51:49,310 INFO  [main]: service.AbstractService (AbstractService.java:start(104))
- Service:SessionManager is started.
2014-05-16 13:51:49,310 INFO  [main]: service.AbstractService (AbstractService.java:start(104))
- Service:CLIService is started.
2014-05-16 13:51:49,472 INFO  [main]: hive.metastore (HiveMetaStoreClient.java:open(297))
- Trying to connect to metastore with URI thrift://byn001-1:9083
2014-05-16 13:51:49,476 INFO  [main]: hive.metastore (HiveMetaStoreClient.java:open(385))
- Connected to metastore.
2014-05-16 13:51:49,525 INFO  [main]: service.AbstractService (AbstractService.java:start(104))
- Service:ThriftBinaryCLIService is started.
2014-05-16 13:51:49,526 INFO  [main]: service.AbstractService (AbstractService.java:start(104))
- Service:HiveServer2 is started.
2014-05-16 13:51:49,701 INFO  [Thread-6]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(88))
- ThriftBinaryCLIService listening on 0.0.0.0/0.0.0.0:10000
{code}

> Race condition when starting all services causing Hive service check to fail
> ----------------------------------------------------------------------------
>
>                 Key: AMBARI-5800
>                 URL: https://issues.apache.org/jira/browse/AMBARI-5800
>             Project: Ambari
>          Issue Type: Bug
>    Affects Versions: 1.6.0
>         Environment: SLES11
> ambari-server-1.6.0-39
> hive-0.13.0.2.1.2.0-402
>            Reporter: Sudhir Prakash
>            Priority: Critical
>
> # I performed an install on a 7 node cluster
> # During the install, I noticed that the Hive service check failed with the error: {{Test
connectivity to hive server Connection to byn001-1 on port 10000 failed: [Errno 111] Connection
refused}}
> # I proceeded through the rest of the install wizard
> # Stop All
> # Start All and noticed the same error again
> I retried stop all/start all this time monitoring the Ambari start progess, the Hive
Server2 logs, and a netstat of port 10000. What I noticed is that immediately after the start
Hive is issued, the service check is run and fails. However, it takes about 55 seconds for
HiveServer2 to actually start and claim port 10000. 
> The start up sequence needs to be modified to wait for Hive to finish starting before
running the service check.
> This issue is easily reproducible and has been seen by multiple people there.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message