ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Boesch <java...@gmail.com>
Subject Re: Resolved blueprints topology cluster is having long delays "Looking for keyName .."
Date Fri, 16 Oct 2015 02:36:42 GMT
I neglected to mention: the ambari console shows   the attempted cluster
launch as   "Logical REquest: Provision Cluster <name>:  Not started"


1 Background Operation Running
Operations
Start Time
Duration
Show: All (1)Pending (1)In Progress (0)Failed (0)Success (0)Aborted (0)Timedout
(0)
 Logical Request: Provision Cluster 'eazcluster'
<http://10.103.177.55:8080/#>
Not started

2015-10-15 19:33 GMT-07:00 Stephen Boesch <javadba@gmail.com>:

> We have a blueprints defined cluster that has an accepted topology:
>
> ambari-server.log" 6839L, 998638C
> 16 Oct 2015 01:31:40,374  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type hdfs-site is resolved in the cluster config.
> 16 Oct 2015 01:31:40,374  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type hadoop-env is resolved in the cluster config.
> 16 Oct 2015 01:31:40,375  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type hive-site is resolved in the cluster config.
> 16 Oct 2015 01:31:40,375  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type yarn-site is resolved in the cluster config.
> 16 Oct 2015 01:31:40,375  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type oozie-site is resolved in the cluster config.
> 16 Oct 2015 01:31:40,376  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type webhcat-site is resolved in the cluster config.
> 16 Oct 2015 01:31:40,376  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type cluster-env is resolved in the cluster config.
> 16 Oct 2015 01:31:40,376  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type mapred-site is resolved in the cluster config.
> 16 Oct 2015 01:31:40,377  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type core-site is resolved in the cluster config.
> 16 Oct 2015 01:31:40,377  INFO [pool-2-thread-1] AmbariContext:375 -
> Config type oozie-env is resolved in the cluster config.
> 16 Oct 2015 01:31:40,377  INFO [pool-2-thread-1] AmbariContext:381 - All
> required configuration types are in the* TOPOLOGY_RESOLVED state.
> Blueprint deployment can now continue.*
> 16 Oct 2015 01:31:40,377  INFO [pool-2-thread-1] TopologyManager:558 -
> TopologyManager.ConfigureClusterTask: Exiting
>
>
> We then have a few messages that seem healthy:
>
> 16 Oct 2015 01:32:00,945  INFO [Thread-25]
> AbstractPoolBackedDataSource:462 - Initializing c3p0 pool...
> 16 Oct 2015 01:32:00,985  INFO [Thread-25] JobStoreTX:861 - Freed 0
> triggers from 'acquired' / 'blocked' state.
> 16 Oct 2015 01:32:00,992  INFO [Thread-25] JobStoreTX:871 - Recovering 0
> jobs that were in-progress at the time of the last shut-down.
> 16 Oct 2015 01:32:00,992  INFO [Thread-25] JobStoreTX:884 - Recovery
> complete.
> 16 Oct 2015 01:32:00,998  INFO [Thread-25] QuartzScheduler:575 - Scheduler
> ExecutionScheduler_$_NON_CLUSTERED started.
> 16 Oct 2015 01:33:34,658  INFO [qtp-client-72] JvmInformation:446 -
> Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed
> OOPs and Concurrent Mark-and-Sweep GC
> 16 Oct 2015 01:33:34,928  INFO [qtp-client-72]
> TimelineMetricCacheProvider:73 - ..
>
> However we then have a series of LONG (multiple minutes) delays, in
> between which there are some "Looking for keyname <foo>" :
>
>
> 016.novalocal/10.103.177.55-9ac6f6e0-9d4a-447e-b3ee-53926d6e16bf
> 16 Oct 2015 01:33:35,088  INFO [qtp-client-76] PersistKeyValueService:82 -
> Looking for keyName CLUSTER_CURRENT_STATUS
> 16 Oct 2015 01:33:36,272  INFO [qtp-client-72] PersistKeyValueService:82 -
> Looking for keyName user-pref-admin-dashboard
> 16 Oct 2015 01:33:43,550  INFO [qtp-client-72] PersistKeyValueService:82 -
> Looking for keyName time-range-service-HDFS
> 16 Oct 2015 01:33:44,995  WARN [pool-7-thread-1] JMXPropertyProvider:202 -
> Unable to get JMX metrics.  No host name for NAMENODE
> 16 Oct 2015 01:34:00,930  INFO [qtp-client-72] PersistKeyValueService:82 -
> Looking for keyName admin-settings-show-bg-admin
> 16 Oct 2015 01:34:01,010  INFO [qtp-client-76] PersistKeyValueService:82 -
> Looking for keyName hostPopup-pagination-displayLength-admin
> 16 Oct 2015 01:36:34,123  INFO [qtp-client-82] PersistKeyValueService:82 -
> Looking for keyName time-range-service-AMBARI_METRICS
> 16 Oct 2015 01:36:42,903  INFO [qtp-client-125] PersistKeyValueService:82
> - Looking for keyName time-range-service-HDFS
> 16 Oct 2015 01:36:44,138  WARN [pool-7-thread-35] JMXPropertyProvider:202
> - Unable to get JMX metrics.  No host name for NAMENODE
> 16 Oct 2015 01:36:57,590  INFO [qtp-client-75] PersistKeyValueService:82 -
> Looking for keyName admin-settings-show-bg-admin
> 16 Oct 2015 01:48:10,901  INFO [qtp-client-192] PersistKeyValueService:82
> - Looking for keyName admin-settings-show-bg-admin
> 16 Oct 2015 01:53:06,487  INFO [qtp-client-214] PersistKeyValueService:82
> - Looking for keyName user-pref-admin-dashboard
>
>
>
> After that last message there was another 15 minute delay and then nothing
> (another sixty minutes has passed)..
>
> How to determine what is happening here and why?
>
>
>

Mime
View raw message