ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Boesch <java...@gmail.com>
Subject Re: Resolved blueprints topology cluster is having long delays "Looking for keyName .."
Date Fri, 16 Oct 2015 02:59:24 GMT
Digging more the Ambari agents are not registered with the driver.  I can
not find documentation on how to cause the agents (which *are *running on
each of the hosts) register themselves with the driver.  Tips appreciated.


2015-10-15 19:36 GMT-07:00 Stephen Boesch <javadba@gmail.com>:

> I neglected to mention: the ambari console shows   the attempted cluster
> launch as   "Logical REquest: Provision Cluster <name>:  Not started"
>
>
> 1 Background Operation Running
> Operations
> Start Time
> Duration
> Show: All (1)Pending (1)In Progress (0)Failed (0)Success (0)Aborted (0)Timedout
> (0)
>  Logical Request: Provision Cluster 'eazcluster'
> <http://10.103.177.55:8080/#>
> Not started
>
> 2015-10-15 19:33 GMT-07:00 Stephen Boesch <javadba@gmail.com>:
>
>> We have a blueprints defined cluster that has an accepted topology:
>>
>> ambari-server.log" 6839L, 998638C
>> 16 Oct 2015 01:31:40,374  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type hdfs-site is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,374  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type hadoop-env is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,375  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type hive-site is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,375  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type yarn-site is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,375  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type oozie-site is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,376  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type webhcat-site is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,376  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type cluster-env is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,376  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type mapred-site is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,377  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type core-site is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,377  INFO [pool-2-thread-1] AmbariContext:375 -
>> Config type oozie-env is resolved in the cluster config.
>> 16 Oct 2015 01:31:40,377  INFO [pool-2-thread-1] AmbariContext:381 - All
>> required configuration types are in the* TOPOLOGY_RESOLVED state.
>> Blueprint deployment can now continue.*
>> 16 Oct 2015 01:31:40,377  INFO [pool-2-thread-1] TopologyManager:558 -
>> TopologyManager.ConfigureClusterTask: Exiting
>>
>>
>> We then have a few messages that seem healthy:
>>
>> 16 Oct 2015 01:32:00,945  INFO [Thread-25]
>> AbstractPoolBackedDataSource:462 - Initializing c3p0 pool...
>> 16 Oct 2015 01:32:00,985  INFO [Thread-25] JobStoreTX:861 - Freed 0
>> triggers from 'acquired' / 'blocked' state.
>> 16 Oct 2015 01:32:00,992  INFO [Thread-25] JobStoreTX:871 - Recovering 0
>> jobs that were in-progress at the time of the last shut-down.
>> 16 Oct 2015 01:32:00,992  INFO [Thread-25] JobStoreTX:884 - Recovery
>> complete.
>> 16 Oct 2015 01:32:00,998  INFO [Thread-25] QuartzScheduler:575 -
>> Scheduler ExecutionScheduler_$_NON_CLUSTERED started.
>> 16 Oct 2015 01:33:34,658  INFO [qtp-client-72] JvmInformation:446 -
>> Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed
>> OOPs and Concurrent Mark-and-Sweep GC
>> 16 Oct 2015 01:33:34,928  INFO [qtp-client-72]
>> TimelineMetricCacheProvider:73 - ..
>>
>> However we then have a series of LONG (multiple minutes) delays, in
>> between which there are some "Looking for keyname <foo>" :
>>
>>
>> 016.novalocal/10.103.177.55-9ac6f6e0-9d4a-447e-b3ee-53926d6e16bf
>> 16 Oct 2015 01:33:35,088  INFO [qtp-client-76] PersistKeyValueService:82
>> - Looking for keyName CLUSTER_CURRENT_STATUS
>> 16 Oct 2015 01:33:36,272  INFO [qtp-client-72] PersistKeyValueService:82
>> - Looking for keyName user-pref-admin-dashboard
>> 16 Oct 2015 01:33:43,550  INFO [qtp-client-72] PersistKeyValueService:82
>> - Looking for keyName time-range-service-HDFS
>> 16 Oct 2015 01:33:44,995  WARN [pool-7-thread-1] JMXPropertyProvider:202
>> - Unable to get JMX metrics.  No host name for NAMENODE
>> 16 Oct 2015 01:34:00,930  INFO [qtp-client-72] PersistKeyValueService:82
>> - Looking for keyName admin-settings-show-bg-admin
>> 16 Oct 2015 01:34:01,010  INFO [qtp-client-76] PersistKeyValueService:82
>> - Looking for keyName hostPopup-pagination-displayLength-admin
>> 16 Oct 2015 01:36:34,123  INFO [qtp-client-82] PersistKeyValueService:82
>> - Looking for keyName time-range-service-AMBARI_METRICS
>> 16 Oct 2015 01:36:42,903  INFO [qtp-client-125] PersistKeyValueService:82
>> - Looking for keyName time-range-service-HDFS
>> 16 Oct 2015 01:36:44,138  WARN [pool-7-thread-35] JMXPropertyProvider:202
>> - Unable to get JMX metrics.  No host name for NAMENODE
>> 16 Oct 2015 01:36:57,590  INFO [qtp-client-75] PersistKeyValueService:82
>> - Looking for keyName admin-settings-show-bg-admin
>> 16 Oct 2015 01:48:10,901  INFO [qtp-client-192] PersistKeyValueService:82
>> - Looking for keyName admin-settings-show-bg-admin
>> 16 Oct 2015 01:53:06,487  INFO [qtp-client-214] PersistKeyValueService:82
>> - Looking for keyName user-pref-admin-dashboard
>>
>>
>>
>> After that last message there was another 15 minute delay and then
>> nothing (another sixty minutes has passed)..
>>
>> How to determine what is happening here and why?
>>
>>
>>
>

Mime
View raw message