ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayesh Thakrar <j_thak...@yahoo.com>
Subject Re: Kafka broker metrics not appearing in REST API
Date Thu, 07 May 2015 03:54:56 GMT
Sure Harsha, here it is -
auto.leader.rebalance.enable=truekafka.ganglia.metrics.port=8671socket.send.buffer.bytes=104857600message.max.bytes=104857600num.network.threads=3log.segment.bytes=1073741824kafka.ganglia.metrics.host=localhostlog.cleanup.interval.mins=10replica.lag.time.max.ms=10000fetch.purgatory.purge.interval.requests=10000kafka.ganglia.metrics.group=kafkareplica.lag.max.messages=4000port=9092log.dirs=/localpart0/kafka/datalog.retention.bytes=-1producer.purgatory.purge.interval.requests=10000log.flush.scheduler.interval.ms=3000default.replication.factor=3replica.high.watermark.checkpoint.interval.ms=5000zookeeper.connect=dtord01flm03p.dc.dotomi.net:2181,dtord01flm01p.dc.dotomi.net:2181,dtord01flm02p.dc.dotomi.net:2181controlled.shutdown.retry.backoff.ms=5000kafka.timeline.metrics.host=dtord01flm03p.dc.dotomi.netkafka.timeline.metrics.reporter.sendInterval=5900num.partitions=1log.flush.interval.messages=10000controller.socket.timeout.ms=30000zookeeper.sync.time.ms=2000kafka.timeline.metrics.reporter.enabled=truecontrolled.shutdown.max.retries=3replica.fetch.wait.max.ms=500controlled.shutdown.enable=truelog.roll.hours=168replica.socket.receive.buffer.bytes=104857600kafka.ganglia.metrics.reporter.enabled=truezookeeper.connection.timeout.ms=6000replica.fetch.max.bytes=114000000num.replica.fetchers=1socket.request.max.bytes=104857600kafka.timeline.metrics.maxRowCacheSize=10000kafka.timeline.metrics.port=6188queued.max.requests=500broker.id=0socket.receive.buffer.bytes=104857600controller.message.queue.size=10num.io.threads=8log.flush.interval.ms=3000kafka.metrics.reporters=org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporterreplica.fetch.min.bytes=1replica.socket.timeout.ms=30000zookeeper.session.timeout.ms=30000auto.create.topics.enable=falselog.index.size.max.bytes=104857600host.name=dtord01flm01p.dc.dotomi.netlog.index.interval.bytes=4096log.retention.hours=48


      From: Sriharsha Chintalapani <schintalapani@hortonworks.com>
 To: "user@ambari.apache.org" <user@ambari.apache.org>; Jayesh Thakrar <j_thakrar@yahoo.com>

 Sent: Wednesday, May 6, 2015 10:50 PM
 Subject: Re: Kafka broker metrics not appearing in REST API
   
#yiv7951455516 body{font-family:Helvetica, Arial;font-size:13px;}If possible post /etc/kafka/conf/server.properties
file. The only issue might be there are no metrics reporters registered
-- 
Harsha




On May 6, 2015 at 8:43:22 PM, Jayesh Thakrar (j_thakrar@yahoo.com) wrote:

There is no apparent issue in Kafka.Its running fine and data is flowing in and out without
any issue.
My topic partitions are all in-sync also.I think its more likely an issue in extracting the
metrics out of the metrics store, but I am not so familiar with the architecture.
[jthakrar@dtord01flm01p ~]$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
dtord01flm01p --describeTopic:DmmAccess_v12     PartitionCount:5        ReplicationFactor:3
    Configs:        Topic: DmmAccess_v12    Partition: 0    Leader: 8       Replicas:
8,15,16       Isr: 15,16,8        Topic: DmmAccess_v12    Partition: 1    Leader:
2       Replicas: 2,16,9        Isr: 16,9,2        Topic: DmmAccess_v12    Partition:
2    Leader: 3       Replicas: 3,14,10       Isr: 3,14,10        Topic: DmmAccess_v12
   Partition: 3    Leader: 4       Replicas: 4,18,21       Isr: 21,18,4     
  Topic: DmmAccess_v12    Partition: 4    Leader: 5       Replicas: 5,19,20    
  Isr: 19,5,20Topic:slimlogs_v1       PartitionCount:3        ReplicationFactor:3
    Configs:        Topic: slimlogs_v1      Partition: 0    Leader: 12      Replicas:
12,2,3        Isr: 12,3,2        Topic: slimlogs_v1      Partition: 1    Leader:
14      Replicas: 14,3,4        Isr: 14,3,4        Topic: slimlogs_v1      Partition:
2    Leader: 15      Replicas: 15,4,5        Isr: 15,4,5


From: Siddharth Wagle <swagle@hortonworks.com>
To: "user@ambari.apache.org" <user@ambari.apache.org>; Jayesh Thakrar <j_thakrar@yahoo.com>
Cc: Sriharsha Chintalapani <schintalapani@hortonworks.com>
Sent: Wednesday, May 6, 2015 10:30 PM
Subject: Re: Kafka broker metrics not appearing in REST API

Hi Jayesh,
There is little to go by as far as debugging info here since based on what you observed host
and cluster metrics are being reported.
Could you take a look at Kafka logs and check for exception if any?
BR,Sid




From: Jayesh Thakrar <j_thakrar@yahoo.com>
Sent: Wednesday, May 06, 2015 8:07 PM
To: Siddharth Wagle; user@ambari.apache.org
Subject: Re: Kafka broker metrics not appearing in REST API Hi Siddharth,

Yes, I am using Ambari 2.0 with Ambari Metrics service.The interesting thing is that I got
them for some time and not anymore.And I also know that the metrics are being collected since
i can see them on the dashboard.Any pointer for troubleshooting?
And btw, it would be nice to have a count of messages received and not a computed metric count
/ min.TSDB does a good job of giving me cumulative and rate-per-sec graphs and numbers.
Thanks in advance,Jayesh
From: Siddharth Wagle <swagle@hortonworks.com>
To: "user@ambari.apache.org" <user@ambari.apache.org>; Jayesh Thakrar <j_thakrar@yahoo.com>
Sent: Wednesday, May 6, 2015 10:03 PM
Subject: Re: Kafka broker metrics not appearing in REST API

Hi Jayesh,
Are you using Ambari 2.0 with Ambari Metrics service?
BR,Sid


From: Jayesh Thakrar <j_thakrar@yahoo.com>
Sent: Wednesday, May 06, 2015 7:53 PM
To: user@ambari.apache.org
Subject: Kafka broker metrics not appearing in REST API Hi,
I have installed 2 clusters with Ambari and Storm and Kafka.After the install, I was able
to get metrics for both Storm and Kafka via REST API.This worked fine for a week, but since
the past 2 days, I have not been getting Kafka metrics.
I need the metrics to push to an OpenTSDB cluster.I do get host metrics and Nimbus metrics
but not KAFKA_BROKER metrics.

I did have maintenance turned on for some time, but maintenance is turned off now.
[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/NIMBUS?fields=metrics'{ 
"href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/NIMBUS?fields=metrics", 
"ServiceComponentInfo" : {    "cluster_name" : "ord_flume_kafka_prod",    "component_name"
: "NIMBUS",    "service_name" : "STORM"  },  "metrics" : {    "storm" : {      "nimbus"
: {        "freeslots" : 54.0,        "supervisors" : 27.0,        "topologies"
: 0.0,        "totalexecutors" : 0.0,        "totalslots" : 54.0,        "totaltasks"
: 0.0,        "usedslots" : 0.0      }    }  }}
[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/KAFKA_BROKER?fields=metrics'{ 
"href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/KAFKA_BROKER?fields=metrics", 
"ServiceComponentInfo" : {    "cluster_name" : "ord_flume_kafka_prod",    "component_name"
: "KAFKA_BROKER",    "service_name" : "KAFKA"  }}
[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/SUPERVISOR?fields=metrics'{ 
"href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/SUPERVISOR?fields=metrics", 
"ServiceComponentInfo" : {    "cluster_name" : "ord_flume_kafka_prod",    "component_name"
: "SUPERVISOR",    "service_name" : "STORM"  }






  
Mime
View raw message