ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Wagle <swa...@hortonworks.com>
Subject Re: Kafka broker metrics not appearing in REST API
Date Thu, 07 May 2015 03:49:25 GMT
Since you say Metric collector crashed, one possibility is that there less physical memory
on the metric collector host than what is configured for hbase.

Memory settings are present in, ams-hbase-env.

-Sid


Sent by Outlook<http://taps.io/outlookmobile> for Android



On Wed, May 6, 2015 at 8:43 PM -0700, "Jayesh Thakrar" <j_thakrar@yahoo.com<mailto:j_thakrar@yahoo.com>>
wrote:

There is no apparent issue in Kafka.
Its running fine and data is flowing in and out without any issue.

My topic partitions are all in-sync also.
I think its more likely an issue in extracting the metrics out of the metrics store, but I
am not so familiar with the architecture.

[jthakrar@dtord01flm01p ~]$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
dtord01flm01p --describe
Topic:DmmAccess_v12     PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: DmmAccess_v12    Partition: 0    Leader: 8       Replicas: 8,15,16       Isr:
15,16,8
        Topic: DmmAccess_v12    Partition: 1    Leader: 2       Replicas: 2,16,9        Isr:
16,9,2
        Topic: DmmAccess_v12    Partition: 2    Leader: 3       Replicas: 3,14,10       Isr:
3,14,10
        Topic: DmmAccess_v12    Partition: 3    Leader: 4       Replicas: 4,18,21       Isr:
21,18,4
        Topic: DmmAccess_v12    Partition: 4    Leader: 5       Replicas: 5,19,20       Isr:
19,5,20
Topic:slimlogs_v1       PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: slimlogs_v1      Partition: 0    Leader: 12      Replicas: 12,2,3        Isr:
12,3,2
        Topic: slimlogs_v1      Partition: 1    Leader: 14      Replicas: 14,3,4        Isr:
14,3,4
        Topic: slimlogs_v1      Partition: 2    Leader: 15      Replicas: 15,4,5        Isr:
15,4,5



________________________________
From: Siddharth Wagle <swagle@hortonworks.com>
To: "user@ambari.apache.org" <user@ambari.apache.org>; Jayesh Thakrar <j_thakrar@yahoo.com>
Cc: Sriharsha Chintalapani <schintalapani@hortonworks.com>
Sent: Wednesday, May 6, 2015 10:30 PM
Subject: Re: Kafka broker metrics not appearing in REST API

Hi Jayesh,

There is little to go by as far as debugging info here since based on what you observed host
and cluster metrics are being reported.

Could you take a look at Kafka logs and check for exception if any?

BR,
Sid




________________________________
From: Jayesh Thakrar <j_thakrar@yahoo.com>
Sent: Wednesday, May 06, 2015 8:07 PM
To: Siddharth Wagle; user@ambari.apache.org
Subject: Re: Kafka broker metrics not appearing in REST API

Hi Siddharth,

Yes, I am using Ambari 2.0 with Ambari Metrics service.
The interesting thing is that I got them for some time and not anymore.
And I also know that the metrics are being collected since i can see them on the dashboard.
Any pointer for troubleshooting?

And btw, it would be nice to have a count of messages received and not a computed metric count
/ min.
TSDB does a good job of giving me cumulative and rate-per-sec graphs and numbers.

Thanks in advance,
Jayesh

________________________________
From: Siddharth Wagle <swagle@hortonworks.com>
To: "user@ambari.apache.org" <user@ambari.apache.org>; Jayesh Thakrar <j_thakrar@yahoo.com>
Sent: Wednesday, May 6, 2015 10:03 PM
Subject: Re: Kafka broker metrics not appearing in REST API

Hi Jayesh,

Are you using Ambari 2.0 with Ambari Metrics service?

BR,
Sid


________________________________
From: Jayesh Thakrar <j_thakrar@yahoo.com>
Sent: Wednesday, May 06, 2015 7:53 PM
To: user@ambari.apache.org
Subject: Kafka broker metrics not appearing in REST API

Hi,

I have installed 2 clusters with Ambari and Storm and Kafka.
After the install, I was able to get metrics for both Storm and Kafka via REST API.
This worked fine for a week, but since the past 2 days, I have not been getting Kafka metrics.

I need the metrics to push to an OpenTSDB cluster.
I do get host metrics and Nimbus metrics but not KAFKA_BROKER metrics.

I did have maintenance turned on for some time, but maintenance is turned off now.

[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/NIMBUS?fields=metrics'
{
  "href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/NIMBUS?fields=metrics",
  "ServiceComponentInfo" : {
    "cluster_name" : "ord_flume_kafka_prod",
    "component_name" : "NIMBUS",
    "service_name" : "STORM"
  },
  "metrics" : {
    "storm" : {
      "nimbus" : {
        "freeslots" : 54.0,
        "supervisors" : 27.0,
        "topologies" : 0.0,
        "totalexecutors" : 0.0,
        "totalslots" : 54.0,
        "totaltasks" : 0.0,
        "usedslots" : 0.0
      }
    }
  }
}

[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/KAFKA_BROKER?fields=metrics'
{
  "href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/KAFKA_BROKER?fields=metrics",
  "ServiceComponentInfo" : {
    "cluster_name" : "ord_flume_kafka_prod",
    "component_name" : "KAFKA_BROKER",
    "service_name" : "KAFKA"
  }
}

[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/SUPERVISOR?fields=metrics'
{
  "href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/SUPERVISOR?fields=metrics",
  "ServiceComponentInfo" : {
    "cluster_name" : "ord_flume_kafka_prod",
    "component_name" : "SUPERVISOR",
    "service_name" : "STORM"
  }






Mime
View raw message