ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Wagle <>
Subject Re: Posting Metrics to Ambari
Date Tue, 28 Jul 2015 02:50:17 GMT
For Step1, when you say exposing metrics through the Ambari REST API... are you talking about
the metrics collector REST API, or through the Ambari Server REST API?

Answer: Ambari REST API: Note that this is intended use because this is what ties the metrics
to you your cluster resources, example: You can query for say give me metrics for the active
Namenode only using Ambari's API.

Is SERVER_TIME the field that has to fall between startTime and endTime?

Yes. That is correct

There is nothing special about the query  you seem to have the fragments right, only this
is you are query for a large time window, AMS would not return data from METRIC_RECORD table
for a such a large time window it would try to find this in the aggregate table, METRIC_RECORD_MINUTE
or HOURLY. Try reducing you time range, also check the aggregate tables, the data should still
be present in those tables.

Precision params:


From: Bryan Bende <>
Sent: Monday, July 27, 2015 6:21 PM
Subject: Re: Posting Metrics to Ambari

Hi Jaimin,

For Step1, when you say exposing metrics through the Ambari REST API... are you talking about
the metrics collector REST API, or through the Ambari Server REST API?

I am able to see data through Phoenix, as an example:

|               METRIC_NAME                |                 HOSTNAME                 |  
            SERVER_TIME                |                  APP_ID                  |


| FlowFiles_Received_Last_5_mins           | localhost                          | 1438045869329
                           | NIFI                                     |

Then I try to use this API call:


and I get: {"metrics":[]}

Something must not be lining up with what I am sending over. Is SERVER_TIME the field that
has to fall between startTime and endTime?


On Mon, Jul 27, 2015 at 1:40 PM, Jaimin Jetly <<>>

Hi Bryan,

There are 2 steps in this that needs to be achieved.

STEP-1:  Exposing service metrics successfully through Ambari REST API

STEP-2:  Ambari UI displaying widgets comprised from newly exposed metrics via Ambari server.

As step-1 is pre-requisite to step-2, can you confirm that you were able to achieve step-1
(exposing service metrics successfully through Ambari REST API) ?

NOTE: /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/metrics.json are
the metrics specific to Ambari metrics service. If the new metrics that you want to expose
are related to any other service then please edit/create metrics.json file in that specific
service package and not in Ambari metrics service package. widgets.json also needs to be changed/added
in the same service package and not at /var/lib/ambari-server/resources/stacks/HDP/2.0.6/widgets.json
(unless you want to add system heatmaps for a stack that inherits HDP-2.0.6 stack).

-- Thanks


From: Bryan Bende <<>>
Sent: Sunday, July 26, 2015 2:10 PM

Subject: Re: Posting Metrics to Ambari

Hi Sid,

Thanks for the pointers about how to add a metric to the UI. Based on those instructions I
modified /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/metrics.json
and added the following based on the test metrics I posted:

"metrics/SmokeTest/FakeMetric": {

              "metric": "AMBARI_METRICS.SmokeTest.FakeMetric",

              "pointInTime": true,

              "temporal": true


>From digging around the filesystem there appears to be a widgets.json in /var/lib/ambari-server/resources/stacks/HDP/2.0.6/widgets.json.
It looks like this file only contained the definitions of the heatmaps, so I wasn't sure if
this was the right place, but just to see what happened I modified it as follows:

1) Added a whole new layout:

2) Added a heatmap for the test metric:

Then I restarted the HDP VM but I don't see anything in the UI under Metric Actions ->
Add, or under Heatmaps. Anything that seems completely wrong about what I did? Maybe I should
be going down the route of defining a new service type for system I will be sending metrics

Sorry to keep bothering with all these questions, I just don't have any previous experience
with Ambari.



On Sun, Jul 26, 2015 at 12:10 AM, Siddharth Wagle <<>>

The AMS API does not allow open ended queries so startTime and endTime are required fields,
the curl call should return the error code with the apt response.

If this doesn't happen please go ahead and file a Jira.

Using AMS through Ambari UI after getting the plumbing work with metrics.json completed would
be much easier. The AMS API does need some refinement. Jiras / Bugs are welcome.


From: Siddharth Wagle <<>>
Sent: Saturday, July 25, 2015 9:01 PM

Subject: Re: Posting Metrics to Ambari

No dev work need only need to modify metrics.json file and then add widget from UI.

Stack details:

UI specifics:


From: Bryan Bende <<>>
Sent: Saturday, July 25, 2015 7:10 PM
Subject: Re: Posting Metrics to Ambari

Quick update, I was able to connect with the phoenix 4.2.2 client and I did get results querying
by SERVER_TIME desc limit 10;

Now that I know the metrics are posting, I am less concerned about querying through the REST

Is there any way to get a custom metric added to the main page of Ambari? or does this require
development work?



On Sat, Jul 25, 2015 at 9:42 PM, Bryan Bende <<>>
Hi Sid,

Thanks for the suggestions. I turned on DEBUG for the metrics collector (had to do this through
the Ambari UI configs section) and now I can see some activity... When I post a metric I see:

01:30:18,372 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics] TimelineWebServices:270
- Storing metrics: {

  "metrics" : [ {

    "timestamp" : 1432075898000,

    "metricname" : "AMBARI_METRICS.SmokeTest.FakeMetric",

    "appid" : "amssmoketestfake",

    "hostname" : "localhost",

    "starttime" : 1432075898000,

    "metrics" : {

      "1432075898000" : 0.963781711428,

      "1432075899000" : 1.432075898E12


  } ]


01:30:18,372 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics] DefaultPhoenixDataSource:67
- Metric store connection url: jdbc:phoenix:localhost:61181:/hbase

01:30:18,376 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics] MutationState:361
- Sending 2 mutations for METRIC_RECORD with 8 key values of total size 925 bytes

01:30:18,380 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics] MutationState:436
- Total time for batch call of  2 mutations into METRIC_RECORD: 3 ms

01:30:18,381 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics] log:40 - RESPONSE
/ws/v1/timeline/metrics  200

So it looks like it posted successfully. Then I hit:


and I see...

01:31:16,952 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric]
ParallelIterators:412 - Guideposts: ]

01:31:16,953 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric]
ParallelIterators:481 - The parallelScans: [[{"timeRange":[0,1437874276946],"batch":-1,"startRow":"AMBARI_METRICS.SmokeTest.FakeMetric","stopRow":"AMBARI_METRICS.SmokeTest.FakeMetric\\x01","loadColumnFamiliesOnDemand":null,"totalColumns":1,"cacheBlocks":true,"families":{"0":["ALL"]},"maxResultSize":-1,"maxVersions":1,"filter":"","caching":-1}]]

01:31:16,953 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric]
BaseQueryPlan:243 - Iterator ready: MergeSortTopNResultIterator [limit=1, count=0, orderByColumns=[METRIC_NAME
DESC, SERVER_TIME DESC], ptr1=, ptr2=]

01:31:16,957 DEBUG [phoenix-1-thread-171] ParallelIterators:629 - Id: d0c9c381-f35f-48e6-b970-8b6d5997684b,
Time: 3ms, Scan: {"timeRange":[0,1437874276946],"batch":-1,"startRow":"AMBARI_METRICS.SmokeTest.FakeMetric","stopRow":"AMBARI_METRICS.SmokeTest.FakeMetric\\x01","loadColumnFamiliesOnDemand":null,"totalColumns":1,"cacheBlocks":true,"families":{"0":["ALL"]},"maxResultSize":-1,"maxVersions":1,"filter":"","caching":10000}

01:31:16,959 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric]
PhoenixHBaseAccessor:552 - Aggregate records size: 0

I'll see if I can get the phoenix client working and see what that returns.



On Fri, Jul 24, 2015 at 5:44 PM, Siddharth Wagle <<>>

Hi Bryan,

Few things you can do:

1. Turn on DEBUG mode by changing at, /etc/ambari-metrics-collector/conf/

This might reveal more info, I don't think we print every metrics received to the log in 2.0
or 2.1, I did add this option if TRACE is enabled to trunk recently.

2. Connect using Phoenix directly and you can do a SELECT query like this:

SELECT * from METRIC_RECORD WHERE METRIC_NAME = '<your-metric-name>' order by SERVER_TIME
desc limit 10;

Instructions for connecting to Phoenix:

3. What API call are you making to get metrics?

E.g.: http://<ams-collector>:6188/ws/v1/timeline/metrics?metricNames=<your-metric-name>&startTime=<epoch>&endTime=<epoch>&hostname=<hostname>


From: Bryan Bende <<>>
Sent: Friday, July 24, 2015 2:03 PM
Subject: Posting Metrics to Ambari

I'm interested in sending metrics to Ambari and I've been looking at the Metrics Collector
REST API described here:

I figured the easiest way to test it would be to get the latest HDP Sandbox... so I downloaded
and started it up. The Metrics Collector service wasn't running so I started it, and also
added port 6188 to the VM port forwarding. From there I used the example POST on the Wiki
page and made a successful POST which got a 200 response. After that I tried the query, but
could never get any results to come back.

I know this list is not specific to HDP, but I was wondering if anyone has any suggestions
as to what I can look at to figure out what is happening with the data I am posting.

I was watching the metrics collector log while posting and querying and didn't see any activity
besides the periodic aggregation.

Any suggestions would be greatly appreciated.



View raw message