hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bingxue Qiu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5814) Add druid as storage backend in YARN Timeline Service
Date Wed, 16 Nov 2016 10:20:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670054#comment-15670054

Bingxue Qiu commented on YARN-5814:

Thanks [~gtCarrera9] for your suggestions:

1. For the writer design issues, We have implemented writer by kafka and a mr job (for HA)
pull data to the realtime nodes of druid. But I'm not so sure this method is also fit for
others. After all tranquility is more simple.  I will give the design of them later. we can
choose to implement one or both of them.

2. For the table design, it may not be fit for using timeline.entity table to hold general
timeline entities including container data in druid implementation. In HBase implementation,
we can store general timeline entities with column family in entity table and scan them by
rowkey. But druid is fixed schema column storage, if we need ad-hoc/agg in real-time, timeline.entity
table maybe a wide table with many columns. It would bring the data redundancy  and generate
many rows and increase cache miss. That's why we consider to add these tables but not timeline.entity
Please feel free to give your suggestions. Thanks!

>  Add druid as storage backend in YARN Timeline Service
> ------------------------------------------------------
>                 Key: YARN-5814
>                 URL: https://issues.apache.org/jira/browse/YARN-5814
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: ATSv2
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Bingxue Qiu
>         Attachments: Add-Druid-in-YARN-Timeline-Service.pdf
> h3. Introduction
> I propose to add druid as storage backend in YARN Timeline Service.
> We run more than 6000 applications and generate 450 million metrics daily in Alibaba
Clusters with thousands of nodes. We need to collect and store meta/events/metrics data, online
analyze the utilization reports of various dimensions and display the trends of allocation/usage
resources for cluster by joining and aggregating data. It helps us to manage and optimize
the cluster by tracking resource utilization.
> To achieve our goal we have changed to use druid as the storage instead of HBase and
have achieved sub-second OLAP performance in our production environment for few months. 
> h3. Analysis
> Currently YARN Timeline Service only supports aggregating metrics at a) flow level by
FlowRunCoprocessor and b) application level metrics aggregating by AppLevelTimelineCollector,
offline (time-based periodic) aggregation for flows/users/queues for reporting and analysis
is planned but not yet implemented. YARN Timeline Service chooses Apache HBase as the primary
storage backend. As we all know that HBase doesn't fit for OLAP.
>  For arbitrary exploration of data,such as online analyze the utilization reports of
various dimensions(Queue,Flow,Users,Application,CPU,Memory) by joining and aggregating data,
Druid's custom column format enables ad-hoc queries without pre-computation. The format also
enables fast scans on columns, which is important for good aggregation performance.
> To achieve our goal that support to online analyze the utilization reports of various
dimensions, display the variation trends of allocation/usage resources for cluster, and arbitrary
exploration of data, we propose to add druid storage and implement DruidWriter /DruidReader
in YARN Timeline Service.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message