hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1530) [Umbrella] Store, manage and serve per-framework application-timeline data
Date Wed, 12 Mar 2014 23:11:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932623#comment-13932623

Zhijie Shen commented on YARN-1530:

I've done some stress test about the timeline service. I've taken two steps:

1. I setup a yarn cluster of 5 nodes (1 master and 4 slaves). I ran 7200 mapreduce example
jobs with tez framework (which will post tez entities to the timeline service) in 10 hours.
The max concurrent job was 7, because it was not a big cluster with 32G memory only, but the
real concurrency should be higher because of multiple mappers/reducers. The workload was kept
almost full in 10 hours. All the job were succeeded, and the tez entities were stored in the
timeline store without exceptions. The leveldb based timeline store has grown to about 220MB
(not very big because of small example jobs).

2. Afterwards, I tested the the concurrent reads/writes together. On the write part, I did
the same thing as step 1. On the read part, I set up 4 timeline query clients, one on each
slave node. Each client starts 10 parallel threads to send requests to the timeline service
for 10 hours as well. Each client sent more than 6 million queries during the 10 hours with
the combination of three RESTful APIs (24+ million total for 4 clients). In general, the timeline
service was still working well. I just saw one query  was responded with not found exception,
and some other JVM warnings. The query of get entities takes 0.X on average while the query
of get entity/events take 0.0x.

Therefore, the timeline service with leveldb store works so far so good. I'll do more stress
testing with big entity, and update to you once I've some metrics.

> [Umbrella] Store, manage and serve per-framework application-timeline data
> --------------------------------------------------------------------------
>                 Key: YARN-1530
>                 URL: https://issues.apache.org/jira/browse/YARN-1530
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>         Attachments: application timeline design-20140108.pdf, application timeline design-20140116.pdf,
application timeline design-20140130.pdf, application timeline design-20140210.pdf
> This is a sibling JIRA for YARN-321.
> Today, each application/framework has to do store, and serve per-framework data all by
itself as YARN doesn't have a common solution. This JIRA attempts to solve the storage, management
and serving of per-framework data from various applications, both running and finished. The
aim is to change YARN to collect and store data in a generic manner with plugin points for
frameworks to do their own thing w.r.t interpretation and serving.

This message was sent by Atlassian JIRA

View raw message