hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3378) a load test client that can replay a volume of history files
Date Tue, 31 Mar 2015 01:15:54 GMT

    [ https://issues.apache.org/jira/browse/YARN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387762#comment-14387762

Sangjin Lee commented on YARN-3378:

Looking into this bit more, I realize that converting this into something against the v.2
API is a little more involved than I originally thought. Basically it has to do with the distributed
timeline collectors and the fact we need to use non-existent apps to be able to pump in data
into the timeline service.

Some challenges:
- we need to trigger creating app-level timeline collectors for these fake apps without real
AMs: we may need to add RPC for the per-node timeline collector to trigger adding and removing
app-level timeline collectors
- when the app-level timeline collectors are added to the timeline collector manager, the
collector manager needs to talk to NM to get the context but NM won't know the context as
these are for non-existent apps
- we also need to remove these fake apps properly from the per-node timeline collectors, NMs,
and RMs
- we need a way to spray the traffic to various NMs

Another idea may be to create per-node timeline collectors in a special manner for these load
tests so that you can simply direct traffic to them. We'll need to make sure this mode is
removed or cleaned up properly before we merge this.

Thoughts? Comments?

> a load test client that can replay a volume of history files
> ------------------------------------------------------------
>                 Key: YARN-3378
>                 URL: https://issues.apache.org/jira/browse/YARN-3378
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
> It might be good to create a load test client that can replay a large volume of history
files into the timeline service. One can envision running such a load test client as a mapreduce
job and generate a fair amount of load. It would be useful to spot check correctness, and
more importantly observe performance characteristic.

This message was sent by Atlassian JIRA

View raw message