Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 994DF18D1C for ; Fri, 11 Dec 2015 17:48:52 +0000 (UTC) Received: (qmail 5225 invoked by uid 500); 11 Dec 2015 17:48:52 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 5163 invoked by uid 500); 11 Dec 2015 17:48:52 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 5154 invoked by uid 99); 11 Dec 2015 17:48:52 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Dec 2015 17:48:52 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2CA0AE05D3; Fri, 11 Dec 2015 17:48:52 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: sjlee@apache.org To: common-commits@hadoop.apache.org Message-Id: <51d51cf9f2f84bcd8a57cfe2ff2f7312@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: YARN-4341. add doc about timeline performance tool usage (Chang Li via sjlee) Date: Fri, 11 Dec 2015 17:48:52 +0000 (UTC) Repository: hadoop Updated Branches: refs/heads/trunk a48301791 -> e8964589d YARN-4341. add doc about timeline performance tool usage (Chang Li via sjlee) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e8964589 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e8964589 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e8964589 Branch: refs/heads/trunk Commit: e8964589de27a507c31203a87c3cbe0a8d68d3dd Parents: a483017 Author: Sangjin Lee Authored: Fri Dec 11 09:46:32 2015 -0800 Committer: Sangjin Lee Committed: Fri Dec 11 09:46:32 2015 -0800 ---------------------------------------------------------------------- hadoop-yarn-project/CHANGES.txt | 3 + .../src/site/markdown/TimelineServer.md | 78 ++++++++++++++++++++ 2 files changed, 81 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/e8964589/hadoop-yarn-project/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 32d76f4..a29194a 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -607,6 +607,9 @@ Release 2.8.0 - UNRELEASED YARN-4248. REST API for submit/update/delete Reservations. (curino) + YARN-4341. add doc about timeline performance tool usage (Chang Li via + sjlee) + OPTIMIZATIONS YARN-3339. TestDockerContainerExecutor should pull a single image and not http://git-wip-us.apache.org/repos/asf/hadoop/blob/e8964589/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md index 2048012..3e589d3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md @@ -26,6 +26,10 @@ The YARN Timeline Server * [Publishing of application specific data](#Publishing_of_application_specific_data) * [Timeline Server REST API](#Timeline_Server_REST_API_v1) * [Generic Data REST APIs](#GENERIC_DATA_REST_APIS) +* [Timelnine Server Performance Test Tool](#TIMELINE_SERVER_PERFORMANCE_TEST_TOOL) + * [Highlights](#HIGHLIGHTS) + * [Usage](#USAGE) + * [Sample Runs](#SAMPLE_RUNS) Overview --------- @@ -2033,3 +2037,77 @@ This hides details of other domains from an unauthorized caller. this failure *will not* result in an HTTP error code being retured. A status code of 200 will be returned —however, there will be an error code in the list of failed entities for each entity which could not be added. + + Timelnine Server Performance Test Tool +---------- +###Highlights + +The timeline server performance test tool helps measure timeline server's write performance. The test +launches SimpleEntityWriter mappers or JobHistoryFileReplay mappers to write timeline +entities to the timeline server. At the end, the transaction rate(ops/s) per mapper and the total transaction rate +will be measured and printed out. Running the test with SimpleEntityWriter mappers +will also measure and show the IO rate(KB/s) per mapper and the total IO rate. + +###Usage + +Mapper Types Description: + + 1. SimpleEntityWriter mapper + Each mapper writes a user-specified number of timeline entities + with a user-specified size to the timeline server. + + 2. JobHistoryFileReplay mapper + Each mapper replays jobhistory files under a specified directory + (both the jhist file and its corresponding conf.xml are required to + be present in order to be replayed. The number of mappers should be no more + than the number of jobhistory files). + Each mapper will get assigned some jobhistory files to replay. For each + job history file, a mapper will parse it to get jobinfo and then create + timeline entities. Each mapper also has the choice to write all the + timeline entities created at once or one at a time. + +Options: + + [-m ] number of mappers (default: 1) + [-v] timeline service version + [-mtype ] + 1. simple entity write mapper + 2. jobhistory files replay mapper + [-s <(KBs)test>] number of KB per put (mtype=1, default: 1 KB) + [-t] package sending iterations per mapper (mtype=1, default: 100) + [-d ] root path of job history files (mtype=2) + [-r ] (mtype=2) + 1. write all entities for a job in one put (default) + 2. write one entity at a time + +###Sample Runs + +Run SimpleEntityWriter test: + + bin/hadoop jar performanceTest.jar timelineperformance -m 4 -mtype 1 -s 3 -t 200 + +Example output of SimpleEntityWriter test : + + TRANSACTION RATE (per mapper): 20000.0 ops/s + IO RATE (per mapper): 60000.0 KB/s + TRANSACTION RATE (total): 80000.0 ops/s + IO RATE (total): 240000.0 KB/s + +Run JobHistoryFileReplay mapper test + + $ bin/hadoop jar performanceTest.jar timelineperformance -m 2 -mtype 2 -d /testInput -r 2 + +Example input of JobHistoryFileReplay mapper test: + + $ bin/hadoop fs -ls /testInput + /testInput/job_1.jhist + /testInput/job_1_conf.xml + /testInput/job_2.jhist + /testInput/job_2_conf.xml + +Eample output of JobHistoryFileReplay test: + + TRANSACTION RATE (per mapper): 4000.0 ops/s + IO RATE (per mapper): 0.0 KB/s + TRANSACTION RATE (total): 8000.0 ops/s + IO RATE (total): 0.0 KB/s