Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Tue, 5 May 2015 18:18:01 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12788133.1428103029000.5762.1430849881812@Atlassian.JIRA>
In-Reply-To: <JIRA.12788133.1428103029000@Atlassian.JIRA>
References: <JIRA.12788133.1428103029000@Atlassian.JIRA>
 <JIRA.12788133.1428103029326@arcas>
Subject: [jira] [Commented] (YARN-3448) Add Rolling Time To Lives Level DB
 Plugin Capabilities
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528969#comment-14528969 ] 

Hadoop QA commented on YARN-3448:
---------------------------------

\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  14m 42s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 4 new or modified test files. |
| {color:green}+1{color} | javac |   7m 34s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |   9m 35s | There were no new javadoc warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   1m 40s | There were no new checkstyle issues. |
| {color:green}+1{color} | whitespace |   0m  2s | The patch has no lines that end in whitespace. |
| {color:green}+1{color} | install |   1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   2m 50s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native |   3m 14s | Pre-build of native portion |
| {color:green}+1{color} | yarn tests |   0m 22s | Tests passed in hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests |   3m 26s | Tests passed in hadoop-yarn-server-applicationhistoryservice. |
| {color:green}+1{color} | hdfs tests |   0m 15s | Tests passed in hadoop-hdfs-client. |
| | |  46m 17s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12730542/YARN-3448.16.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9b01f81 |
| hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-yarn-api.txt |
| hadoop-yarn-server-applicationhistoryservice test log | https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt |
| hadoop-hdfs-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-hdfs-client.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7710/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7710/console |


This message was automatically generated.

> Add Rolling Time To Lives Level DB Plugin Capabilities
> ------------------------------------------------------
>
>                 Key: YARN-3448
>                 URL: https://issues.apache.org/jira/browse/YARN-3448
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>         Attachments: YARN-3448.1.patch, YARN-3448.10.patch, YARN-3448.12.patch, YARN-3448.13.patch, YARN-3448.14.patch, YARN-3448.15.patch, YARN-3448.16.patch, YARN-3448.2.patch, YARN-3448.3.patch, YARN-3448.4.patch, YARN-3448.5.patch, YARN-3448.7.patch, YARN-3448.8.patch, YARN-3448.9.patch
>
>
> For large applications, the majority of the time in LeveldbTimelineStore is spent deleting old entities record at a time. An exclusive write lock is held during the entire deletion phase which in practice can be hours. If we are to relax some of the consistency constraints, other performance enhancing techniques can be employed to maximize the throughput and minimize locking time.
> Split the 5 sections of the leveldb database (domain, owner, start time, entity, index) into 5 separate databases. This allows each database to maximize the read cache effectiveness based on the unique usage patterns of each database. With 5 separate databases each lookup is much faster. This can also help with I/O to have the entity and index databases on separate disks.
> Rolling DBs for entity and index DBs. 99.9% of the data are in these two sections 4:1 ration (index to entity) at least for tez. We replace DB record removal with file system removal if we create a rolling set of databases that age out and can be efficiently removed. To do this we must place a constraint to always place an entity's events into it's correct rolling db instance based on start time. This allows us to stitching the data back together while reading and artificial paging.
> Relax the synchronous writes constraints. If we are willing to accept losing some records that we not flushed in the operating system during a crash, we can use async writes that can be much faster.
> Prefer Sequential writes. sequential writes can be several times faster than random writes. Spend some small effort arranging the writes in such a way that will trend towards sequential write performance over random write performance.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)