hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shiwei Guo (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-4199) Minimize lock time in LeveldbTimelineStore.discardOldEntities
Date Tue, 22 Sep 2015 04:31:04 GMT
Shiwei Guo created YARN-4199:
--------------------------------

             Summary: Minimize lock time in LeveldbTimelineStore.discardOldEntities
                 Key: YARN-4199
                 URL: https://issues.apache.org/jira/browse/YARN-4199
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: timelineserver, yarn
            Reporter: Shiwei Guo


I current implementation, LeveldbTimelineStore.discardOldEntities holds a writeLock on deleteLock,
which will block other put operation, which eventually block the execution of YARN jobs(e.g.
TEZ). When there is lots of history jobs in timelinestore, the block time will be very long.
In our observation, it block all the TEZ jobs for several hours or longer. 

The possible solutions are:
- Optimize leveldb configuration,  so a full scan won't take long time.
- Take a snapshot of leveldb, and scan the snapshot, so we only need to hold lock while getSnapshot.
One question is that whether snapshot will take long time or not, cause I have no experience
with leveldb.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message