Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <26260818.1193251130993.JavaMail.jira@brutus>
Date: Wed, 24 Oct 2007 11:38:50 -0700 (PDT)
From: "Mukund Madhugiri (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-1942) Increase the concurrency of
 transaction logging to edits log
In-Reply-To: <32101259.1190700290842.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537381 ] 

Mukund Madhugiri commented on HADOOP-1942:
------------------------------------------

Please ignore my previous comment. The run did not actually go thru with 120 threads. It was run with 40 threads as well.

Here is new data from a 100 node benchmark run:

#1. TPS without this fix, 40 threads
CreateWrite: 1023
OpenRead: 46728
Rename: 1169
Delete: 962

#2. TPS with this fix, 40 threads
CreateWrite: 3533 (3 times better than #1)
OpenRead: 43243
Rename: 9090 (7 times better than #1)
Delete: 7142 (7 times better than #1)

#3. TPS with this fix, 120 threads
CreateWrite: 4138 (4 times better than #1)
OpenRead: 50251
Rename: 10097 (8 times better than #1)
Delete: 7210 (7 times better than #1)

> Increase the concurrency of transaction logging to edits log
> ------------------------------------------------------------
>
>                 Key: HADOOP-1942
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1942
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.15.0
>
>         Attachments: transactionLogSync.patch, transactionLogSync2.patch, transactionLogSync3.patch, transactionLogSync4.patch, transactionLogSync5.patch, transactionLogSync6.patch, transactionLogSync8.patch, transactionLogSync9.patch
>
>
> For some typical workloads, the throughput of the namenode is bottlenecked by the rate of transactions that are being logged into tghe edits log. In the current code, a batching scheme implies that all transactions do not have to incur a sync of the edits log to disk. However, the existing batch-ing scheme can be improved.
> One option is to keep two buffers associated with edits file. Threads write to the primary buffer while holding the FSNamesystem lock. Then the thread release the FSNamesystem lock, acquires a new lock called the syncLock, swaps buffers, and flushes the old buffer to the persistent store. Since the buffers are swapped, new transactions continue to get logged into the new buffer. (Of course, the new transactions cannot complete before this new buffer is sync-ed).
> This approach does a better job of batching syncs to disk, thus improving performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.