hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingjian Deng <koven2...@gmail.com>
Subject performance: HLog flush to disk each thread, can we decrease IO calls?
Date Wed, 29 Jun 2011 10:27:15 GMT
Hi:
    We found that the hlog sync to disk each time. When one thread exec
"doWrite(info, logKey, edit);", the others wait for "updateLock" in
HLog.java.
    Why not the others add their edits into a list and wait. When sync's
time, the whole list sync to disk once. I think it will decrease the IO
calls.

    So Maybe we will make two lists for edits. Each thread write to the
"waledits" and wait for "updateLock". Each thread can copy the "waledits" to
"flushedits" and flush the "flushedits" to
disk once it gets "updateLock".

    In my test, it can increase the write speed of 40%.

    Just see the HLog.patch.

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message