hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1394) Uploads sometimes fall to 0 requests/second (Binding up on HLog#append?)
Date Sun, 10 May 2009 21:56:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707845#action_12707845

stack commented on HBASE-1394:

Looking at log sizes, they are 110MB regularly in this test.  So, there is at least one block
transition inline.  That probably costs.  

So, here are some tests to try using HLog and HDFS only -- can be done without involving regionserver
-- just write a harness:

1. See if writing logs < hdfs block size improves things
2. In my measurements, it doesn't seem to be an issue but close of a log file during roll
is inline -- all writes stop while the close is being done.  Measure doing close out of band.
 Measure too opening the new log file before we need it so that its read to go rather than
hold up incoming edits while new file is opened.

I don't think the above will address the long time I'm seeing doing appends at certain times
during a bulk upload.  Talking with Ryan, maybe we need a pool of loggers where we scatter
edits over the pool -- if one is off waiting on a DataStreamer to catchup then that one write
will take some time rather than have all blocked.  This'll make the recovery harder in that
we'll have to sort regionservers's log pool but we're going to have to rewrite that anyways.

> Uploads sometimes fall to 0 requests/second (Binding up on HLog#append?)
> ------------------------------------------------------------------------
>                 Key: HBASE-1394
>                 URL: https://issues.apache.org/jira/browse/HBASE-1394
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>         Attachments: loghowlongittookappendinghlog.patch
> Trying to figure why rate sometimes goes to zero.
> Studying the reginoserver, HLog#append looks like a possible culprit.
> {code}
> "IPC Server handler 7 on 60021" daemon prio=10 tid=0x000000004057dc00 nid=0x1bc4 in Object.wait()
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:2964)
>         - locked <0x00007f9e3e449ff0> (a java.util.LinkedList)
>         - locked <0x00007f9e3e449e18> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
>         at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
>         at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
>         at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
>         - locked <0x00007f9e3e449e18> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
>         at java.io.DataOutputStream.write(DataOutputStream.java:90)
>         - locked <0x00007f9e434e5588> (a org.apache.hadoop.fs.FSDataOutputStream)
>         at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1020)
>         - locked <0x00007f9e434e55c0> (a org.apache.hadoop.io.SequenceFile$Writer)
>         at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:984)
>         - locked <0x00007f9e434e55c0> (a org.apache.hadoop.io.SequenceFile$Writer)
>         at org.apache.hadoop.hbase.regionserver.HLog.doWrite(HLog.java:565)
>         at org.apache.hadoop.hbase.regionserver.HLog.append(HLog.java:521)
>         - locked <0x00007f9dfa376f70> (a java.lang.Object)
>         at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1777)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1348)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1289)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1727)
>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
> {code}

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message