hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1394) Uploads sometimes fall to 0 requests/second (Binding up on HLog#append?)
Date Tue, 12 May 2009 06:10:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708338#action_12708338

stack commented on HBASE-1394:

Thinking more, the log split involves reading in a regionservers commit logs and dividing
up the edits by region.   I think we can have an HLog that opens a pool of N commit logs now
with the logger distributing edits over the pool of log writers.  What it would mean is that
edits while written in order, as we spray them across N files, they could be processed out
of order as we read them in; e.g. If 3 logs in the pool, let edit A go into file 1, edit B
into file 2 and edit C into file 3 and so one.  Since our current basic split process reads
a file at a time, we'd read file 1 and find edits A, then D, and so on.  Processing these
files, serially, we'd write the region-scoped edits into new SequenceFiles -- not sorted MapFiles
or hfiles -- of one per region.   These files are then picked up on region open and the edits
inserted into memcache (We read the log on the open of each individual store -- which seems
profligate).  The insertion into memcache will put them back into the 'right' order -- newer
edits will sort before older, etc.  The only place I see there possibly being a problem is
if you insert a KV with one value and then afterward insert a  KV with exact same r/c/ts coordinates
but with different value.  Its possible that this ordering will not be honored on crash. 
Is that OK?

> Uploads sometimes fall to 0 requests/second (Binding up on HLog#append?)
> ------------------------------------------------------------------------
>                 Key: HBASE-1394
>                 URL: https://issues.apache.org/jira/browse/HBASE-1394
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Critical
>             Fix For: 0.20.0
>         Attachments: loghowlongittookappendinghlog.patch
> Trying to figure why rate sometimes goes to zero.
> Studying the reginoserver, HLog#append looks like a possible culprit.
> {code}
> "IPC Server handler 7 on 60021" daemon prio=10 tid=0x000000004057dc00 nid=0x1bc4 in Object.wait()
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:2964)
>         - locked <0x00007f9e3e449ff0> (a java.util.LinkedList)
>         - locked <0x00007f9e3e449e18> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
>         at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
>         at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
>         at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
>         - locked <0x00007f9e3e449e18> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
>         at java.io.DataOutputStream.write(DataOutputStream.java:90)
>         - locked <0x00007f9e434e5588> (a org.apache.hadoop.fs.FSDataOutputStream)
>         at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1020)
>         - locked <0x00007f9e434e55c0> (a org.apache.hadoop.io.SequenceFile$Writer)
>         at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:984)
>         - locked <0x00007f9e434e55c0> (a org.apache.hadoop.io.SequenceFile$Writer)
>         at org.apache.hadoop.hbase.regionserver.HLog.doWrite(HLog.java:565)
>         at org.apache.hadoop.hbase.regionserver.HLog.append(HLog.java:521)
>         - locked <0x00007f9dfa376f70> (a java.lang.Object)
>         at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1777)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1348)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1289)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1727)
>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
> {code}

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message