hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
Date Thu, 27 Oct 2011 22:10:33 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137613#comment-13137613
] 

Ted Yu commented on HBASE-4528:
-------------------------------

Found the following in output file for a failed test:
{code}
2011-10-27 14:25:15,878 DEBUG [10.246.204.28,52533,1319750687794.splitLogManagerTimeoutMonitor]
master.SplitLogManager$TimeoutMonitor(829): total tasks = 1 unassigned = 1
2011-10-27 14:25:16,126 INFO  [SplitLogWorker-10.246.204.28,52535,1319750687853] regionserver.SplitLogWorker(308):
worker 10.246.204.28,52535,1319750687853 done with task /hbase/splitlog/hdfs%3A%2F%2Flocalhost%3A52522%2Fuser%2Fzhihyu%2F.logs%2F10.246.204.28%2C52535%2C1319750687853%2F10.246.204.28%252C52535%252C1319750687853.1319750690240
in 5049ms
2011-10-27 14:25:16,126 ERROR [SplitLogWorker-10.246.204.28,52535,1319750687853] regionserver.SplitLogWorker(169):
unexpected error
java.lang.NullPointerException
  at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeThreads(DFSClient.java:3648)
  at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3691)
  at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3626)
  at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
  at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
  at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
  at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.close(SequenceFileLogWriter.java:177)
  at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:458)
  at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:352)
  at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:113)
  at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:266)
  at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:197)
  at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:165)
  at java.lang.Thread.run(Thread.java:680)
2011-10-27 14:25:16,126 INFO  [SplitLogWorker-10.246.204.28,52535,1319750687853] regionserver.SplitLogWorker(171):
SplitLogWorker 10.246.204.28,52535,1319750687853 exiting
2011-10-27 14:25:16,878 DEBUG [10.246.204.28,52533,1319750687794.splitLogManagerTimeoutMonitor]
master.SplitLogManager$TimeoutMonitor(829): total tasks = 1 unassigned = 1
{code}
I don't think NPE out of DFSClient.java for closing is critical.
The following patch allowed me to loop through TestDistributedLogSplitting on MacBook 9 times
without hitting test failure.
{code}
Index: src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
===================================================================
--- src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java	(revision 1190003)
+++ src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java	(working copy)
@@ -455,7 +455,11 @@
         }
         n++;
         WriterAndPath wap = (WriterAndPath)o;
-        wap.w.close();
+        try {
+          wap.w.close();
+        } catch (Exception e) {
+          LOG.debug("splitLogFileToTemp closing SequenceFileLogWriter", e);
+        }
         LOG.debug("Closed " + wap.p);
       }
       String msg = ("processed " + editsCount + " edits across " + n + " regions" +
{code}

                
> The put operation can release the rowlock before sync-ing the Hlog
> ------------------------------------------------------------------
>
>                 Key: HBASE-4528
>                 URL: https://issues.apache.org/jira/browse/HBASE-4528
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.94.0
>
>         Attachments: HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt,
appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, appendNoSyncPut6.txt,
appendNoSyncPut7.txt, appendNoSyncPut8.txt
>
>
> This allows for better throughput when there are hot rows. A single row update improves
from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message