Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2A344F097 for ; Tue, 2 Apr 2013 21:03:19 +0000 (UTC) Received: (qmail 95205 invoked by uid 500); 2 Apr 2013 21:03:18 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 95145 invoked by uid 500); 2 Apr 2013 21:03:18 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 95136 invoked by uid 99); 2 Apr 2013 21:03:18 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Apr 2013 21:03:18 +0000 Date: Tue, 2 Apr 2013 21:03:18 +0000 (UTC) From: "Jeffrey Zhong (JIRA)" To: dev@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HBASE-8246) Backport HBase-6138 to 0.94 where SplitLogWorker exits due to ConcurrentModificationException MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Jeffrey Zhong created HBASE-8246: ------------------------------------ Summary: Backport HBase-6138 to 0.94 where SplitLogWorker exits due to ConcurrentModificationException Key: HBASE-8246 URL: https://issues.apache.org/jira/browse/HBASE-8246 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.6 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 0.94.7 Today we found the following error in our tests. Later I found we already fixed the issue in trunk. I think we should backpor the fix because the consequence of the issue is high and the fix isn't complicated. {code} 2013-04-01 21:23:21,864 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker ip-10-143-160-121.ec2.internal,60020,1364849529986 done with task /hbase/splitlog/hdfs%3A%2F%2Fip-10-137-16-140.ec2.internal%3A8020%2Fapps%2Fhbase%2Fdata%2F.logs%2Fip-10-137-20-188.ec2.internal%2C60020%2C1364849530779-splitting%2Fip-10-137-20-188.ec2.internal%252C60020%252C1364849530779.1364865556657 in 67129ms 2013-04-01 21:23:21,864 ERROR org.apache.hadoop.hbase.regionserver.SplitLogWorker: unexpected error java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100) at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$OutputSink.closeLogWriters(HLogSplitter.java:1279) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$OutputSink.finishWritingAndClose(HLogSplitter.java:1170) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:475) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:111) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:195) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:163) at java.lang.Thread.run(Thread.java:662) 2013-04-01 21:23:21,865 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker ip-10-143-160-121.ec2.internal,60020,1364849529986 exiting {code} The impact of this issue is that SplitLogWorker exits so does the region server recovering mechanism of HBase. If any RS failed after all SplitLogWorkers in te cluster exit due to the issue, you'll see a hang log splitting job and the failed RS won't be recovered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira