Return-Path: Delivered-To: apmail-hadoop-hbase-dev-archive@minotaur.apache.org Received: (qmail 16276 invoked from network); 27 Jul 2009 03:05:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 27 Jul 2009 03:05:32 -0000 Received: (qmail 54618 invoked by uid 500); 27 Jul 2009 03:06:37 -0000 Delivered-To: apmail-hadoop-hbase-dev-archive@hadoop.apache.org Received: (qmail 54566 invoked by uid 500); 27 Jul 2009 03:06:37 -0000 Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-dev@hadoop.apache.org Received: (qmail 54551 invoked by uid 99); 27 Jul 2009 03:06:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 03:06:37 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 03:06:35 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id C4F57234C004 for ; Sun, 26 Jul 2009 20:06:14 -0700 (PDT) Message-ID: <2129317864.1248663974791.JavaMail.jira@brutus> Date: Sun, 26 Jul 2009 20:06:14 -0700 (PDT) From: "Andrew Purtell (JIRA)" To: hbase-dev@hadoop.apache.org Subject: [jira] Updated: (HBASE-1707) master fails to reassign region after failed deployment In-Reply-To: <2075991104.1248639254879.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-1707: ---------------------------------- Attachment: master+regionserver224.rar > master fails to reassign region after failed deployment > ------------------------------------------------------- > > Key: HBASE-1707 > URL: https://issues.apache.org/jira/browse/HBASE-1707 > Project: Hadoop HBase > Issue Type: Bug > Affects Versions: 0.19.1 > Reporter: Andrew Purtell > Attachments: master+regionserver224.rar > > > From a Trend 0.19.3 cluster: > All the crawlers kept on printing log like this around 11:00 (UTC). > ================================================ > 09/07/23 11:02:06 ERROR HBaseWriter:112 - org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server Some server for region , row '092501240723072009-0000000000000000308-10.3.134.228-d822aab6a9fdf4c216e397f00cf89d77', but failed after 11 attempts. > I think this is caused by regionserver problem; from regionserver log the key should be online after 10:44 > ========================================= > 2009-07-23 10:44:49,789 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,092501240723072009-0000000000000000308-10.3.134.228-a3e3be27683ba154539d1c055890e781,1248345884988: safeMode=false > 2009-07-23 10:44:49,789 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,092501240723072009-0000000000000000308-10.3.134.228-746d6fcc693a34536215d76dee0fabee,1248345884988 > 2009-07-23 10:44:49,863 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,092501240723072009-0000000000000000308-10.3.134.228-a3e3be27683ba154539d1c055890e781,1248345884988/694893409 available > 2009-07-23 10:44:52,736 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 0) > java.util.ConcurrentModificationException > at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100) > at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145) > at org.apache.hadoop.hbase.regionserver.HStore.getStorefilesIndexSize(HStore.java:2187) > at org.apache.hadoop.hbase.regionserver.HRegionServer.doMetrics(HRegionServer.java:948) > at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:338) > at java.lang.Thread.run(Thread.java:619) > ========================================= -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.