Return-Path: X-Original-To: apmail-accumulo-notifications-archive@minotaur.apache.org Delivered-To: apmail-accumulo-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 219E918852 for ; Sat, 9 May 2015 00:31:00 +0000 (UTC) Received: (qmail 17470 invoked by uid 500); 9 May 2015 00:31:00 -0000 Delivered-To: apmail-accumulo-notifications-archive@accumulo.apache.org Received: (qmail 17432 invoked by uid 500); 9 May 2015 00:31:00 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 17415 invoked by uid 99); 9 May 2015 00:30:59 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 May 2015 00:30:59 +0000 Date: Sat, 9 May 2015 00:30:59 +0000 (UTC) From: "Josh Elser (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ACCUMULO-3787) Multiple locations for Metadata tablet MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ACCUMULO-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536025#comment-14536025 ] Josh Elser commented on ACCUMULO-3787: -------------------------------------- Root tablet rfile. Contents from {{accumulo rfile-info}} {noformat} !0;~ last:14d34e74806005f [] 175 false -> c6402.ambari.apache.org:9997 !0;~ last:14d34e748060080 [] 476 false -> c6402.ambari.apache.org:9997 !0< last:14d34e748060060 [] 177 false -> c6403.ambari.apache.org:9997 !0< last:14d34e748060080 [] 478 false -> c6402.ambari.apache.org:9997 ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 176 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 469 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 477 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 470 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 172 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 174 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 473 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 475 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 171 false -> ~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 472 false -> !0;~ loc:14d34e748060080 [] 258 false -> c6402.ambari.apache.org:9997 !0;~ ~tab:~pr [] 0 false -> !0< loc:14d34e748060065 [] 183 false -> c6401.ambari.apache.org:9997 !0< loc:14d34e748060080 [] 447 false -> c6402.ambari.apache.org:9997 !0< ~tab:~pr [] 0 false -> ~ !0;~ file:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0... TRUNCATED [] 175 false -> 1520,93 !0;~ file:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0... TRUNCATED [] 476 false -> 1963,117 !0;~ srv:compact [] 476 false -> 20 !0;~ srv:dir [] 0 false -> hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0/table_info !0;~ srv:flush [] 468 false -> 20 !0;~ srv:lock [] 476 false -> tservers/c6402.ambari.apache.org:9997/zlock-0000000001$14d34e748060080 !0;~ srv:time [] 468 false -> L808 !0< file:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0... TRUNCATED [] 177 false -> 570,2 !0< file:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0... TRUNCATED [] 478 false -> 638,4 !0< srv:compact [] 478 false -> 20 !0< srv:dir [] 0 false -> hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0/default_tablet !0< srv:flush [] 467 false -> 20 !0< srv:lock [] 478 false -> tservers/c6402.ambari.apache.org:9997/zlock-0000000001$14d34e748060080 !0< srv:time [] 467 false -> L1196 ~wal+c6402.ambari.apache.org:9997[14d34e748060080] log:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/wal/c6402... TRUNCATED [] 439 false -> ~wal+c6402.ambari.apache.org:9997[14d34e748060080] log:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/wal/c6402... TRUNCATED [] 436 false -> ~wal+c6403.ambari.apache.org:9997[14d34e748060072] log:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/wal/c6403... TRUNCATED [] 443 false -> unused ~wal+c6403.ambari.apache.org:9997[14d34e748060072] log:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/wal/c6403... TRUNCATED [] 443 false -> unused {noformat} > Multiple locations for Metadata tablet > -------------------------------------- > > Key: ACCUMULO-3787 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3787 > Project: Accumulo > Issue Type: Bug > Components: master, tserver > Reporter: Josh Elser > Assignee: Josh Elser > Priority: Blocker > Labels: 1.7.0_QA > Fix For: 1.7.0 > > > Running continuous ingest with agitation > {noformat} > 2015-05-08 23:59:44,381 [state.ZooTabletStateStore] DEBUG: Returning root tablet state: +r<<@(null,c6402.ambari.apache.org:9997[14d34e748060089],c6402.ambari.apache.org:9997[14d34e748060089]) > 2015-05-08 23:59:44,381 [master.Master] DEBUG: Root Table location State: +r<<@(null,c6402.ambari.apache.org:9997[14d34e748060089],c6402.ambari.apache.org:9997[14d34e748060089]) > 2015-05-08 23:59:44,381 [master.Master] DEBUG: [Root Table]: scan time 0.00 seconds > 2015-05-08 23:59:44,381 [master.Master] DEBUG: [Root Table] sleeping for 60.00 seconds > 2015-05-08 23:59:44,412 [master.Master] ERROR: Error processing table state for store Normal Tablets > java.lang.RuntimeException: java.lang.IllegalStateException: Tablet has multiple locations : !0< > at org.apache.accumulo.server.master.state.MetaDataTableScanner.(MetaDataTableScanner.java:72) > at org.apache.accumulo.server.master.state.MetaDataStateStore.iterator(MetaDataStateStore.java:70) > at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:173) > Caused by: java.lang.IllegalStateException: Tablet has multiple locations : !0< > at org.apache.accumulo.core.metadata.MetadataLocationObtainer.getMetadataLocationEntries(MetadataLocationObtainer.java:227) > at org.apache.accumulo.core.metadata.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:114) > at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:463) > at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocationAndCheckLock(TabletLocatorImpl.java:634) > at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:625) > at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:280) > at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:355) > at org.apache.accumulo.core.client.impl.TimeoutTabletLocator.binRanges(TimeoutTabletLocator.java:100) > at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:233) > at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.lookup(TabletServerBatchReaderIterator.java:220) > at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.(TabletServerBatchReaderIterator.java:154) > at org.apache.accumulo.core.client.impl.TabletServerBatchReader.iterator(TabletServerBatchReader.java:115) > at org.apache.accumulo.server.master.state.MetaDataTableScanner.(MetaDataTableScanner.java:66) > ... 2 more > 2015-05-08 23:59:44,412 [master.Master] DEBUG: Metadata Tablets location State: !0;~<@(null,c6402.ambari.apache.org:9997[14d34e748060080],c6402.ambari.apache.org:9997[14d34e748060080]) > 2015-05-08 23:59:44,439 [master.Master] ERROR: Error processing table state for store Metadata Tablets > java.lang.RuntimeException: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException: found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065] and c6402.ambari.apache.org:9997[14d34e748060080] > at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:191) > at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:130) > at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:49) > at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:175) > Caused by: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException: found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065] and c6402.ambari.apache.org:9997[14d34e748060080] > at org.apache.accumulo.server.master.state.MetaDataTableScanner.createTabletLocationState(MetaDataTableScanner.java:160) > at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:187) > ... 3 more > 2015-05-08 23:59:44,439 [master.Master] DEBUG: Attempting repair on !0< > 2015-05-08 23:59:44,450 [master.Master] WARN : Found a tablet hosted on multiple servers, attempting to repair > 2015-05-08 23:59:44,451 [master.Master] ERROR: Metadata table is inconsistent at !0< and all assigned/future tservers are still online. > 2015-05-08 23:59:44,577 [master.Master] DEBUG: Metadata Tablets location State: !0;~<@(null,c6402.ambari.apache.org:9997[14d34e748060080],c6402.ambari.apache.org:9997[14d34e748060080]) > 2015-05-08 23:59:44,590 [master.Master] ERROR: Error processing table state for store Metadata Tablets > java.lang.RuntimeException: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException: found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065] and c6402.ambari.apache.org:9997[14d34e748060080] > at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:191) > at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:130) > at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:49) > at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:175) > Caused by: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException: found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065] and c6402.ambari.apache.org:9997[14d34e748060080] > at org.apache.accumulo.server.master.state.MetaDataTableScanner.createTabletLocationState(MetaDataTableScanner.java:160) > at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:187) > ... 3 more > {noformat} > Agitation had stopped all of Accumulo. When I brought it back up, root came back online but metadata never did due to the above issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)