accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ACCUMULO-3787) Multiple locations for Metadata tablet
Date Sat, 09 May 2015 00:05:59 GMT
Josh Elser created ACCUMULO-3787:
------------------------------------

             Summary: Multiple locations for Metadata tablet
                 Key: ACCUMULO-3787
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3787
             Project: Accumulo
          Issue Type: Bug
          Components: master, tserver
            Reporter: Josh Elser
            Assignee: Josh Elser
            Priority: Blocker
             Fix For: 1.7.0


Running continuous ingest with agitation

{noformat}
2015-05-08 23:59:44,381 [state.ZooTabletStateStore] DEBUG: Returning root tablet state: +r<<@(null,c6402.ambari.apache.org:9997[14d34e748060089],c6402.ambari.apache.org:9997[14d34e748060089])
2015-05-08 23:59:44,381 [master.Master] DEBUG: Root Table location State: +r<<@(null,c6402.ambari.apache.org:9997[14d34e748060089],c6402.ambari.apache.org:9997[14d34e748060089])
2015-05-08 23:59:44,381 [master.Master] DEBUG: [Root Table]: scan time 0.00 seconds
2015-05-08 23:59:44,381 [master.Master] DEBUG: [Root Table] sleeping for 60.00 seconds
2015-05-08 23:59:44,412 [master.Master] ERROR: Error processing table state for store Normal
Tablets
java.lang.RuntimeException: java.lang.IllegalStateException: Tablet has multiple locations
: !0<
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.<init>(MetaDataTableScanner.java:72)
        at org.apache.accumulo.server.master.state.MetaDataStateStore.iterator(MetaDataStateStore.java:70)
        at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:173)
Caused by: java.lang.IllegalStateException: Tablet has multiple locations : !0<
        at org.apache.accumulo.core.metadata.MetadataLocationObtainer.getMetadataLocationEntries(MetadataLocationObtainer.java:227)
        at org.apache.accumulo.core.metadata.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:114)
        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:463)
        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocationAndCheckLock(TabletLocatorImpl.java:634)
        at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:625)
        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:280)
        at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:355)
        at org.apache.accumulo.core.client.impl.TimeoutTabletLocator.binRanges(TimeoutTabletLocator.java:100)
        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:233)
        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.lookup(TabletServerBatchReaderIterator.java:220)
        at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.<init>(TabletServerBatchReaderIterator.java:154)
        at org.apache.accumulo.core.client.impl.TabletServerBatchReader.iterator(TabletServerBatchReader.java:115)
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.<init>(MetaDataTableScanner.java:66)
        ... 2 more
2015-05-08 23:59:44,412 [master.Master] DEBUG: Metadata Tablets location State: !0;~<@(null,c6402.ambari.apache.org:9997[14d34e748060080],c6402.ambari.apache.org:9997[14d34e748060080])
2015-05-08 23:59:44,439 [master.Master] ERROR: Error processing table state for store Metadata
Tablets
java.lang.RuntimeException: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065]
and c6402.ambari.apache.org:9997[14d34e748060080]
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:191)
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:130)
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:49)
        at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:175)
Caused by: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065]
and c6402.ambari.apache.org:9997[14d34e748060080]
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.createTabletLocationState(MetaDataTableScanner.java:160)
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:187)
        ... 3 more
2015-05-08 23:59:44,439 [master.Master] DEBUG: Attempting repair on !0<
2015-05-08 23:59:44,450 [master.Master] WARN : Found a tablet hosted on multiple servers,
attempting to repair
2015-05-08 23:59:44,451 [master.Master] ERROR: Metadata table is inconsistent at !0< and
all assigned/future tservers are still online.
2015-05-08 23:59:44,577 [master.Master] DEBUG: Metadata Tablets location State: !0;~<@(null,c6402.ambari.apache.org:9997[14d34e748060080],c6402.ambari.apache.org:9997[14d34e748060080])
2015-05-08 23:59:44,590 [master.Master] ERROR: Error processing table state for store Metadata
Tablets
java.lang.RuntimeException: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065]
and c6402.ambari.apache.org:9997[14d34e748060080]
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:191)
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:130)
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:49)
        at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:175)
Caused by: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065]
and c6402.ambari.apache.org:9997[14d34e748060080]
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.createTabletLocationState(MetaDataTableScanner.java:160)
        at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:187)
        ... 3 more
{noformat}

Agitation had stopped all of Accumulo. When I brought it back up, root came back online but
metadata never did due to the above issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message