accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3787) Multiple locations for Metadata tablet
Date Sat, 09 May 2015 00:30:59 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536025#comment-14536025
] 

Josh Elser commented on ACCUMULO-3787:
--------------------------------------

Root tablet rfile. Contents from {{accumulo rfile-info}}

{noformat}
!0;~ last:14d34e74806005f [] 175 false -> c6402.ambari.apache.org:9997
!0;~ last:14d34e748060080 [] 476 false -> c6402.ambari.apache.org:9997
!0< last:14d34e748060060 [] 177 false -> c6403.ambari.apache.org:9997
!0< last:14d34e748060080 [] 478 false -> c6402.ambari.apache.org:9997
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 176 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 469 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 477 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 470 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 172 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 174 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 473 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 475 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 171 false
->
~delhdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/table... TRUNCATED : [] 472 false
->
!0;~ loc:14d34e748060080 [] 258 false -> c6402.ambari.apache.org:9997
!0;~ ~tab:~pr [] 0 false ->
!0< loc:14d34e748060065 [] 183 false -> c6401.ambari.apache.org:9997
!0< loc:14d34e748060080 [] 447 false -> c6402.ambari.apache.org:9997
!0< ~tab:~pr [] 0 false -> ~
!0;~ file:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0... TRUNCATED []
175 false -> 1520,93
!0;~ file:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0... TRUNCATED []
476 false -> 1963,117
!0;~ srv:compact [] 476 false -> 20
!0;~ srv:dir [] 0 false -> hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0/table_info
!0;~ srv:flush [] 468 false -> 20
!0;~ srv:lock [] 476 false -> tservers/c6402.ambari.apache.org:9997/zlock-0000000001$14d34e748060080
!0;~ srv:time [] 468 false -> L808
!0< file:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0... TRUNCATED
[] 177 false -> 570,2
!0< file:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0... TRUNCATED
[] 478 false -> 638,4
!0< srv:compact [] 478 false -> 20
!0< srv:dir [] 0 false -> hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/tables/!0/default_tablet
!0< srv:flush [] 467 false -> 20
!0< srv:lock [] 478 false -> tservers/c6402.ambari.apache.org:9997/zlock-0000000001$14d34e748060080
!0< srv:time [] 467 false -> L1196
~wal+c6402.ambari.apache.org:9997[14d34e748060080] log:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/wal/c6402...
TRUNCATED [] 439 false ->
~wal+c6402.ambari.apache.org:9997[14d34e748060080] log:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/wal/c6402...
TRUNCATED [] 436 false ->
~wal+c6403.ambari.apache.org:9997[14d34e748060072] log:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/wal/c6403...
TRUNCATED [] 443 false -> unused
~wal+c6403.ambari.apache.org:9997[14d34e748060072] log:hdfs://c6401.ambari.apache.org:8020/apps/accumulo/data/wal/c6403...
TRUNCATED [] 443 false -> unused
{noformat}

> Multiple locations for Metadata tablet
> --------------------------------------
>
>                 Key: ACCUMULO-3787
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3787
>             Project: Accumulo
>          Issue Type: Bug
>          Components: master, tserver
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>            Priority: Blocker
>              Labels: 1.7.0_QA
>             Fix For: 1.7.0
>
>
> Running continuous ingest with agitation
> {noformat}
> 2015-05-08 23:59:44,381 [state.ZooTabletStateStore] DEBUG: Returning root tablet state:
+r<<@(null,c6402.ambari.apache.org:9997[14d34e748060089],c6402.ambari.apache.org:9997[14d34e748060089])
> 2015-05-08 23:59:44,381 [master.Master] DEBUG: Root Table location State: +r<<@(null,c6402.ambari.apache.org:9997[14d34e748060089],c6402.ambari.apache.org:9997[14d34e748060089])
> 2015-05-08 23:59:44,381 [master.Master] DEBUG: [Root Table]: scan time 0.00 seconds
> 2015-05-08 23:59:44,381 [master.Master] DEBUG: [Root Table] sleeping for 60.00 seconds
> 2015-05-08 23:59:44,412 [master.Master] ERROR: Error processing table state for store
Normal Tablets
> java.lang.RuntimeException: java.lang.IllegalStateException: Tablet has multiple locations
: !0<
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.<init>(MetaDataTableScanner.java:72)
>         at org.apache.accumulo.server.master.state.MetaDataStateStore.iterator(MetaDataStateStore.java:70)
>         at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:173)
> Caused by: java.lang.IllegalStateException: Tablet has multiple locations : !0<
>         at org.apache.accumulo.core.metadata.MetadataLocationObtainer.getMetadataLocationEntries(MetadataLocationObtainer.java:227)
>         at org.apache.accumulo.core.metadata.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:114)
>         at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:463)
>         at org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocationAndCheckLock(TabletLocatorImpl.java:634)
>         at org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:625)
>         at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:280)
>         at org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:355)
>         at org.apache.accumulo.core.client.impl.TimeoutTabletLocator.binRanges(TimeoutTabletLocator.java:100)
>         at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:233)
>         at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.lookup(TabletServerBatchReaderIterator.java:220)
>         at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.<init>(TabletServerBatchReaderIterator.java:154)
>         at org.apache.accumulo.core.client.impl.TabletServerBatchReader.iterator(TabletServerBatchReader.java:115)
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.<init>(MetaDataTableScanner.java:66)
>         ... 2 more
> 2015-05-08 23:59:44,412 [master.Master] DEBUG: Metadata Tablets location State: !0;~<@(null,c6402.ambari.apache.org:9997[14d34e748060080],c6402.ambari.apache.org:9997[14d34e748060080])
> 2015-05-08 23:59:44,439 [master.Master] ERROR: Error processing table state for store
Metadata Tablets
> java.lang.RuntimeException: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065]
and c6402.ambari.apache.org:9997[14d34e748060080]
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:191)
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:130)
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:49)
>         at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:175)
> Caused by: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065]
and c6402.ambari.apache.org:9997[14d34e748060080]
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.createTabletLocationState(MetaDataTableScanner.java:160)
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:187)
>         ... 3 more
> 2015-05-08 23:59:44,439 [master.Master] DEBUG: Attempting repair on !0<
> 2015-05-08 23:59:44,450 [master.Master] WARN : Found a tablet hosted on multiple servers,
attempting to repair
> 2015-05-08 23:59:44,451 [master.Master] ERROR: Metadata table is inconsistent at !0<
and all assigned/future tservers are still online.
> 2015-05-08 23:59:44,577 [master.Master] DEBUG: Metadata Tablets location State: !0;~<@(null,c6402.ambari.apache.org:9997[14d34e748060080],c6402.ambari.apache.org:9997[14d34e748060080])
> 2015-05-08 23:59:44,590 [master.Master] ERROR: Error processing table state for store
Metadata Tablets
> java.lang.RuntimeException: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065]
and c6402.ambari.apache.org:9997[14d34e748060080]
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:191)
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:130)
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:49)
>         at org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:175)
> Caused by: org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
found two locations for the same extent !0<: c6401.ambari.apache.org:9997[14d34e748060065]
and c6402.ambari.apache.org:9997[14d34e748060080]
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.createTabletLocationState(MetaDataTableScanner.java:160)
>         at org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:187)
>         ... 3 more
> {noformat}
> Agitation had stopped all of Accumulo. When I brought it back up, root came back online
but metadata never did due to the above issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message