accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3276) Shard.xml hung with no client output
Date Thu, 30 Oct 2014 19:46:33 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190692#comment-14190692
] 

Josh Elser commented on ACCUMULO-3276:
--------------------------------------

A full picture for {{3;r03f53;r0155f}}:

{noformat}
2014-10-28 18:37:23,313 [tserver.Tablet] DEBUG: Files for low split 3;r03f53;r0155f  [hdfs://nn:8020/apps/accumulo/tables/3/b-00000f6/I00000f9.rf,
hdfs://nn:8020/apps/accumulo/tables/3/b-00000fd/I00000fe.rf, hdfs://nn:8020/apps/accumulo/tables/3/b-00000fd/I00000fg.rf,
hdfs://nn:8020/apps/accumulo/tables/3/b-00000ff/I00000fi.rf, hdfs://nn:8020/apps/accumulo/tables/3/b-00000fo/I00000fp.rf,
hdfs://nn:8020/apps/accumulo/tables/3/b-00000fq/I00000fs.rf, hdfs://nn:8020/apps/accumulo/tables/3/b-00000g1/I00000g2.rf,
hdfs://nn:8020/apps/accumulo/tables/3/b-00000g4/I00000g5.rf, hdfs://nn:8020/apps/accumulo/tables/3/b-00000g8/I00000g9.rf,
hdfs://nn:8020/apps/accumulo/tables/3/b-00000gg/I00000gh.rf, hdfs://nn:8020/apps/accumulo/tables/3/b-00000gm/I00000go.rf,
hdfs://nn:8020/apps/accumulo/tables/3/b-00000gn/I00000gp.rf, hdfs://nn:8020/apps/accumulo/tables/3/b-00000gn/I00000gs.rf,
hdfs://nn:8020/apps/accumulo/tables/3/b-00000gz/I00000h0.rf, hdfs://nn:8020/apps/accumulo/tables/3/b-00000h9/I00000he.rf,
hdfs://nn:8020/apps/accumulo/tables/3/b-00000ha/I00000hc.rf, hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hq.rf,
hdfs://nn:8020/apps/accumulo/tables/3/default_tablet/A000005m.rf]
2014-10-28 18:37:23,389 [tserver.Tablet] TABLET_HIST: 3;r04d6f;r0155f split 3;r03f53;r0155f
3;r04d6f;r03f53
2014-10-28 18:37:23,964 [tserver.Tablet] TABLET_HIST: 3;r03f53;r0155f opened
2014-10-28 18:37:24,553 [master.EventCoordinator] INFO : old_tserver:9997 reported split 3;r03f53;r0155f,
3;r04d6f;r03f53
2014-10-28 18:37:26,123 [client.BulkImporter] DEBUG: Asking old_tserver:9997 to bulk load
{3;r05b25;r05728={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hr.rf=MapFileInfo(estimatedSize:1128)},
3;r03f53;r0155f={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hr.rf=MapFileInfo(estimatedSize:11286)},
3;r07dad;r06c0f={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hr.rf=MapFileInfo(estimatedSize:6772)},
3;r08ff0;r07dad={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hr.rf=MapFileInfo(estimatedSize:6772)},
3;r04d6f;r03f53={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hr.rf=MapFileInfo(estimatedSize:5643)},
3;r05728;r04d6f={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hr.rf=MapFileInfo(estimatedSize:3386)},
3;r06c0f;r05b25={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hr.rf=MapFileInfo(estimatedSize:6772)}}
2014-10-28 18:37:26,249 [tserver.Tablet] TABLET_HIST: 3;r03f53;r0155f import hdfs://nn:8020/apps/accumulo/tables/3/b-00000hp/I00000hr.rf
11286 0
2014-10-28 18:37:26,755 [client.BulkImporter] DEBUG: Asking old_tserver:9997 to bulk load
{3;r05b25;r05728={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:2272)},
3;r03f53;r0155f={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:15910)},
3;r0b1c1;r08ff0={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:13637)},
3;r07dad;r06c0f={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:6818)},
3;r08ff0;r07dad={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:6818)},
3;r04d6f;r03f53={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:5682)},
3;r05728;r04d6f={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:3409)},
3;r0155f<={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:7955)},
3;r06c0f;r05b25={hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf=MapFileInfo(estimatedSize:5682)}}
2014-10-28 18:37:26,937 [client.BulkImporter] DEBUG: Asking old_tserver:9997 to bulk load
{3;r03f53;r0155f={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hz/I00000i0.rf=MapFileInfo(estimatedSize:13536)},
3;r04d6f;r03f53={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hz/I00000i0.rf=MapFileInfo(estimatedSize:5206)},
3;r05728;r04d6f={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hz/I00000i0.rf=MapFileInfo(estimatedSize:1041)},
3;r0155f<={hdfs://nn:8020/apps/accumulo/tables/3/b-00000hz/I00000i0.rf=MapFileInfo(estimatedSize:6247)}}
2014-10-28 18:37:26,972 [tserver.Tablet] TABLET_HIST: 3;r03f53;r0155f import hdfs://nn:8020/apps/accumulo/tables/3/b-00000hz/I00000i0.rf
13536 0
2014-10-28 18:37:27,015 [tserver.Tablet] TABLET_HIST: 3;r03f53;r0155f import hdfs://nn:8020/apps/accumulo/tables/3/b-00000i2/I00000i3.rf
15910 0
2014-10-28 18:37:27,261 [master.Master] DEBUG: migration 3;r03f53;r0155f: old_tserver:9997[24957edcb020003]
-> hung_tserver:9997[34957ee474a0002]
2014-10-28 18:37:27,534 [tserver.Tablet] DEBUG: completeClose(saveState=true completeClose=true)
3;r03f53;r0155f
2014-10-28 18:37:27,534 [tserver.Tablet] DEBUG: initiateClose(saveState=true queueMinC=false
disableWrites=false) 3;r03f53;r0155f
2014-10-28 18:37:28,053 [tserver.TabletServer] DEBUG: Unassigning 3;r03f53;r0155f@(null,old_tserver:9997[24957edcb020003],null)
2014-10-28 18:37:28,053 [tserver.Tablet] TABLET_HIST: 3;r03f53;r0155f closed
2014-10-28 18:37:28,156 [tserver.TabletServer] INFO : unloaded 3;r03f53;r0155f
2014-10-28 18:37:28,178 [master.EventCoordinator] INFO : tablet 3;r03f53;r0155f was unloaded
from hung_tserver:9997
2014-10-28 18:37:28,475 [tserver.TabletServer] DEBUG: Loading extent: 3;r03f53;r0155f
2014-10-28 18:37:28,475 [tserver.TabletServer] INFO : Loading tablet 3;r03f53;r0155f
2014-10-28 18:37:28,475 [tserver.TabletServer] INFO : hung_tserver:9997: got assignment from
master: 3;r03f53;r0155f
2014-10-28 18:37:28,476 [tserver.TabletServer] DEBUG: verifying extent 3;r03f53;r0155f
2014-10-28 18:37:28,535 [tserver.Tablet] DEBUG: got [] for logs for 3;r03f53;r0155f
{noformat}

"hung_tserver" is the tserver which eventually got stuck trying to make this assignment. "old_tserver"
was the tserver which this tablet originated on (from a split).

> Shard.xml hung with no client output
> ------------------------------------
>
>                 Key: ACCUMULO-3276
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3276
>             Project: Accumulo
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 1.6.1
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.6.2, 1.7.0
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> Ran Shard.xml over a 5 node instance. The only line of client output I got was that ZooSession
connected to the quorum.
> 45 minutes later, my test runner timed out the module. We need more information in the
client test log to actually determine where it got stuck.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message