accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Fuchs (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-1364) Silent failure after power outage
Date Wed, 08 May 2013 22:01:15 GMT


Adam Fuchs commented on ACCUMULO-1364:

Regarding whether it is Accumulo's problem if one of the underlying systems has a failure,
it absolutely is. We have to take responsibility for the entire stack below us. The action
we take could be to fix the problem in Accumulo despite shortcomings of underlying systems,
we could document the suggestion that users "don't run it that way" (which doesn't really
work if it's an HDFS problem), or we could take responsibility for pushing the fix to the
underlying layers. It would make me sad if we just pass this off as someone else's problem
when our system clearly doesn't meet the level of reliability that we claim it meets.

That said, at this point we're still speculating on the root cause, so we need to find a way
to recreate this problem before we diagnose it. I believe John is going to try to replicate
this in a VM, and will escalate that to hardware if the VM attempt fails. My open question
is whether we should hold the release of 1.5.0 until we can diagnose (or fail to replicate).

> Silent failure after power outage
> ---------------------------------
>                 Key: ACCUMULO-1364
>                 URL:
>             Project: Accumulo
>          Issue Type: Sub-task
>          Components: master, tserver
>         Environment: hadoop-1.0.4, accumulo-1.5-SNAPSHOT svn version 1470047
>            Reporter: John Vines
>            Assignee: Eric Newton
>            Priority: Blocker
>             Fix For: 1.5.0
> We were doing some testing on an Accumulo snapshot using continuous ingest when the power
went out. When it came back we noticed some corrupt blocks in HDFS, mostly around the WAL.
I wasn't certain if that was a happenstance of how the sync blocks can turn out, so I went
ahead and started Accumulo to see if it could handle it. What I got wasn't what I expected.
> There are 0 errors reported on the monitor. It just sits with 5 tservers available and
no tablets online. The master appears it attempted to assign and then is waiting for the walog
to close, which never happens-
> {quote} 2013-04-30 10:38:23,648 [master.EventCoordinator] INFO : There are now 5 tablet
> 2013-04-30 10:38:23,719 [state.ZooTabletStateStore] DEBUG: root tablet logSet []
> 2013-04-30 10:38:23,720 [state.ZooTabletStateStore] DEBUG: root tablet logSet []
> 2013-04-30 10:38:23,725 [state.ZooTabletStateStore] DEBUG: Returning root tablet state:
> 2013-04-30 10:38:23,740 [master.Master] INFO : Loaded class : org.apache.accumulo.server.master.recovery.HadoopLogCloser
> 2013-04-30 10:38:23,741 [recovery.RecoveryManager] INFO : Starting recovery of ed30bd24-b348-4344-8614-a2d79f933462
(in : 10s) created for, tablet !0;!0<< holds a reference
> 2013-04-30 10:38:23,751 [master.Master] DEBUG: [Root Tablet]: scan time 0.04 seconds
> 2013-04-30 10:38:23,751 [master.Master] DEBUG: [Root Tablet] sleeping for 60.00 seconds
> 2013-04-30 10:38:23,823 [metrics.MetricsConfiguration] DEBUG: Loading config file: /cloud/accumulo/apache-accumulo-1.5.0-SNAPSHOT_1470047/conf/accumulo-metrics.xml
> 2013-04-30 10:38:23,838 [master.Master] DEBUG: Finished gathering information from 5
servers in 0.21 seconds
> 2013-04-30 10:38:23,841 [master.Master] DEBUG: not balancing because there are unhosted
> 2013-04-30 10:38:23,852 [master.Master] DEBUG: Finished gathering information from 5
servers in 0.01 seconds
> 2013-04-30 10:38:23,852 [master.Master] DEBUG: not balancing because there are unhosted
> 2013-04-30 10:38:23,861 [metrics.MetricsConfiguration] DEBUG: Metrics collection enabled=false
> 2013-04-30 10:38:23,874 [impl.ThriftScanner] DEBUG: Error getting transport to
: NotServingTabletException(extent:TKeyExtent(table:21 30, endRow:21 30 3C, prevEndRow:null))
>  {quote}
> That Exception repeats endlessly with periodic
> bq. 2013-04-30 10:38:34,756 [recovery.HadoopLogCloser] INFO : Waiting for file to be
closed /accumulo/wal/
> On the tserver in question, it seems to have no idea that it's supposed to be recovering
the root tablet though
> {quote}
> 2013-04-30 10:38:22,432 [tabletserver.TabletServer] DEBUG: org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler
> 2013-04-30 10:38:22,544 [metrics.MetricsConfiguration] DEBUG: Loading config file: /cloud/accumulo/apache-accumulo-1.5.0-SNAPSHOT_1470047/conf/accumu
> lo-metrics.xml
> 2013-04-30 10:38:22,549 [metrics.MetricsConfiguration] DEBUG: Metrics collection enabled=false
> 2013-04-30 10:38:22,551 [tabletserver.TabletServer] INFO : port = 9997
> 2013-04-30 10:38:22,621 [tabletserver.TabletServer] DEBUG: Obtained tablet server lock
> .102.202:9997/zlock-0000000000
> 2013-04-30 10:38:23,266 [tabletserver.TabletServer] DEBUG: gc ParNew=0.00(+0.00) secs
ConcurrentMarkSweep=0.00(+0.00) secs freemem=8,486,794,504(+45,
> 036,880) totalmem=8,536,260,608
> 2013-04-30 10:38:23,947 [tabletserver.TabletServer] DEBUG: MultiScanSess
0 entries in 0.07 secs (lookup_time:0.00 secs tablets:1
>  ranges:1) 
> 2013-04-30 10:38:23,986 [tabletserver.TabletServer] DEBUG: MultiScanSess
0 entries in 0.00 secs (lookup_time:0.00 secs tablets:1
>  ranges:1) 
> {quote}
> With that debug message repeating endlessly. Out and err files on the master and that
tserver are empty.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message