accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dickson, Matt MR" <matt.dick...@defence.gov.au>
Subject RE: ASSIGNED_TO_DEAD_SERVER #walogs:2 [SEC=UNOFFICIAL]
Date Tue, 17 Dec 2019 01:53:05 GMT
UNOFFICIAL

A correction to my description:

Looking at the Accumulo gui on the 'Table Problems' section there are 8K errors stating:

Table                     Problem Type                    Server                   Time  
                   Resource                             Exception
Table                     TABLET_LOAD                   host                       datetime
             resourceUUID                    java.io.IOException: ....FileNoteFoundException:
File does not exist: hdfs://system/accumulo/recovery/434adfsdf124312f/failed/data

These seem to correspond to records in the accumulo.metadata table:

Scan -t accumulo.metadata -b ~err

~err_zxn  TABLET_LOAD: ............

From: Dickson, Matt MR <matt.dickson@defence.gov.au>
Sent: Tuesday, 17 December 2019 12:29 PM
To: user@accumulo.apache.org
Subject: ASSIGNED_TO_DEAD_SERVER #walogs:2 [SEC=UNOFFICIAL]

UNOFFICIAL

Hi,

I'm trying to recover from an issue that was caused by the table.split.threshold being set
to a very low size that then generated a massive load on zookeeper and cluster nodes timing
out communicating with zookeeper while Accumulo was splitting tablets.  This was noticed when
tablet servers were being declared dead.

I've corrected the threshold and Accumulo is back online however there are 8K unhosted tablets
that are not coming online.

Running check the checkTablets script produces the exact number of errors as there are unhosted
tablets with a message like:

4d4;blah::words::4gfv43@(host:9997[23423442344234f23fd],null,null_ is ASSIGNED_TO_DEAD_SERVER
#walogs:2

I'm not concerned if there is data in these tablets and it is lost in returning the system
to a healthy state because I suspect other Accumulo operations can't proceed while tablets
are unhosted so just need to remove these issues.

Any advice would be great.

Thanks in advance,
Matt

Mime
View raw message