accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dickson, Matt MR" <matt.dick...@defence.gov.au>
Subject RE: accumulo.root invalid table reference [SEC=UNOFFICIAL]
Date Tue, 21 Feb 2017 03:18:38 GMT
UNOFFICIAL

It looks like an issue with one of the metadata table tablets. On startup the server that
hosts a particular metadata tablet gets scanned by all other tablet servers in the cluster.
 This then crashes that tablet server with an error in the tserver log;

... [zookeeper.ZooCache] WARN: Saw (possibly) transient exception communicating with ZooKeeper,
will retry
SessionExpiredException: KeeperErrorCode = Session expired for /accumulo/4234234234234234/namespaces/+accumulo/conf/table.scan.max.memory

That metadata table tablet is then transferred to another host which then fails also, and
so on.

While the server is hosting this metadata tablet, we see the following log statement from
all tserver.logs in the cluster:

.... [impl.ThriftScanner] DEBUG: Scan failed, thrift error org.apache.thrift.transport.TTransportException
 null (!0;1vm\\;125.323.233.23::2016103<,server.com.org:9997<http://server.com.org:9997/>,2342423df12341d)
Hope that helps complete the picture.


________________________________
From: Christopher [mailto:ctubbsii@apache.org]
Sent: Tuesday, 21 February 2017 13:17
To: user@accumulo.apache.org
Subject: Re: accumulo.root invalid table reference [SEC=UNOFFICIAL]


Removing them is probably a bad idea. The root table entries correspond to split points in
the metadata table. There is no need for the tables which existed when the metadata table
split to still exist for this to continue to act as a valid split point.

Would need to see the exception stack trace, or at least an error message, to troubleshoot
the shell scanning error you saw.

On Mon, Feb 20, 2017, 20:00 Dickson, Matt MR <matt.dickson@defence.gov.au<mailto:matt.dickson@defence.gov.au>>
wrote:
UNOFFICIAL

In case it is ok to remove these from the root table, how can I scan the root table for rows
with a rowid starting with !0;1vm?

Running "scan -b !0;1vm" throws an exception and exits the shell.


-----Original Message-----
From: Dickson, Matt MR [mailto:matt.dickson@defence.gov.au<mailto:matt.dickson@defence.gov.au>]
Sent: Tuesday, 21 February 2017 09:30
To: 'user@accumulo.apache.org<mailto:user@accumulo.apache.org>'
Subject: RE: accumulo.root invalid table reference [SEC=UNOFFICIAL]

UNOFFICIAL


Does that mean I should have entries for 1vm in the metadata table corresponding to the root
table?

We are running 1.6.5


-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com<mailto:josh.elser@gmail.com>]
Sent: Tuesday, 21 February 2017 09:22
To: user@accumulo.apache.org<mailto:user@accumulo.apache.org>
Subject: Re: accumulo.root invalid table reference [SEC=UNOFFICIAL]

The root table should only reference the tablets in the metadata table.
It's a hierarchy: like metadata is for the user tables, root is for the metadata table.

What version are ya running, Matt?

Dickson, Matt MR wrote:
> *UNOFFICIAL*
>
> I have a situation where all tablet servers are progressively being
> declared dead. From the logs the tservers report errors like:
> 2017-02-.... DEBUG: Scan failed thrift error
> org.apache.thrift.trasport.TTransportException null
> (!0;1vm\\125.323.233.23::2016103<,server.com.org:9997<http://server.com.org:9997>,2342423df12341d)
> 1vm was a table id that was deleted several months ago so it appears
> there is some invalid reference somewhere.
> Scanning the metadata table "scan -b 1vm" returns no rows returned for 1vm.
> A scan of the accumulo.root table returns approximately 15 rows that
> start with; !0:1vm;<i/p addr>/::2016103 /blah/ // How are the root
> table entries used and would it be safe to remove these entries since
> they reference a deleted table?
> Thanks in advance,
> Matt
> //
--
Christopher

Mime
View raw message