accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <>
Subject Re: Orphaned FATE Locks [SEC=UNOFFICIAL]
Date Wed, 21 Sep 2016 14:52:17 GMT
On Wed, Sep 7, 2016 at 8:55 PM, Dickson, Matt MR
<> wrote:
> In Zookeeper there don't appear to be any locks with the same txid that is listed via
Accumulo.  However under /accumulo/xxxxxxxx/table_locks/+default/ there are the same number
of files as orphaned locks labelled 'lock-000000xx', are these the locks I can delete?

Every table lock should have an associated fate transaction.   The
fate print tool has analysis[1] to look for situations when this is
not the case.  It seems like this analysis is finding locks w/o
transactions.   Looking at the code I see there is slight possibility
of race condition.  Its reads the list of locks into memory and then
reads the transactions.  So if a transaction completed and deleted
lock between those steps the tool could report a false positive.  If
you run the tool multiple times and it reports the same thing then
that race condition is not the cause.

Yes, you should delete the locks.  These orphaned locks can hold up
future table operations.  It would be nice to find out what caused
this.  You can grep for the fate txid related to the locks in the
master log to gather info about the fate tx that created the lock.

Do you know if the tables ids its complaining about as having locks
were deleted?  Do you know if fate transactions were deleted (manually
in ZK or using Accumulo's tool)?


> I should note that while investigating this there were no other fate transactions being
listed by Accumulo for this table, +default, so the system was in a stable state.
> -----Original Message-----
> From: Josh Elser []
> Sent: Thursday, 8 September 2016 01:07
> To:
> Subject: Re: Orphaned FATE Locks [SEC=UNOFFICIAL]
> Hi Matt,
> What version of Accumulo are you using? Figuring out why those transactions aren't automatically
get removed is something else we would want to look into.
> It sounds like these transactions are just vestigial (not actually running), so I wouldn't
think that they would affect current bulk loads.
> I believe you could just stop the Master and remove the corresponding nodes in ZooKeeper
(as that's where the txns are stored and `fate print` is reading from), but I would defer
to Keith for confirmation first :)
> Dickson, Matt MR wrote:
>> When running 'fate print -t IN_PROGRESS' to list fate transactions
>> there are approximately eight orphaned locks listed as:
>> txid: xxxxxxxxxxxxxxx locked: [R:+default] I'm looking into these
>> because bulk ingests are failing and there are a lot of CopyFailed
>> transactions in the fate lock list. Could these orphaned locks block
>> further bulk ingests and is there a way to kill them?
>> When I run 'fate fail xxxxxxxx' it states there is no fate transaction
>> associated with the transaction id.
>> Thanks advance,
>> Matt

View raw message