accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Popp (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ACCUMULO-2166) tserver recovering tablet on startup fails with "Garbage collection may be interfering with lock keep-alive"
Date Thu, 09 Jan 2014 21:33:51 GMT
Ben Popp created ACCUMULO-2166:
----------------------------------

             Summary: tserver recovering tablet on startup fails with "Garbage collection
may be interfering with lock keep-alive"
                 Key: ACCUMULO-2166
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2166
             Project: Accumulo
          Issue Type: Bug
            Reporter: Ben Popp


on a locally built 1.6.0-SNAPSHOT based on the SHA 417902e218c566333b6ea5ac492186ae305e5e16
(Dec 31st) alongside an apache hadoop 1.2.1 install. 

I had a single node deployment on my laptop, which had probably been interrupted by my laptop
sleep during some ingest.  

When i restarted the tserver, I got a "FATAL: Garbage collection may be interfering with lock
keep-alive.  Halting." log message after some recovery attempts and the tserver shut down.
 

I restarted and had similar results twice.  [~vines] and I tried to attach a debugger to the
tserver on startup the next time, and recovery completed without any failures!  I have restarted
the tserver since then with no issues. 

The only notable thing special about my workspace (since it affects the startup paths) is
that i'm using some custom Authorizer and Authenticator plugins in Accumulo. 

Log excerpts: 

{noformat} 
==============
FIRST TIME 
==============

2014-01-09 12:42:58,241 [server.Accumulo] INFO : tserver starting
...snip...
2014-01-09 12:44:47,820 [tserver.Tablet] INFO : Starting Write-Ahead Log recovery for d<;7006d
2014-01-09 12:44:47,821 [tserver.TabletServer] INFO : Looking for hdfs://localhost:9000/accumulo160b/recovery/2dceae9f-98c2-4f0c-9c66-27f4ab52113a/finished
2014-01-09 12:44:47,822 [tserver.TabletServer] INFO : Looking for hdfs://localhost:9000/accumulo160b/recovery/61b9709a-c945-41ae-946c-fc6425f4c96a/finished
2014-01-09 12:44:47,823 [log.SortedLogRecovery] INFO : Looking at mutations from hdfs://localhost:9000/accumulo160b/recovery/2dceae9f-98c2-4f0c-9c66-27f4ab52113a
for d<;7006d
2014-01-09 12:44:47,900 [log.SortedLogRecovery] INFO : Looking at mutations from hdfs://localhost:9000/accumulo160b/recovery/61b9709a-c945-41ae-946c-fc6425f4c96a
for d<;7006d
2014-01-09 12:44:47,990 [log.SortedLogRecovery] INFO : Scanning for mutations starting at
sequence number 2 for tid 110
2014-01-09 12:45:32,510 [util.Halt] FATAL: Garbage collection may be interfering with lock
keep-alive.  Halting.

==============
SECOND TIME 
==============

2014-01-09 13:55:17,890 [server.Accumulo] INFO : tserver starting
...snip...
2014-01-09 13:55:33,687 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/7e36fa46-95df-4b58-a1e4-6a29f43843a8
to hdfs://localhost:9000/accumulo160b/recovery/7e36fa46-95df-4b58-a1e4-6a29f43843a8
2014-01-09 13:55:33,688 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/e3428689-9011-4908-8990-9bd079978983
to hdfs://localhost:9000/accumulo160b/recovery/e3428689-9011-4908-8990-9bd079978983
2014-01-09 13:56:31,628 [util.Halt] FATAL: Garbage collection may be interfering with lock
keep-alive.  Halting.


==============
THIRD TIME 
==============

2014-01-09 14:04:16,315 [server.Accumulo] INFO : tserver starting
...snip...
2014-01-09 14:04:32,078 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/e3428689-9011-4908-8990-9bd079978983
to hdfs://localhost:9000/accumulo160b/recovery/e3428689-9011-4908-8990-9bd079978983
2014-01-09 14:04:32,079 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/a7467054-ec80-4734-974d-7c55c201b332
to hdfs://localhost:9000/accumulo160b/recovery/a7467054-ec80-4734-974d-7c55c201b332
2014-01-09 14:04:32,370 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
2014-01-09 14:04:32,811 [log.LogSorter] INFO : Finished log sort a7467054-ec80-4734-974d-7c55c201b332
11543822 bytes 1 parts in 732ms
2014-01-09 14:04:32,817 [log.LogSorter] INFO : Copying hdfs://localhost:9000/accumulo160b/wal/localhost+9997/7e36fa46-95df-4b58-a1e4-6a29f43843a8
to hdfs://localhost:9000/accumulo160b/recovery/7e36fa46-95df-4b58-a1e4-6a29f43843a8
2014-01-09 14:05:27,482 [util.Halt] FATAL: Garbage collection may be interfering with lock
keep-alive.  Halting.
{noformat} 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message