accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ACCUMULO-3351) Tracer can't write traces after offline and online of trace table
Date Thu, 20 Nov 2014 18:42:34 GMT
Josh Elser created ACCUMULO-3351:
------------------------------------

             Summary: Tracer can't write traces after offline and online of trace table
                 Key: ACCUMULO-3351
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3351
             Project: Accumulo
          Issue Type: Bug
          Components: trace
    Affects Versions: 1.6.1
            Reporter: Josh Elser
            Assignee: Josh Elser
             Fix For: 1.6.2


While running tests for ACCUMULO-3167, I updated one of the tests to offline the trace table
to reduce the possibility that any active logs for the trace table would exist in the metadata
table.

A later test went to validate that traces were found for some conditional update sessions
and hung indefinitely.

Inspecting the tracer log, the batchwriter had two exceptions due to the trace table being
offline (as expected), but never recovered when the trace table came back online.

{noformat}
2014-11-20 13:08:28,717 [impl.TabletServerBatchWriter] DEBUG: Table trace (in) is offline
org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
2014-11-20 13:08:28,720 [tracer.TraceServer] WARN : Problem flushing traces, resetting writer.
Set log level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.MutationsRejectedException:
# constraint violations : 0  security codes: {}  # server errors 0 # exceptions 1
2014-11-20 13:08:28,720 [tracer.TraceServer] DEBUG: flushing traces failed due to exception
org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0  security
codes: {}  # server errors 0 # exceptions 1
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.flush(TabletServerBatchWriter.java:331)
	at org.apache.accumulo.core.client.impl.BatchWriterImpl.flush(BatchWriterImpl.java:59)
	at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:245)
	at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
	at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
	at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
	... 2 more
2014-11-20 13:08:28,722 [tracer.TraceServer] WARN : Unable to create a batch writer, will
retry. Set log level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.TableOfflineException:
Table trace (in) is offline
2014-11-20 13:08:28,722 [tracer.TraceServer] DEBUG: batch writer creation failed with exception.
org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
	at org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:86)
	at org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
	at org.apache.accumulo.tracer.TraceServer.resetWriter(TraceServer.java:262)
	at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:250)
	at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
	at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
	at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
2014-11-20 13:08:28,723 [tracer.TraceServer] WARN : Problem closing batch writer. Set log
level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.MutationsRejectedException:
# constraint violations : 0  security codes: {}  # server errors 0 # exceptions 1
2014-11-20 13:08:28,723 [tracer.TraceServer] DEBUG: batch writer close failed with exception
org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0  security
codes: {}  # server errors 0 # exceptions 1
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:354)
	at org.apache.accumulo.core.client.impl.BatchWriterImpl.close(BatchWriterImpl.java:54)
	at org.apache.accumulo.tracer.TraceServer.resetWriter(TraceServer.java:271)
	at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:250)
	at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
	at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
	at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
	... 2 more
2014-11-20 13:10:12,929 [tracer.TraceServer] WARN : writer is not ready; discarding span.
2014-11-20 13:10:12,930 [tracer.TraceServer] WARN : writer is not ready; discarding span.
{noformat}

"writer is not ready; discarding span." repeats indefinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message