accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (ACCUMULO-3351) Tracer can't write traces after offline and online of trace table
Date Fri, 21 Nov 2014 04:05:33 GMT

     [ https://issues.apache.org/jira/browse/ACCUMULO-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Elser resolved ACCUMULO-3351.
----------------------------------
    Resolution: Fixed

> Tracer can't write traces after offline and online of trace table
> -----------------------------------------------------------------
>
>                 Key: ACCUMULO-3351
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3351
>             Project: Accumulo
>          Issue Type: Bug
>          Components: trace
>    Affects Versions: 1.6.1
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.6.2, 1.7.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> While running tests for ACCUMULO-3167, I updated one of the tests to offline the trace
table to reduce the possibility that any active logs for the trace table would exist in the
metadata table.
> A later test went to validate that traces were found for some conditional update sessions
and hung indefinitely.
> Inspecting the tracer log, the batchwriter had two exceptions due to the trace table
being offline (as expected), but never recovered when the trace table came back online.
> {noformat}
> 2014-11-20 13:08:28,717 [impl.TabletServerBatchWriter] DEBUG: Table trace (in) is offline
> org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> 	at java.util.TimerThread.mainLoop(Timer.java:555)
> 	at java.util.TimerThread.run(Timer.java:505)
> 2014-11-20 13:08:28,720 [tracer.TraceServer] WARN : Problem flushing traces, resetting
writer. Set log level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.MutationsRejectedException:
# constraint violations : 0  security codes: {}  # server errors 0 # exceptions 1
> 2014-11-20 13:08:28,720 [tracer.TraceServer] DEBUG: flushing traces failed due to exception
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations :
0  security codes: {}  # server errors 0 # exceptions 1
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.flush(TabletServerBatchWriter.java:331)
> 	at org.apache.accumulo.core.client.impl.BatchWriterImpl.flush(BatchWriterImpl.java:59)
> 	at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:245)
> 	at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> 	at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> 	at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> 	at java.util.TimerThread.mainLoop(Timer.java:555)
> 	at java.util.TimerThread.run(Timer.java:505)
> Caused by: org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is
offline
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> 	... 2 more
> 2014-11-20 13:08:28,722 [tracer.TraceServer] WARN : Unable to create a batch writer,
will retry. Set log level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.TableOfflineException:
Table trace (in) is offline
> 2014-11-20 13:08:28,722 [tracer.TraceServer] DEBUG: batch writer creation failed with
exception.
> org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is offline
> 	at org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:86)
> 	at org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
> 	at org.apache.accumulo.tracer.TraceServer.resetWriter(TraceServer.java:262)
> 	at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:250)
> 	at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> 	at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> 	at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> 	at java.util.TimerThread.mainLoop(Timer.java:555)
> 	at java.util.TimerThread.run(Timer.java:505)
> 2014-11-20 13:08:28,723 [tracer.TraceServer] WARN : Problem closing batch writer. Set
log level to DEBUG to see stacktrace. cause: org.apache.accumulo.core.client.MutationsRejectedException:
# constraint violations : 0  security codes: {}  # server errors 0 # exceptions 1
> 2014-11-20 13:08:28,723 [tracer.TraceServer] DEBUG: batch writer close failed with exception
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations :
0  security codes: {}  # server errors 0 # exceptions 1
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:354)
> 	at org.apache.accumulo.core.client.impl.BatchWriterImpl.close(BatchWriterImpl.java:54)
> 	at org.apache.accumulo.tracer.TraceServer.resetWriter(TraceServer.java:271)
> 	at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:250)
> 	at org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> 	at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> 	at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> 	at java.util.TimerThread.mainLoop(Timer.java:555)
> 	at java.util.TimerThread.run(Timer.java:505)
> Caused by: org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is
offline
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> 	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> 	... 2 more
> 2014-11-20 13:10:12,929 [tracer.TraceServer] WARN : writer is not ready; discarding span.
> 2014-11-20 13:10:12,930 [tracer.TraceServer] WARN : writer is not ready; discarding span.
> {noformat}
> "writer is not ready; discarding span." repeats indefinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message