accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2213) tracer reports: IllegalStateException: Closed
Date Fri, 17 Jan 2014 15:23:22 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874853#comment-13874853
] 

Sean Busbey commented on ACCUMULO-2213:
---------------------------------------

I can make this happen reliably on 1.4.5 while using gremlins to disrupt the node the tracer
is on.

I also get exceptions in the main loop, which blows up the log for the service:

{noformat}
2014-01-15 23:41:10,533 [trace.TraceServer] ERROR: Unable to write mutation to table: org.apache.accumulo.core.data.Mutation@0
java.lang.IllegalStateException: Closed
        at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:192)
        at org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:40)
        at org.apache.accumulo.server.trace.TraceServer$Receiver.span(TraceServer.java:136)
        at org.apache.accumulo.cloudtrace.thrift.SpanReceiver$Processor$span.process(SpanReceiver.java:205)
        at org.apache.accumulo.cloudtrace.thrift.SpanReceiver$Processor.process(SpanReceiver.java:185)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2014-01-15 23:41:10,533 [trace.TraceServer] ERROR: Unable to write mutation to table: org.apache.accumulo.core.data.Mutation@0
java.lang.IllegalStateException: Closed
        at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:192)
        at org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:40)
        at org.apache.accumulo.server.trace.TraceServer$Receiver.span(TraceServer.java:137)
        at org.apache.accumulo.cloudtrace.thrift.SpanReceiver$Processor$span.process(SpanReceiver.java:205)
        at org.apache.accumulo.cloudtrace.thrift.SpanReceiver$Processor.process(SpanReceiver.java:185)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
{noformat}

I think there are two issues here: that the writer gets closed and that our error handling
to reset doesn't properly account for enough error conditions (right now it's just Mutations
Rejected).

I've been talking in IRC as I try to figure out how the tracer's writer could get closed.
Can we coordinate there?

> tracer reports: IllegalStateException: Closed
> ---------------------------------------------
>
>                 Key: ACCUMULO-2213
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2213
>             Project: Accumulo
>          Issue Type: Bug
>          Components: trace
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>            Priority: Minor
>             Fix For: 1.4.5, 1.5.1, 1.6.0
>
>
> During a 24 hour continuous ingest test with agitation, the following was reported 42k
times:
> {noformat}
> Timer task failed org.apache.accumulo.tracer.TraceServer$1 Closed
> 	java.lang.IllegalStateException: Closed
> 		at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.flush(TabletServerBatchWriter.java:302)
> 		at org.apache.accumulo.core.client.impl.BatchWriterImpl.flush(BatchWriterImpl.java:59)
> 		at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:225)
> 		at org.apache.accumulo.tracer.TraceServer.access$400(TraceServer.java:75)
> 		at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:217)
> 		at org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> 		at java.util.TimerThread.mainLoop(Timer.java:512)
> 		at java.util.TimerThread.run(Timer.java:462)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message