Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Message-ID: <58F7A777.6040100@gmail.com>
Date: Wed, 19 Apr 2017 14:07:51 -0400
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Postbox 3.0.11 (Macintosh/20140602)
MIME-Version: 1.0
To: user@accumulo.apache.org
Subject: Re: Batchscanner - error closing output stream because stream is
 closed
References: <ec05167a-6231-ae0a-a7ef-77d8c2de0c37@incadencecorp.com> <58F7A1AF.80803@gmail.com> <9ddb848c-d1a9-4330-a164-36c8d379a7da@incadencecorp.com>
In-Reply-To: <9ddb848c-d1a9-4330-a164-36c8d379a7da@incadencecorp.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Wed, 19 Apr 2017 18:08:01 -0000

19824 is an identifier that would tie it back to a specific BatchScanner 
instance, but that's not going to tell you where in the code it was created.

I remember something with a finalize() method being invoked in a 
try-with-resources block (the finalizer closes the BS, and then the 
finally for the try-with-resources also tries to close() it). Let me see 
if I can find that issue -- we might have made a code-change in 1.7.3 
for it.

David Boyd wrote:
> Josh:
>
> Thanks. I am using 1.7.2. That is why I asked about how to trace back to
> a specific scan in
> my code. I agree this sounds like a race condition but I have multiple
> batch scanners potentially happening
> including some down inside Geomesa. So figuring out from the trace which
> scanner spawned the thread would
> be helpful to generate an snippit.
>
>
>
> On 4/19/17 1:43 PM, Josh Elser wrote:
>> David --
>>
>> What version of Accumulo? Can you share a snippet of code (or,
>> ideally, a executable example we can run) in which you're using the
>> BatchScanner?
>>
>> Generally speaking, it appears that one of the BatchScanner threads
>> was finished, trying to return the Thrift connection to the pool, but
>> the underlying socket was already closed (assumably, by someone else).
>> Sounds like a race condition, multiple close()'s happening perhaps.
>>
>> David Boyd wrote:
>>> All:
>>>
>>> I am getting this stack trace periodically based on no pattern I can
>>> determine from my application.
>>>
>>> Is this a message I should be worried about?
>>>
>>> What is a technique to trace this back to my code and the cause?
>>>
>>> Obviously something is closing things before the thread closes it.
>>>
>>>
>>>> 2017-04-19 12:33:53,423 | WARN | [batch scanner 19824- 8 looking up 1
>>>> ranges at accumulodev:9997] | (TIOStreamTransport.java:112) - Error
>>>> closing output stream.
>>>> java.io.IOException: The stream is closed
>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
>>>>
>>>>
>>>> at
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
>>>> at
>>>> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
>>>>
>>>>
>>>> at
>>>> org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)
>>>>
>>>>
>>>> at
>>>> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309)
>>>>
>>>>
>>>> at
>>>> org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571)
>>>>
>>>>
>>>> at
>>>> org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151)
>>>>
>>>> at
>>>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:686)
>>>>
>>>>
>>>> at
>>>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:349)
>>>>
>>>>
>>>> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>
>>>>
>>>> at
>>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>>>
>>>>
>>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>