hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: nosuchelementexception causing I/O reactor to be terminated
Date Wed, 29 Nov 2017 21:45:57 GMT
On Wed, Nov 29, 2017 at 6:44 PM, Nitu Choudhary <nitu@oath.com.INVALID> 
wrote:
> Thanks Oleg. I looked at your example, but it was not clear to me as 
> to how
> I can avoid passing an input stream from the http request. So in our 
> case
> the incoming data from the client is obtained in form of an input 
> stream
> and we create a HttpAsyncRequestProducer and pass the entity to it. 
> Do you
> have any examples for such a use case to see how we can improve our 
> code?

Nitu

InputStream / OutputStream interfaces are inherently blocking. If you 
are absolutely sure the request data cannot be represented in any other 
form you will likely be better off using the classic (blocking) Apache 
HttpClient.

Oleg


> 
> regards,
> Nitu
> 
> On Tue, Nov 28, 2017 at 1:18 PM, Oleg Kalnichevski <olegk@apache.org> 
> wrote:
> 
>>  On Tue, Nov 28, 2017 at 7:43 PM, Nitu Choudhary 
>> <nitu@oath.com.INVALID>
>>  wrote:
>> 
>>>  sorry, I am still a little confused. Can you explain the message 
>>> exchange
>>>  in more detail?
>>> 
>>>  Say I get a future when I make a request. Now I will typically do a
>>>  future.get and get the future http response and consume it. 
>>> However, in
>>>  some cases I might not be able to call a get on the future, but 
>>> the future
>>>  is still holding the response. When does this memory get released?
>>> 
>>>  regards,
>>>  Nitu
>>> 
>> 
>>  Hi Nitu
>> 
>>  The response future does not necessarily hold a reference to the 
>> entire
>>  response message (though it might in the most over-simplistic 
>> scenario,
>>  which I would recommend to avoid). It rather holds reference to a 
>> result of
>>  response processing generated by the associated 
>> HttpAsyncResponseConsumer.
>>  See this sample for a reference
>> 
>>  http://hc.apache.org/httpcomponents-asyncclient-4.1.x/
>>  httpasyncclient/examples/org/apache/http/examples/nio/clien
>>  t/AsyncClientHttpExchangeStreaming.java
>> 
>>  HttpAsyncResponseConsumer is responsible for resource allocation and
>>  deallocation. HttpAsyncClient makes sure 
>> HttpAsyncResponseConsumer-s get to
>>  complete their work (normally or abnormally) regardless of what 
>> happens to
>>  the response future. As long as the result object does not hold any
>>  resources beyond objects in the heap, it does not need to be 
>> explicitly
>>  closed or released.
>> 
>>  Hope this helps
>> 
>>  Oleg
>> 
>> 
>> 
>> 
>>>  On Tue, Nov 28, 2017 at 12:15 AM, Oleg Kalnichevski 
>>> <olegk@apache.org>
>>>  wrote:
>>> 
>>> 
>>>> 
>>>>   On Mon, Nov 27, 2017 at 6:57 PM, Nitu Choudhary 
>>>> <nitu@oath.com.INVALID>
>>>>   wrote:
>>>> 
>>>>   thanks Oleg. What happens if in a successful case the future is 
>>>> not
>>>>>   consumed by the application? When will that memory be released?
>>>>> 
>>>>> 
>>>>   Upon completion of the message exchange (assuming both request 
>>>> producer
>>>>   and response consumer correctly deallocate their resources).
>>>> 
>>>>   Oleg
>>>> 
>>>> 
>>>>   regards,
>>>>>   Nitu
>>>>> 
>>>>>   On Thu, Nov 23, 2017 at 3:08 AM, Oleg Kalnichevski 
>>>>> <olegk@apache.org>
>>>>>   wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>>    On Wed, Nov 22, 2017 at 9:19 PM, Nitu Choudhary
>>>>>>  <nitu@oath.com.INVALID>
>>>>>>    wrote:
>>>>>> 
>>>>>>    Hi,
>>>>>> 
>>>>>>> 
>>>>>>>    Under high concurrency we see cases where I/O reactor is 
>>>>>>> getting
>>>>>>>    terminated
>>>>>>>    when closing the input stream. In our application we also

>>>>>>> close the
>>>>>>>   input
>>>>>>>    stream as we want to cleanup the memory under various 
>>>>>>> exception
>>>>>>>    conditions.
>>>>>>>    What is the recommended way to consume the http entity? It

>>>>>>> shows
>>>>>>>  that
>>>>>>>   on
>>>>>>>    timeout the nio client is trying to consume the entity, but

>>>>>>> it has
>>>>>>>   already
>>>>>>>    been consumed by our application as it got a done signal,

>>>>>>> and we
>>>>>>>   retrieved
>>>>>>>    the entity and closed it.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>    Hi Nitu
>>>>>> 
>>>>>>    Unlike its blocking counterpart HttpAsyncClient automatically
>>>>>>   deallocates
>>>>>>    resources once a message exchange is complete (successfully or
>>>>>>    unsuccessfully). One does not need to ensure the response 
>>>>>> entity is
>>>>>>   fully
>>>>>>    consumed in order to ensure proper deallocation of the 
>>>>>> connection
>>>>>>    associated with it. This is taken care of by the framework.
>>>>>> 
>>>>>>    However I would strongly recommend to avoid mixing 
>>>>>> non-blocking I/O
>>>>>>  with
>>>>>>    inherently blocking InputStream / OutputStream based APIs. I 
>>>>>> would
>>>>>>    recommend using proper non-blocking HttpAsyncRequestProducer 
>>>>>> and
>>>>>>    HttpAsyncResponseConsumer implementations for any non-trivial
>>>>>>  request /
>>>>>>    response exchanges with HttpAsyncClient. In other words try 
>>>>>> getting
>>>>>>  rid
>>>>>>   of
>>>>>>    SequenceInputStream entirely.
>>>>>> 
>>>>>>    Hope this helps
>>>>>> 
>>>>>>    Oleg
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>    Here is the stack for your reference:
>>>>>> 
>>>>>>> 
>>>>>>>    org.apache.http.impl.nio.client.InternalHttpAsyncClient -
I/O
>>>>>>>  reactor
>>>>>>>    terminated abnormally java.util.NoSuchElementException: null

>>>>>>> at
>>>>>>>    java.util.
>>>>>>>    ArrayList$Itr.next(ArrayList.java:854) at
>>>>>>>   java.util.Collections$3.nextEl
>>>>>>>    ement(Collections.java:5216) at 
>>>>>>> java.io.SequenceInputStream.ne
>>>>>>>   xtStream(
>>>>>>>    SequenceInputStream.java:110) at 
>>>>>>> java.io.SequenceInputStream.close(
>>>>>>>    SequenceInputStream.java:232) at 
>>>>>>> org.apache.http.nio.entity.Ent
>>>>>>>    
>>>>>>> ityAsyncContentProducer.close(EntityAsyncContentProducer.java:92)
>>>>>>>  at
>>>>>>>   org.
>>>>>>>    apache.http.nio.protocol.BasicAsyncRequestProducer.close(
>>>>>>>    BasicAsyncRequestProducer.java:155) at
>>>>>>>  org.apache.http.impl.nio.clien
>>>>>>>   t.
>>>>>>>    DefaultClientExchangeHandlerImpl.releaseResources(DefaultC
>>>>>>>    lientExchangeHandlerImpl.java:84) at 
>>>>>>> org.apache.http.impl.nio.clien
>>>>>>>  t.
>>>>>>>    AbstractClientExchangeHandler.failed(AbstractClientExchangeH
>>>>>>>    andler.java:429)
>>>>>>>    at 
>>>>>>> org.apache.http.impl.nio.client.AbstractClientExchangeHandler.
>>>>>>>    
>>>>>>> connectionRequestFailed(AbstractClientExchangeHandler.java:348)

>>>>>>> at
>>>>>>>   org.
>>>>>>>    apache.http.impl.nio.client.AbstractClientExchangeHandler.ac
>>>>>>>  cess$100(
>>>>>>>    AbstractClientExchangeHandler.java:62) at
>>>>>>>   org.apache.http.impl.nio.clien
>>>>>>>    t.
>>>>>>>    AbstractClientExchangeHandler$1.failed(AbstractClientExchang
>>>>>>>    eHandler.java:
>>>>>>>    392) at 
>>>>>>> org.apache.http.concurrent.BasicFuture.failed(BasicFuture.
>>>>>>> 
>>>>>>>    java:134)
>>>>>>>    at 
>>>>>>> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionMa
>>>>>>>   nager$1.
>>>>>>>    failed(PoolingNHttpClientConnectionManager.java:289) at
>>>>>>>   org.apache.http.
>>>>>>>    concurrent.BasicFuture.failed(BasicFuture.java:134) at
>>>>>>>    org.apache.http.nio.
>>>>>>>    pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at
>>>>>>>   org.apache.
>>>>>>>    http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIO
>>>>>>>    ConnPool.java:
>>>>>>>    584) at org.apache.http.nio.pool.Abstr
>>>>>>>  actNIOConnPool$InternalSession
>>>>>>>    RequestCallback.timeout(AbstractNIOConnPool.java:851) at
>>>>>>>   org.apache.http.
>>>>>>>    impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestI
>>>>>>>   mpl.java:183)
>>>>>>>    at
>>>>>>>    org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.
>>>>>>>    processTimeouts(
>>>>>>>    DefaultConnectingIOReactor.java:210) at
>>>>>>>  org.apache.http.impl.nio.react
>>>>>>>   or.
>>>>>>>    DefaultConnectingIOReactor.processEvents(DefaultConnectingIO
>>>>>>>    Reactor.java:155)
>>>>>>>    at 
>>>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReacto
>>>>>>>   r.execute(
>>>>>>>    AbstractMultiworkerIOReactor.java:348) at
>>>>>>>   org.apache.http.impl.nio.conn.
>>>>>>>    PoolingNHttpClientConnectionManager.execute(PoolingNHttpClie
>>>>>>>    ntConnectionManager.java:194) at 
>>>>>>> org.apache.http.impl.nio.client.
>>>>>>>    CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClient
>>>>>>>   Base.java:64)
>>>>>>>    at
>>>>>>>    java.lang.Thread.run(Thread.java:748)
>>>>>>> 
>>>>>>>    regards,
>>>>>>>    nitu
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>    ------------------------------------------------------------
>>>>>>  ---------
>>>>>>    To unsubscribe, e-mail: 
>>>>>> httpclient-users-unsubscribe@hc.apache.org
>>>>>>    For additional commands, e-mail: 
>>>>>> httpclient-users-help@hc.apache.org
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>   
>>>> ---------------------------------------------------------------------
>>>>   To unsubscribe, e-mail: 
>>>> httpclient-users-unsubscribe@hc.apache.org
>>>>   For additional commands, e-mail: 
>>>> httpclient-users-help@hc.apache.org
>>>> 
>>>> 
>>>> 
>> 
>>  
>> ---------------------------------------------------------------------
>>  To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>  For additional commands, e-mail: httpclient-users-help@hc.apache.org
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message