hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: how to terminate data consuming by asyncclient without declining of the result
Date Tue, 09 Aug 2011 19:24:42 GMT
On Tue, 2011-08-09 at 19:24 +0300, Xasima wrote:
> Hello. I 'm performing  numerous requests to the search API, but interested
> only in certain field (JSON / HTML attribute) that is presented usually in
> the beginning of the document with search response. The parsing is performed
> with the help of  findWithinHorizon() method of Scanner, that allows to
> efficiently iterate over the stream. I have decided to drop the connection
> as far as the necessary field has been obtained. To perform this I close the
> source endpoint of pipe (that is used as bridge between httclient consumer
> and actual parsing logic).
> 
>      Parsing code
>             Scanner s = new Scanner(pipe.source());
> String match = s.findWithinHorizon(patternTotalResults, 0);
>             this.buildResult = Integer.parseInt(match);
> 
>             // Send close() to the Pipe
>            * pipe.source().close();*
> 
> As far as the "bridge" pipe is closed, onByteReceived() methods that
> performs data consuming obtains the IOException on write and going to drop
> the active connection.
> 
> 
> Consumer code
> 
> public abstract class CrawlerParsingConsumer extends
> AsyncByteConsumer<Integer> {
> 
> @Override
> protected void onByteReceived(ByteBuffer buffer, IOControl control)
> throws IOException {
> 
> boolean isForcedTermination = false;
> log.debug("Try to process block of " + buffer.remaining() + " size");
> try {
> while (buffer.hasRemaining()) {
> pipe.sink().write(buffer);
> }
> } catch (IOException e) {
> log.debug("Seems that pipe has been intensionally closed: " + e);
> isForcedTermination = true;
> } finally {
> if (isForcedTermination) {
> log.debug("Some data of block may been discarded. "
> + "Drop the connection as well since the rest of data is not needed");
> *super.cancel(); //HERE I WANT TO DROP THE CONNECTION *
> } else
> log.debug("Block has been processed");
> }
> 
> }
> 
> 
> The problem is that despite of already calculated result (buildResult), the
> future returned by httpclient.execute() is NULL, due to the NPE inside the
> HttpClient.
> 
>       Execution code
>                 HttpAsyncGet httpAsyncGet = new HttpAsyncGet(request);
> Future<Integer> future =  httpclient.execute(httpAsyncGet, new
> CrawlerParsingConsumer(), null);
>                 ....
>                 *future.get(); //NPE here, since the connection has been
> actually terminated in the middle of data tranfer*

...

> 
> My question is the following. Can I somehow terminate the further processing
> (onByteReceived()) after an event, but remains the future returned from
> httpclient  normal.

You can use IOControl#shutdown() to shut down the underlying connection.
You will also have to build a custom HttpAsyncResponseConsumer
implementation that retains its state even if the underlying connection
gets terminated. 


> Can you recommend another approach to perform partial nio data consuming (n
> threads) and parsing (m threads).

>From what I can gather you would much better off using an HTTP client
based on a blocking I/O model.

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message