hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: HttpAsyncClient and cookies
Date Wed, 19 Dec 2012 16:56:24 GMT
On Tue, 2012-12-18 at 15:36 -0800, vigna wrote:
> We are trying to use DefaultHttpAsyncClient in our new crawler. We need to
> handle a few hundred connections per thread asynchronously, and it seems the
> right candidate. 
> 
> In the last weeks we experimented with many DefaultHttpClient on a few
> thousand threads and it worked well (actually, we found a couple of bugs, as
> our crawls are very wide and meet any kind of server configuration errors).
> Consider that we crawl URLs from different sites continuously, so we need to
> change at each request the cookie store, which we do by direct management of
> the store itself.
> 
> After digging the (little) documentation, I really couldn't figure out how
> to manage cookies with HttpAsyncClient. Any suggestion or code snipped would
> be really welcome: what we need to do, basically, is:
> 
> - keep a few hundred GET requests open in parallel.
> - use for each request an AsyncByteConsumer to accumulate in a buffer the
> content, and in some data structure headers, cookies, etc.
> - on completion, schedule the received data for analysis.
> 
> All this requires however to manage cookies, and I could not understand how
> to modify the cookie store for each async request, and how to get the cookie
> store in onResponseReceived().
> 
> Any help appreciated!
> 
> seba
> 

---
static class MyResponseConsumer extends AsyncCharConsumer<Boolean> {

    @Override
    protected void onResponseReceived(final HttpResponse response) {
    }

    @Override
    protected void onCharReceived(final CharBuffer buf, final IOControl
ioctrl) throws IOException {
        while (buf.hasRemaining()) {
            System.out.print(buf.get());
        }
    }

    @Override
    protected void releaseResources() {
    }

    @Override
    protected Boolean buildResult(final HttpContext context) {
        CookieStore cookieStore = (CookieStore) context.getAttribute(
                ClientContext.COOKIE_STORE);
        List<Cookie> cookies = cookieStore.getCookies();
        for (Cookie cookie: cookies) {
            System.out.println(cookie);
        }
        return Boolean.TRUE;
    }

}

public static void main(String[] args) throws Exception {
    
    HttpAsyncClient httpclient = new DefaultHttpAsyncClient();
    httpclient.start();
    try {
        BasicCookieStore cookieStore = new BasicCookieStore();
        BasicHttpContext context = new BasicHttpContext();
        context.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
        Future<Boolean> future = httpclient.execute(
                HttpAsyncMethods.createGet("http://www.google.com/"),
                new MyResponseConsumer(), null);
        Boolean result = future.get();
        if (result != null && result.booleanValue()) {
            System.out.println("Request successfully executed");
        } else {
            System.out.println("Request failed");
        }
        System.out.println("Shutting down");
    } finally {
        httpclient.shutdown();
    }
    System.out.println("Done");

}
---

Oleg



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message