hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: NullPointerException in BestMatchSpec.formatCookies
Date Thu, 02 Aug 2012 19:46:37 GMT
So it's can be that neither... Thanks for the links.

I have restarted the application. I have added the debug level for
RequestAddCookies and asked the application to stop as soon as there
is an exception. I just need to let it run for some time. Last time I
saw the exception it was running for few days. So it might take time
before it happens again.

2012/8/2, sebb <sebbaz@gmail.com>:
> On 2 August 2012 19:36, Jean-Marc Spaggiari <jean-marc@spaggiari.org>
> wrote:
>> I'm using 1.7.0_05 from Sun.
>>
>> I have enabled the RequestAddCookies debug logs and will see when I
>> will get the next NPE.
>>
>> One the line 193 (I don't have the source) is version an int? Or an
>> Integer?
>
> int, see:
>
> http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/impl/cookie/BestMatchSpec.html#193
> and
> http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/cookie/Cookie.html#126
>
>> I agree with you, cookie can't be null, else we will have got an NPE
>> at  "if (!matchedCookies.isEmpty())"
>>
>> I will restart my tool with the debug mode enabled and keep you posted.
>>
>> 2012/8/2, sebb <sebbaz@gmail.com>:
>>> On 2 August 2012 19:03, Jean-Marc Spaggiari <jean-marc@spaggiari.org>
>>> wrote:
>>>> I'm using version 4.2.1 binary distribution.
>>>
>>> In which case the NPE is very odd.
>>>
>>> The stack dump shows:
>>>
>>> java.lang.NullPointerException
>>>         at
>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>>         at
>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>>
>>> Now the code in question is:
>>>
>>> BestMatchSpec.java:193: if (cookie.getVersion() < version) {
>>>
>>> AFAICT, this can only cause NPE if "cookie" is null.
>>>
>>> The cookie variable is derived from the list created in
>>> RequestAddCookies.java:
>>>
>>>         List<Cookie> cookies = new
>>> ArrayList<Cookie>(cookieStore.getCookies());
>>>         // Find cookies matching the given origin
>>>         List<Cookie> matchedCookies = new ArrayList<Cookie>();
>>>         Date now = new Date();
>>>         for (Cookie cookie : cookies) {
>>>             if (!cookie.isExpired(now)) { // <== cookie cannot be null
>>> here
>>>                 if (cookieSpec.match(cookie, cookieOrigin)) {
>>>                     if (this.log.isDebugEnabled()) {
>>>                         this.log.debug("Cookie " + cookie + " match "
>>> + cookieOrigin);
>>>                     }
>>>                     matchedCookies.add(cookie); // <== so cannot be null
>>> here
>>>                 }
>>>             } else {
>>>                 if (this.log.isDebugEnabled()) {
>>>                     this.log.debug("Cookie " + cookie + " expired");
>>>                 }
>>>             }
>>>         }
>>>         // Generate Cookie request headers
>>>         if (!matchedCookies.isEmpty()) {
>>>             List<Header> headers =
>>> cookieSpec.formatCookies(matchedCookies); // <== 196
>>>             for (Header header : headers) {
>>>                 request.addHeader(header);
>>>             }
>>>         }
>>>
>>> I cannot see how a null cookie can be added to the list, nor how a
>>> list entry could be set to null later.
>>>
>>> It would be worth enabling Debug logging for the RequestAddCookies
>>> class.
>>>
>>> What JVM are you using?
>>>
>>>> 2012/8/2, sebb <sebbaz@gmail.com>:
>>>>> On 1 August 2012 12:08, Jean-Marc Spaggiari <jean-marc@spaggiari.org>
>>>>> wrote:
>>>>>> Hi Oleg,
>>>>>>
>>>>>> My application is reading many other URLs before this one. So maybe
>>>>>> it's a combination of multiple loads.
>>>>>>
>>>>>> I will try to dump more when I will get the issue again.
>>>>>>
>>>>>> In the meantime, here is how I'm initializing the HttpClient:
>>>>>>
>>>>>> // Creation of the HTTP Client
>>>>>> SchemeRegistry schemeRegistry = new SchemeRegistry();
>>>>>> schemeRegistry.register(new Scheme("http", 80,
>>>>>> PlainSocketFactory.getSocketFactory()));
>>>>>> schemeRegistry.register(new Scheme("https", 443,
>>>>>> SSLSocketFactory.getSocketFactory()));
>>>>>> PoolingClientConnectionManager cm = new
>>>>>> PoolingClientConnectionManager(schemeRegistry);
>>>>>> cm.setMaxTotal(200);
>>>>>> cm.setDefaultMaxPerRoute(20);
>>>>>>
>>>>>> HttpParams params = new BasicHttpParams();
>>>>>> if (useProxy)
>>>>>>         params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new
>>>>>> HttpHost("proxy", 80));
>>>>>> client = new DefaultHttpClient(cm, params);
>>>>>> client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT,
new
>>>>>> Integer(30000));
>>>>>> client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
>>>>>> new Integer(30000));
>>>>>> client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY,
>>>>>> false);
>>>>>>
>>>>>> It's not the first time I'm getting this exception, so I might be
>>>>>> able
>>>>>> to reproduce that in a near futur.
>>>>>
>>>>> What version of HttpClient are you using?
>>>>>
>>>>>> JM
>>>>>>
>>>>>> 2012/8/1, Oleg Kalnichevski <olegk@apache.org>:
>>>>>>> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm getting the exception below when I'm trying to load
>>>>>>>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
>>>>>>>> ... It's not critical since I'm catching everything, but
I'm
>>>>>>>> wondering
>>>>>>>> if this is normal or if this is something which need to be
fixed...
>>>>>>>>
>>>>>>>> java.lang.NullPointerException
>>>>>>>>      at
>>>>>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>>>>>>>      at
>>>>>>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>>>>>>>      at
>>>>>>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
>>>>>>>>      at
>>>>>>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
>>>>>>>>      at
>>>>>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
>>>>>>>>      at
>>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>>>>>>>      at
>>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>>>>>>>>      at
>>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>>>>>>>>      at
>>>>>>>> org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> JM
>>>>>>>>
>>>>>>>
>>>>>>> Jean-Marc
>>>>>>>
>>>>>>> I am not able to reproduce the problem just by executing HTTP
GET to
>>>>>>> the
>>>>>>> URL in question. All looks perfectly normal.
>>>>>>>
>>>>>>> [DEBUG] BasicClientConnectionManager - Get connection for route
>>>>>>> {}->http://www.expedia.ca
>>>>>>> [DEBUG] DefaultClientConnectionOperator - Connecting to
>>>>>>> www.expedia.ca:80
>>>>>>> [DEBUG] RequestAddCookies - CookieSpec selected: best-match
>>>>>>> [DEBUG] RequestAuthCache - Auth cache not set in the context
>>>>>>> [DEBUG] RequestTargetAuthentication - Target auth state:
>>>>>>> UNCHALLENGED
>>>>>>> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
>>>>>>> [DEBUG] DefaultHttpClient - Attempt 1 to execute request
>>>>>>> [DEBUG] DefaultClientConnection - Sending request:
>>>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>>>>> [DEBUG] headers - >>
>>>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>>>>> [DEBUG] headers - >> Host: www.expedia.ca
>>>>>>> [DEBUG] headers - >> Connection: Keep-Alive
>>>>>>> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1
(java 1.5)
>>>>>>> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1
200
>>>>>>> OK
>>>>>>> [DEBUG] headers - << HTTP/1.1 200 OK
>>>>>>> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8
>>>>>>> [DEBUG] headers - << Content-Language: en-CA
>>>>>>> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO
DSP IND
>>>>>>> COR
>>>>>>> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
>>>>>>> [DEBUG] headers - << RTSS: 1
>>>>>>> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
>>>>>>> [DEBUG] headers - << Transfer-Encoding:  chunked
>>>>>>> [DEBUG] headers - << Connection: keep-alive
>>>>>>> [DEBUG] headers - << Connection: Transfer-Encoding
>>>>>>> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
>>>>>>> [DEBUG] headers - << Set-Cookie:
>>>>>>> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
>>>>>>> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
>>>>>>> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
>>>>>>> Path=/
>>>>>>> [DEBUG] headers - << Set-Cookie:
>>>>>>> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
>>>>>>> Path=/
>>>>>>> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca;
Path=/
>>>>>>> [DEBUG] headers - << Set-Cookie:
>>>>>>> p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
>>>>>>> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
>>>>>>> 01-Aug-2017 15:54:40 GMT; Path=/
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> MC1][value:
>>>>>>> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path:
>>>>>>> /][expiry:
>>>>>>> Fri Aug 04 04:44:09 GMT+01:00 2017]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry:
>>>>>>> null]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> JSESSION][value:
>>>>>>> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path:
>>>>>>> /][expiry:
>>>>>>> null]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
>>>>>>> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01
>>>>>>> 16:54:40
>>>>>>> GMT+01:00 2017]".
>>>>>>> [DEBUG] DefaultHttpClient - Connection can be kept alive
>>>>>>> indefinitely
>>>>>>>
>>>>>>> Can you put together a small test app that could be used to
>>>>>>> reproduce
>>>>>>> the NPE?
>>>>>>>
>>>>>>> Oleg
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message