hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Httpcomponents Wiki] Update of "LessonsLearned" by CharlesHonton
Date Thu, 04 Sep 2008 19:11:03 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Httpcomponents Wiki" for change notification.

The following page has been changed by CharlesHonton:

New page:
= 100+ websites =

In a recent project which actively queries hundreds of different websites, I rediscovered
some practices which require configuration changes

== User Agent ==
Several websites responded with 500 status code when presented with the default User-Agent
header.  One website sent a 200 status code but the html content of the page was truncated
with "500 server error"  For maximum compatibility, use a standard web browser user-agent

http.useragent = Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/2008070208

== Cookie Policies ==
Very few websites support anything other than base Netscape cookies.

http.protocol.cookie-policy = compatibility

== Cookie Header ==
Although some websites support multiple Cookie headers, many do not.  The documentation for
http.protocol.single-cookie-header is misleading.  This parameter determines how Cookie headers
are sent in the request.  Multiple Set-Cookie headers are always supports.

http.protocol.single-cookie-header = true

== Post/Redirect/Get ==
[http://en.wikipedia.org/wiki/Post/Redirect/Get Post redirecting to Get] turns out to be a
common practice.  Contrary to the [http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.3
RFC 2616] recommendation, this practice relies on the "broken" behavior of major web browsers.
 The query portion for the GET comes strictly from the Location header returned with the 302
response to the POST.

== Certificates ==
The certificate database ($JAVA_HOME/lib/security/cacerts) in the standard java distribution
contains one third of the root certificates that are present in Firefox or Internet Explorer.
 The following script can help you with this task.
# How to use this script
# 1.  Create temp dir
# 2. In firefox, select Tools/Options/Advanced/Encryption/View Certificates
# 3. In the Certificate Manager Dialog, Authorities tab; select all certificates and press
# 4. Select OK and rename files as necessary 
# 5. Add all firefox trusted authorities not already in cacerts by running this file in the
temp directory

rm log
# copy the current certificate database
cp "$JAVA_HOME/jre/lib/security/cacerts" .
# Determine current list of authorities
keytool -list -v -keystore cacerts -storepass changeit |grep "Issuer:"|sort > before

for file in *.crt; do
  echo "================== ${alias} ==================" > "${logfile}"
  echo "==printcert" >> "${logfile}"
  keytool -printcert -file "${file}" -keystore cacerts -storepass changeit >> "${logfile}"
  owner=$(grep "Owner:" "${logfile}" | sed -e "s/Owner: //")
  issuer=$(grep "Issuer:" "${logfile}" | sed -e "s/Issuer: //")
  # is this a root certificate?
  if [ "${owner}" = "${issuer}" ] ; then
    # determine an alias
    for (( i= 0; i<10; i= i+1 )) ; do
      echo "==list ${alias}" >> "${logfile}"
      if ( keytool -list -keystore cacerts -storepass changeit -alias "${alias}" >>
"${logfile}" 2>&1 ) then
    # import the key
    echo "==import" >> "${logfile}"
    keytool -import -file "${file}" -keystore cacerts -storepass changeit -alias ${alias}
>> "${logfile}" 2>&1 <<response
    # delete any key which is a duplicate
    if ( grep "Certificate already exists in keystore under alias" "${logfile}" > /dev/null
); then
      echo "==delete" >> "${logfile}"
      keytool -delete -keystore cacerts -storepass changeit -alias ${alias} >> "${logfile}"
  cat "${logfile}" >> log
#Determine new list of authorities
keytool -list -v -keystore cacerts -storepass changeit |grep "Issuer:"|sort > after
#Determine change list of authorities
diff before after

To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org

View raw message