Mailing-List: contact httpclient-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpClient User Discussion" <httpclient-user@jakarta.apache.org>
Received-SPF: pass (athena.apache.org: domain of raymond@thinkparity.com
 designates 209.85.146.183 as permitted sender)
Message-ID: <c7fd04b40709040908l1279e90j619c1c23c7fafa6d@mail.gmail.com>
Date: Tue, 4 Sep 2007 09:08:39 -0700
From: "Raymond Kroeker" <raymond@thinkparity.com>
To: "HttpClient User Discussion" <httpclient-user@jakarta.apache.org>
Subject: Re: after 60000 requests more than 700 sockets in CLOSE_WAIT
In-Reply-To: <AC908349540F9745AA5AAABCACBE9EE4057856B7@seams042.starbucks.net>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_18044_18374893.1188922119105"
References: <2F2FA11BC93EB2498BAC4718363A9A550493615E@EX5.d.ethz.ch>
	 <AC908349540F9745AA5AAABCACBE9EE4057856B5@seams042.starbucks.net>
	 <2F2FA11BC93EB2498BAC4718363A9A550493619E@EX5.d.ethz.ch>
	 <AC908349540F9745AA5AAABCACBE9EE4057856B7@seams042.starbucks.net>

------=_Part_18044_18374893.1188922119105
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi Armin,
  If your requests are all coming from the same source to the same target,
and the target server supports the option; you might want to take a look at
using a keep alive header.
http://wiki.apache.org/jakarta-httpclient/ConnectionManagementDesign#head-9bd8b93d0d47d5391cff0db414e2593f4cd2f0a3

Raymond

On 9/4/07, Pete Keyes <PKeyes@starbucks.com> wrote:
>
> I think these 2 are the ones you want to look at tuning:
>
> # Solaris TCP_TIME_WAIT_INTERVAL
>
>     * Description: Notifies TCP/IP on how long to keep the connection
> control blocks closed. After the applications complete the TCP/IP
> connection, the control blocks are kept for the specified time. When
> high connection rates occur, a large backlog of the TCP/IP connections
> accumulates and can slow server performance. The server can stall during
> certain peak periods. If the server stalls, the netstat command shows
> that many of the sockets that are opened to the HTTP server are in the
> CLOSE_WAIT or FIN_WAIT_2 state. Visible delays can occur for up to four
> minutes, during which time the server does not send any responses, but
> CPU utilization stays high, with all of the activities in system
> processes.
>     * How to view or set: Use the get command to determine the current
> interval and the set command to specify an interval of 30 seconds. For
> example:
>
>       ndd -get /dev/tcp tcp_time_wait_interval
>       ndd -set /dev/tcp tcp_time_wait_interval 30000
>
>     * Default value: The default time wait interval for a Solaris
> operating system is 240000 milliseconds, which is equal to 4 minutes.
>     * Recommended value: 60000 milliseconds
>
> # Solaris TCP_FIN_WAIT_2_FLUSH_INTERVAL
>
>     * Description: Specifies the timer interval prohibiting a connection
> in the FIN_WAIT_2 state to remain in that state. When high connection
> rates occur, a large backlog of TCP/IP connections accumulates and can
> slow server performance. The server can stall during peak periods. If
> the server stalls, using the netstat command shows that many of the
> sockets opened to the HTTP server are in the CLOSE_WAIT or FIN_WAIT_2
> state. Visible delays can occur for up to four minutes, during which
> time the server does not send any responses, but CPU utilization stays
> high, with all of the activity in system processes.
>     * How to view and set: Use the get command to determine the current
> interval and the set command to specify an interval of 67.5 seconds. For
> example,
>
>       ndd -get /dev/tcp tcp_fin_wait_2_flush_interval
>       ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500
>
>     * Default value: 675000 milliseconds
>     * Recommended value: 67500 milliseconds
>
>
> ...Pete
> Starbucks Coffee Co. - MS IT-5
> 2401 Utah Ave S
> Seattle, WA. 98134
> (w) 206-318-5933
>
> -----Original Message-----
> From: Wittmann Armin [mailto:awittmann@ethz.ch]
> Sent: Tuesday, September 04, 2007 8:16 AM
> To: HttpClient User Discussion
> Subject: RE: after 60000 requests more than 700 sockets in CLOSE_WAIT
>
> Hi Pete
>
> under Linux it is found at a different place (in proc-fs). Look at the
> output below. I guess most settings are in seconds.
> I liked that one for newbies:
> http://www.utdallas.edu/~cantrell/ee6345/pocketguide.pdf
> http://ipsysctl-tutorial.frozentux.net/chunkyhtml/tcpvariables.html
>
> Any suggestions?
> 5 hours after the test stopped (tomcat is still running) none
> of the sockets has been freed.
>
> Thanks, Armin
>
> linux:>$ ls | grep tcp_
> tcp_abc
> tcp_abort_on_overflow
> tcp_adv_win_scale
> tcp_app_win
> tcp_congestion_control
> tcp_dsack
> tcp_ecn
> tcp_fack
> tcp_fin_timeout
> tcp_frto
> tcp_keepalive_intvl
> tcp_keepalive_probes
> tcp_keepalive_time
> tcp_low_latency
> tcp_max_orphans
> tcp_max_syn_backlog
> tcp_max_tw_buckets
> tcp_mem
> tcp_moderate_rcvbuf
> tcp_no_metrics_save
> tcp_orphan_retries
> tcp_reordering
> tcp_retrans_collapse
> tcp_retries1
> tcp_retries2
> tcp_rfc1337
> tcp_rmem
> tcp_sack
> tcp_stdurg
> tcp_synack_retries
> tcp_syncookies
> tcp_syn_retries
> tcp_timestamps
> tcp_tso_win_divisor
> tcp_tw_recycle
> tcp_tw_reuse
> tcp_window_scaling
> tcp_wmem
> linux:>$ cat tcp_fin_timeout
> 60
> linux:>$ cat tcp_keepalive_time
> 7200
> linux:>$ cat tcp_keepalive_probes
> 9
> linux:>$ cat tcp_keepalive_intvl
> 75
> linux:>$ cat tcp_fin_timeout
> 60
>
> > -----Original Message-----
> > From: Pete Keyes [mailto:PKeyes@starbucks.com]
> > Sent: Tuesday, September 04, 2007 4:21 PM
> > To: HttpClient User Discussion
> > Subject: RE: after 60000 requests more than 700 sockets in CLOSE_WAIT
> >
> > I believe you problem has to do with a UNIX network (tcp)
> > configuration
> > setting.  We've seen this often in volume testing.  The socket is left
> > in this state for re-use...
> >
> > Not sure how the configuration is handled on Linux, but on Solaris you
> > can see all the configuration options with the following:
> >       ndd -get /dev/tcp ?
> > tcp_time_wait_interval        (read and write)
> > tcp_conn_req_max_q            (read and write)
> > tcp_conn_req_max_q0           (read and write)
> > tcp_conn_req_min              (read and write)
> > tcp_conn_grace_period         (read and write)
> > tcp_cwnd_max                  (read and write)
> > tcp_debug                     (read and write)
> > tcp_smallest_nonpriv_port     (read and write)
> > tcp_ip_abort_cinterval        (read and write)
> > tcp_ip_abort_linterval        (read and write)
> > tcp_ip_abort_interval         (read and write)
> > tcp_ip_notify_cinterval       (read and write)
> > tcp_ip_notify_interval        (read and write)
> > tcp_ipv4_ttl                  (read and write)
> > tcp_keepalive_interval        (read and write)
> > tcp_maxpsz_multiplier         (read and write)
> > tcp_mss_def_ipv4              (read and write)
> > tcp_mss_max_ipv4              (read and write)
> > tcp_mss_min                   (read and write)
> > tcp_naglim_def                (read and write)
> > tcp_rexmit_interval_initial   (read and write)
> > tcp_rexmit_interval_max       (read and write)
> > tcp_rexmit_interval_min       (read and write)
> > tcp_deferred_ack_interval     (read and write)
> > tcp_snd_lowat_fraction        (read and write)
> > tcp_sth_rcv_hiwat             (read and write)
> > tcp_sth_rcv_lowat             (read and write)
> > tcp_dupack_fast_retransmit    (read and write)
> > tcp_ignore_path_mtu           (read and write)
> > tcp_rcv_push_wait             (read and write)
> > tcp_smallest_anon_port        (read and write)
> > tcp_largest_anon_port         (read and write)
> > tcp_xmit_hiwat                (read and write)
> > tcp_xmit_lowat                (read and write)
> > tcp_recv_hiwat                (read and write)
> > tcp_recv_hiwat_minmss         (read and write)
> > tcp_fin_wait_2_flush_interval (read and write)
> > tcp_co_min                    (read and write)
> > tcp_max_buf                   (read and write)
> > tcp_strong_iss                (read and write)
> > tcp_rtt_updates               (read and write)
> > tcp_wscale_always             (read and write)
> > tcp_tstamp_always             (read and write)
> > tcp_tstamp_if_wscale          (read and write)
> > tcp_rexmit_interval_extra     (read and write)
> > tcp_deferred_acks_max         (read and write)
> > tcp_slow_start_after_idle     (read and write)
> > tcp_slow_start_initial        (read and write)
> > tcp_co_timer_interval         (read and write)
> > tcp_sack_permitted            (read and write)
> > tcp_trace                     (read and write)
> > tcp_compression_enabled       (read and write)
> > tcp_ipv6_hoplimit             (read and write)
> > tcp_mss_def_ipv6              (read and write)
> > tcp_mss_max_ipv6              (read and write)
> > tcp_rev_src_routes            (read and write)
> > tcp_ndd_get_info_interval     (read and write)
> > tcp_rst_sent_rate_enabled     (read and write)
> > tcp_rst_sent_rate             (read and write)
> > tcp_use_smss_as_mss_opt       (read and write)
> > tcp_wroff_xtra                (read and write)
> > tcp_extra_priv_ports          (read only)
> > tcp_extra_priv_ports_add      (write only)
> > tcp_extra_priv_ports_del      (write only)
> > tcp_status                    (read only)
> > tcp_bind_hash                 (read only)
> > tcp_listen_hash               (read only)
> > tcp_conn_hash                 (read only)
> > tcp_acceptor_hash             (read only)
> > tcp_host_param                (read and write)
> > tcp_time_wait_stats           (read only)
> > tcp_host_param_ipv6           (read and write)
> > tcp_1948_phrase               (write only)
> > tcp_reserved_port_list        (read only)
> > tcp_close_wait_interval(obsoleted- use
> > tcp_time_wait_interval) (no read
> > or write)
> >
> > ...Pete
> > Starbucks Coffee Co. - MS IT-5
> > 2401 Utah Ave S
> > Seattle, WA. 98134
> > (w) 206-318-5933
> > -----Original Message-----
> > From: Wittmann Armin [mailto:awittmann@ethz.ch]
> > Sent: Tuesday, September 04, 2007 4:23 AM
> > To: HttpClient User Discussion
> > Subject: after 60000 requests more than 700 sockets in CLOSE_WAIT
> >
> >
> > Hi httpclient-Team
> >
> > over the last 4 days I made a long term test for
> > our new software release.
> >
> > This software ist integrated into an Apache Tomcat 5.5
> > (as a servlet receiving requests, transform them and
> > sending out other GET-requests through http-client) and should
> > run very reliable and without any resource leaks.
> >
> > After finishing this test cycle I noticed that there
> > remained over 700 sockets in CLOSE_WAIT state
> >  (Linux -> netstat -a -p).
> > Due to the identified PID and the destination ip number it is
> > obvious that these sockets have been caused/used
> > by the program part using http-client-3.0.1.
> >
> > I am not a real network crack so I don't know if I need
> > to bother about this. Since the software/tomcat is itended
> > to run months (7x24) without necessarily rebooting it
> > I am not shure if I will run out of network resources.
> >
> > Can somebody help in this subject?
> >
> > By the way: all 60000 http-GET-requests worked well and there
> >       were no other problems at all.
> >
> > Regards
> >
> > Armin
> >
> > --------------------------------------------------------------------
> > My Code (simplified):
> > this code ist executed for every single request
> >
> > HttpConnectionManager connectionManager = new
> > SimpleHttpConnectionManager();
> > HttpClient client = new HttpClient(clientParams, connectionManager);
> > client.setHostConfiguration(hostConfiguration);
> > HttpMethod method =  = new GetMethod();
> > method.setQueryString(pairs);
> > method.setPath(pUrl.getPath());
> > method.setParams(methodParams);
> > try {
> >       client.executeMethod(method);
> > } catch (Exception e) {
> >       failed = true;
> >       throw new Exception(...);
> > } finally {
> >       if (failed) method.abort();
> >
> >       method.releaseConnection();
> >
> >       client.setHttpConnectionManager(null);
> >       client = null;
> > }
> >
> > try {
> >       responseString = method.getResponseBodyAsString();
> > } catch (Exception e) {
> >       throw new Exception(...);
> > } finally {
> >       method.releaseConnection();
> >       method = null;
> > }
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > httpclient-user-help@jakarta.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > httpclient-user-help@jakarta.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>


-- 
--------------------------------------------------------------------------------
Raymond Kroeker
thinkParity Solutions Inc.

------=_Part_18044_18374893.1188922119105--