hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-312) Connections should not be cached
Date Tue, 05 Sep 2006 21:21:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-312?page=comments#action_12432679 ] 
Doug Cutting commented on HADOOP-312:

You're right, Client.running is inapproprate for this use, and the culler does in fact remove
connections (not sure how I missed that).

Re modes: right now there are three modes, each with some separate logic: full-caching, timed
caching, and no caching.  It would simplify the logic (and hence ease testing and increase
reliability) if there were only one mode.  Otherwise, we should add unit tests for all modes.
 But the timed cache seems fairly universal.  Setting the timeout to Integer.MAX_VALUE gives
full-caching, while setting it to zero could cause most connections to terminate after a single
request (although, if some get reused, that should not cause problems).  The default should
be a fairly short timeout, so that some kinds of requests will reuse cached connections and
some will create new connections per request.  This way the default configuration will test
all of the code.  I do not like having modes that might be useful but are not regularly used,
since they'll not be tested and may stop working and then generate bugs.

> Are you saying that we should remove "if (isCachingDisabled)" and instead have the corresponding
logic work under "if (maxIdleTime == 0)" ? 

I don't think we should have either.  MaxIdleTime==0 is simply a very short idle time.  The
culler thread should have a minimum sleep time so that it doesn't busywait.

> Connections should not be cached
> --------------------------------
>                 Key: HADOOP-312
>                 URL: http://issues.apache.org/jira/browse/HADOOP-312
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: no_conn_caching.patch, no_conn_caching.patch, no_conn_caching.patch,
no_conn_caching.patch, no_conn_caching.patch, no_conn_caching.patch, no_connection_caching.patch,
> Servers and clients (client include datanodes, tasktrackers, DFSClients & tasks)
should not cache connections or maybe cache them for very short periods of time. Clients should
set up & tear down connections to the servers everytime they need to contact the servers
(including the heartbeats). If connection is cached, then reuse the existing connection for
a few subsequent transactions until the connection expires. The heartbeat interval should
be more so that many more clients (order of  tens of thousands) can be accomodated within
1 heartbeat interval.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message