hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-312) Connections should not be cached
Date Sun, 03 Sep 2006 15:12:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-312?page=comments#action_12432341 ] 
            
Devaraj Das commented on HADOOP-312:
------------------------------------

> If CLOSE_CONNECTION is a constant, then it should be declared 'static final'. But I think
it can be 
> removed altogether, along with shouldCloseConnection. Why not have the ConnectionCuller

> simply set running=false, so that Connection.run() exits naturally and removes itself
from the 
> connection cache? 
If you are referring to the running boolean field attached to the Client class, the setting
that to false will make all the response receiver threads exit. So instead, we can have another
boolean field attached to the Connection class to signify that the corresponding response
receiver thread should exit.

> The culler doesn't appear to remove the connection from the cache anyway. It should,
so that new 
> requests are not handed to a connection that is exiting. 
It does remove I think. Note there is a i.remove() in the run method of the ConnectionCuller
thread.

> Do we need a separate nocache config parameter, and code to support it? Shouldn't this
be the 
> same as simply setting maxidletime to 0? If so, let's get rid of isCachingDisabled. This
would give 
> only one mode of operation, greatly reducing the number of possible bugs.
So this patch will make the Hadoop clients operate in two modes - one with full caching of
connections, and another idle-time-based caching (connections are culled when they are idle
for some period of time). I am not sure I understood what you mean by one mode of operation.
Are you saying that we should remove "if (isCachingDisabled)" and instead have the corresponding
logic work under "if (maxIdleTime == 0)" ?

Also one more thing - currently, I do a check for isCachingDisabled in a few places and then
execute relevant code. But some of the code, although not required for the mode where we don't
do idle-time-based caching, doesn't really harm the execution of that mode (like incrementRef)
except adding some unecessary code. Does it remain that way or do I remove checks for isCachingDisabled
from those "safe" places ?

> Connections should not be cached
> --------------------------------
>
>                 Key: HADOOP-312
>                 URL: http://issues.apache.org/jira/browse/HADOOP-312
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: no_conn_caching.patch, no_conn_caching.patch, no_conn_caching.patch,
no_conn_caching.patch, no_conn_caching.patch, no_conn_caching.patch, no_connection_caching.patch,
no_connection_caching.patch
>
>
> Servers and clients (client include datanodes, tasktrackers, DFSClients & tasks)
should not cache connections or maybe cache them for very short periods of time. Clients should
set up & tear down connections to the servers everytime they need to contact the servers
(including the heartbeats). If connection is cached, then reuse the existing connection for
a few subsequent transactions until the connection expires. The heartbeat interval should
be more so that many more clients (order of  tens of thousands) can be accomodated within
1 heartbeat interval.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message