hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-9520) PeerCache evicts too frequently causing connection restablishments
Date Tue, 08 Dec 2015 11:39:10 GMT

     [ https://issues.apache.org/jira/browse/HDFS-9520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rajesh Balamohan updated HDFS-9520:
-----------------------------------
    Description: 
Env: 20 node setup
dfs.client.socketcache.capacity = 16

Issue:
======
Monitored PeerCache and it was evicting lots of connections during close. Set "dfs.client.socketcache.capacity=20"
and tested again. Evictions still happened. Ideally it should not be evicting the connections
since "dfs.client.socketcache.capacity" is set to 20 (no. of DN). Screenshot of profiler is
attached in the JIRA.

Workaround:
===========
Temp fix was to set "dfs.client.socketcache.capacity=1000" to prevent eviction. 


Added more debug logs revealed that multimap.size() was 40 instead of 20. LinkedListMultimap
returns the total values instead of key size causing lots of evictions.

{code}
   if (capacity == multimap.size()) {
      evictOldest();
    }
{code}

Should this be (capacity == multimap.keySet().size())  or is it expected that the "dfs.client.socketcache.capacity"
be set to very high value?

\cc [~gopalv], [~sseth]


  was:

Env: 20 node setup
dfs.client.socketcache.capacity = 16

Issue:
======
Monitored PeerCache and it was evicting lots of connections during close. Set "dfs.client.socketcache.capacity=20"
and tested again. Evictions still happened. Ideally it should not be evicting the connections
since "dfs.client.socketcache.capacity" is set to 20 (no. of DN). Screenshot of profiler is
attached in the JIRA.

Workaround:
===========
Temp fix was to set "dfs.client.socketcache.capacity=1000" to prevent eviction. 


Added more debug logs revealed that multimap.size() was 40 instead of 20. LinkedListMultimap
returns the total values instead of key size causing lots of evictions.

{code}
   if (capacity == multimap.size()) {
      evictOldest();
    }
{code}

Should this be (capacity == multimap.keySet().size()) ?

\cc [~gopalv], [~sseth]



> PeerCache evicts too frequently causing connection restablishments
> ------------------------------------------------------------------
>
>                 Key: HDFS-9520
>                 URL: https://issues.apache.org/jira/browse/HDFS-9520
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Rajesh Balamohan
>         Attachments: HDFS-9520.png
>
>
> Env: 20 node setup
> dfs.client.socketcache.capacity = 16
> Issue:
> ======
> Monitored PeerCache and it was evicting lots of connections during close. Set "dfs.client.socketcache.capacity=20"
and tested again. Evictions still happened. Ideally it should not be evicting the connections
since "dfs.client.socketcache.capacity" is set to 20 (no. of DN). Screenshot of profiler is
attached in the JIRA.
> Workaround:
> ===========
> Temp fix was to set "dfs.client.socketcache.capacity=1000" to prevent eviction. 
> Added more debug logs revealed that multimap.size() was 40 instead of 20. LinkedListMultimap
returns the total values instead of key size causing lots of evictions.
> {code}
>    if (capacity == multimap.size()) {
>       evictOldest();
>     }
> {code}
> Should this be (capacity == multimap.keySet().size())  or is it expected that the "dfs.client.socketcache.capacity"
be set to very high value?
> \cc [~gopalv], [~sseth]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message