hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
Date Thu, 29 Mar 2018 16:24:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419288#comment-16419288

Íñigo Goiri commented on HADOOP-13144:

Thanks [~daryn] for checking.

The internal use we have is in the {{RouterRpcClient}} when creating a connection from the
Router to the Namenode.
It looks like the following:
 FederationConnectionId connectionId = new FederationConnectionId(
     socket, ClientNamenodeProtocolPB.class, ugi, timeout, defaultPolicy,
     conf, index);
ClientNamenodeProtocolPB proxy = RPC.getProtocolProxy(
    ClientNamenodeProtocolPB.class, version, connectionId, conf, factory).getProxy();
FederationConnectionId has all the equals, hash, etc, for the cache to track it correctly.

We could mark this as private or something.
In any case, for us, the use is limited to the Router.
Not sure what are the right mechanisms to make this explicit.

> Enhancing IPC client throughput via multiple connections per user
> -----------------------------------------------------------------
>                 Key: HADOOP-13144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13144
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jason Kace
>            Assignee: Íñigo Goiri
>            Priority: Minor
>         Attachments: HADOOP-13144.000.patch, HADOOP-13144.001.patch, HADOOP-13144.002.patch
> The generic IPC client ({{org.apache.hadoop.ipc.Client}}) utilizes a single connection
thread for each {{ConnectionId}}.  The {{ConnectionId}} is unique to the connection's remote
address, ticket and protocol.  Each ConnectionId is 1:1 mapped to a connection thread by the
client via a map cache.
> The result is to serialize all IPC read/write activity through a single thread for a
each user/ticket + address.  If a single user makes repeated calls (1k-100k/sec) to the same
destination, the IPC client becomes a bottleneck.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message