hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sam rash (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
Date Wed, 12 May 2010 18:30:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866680#action_12866680

sam rash commented on HDFS-941:

two other comments:

1. the number of sockets per address is limited, but not the number of addresses.  This may
in practice not be a problem, but the cache can in theory grow very large
2. the usedmap seems like a good place for a memory/object leak:  I can take a socket and
never return it (why again, I vote for getting rid of this data structure period--as far as
a cache is concerned, an entry that someone else owns shouldn't even be there). otherwise,
you've got to periodically clean this map up as well.  Seems like it's only used for stats
which I think you can do w/o actually keeping a hash of used sockets.

> Datanode xceiver protocol should allow reuse of a connection
> ------------------------------------------------------------
>                 Key: HDFS-941
>                 URL: https://issues.apache.org/jira/browse/HDFS-941
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, hdfs client
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: bc Wong
>         Attachments: HDFS-941-1.patch, HDFS-941-2.patch, HDFS-941-3.patch, HDFS-941-3.patch,
> Right now each connection into the datanode xceiver only processes one operation.
> In the case that an operation leaves the stream in a well-defined state (eg a client
reads to the end of a block successfully) the same connection could be reused for a second
operation. This should improve random read performance significantly.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message