hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5574) Remove buffer copy in BlockReader.skip
Date Wed, 18 Dec 2013 19:31:09 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852068#comment-13852068

Colin Patrick McCabe commented on HDFS-5574:

If it's easy, then I would say do {{RemoteBlockReader}} as well, just to avoid divergence.
 There are still some people who have to use the old block reader-- for example, if they are
using SOCKS sockets.

The main point of the old block reader is that it can operate on sockets that don't have channels.
 In the long term, I feel like we could probably wrap those sockets in something that would
provide a channel, or just integrate that functionality into {{RemoteBlockReader2}}.

Your test doesn't really test what you think.  {{DFSInputStream#skip}} is not necessarily
going to call {{BlockReader#skip}}.  If you want to play with a remote block reader directly,
you should call the test {{TestRemoteBlockReader}} or something like that.  There is an example
in {{TestBlockTokenWithDFS}}.  Also, you should set {{DFSInputStream#tcpReadsDisabledForTesting}}
if you want to make sure you're really getting a local block reader.

Finally, {{BlockReaderLocal}} lets you skip past the end of the file without complaining.
 This behavior is consistent with how {{FileInputChannel#position}} works.  Since the {{DFSInputStream}}
knows the length of the block (reader), it never tries to skip past the end.

> Remove buffer copy in BlockReader.skip
> --------------------------------------
>                 Key: HDFS-5574
>                 URL: https://issues.apache.org/jira/browse/HDFS-5574
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Binglin Chang
>            Assignee: Binglin Chang
>            Priority: Trivial
>         Attachments: HDFS-5574.v1.patch, HDFS-5574.v2.patch, HDFS-5574.v3.patch
> BlockReaderLocal.skip and RemoteBlockReader.skip uses a temp buffer to read data to this
buffer, it is not necessary. 

This message was sent by Atlassian JIRA

View raw message