hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2834) ByteBuffer-based read API for DFSInputStream
Date Tue, 20 Mar 2012 16:25:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233524#comment-13233524
] 

jiraposter@reviews.apache.org commented on HDFS-2834:
-----------------------------------------------------



bq.  On 2012-03-20 01:27:50, Todd Lipcon wrote:
bq.  > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java,
line 44
bq.  > <https://reviews.apache.org/r/4212/diff/2/?file=90213#file90213line44>
bq.  >
bq.  >     shouldn't this be true?

Oops, yes. Thankfully the test still passes when it's testing the right path...


bq.  On 2012-03-20 01:27:50, Todd Lipcon wrote:
bq.  > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java,
lines 81-82
bq.  > <https://reviews.apache.org/r/4212/diff/2/?file=90213#file90213line81>
bq.  >
bq.  >     no reason to use DFSClient here. Instead you can just use the filesystem, right?
Then downcast the stream you get back?

Good point - no need even to downcast since FSDataInputStream has the API.


bq.  On 2012-03-20 01:27:50, Todd Lipcon wrote:
bq.  > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java,
line 104
bq.  > <https://reviews.apache.org/r/4212/diff/2/?file=90213#file90213line104>
bq.  >
bq.  >     don't you want an assert on sawException here? You can also use GenericTestUtils.assertExceptionContains()
if you want to check the text of it

Good catch. No particular need to assert the content of the exception - any checksum error
is good enough here. 


bq.  On 2012-03-20 01:27:50, Todd Lipcon wrote:
bq.  > hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java,
lines 562-564
bq.  > <https://reviews.apache.org/r/4212/diff/2/?file=90207#file90207line562>
bq.  >
bq.  >     this comment seems like it's in the wrong spot, since the code that comes after
it doesn't reference offsetFromChunkBoundary.

I removed the comment, it's covered by the comment at line 549.


- Henry


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4212/#review6103
-----------------------------------------------------------


On 2012-03-09 00:47:24, Henry Robinson wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4212/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-03-09 00:47:24)
bq.  
bq.  
bq.  Review request for hadoop-hdfs and Todd Lipcon.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  New patch for HDFS-2834 (I can't update the old review request).
bq.  
bq.  
bq.  This addresses bug HDFS-2834.
bq.      http://issues.apache.org/jira/browse/HDFS-2834
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java
dfab730 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
cc61697 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
4187f1c 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
2b817ff 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
b7da8d4 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
ea24777 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java
9d4f4a2 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java
PRE-CREATION 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelRead.java
bbd0012 
bq.    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
eb2a1d8 
bq.  
bq.  Diff: https://reviews.apache.org/r/4212/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Henry
bq.  
bq.


                
> ByteBuffer-based read API for DFSInputStream
> --------------------------------------------
>
>                 Key: HDFS-2834
>                 URL: https://issues.apache.org/jira/browse/HDFS-2834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Henry Robinson
>            Assignee: Henry Robinson
>         Attachments: HDFS-2834-no-common.patch, HDFS-2834.3.patch, HDFS-2834.4.patch,
HDFS-2834.5.patch, HDFS-2834.6.patch, HDFS-2834.7.patch, HDFS-2834.8.patch, HDFS-2834.9.patch,
HDFS-2834.patch, HDFS-2834.patch, hdfs-2834-libhdfs-benchmark.png
>
>
> The {{DFSInputStream}} read-path always copies bytes into a JVM-allocated {{byte[]}}.
Although for many clients this is desired behaviour, in certain situations, such as native-reads
through libhdfs, this imposes an extra copy penalty since the {{byte[]}} needs to be copied
out again into a natively readable memory area. 
> For these cases, it would be preferable to allow the client to supply its own buffer,
wrapped in a {{ByteBuffer}}, to avoid that final copy overhead. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message