hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "SammiChen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8901) Use ByteBuffer in striping positional read
Date Mon, 05 Sep 2016 08:37:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15464454#comment-15464454
] 

SammiChen commented on HDFS-8901:
---------------------------------

Thanks Zhe!  Thanks Kai!

bq.  Why do we need to flip? In the previous code, arraycopy just starts copying from the
current position of result.

I changed {{readAll(ByteBuffer buf, int len)}} implementation. In Previously implementation,
position of {{buf}} will not be advanced by {{len}} after the all. I think it is not a traditional
way when ByteBuffer parameter is used. SO I changed it. Now {{buf}} will be advanced by {{len}}
after the call. That's why {{result}} should flip before read. 

bq.   Could you also explain why TestPread and TestSnapshotFileLength failed for v17 and v18
patches respectively?

Previously user input buffer is slice()ed in pread, and then the sliced buffer is passed into
the callee.  The sliced buffer has its independent position and length.  This piece of code
is optimized in pread, get rid of the slice(), directly pass the user input buffer to callee.
So user buffer's position and length need to be handled carefully now.  TestPread and TestSnapshotFileLength
failed because user buffer's position and read/write length doesn't took cared correctly in
some execution path. 
    
bq.   Need to be removed:
Oops, will handle it. 

Based on our current discussion, I will reuse the approach in v14 patch to handle the readAll
logic.



> Use ByteBuffer in striping positional read
> ------------------------------------------
>
>                 Key: HDFS-8901
>                 URL: https://issues.apache.org/jira/browse/HDFS-8901
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: SammiChen
>         Attachments: HDFS-8901-v10.patch, HDFS-8901-v17.patch, HDFS-8901-v18.patch, HDFS-8901-v19.patch,
HDFS-8901-v2.patch, HDFS-8901-v3.patch, HDFS-8901-v4.patch, HDFS-8901-v5.patch, HDFS-8901-v6.patch,
HDFS-8901-v7.patch, HDFS-8901-v8.patch, HDFS-8901-v9.patch, HDFS-8901.v11.patch, HDFS-8901.v12.patch,
HDFS-8901.v13.patch, HDFS-8901.v14.patch, HDFS-8901.v15.patch, HDFS-8901.v16.patch, initial-poc.patch
>
>
> Native erasure coder prefers to direct ByteBuffer for performance consideration. To prepare
for it, this change uses ByteBuffer through the codes in implementing striping position read.
It will also fix avoiding unnecessary data copying between striping read chunk buffers and
decode input buffers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message