hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3205) Read multiple chunks directly from FSInputChecker subclass into user buffers
Date Fri, 04 Dec 2009 01:47:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785709#action_12785709
] 

Eli Collins commented on HADOOP-3205:
-------------------------------------

bq. I think you missed the "chunkPos += read;" outside the inner if? Java seems to occasionally
return -1 for EOF for some reason so I was nervous about letting that happen outside the if.
I'd be happy to add an assert read >= 0 though for this case and make it part of the contract
of readChunks to never return negative.

Yup, I think the way it is now is good. 

Patch looks great. Like the new comments and test modifications. Thanks for running the additional
experiments. 

+1

> Read multiple chunks directly from FSInputChecker subclass into user buffers
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-3205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3205
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Raghu Angadi
>            Assignee: Todd Lipcon
>         Attachments: hadoop-3205.txt, hadoop-3205.txt, hadoop-3205.txt, hadoop-3205.txt,
hadoop-3205.txt
>
>
> Implementations of FSInputChecker and FSOutputSummer like DFS do not have access to full
user buffer. At any time DFS can access only up to 512 bytes even though user usually reads
with a much larger buffer (often controlled by io.file.buffer.size). This requires implementations
to double buffer data if an implementation wants to read or write larger chunks of data from
underlying storage.
> We could separate changes for FSInputChecker and FSOutputSummer into two separate jiras.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message