hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2129) Simplify BlockReader to not inherit from FSInputChecker
Date Fri, 05 Aug 2011 18:31:27 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Todd Lipcon updated HDFS-2129:
------------------------------

    Attachment: seq-read-1gb-bench.png

Also ran BenchmarkThroughput to test speed of reading 1GB off the local disk in DFS (a typical
mapper workload). t-test results:
{noformat}

> t.test(subset(d, patch=="with")$time, subset(d, patch=="without")$time)

        Welch Two Sample t-test

data:  subset(d, patch == "with")$time and subset(d, patch == "without")$time 
t = -4.0784, df = 85.684, p-value = 0.0001012
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -0.18340536 -0.06319728 
sample estimates:
mean of x mean of y 
 2.610510  2.733811 
{noformat}

Still need to address the buffer pooling issue as described above, though.

> Simplify BlockReader to not inherit from FSInputChecker
> -------------------------------------------------------
>
>                 Key: HDFS-2129
>                 URL: https://issues.apache.org/jira/browse/HDFS-2129
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs client
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hdfs-2129-benchmark.png, hdfs-2129.txt, hdfs-2129.txt, seq-read-1gb-bench.png
>
>
> BlockReader is currently quite complicated since it has to conform to the FSInputChecker
inheritance structure. It would be much simpler to implement it standalone. Benchmarking indicates
it's slightly faster, as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message