hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6287) Add vecsum test of libhdfs read access times
Date Mon, 05 May 2014 18:19:15 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989791#comment-13989791
] 

Colin Patrick McCabe commented on HDFS-6287:
--------------------------------------------

bq. Hi, Colin. Thanks for posting this. Did you find that you needed to use SSE to get the
addition fast enough so that the benchmark highlights read throughput instead of sum computation?
IOW, could we potentially simplify this patch to not use SSE at all and still have a valid
benchmark?

Without that optimization, the benchmark quickly becomes CPU-bound and you don't get true
numbers for ZCR and other fast read methods.  I just benchmarked 1.5 GB/s for the un-optimized
version versus 5.7 GB/s for the optimized.

bq. I think it would be helpful to add a comment with a high-level summary of what vecsum
does, maybe right before the main.

Added

bq. I have one minor comment on the code itself so far. I think you can remove the hdfsFreeBuilder
call. hdfsBuilderConnect always frees the builder, whether it succeeds or fails. The only
time you would need to call hdfsFreeBuilder directly is if you allocated a builder but then
never attempted to connect with it. I don't see any way for that to happen in the libhdfs_data_create
code.

Yeah, that is deadcode.  Let me remove that

> Add vecsum test of libhdfs read access times
> --------------------------------------------
>
>                 Key: HDFS-6287
>                 URL: https://issues.apache.org/jira/browse/HDFS-6287
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: libhdfs, test
>    Affects Versions: 2.5.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-6282.001.patch, HDFS-6287.002.patch, HDFS-6287.003.patch, HDFS-6287.004.patch,
HDFS-6287.005.patch
>
>
> Add vecsum, a benchmark that tests libhdfs access times.  This includes short-circuit,
zero-copy, and standard libhdfs access modes.  It also has a local filesystem mode for comparison.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message