hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: [jira] [Created] (HDFS-2129) Simplify BlockReader to not inherit from FSInputChecker
Date Mon, 18 Jul 2011 04:55:20 GMT
For benchmarking CPU, I start a pseudo-distributed HDFS cluster, put a
smallish file on the local datanode (such that it fits in buffer cache), and
then use the following script with various parameters to look at CPU usage
to cat the file. for example:

$ REPS_PER_RUN=50 NUM_TRIALS=10 ./read-benchmark.sh
hdfs://localhost/128M-file /tmp/benchmark-results.txt

Script:

#!/bin/sh -x
set -e
BINDIR=$(dirname $0)

INPUT=$1
OUTPUT=$2
NUM_TRIALS=${NUM_TRIALS:-10}
HADOOP=${HADOOP:-./bin/hadoop}
HADOOP_FLAGS=${HADOOP_FLAGS:--Dio.file.buffer.size=$[64*1024]}
REPS_PER_RUN=${REPS_PER_RUN:-1}


HEADER="major\tminor\tfs_in\tfs_out\twall\tuser\tsys\tctx_invol\tctx_vol\n"
TIME_FORMAT="%F\t%R\t%I\t%O\t%e\t%U\t%S\t%c\t%w"

! test -f $OUTPUT && printf $HEADER > $OUTPUT
for x in `seq 1 $NUM_TRIALS` ; do
    /usr/bin/time --append -o $OUTPUT -f $TIME_FORMAT \
        $HADOOP fs $HADOOP_FLAGS -cat $(for rep in $(seq 1 $REPS_PER_RUN) ;
do echo $INPUT ; done) > /dev/null
done


On Wed, Jul 6, 2011 at 1:16 AM, Keren Ouaknine <kereno@gmail.com> wrote:

> Hello,
>
> I am working on the optimization of task scheduling for Hadoop and would
> like to benchmark with* Apache Hadoop's standards benchmarks*. So far, I
> used my own scripts to measure and monitor. Where can I find the
> benchmarking you are referring to please?
>
> Thanks,
> Keren
>
> On Wed, Jul 6, 2011 at 7:32 AM, Todd Lipcon (JIRA) <jira@apache.org>
> wrote:
>
> > Simplify BlockReader to not inherit from FSInputChecker
> > -------------------------------------------------------
> >
> >                 Key: HDFS-2129
> >                 URL: https://issues.apache.org/jira/browse/HDFS-2129
> >             Project: Hadoop HDFS
> >          Issue Type: Sub-task
> >          Components: hdfs client
> >            Reporter: Todd Lipcon
> >            Assignee: Todd Lipcon
> >
> >
> > BlockReader is currently quite complicated since it has to conform to the
> > FSInputChecker inheritance structure. It would be much simpler to
> implement
> > it standalone. Benchmarking indicates it's slightly faster, as well.
> >
> > --
> > This message is automatically generated by JIRA.
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
> >
> >
>
>
> --
> Keren Ouaknine
> Cell: +972 54 2565404
> Web: www.kereno.com
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message