Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates
 209.85.161.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=FBdqbjYpqyESXDlFcidHwlbf/jeYH/sAN7KHk0Q41e4Y/z2KBeR6GK7k6/mWvv17GL
         dSorO+vE+oNI56YXUCMm6JX33QX+2AGrrgMk3xzKFrIjxJ4HF2jjGg982EvclzmI0wA/
         TnJ7IOkR35PN4oNiN80BzdP975jJzMYvfw47M=
MIME-Version: 1.0
In-Reply-To: <AANLkTikSbhKP8Zu8r9r+MfCMZgUBLZZrO=_8ToDCz7T9@mail.gmail.com>
References: <AANLkTik5WS7Qh4F3dt0bO4RcYWUKb08OXyGjsBEHMJPR@mail.gmail.com>
	<AANLkTinQToXUzQTHKeCAjOwLtWE+C-4XgrkWzxSCO4bp@mail.gmail.com>
	<AANLkTi=RAKmJe8aZv1NzOim+LEP_eYKKqELcFEgKRo4-@mail.gmail.com>
	<AANLkTikSbhKP8Zu8r9r+MfCMZgUBLZZrO=_8ToDCz7T9@mail.gmail.com>
Date: Thu, 3 Feb 2011 19:22:46 -0800
Message-ID: <AANLkTin35MWSoFvQzidDkKTwjMP9BMLRtNc0+cM0jNhR@mail.gmail.com>
Subject: Re: "Map input bytes" vs HDFS_BYTES_READ
From: Ted Yu <yuzhihong@gmail.com>
To: mapreduce-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00248c0ef2f268af54049b6c65d2

--00248c0ef2f268af54049b6c65d2
Content-Type: text/plain; charset=ISO-8859-1

>From my limited experiment, I think "Map input bytes" reflects the number of
bytes of local data file(s) when LocalJobRunner is used.

Correct me if I am wrong.

On Tue, Feb 1, 2011 at 7:52 PM, Harsh J <qwertymaniac@gmail.com> wrote:

> Each task counts independently of its attempt/other tasks, thereby
> making the aggregates easier to control. Final counters are aggregated
> only from successfully committed tasks. During the job's run, however,
> counters are shown aggregated from the most successful attempts of a
> task thus far.
>
> On Wed, Feb 2, 2011 at 9:09 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > If map task(s) were retried (mapred.map.max.attempts times), how would
> these
> > two counters be affected ?
> >
> > Thanks
> >
> > On Tue, Feb 1, 2011 at 7:31 PM, Harsh J <qwertymaniac@gmail.com> wrote:
> >
> >> HDFS_BYTES_READ is a FileSystem interface counter. It directly deals
> >> with the FS read (lower level). Map input bytes is what the
> >> RecordReader has processed in number of bytes for records being read
> >> from the input stream.
> >>
> >> For plain text files, I believe both counters must report about the
> >> same value, were entire records being read with no operation performed
> >> on each line. But when you throw in a compressed file, you'll notice
> >> that the HDFS_BYTES_READ would be far lesser than Map input bytes
> >> since the disk read was low, but the total content stored in record
> >> terms was still the same as it would be for an uncompressed file.
> >>
> >> Hope this clears it.
> >>
> >> On Wed, Feb 2, 2011 at 8:06 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> > In hadoop 0.20.2, what's the relationship between "Map input bytes"
> and
> >> > HDFS_BYTES_READ ?
> >> >
> >> > <counter group="FileSystemCounters"
> >> > name="HDFS_BYTES_READ">203446204073</counter>
> >> > <counter group="FileSystemCounters"
> >> > name="HDFS_BYTES_WRITTEN">23413127561</counter>
> >> > <counter group="Map-Reduce Framework" name="Map input
> >> > records">163502600</counter>
> >> > <counter group="Map-Reduce Framework" name="Spilled
> Records">0</counter>
> >> > <counter group="Map-Reduce Framework" name="Map input
> >> > bytes">965922136488</counter>
> >> > <counter group="Map-Reduce Framework" name="Map output
> >> > records">296754600</counter>
> >> >
> >> > Thanks
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> www.harshj.com
> >>
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

--00248c0ef2f268af54049b6c65d2--