Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=s8vsSEzoed2CjtmiEy75fIzSWqL+9i69HP2ZYZ/YRr2d4lJ/lXBtv8xFrw9tVHnx
User-Agent: Microsoft-Entourage/12.19.0.090515
Date: Fri, 26 Jun 2009 11:01:24 -0700
Subject: Re: Doing MapReduce over Har files
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: <core-user@hadoop.apache.org>
Message-ID: <C66A5B04.1F280%mahadev@yahoo-inc.com>
Thread-Topic: Doing MapReduce over Har files
Thread-Index: Acn2iB47GKH5Gv+59kihG+ZeqAN8ZA==
In-Reply-To: <24217500.post@talk.nabble.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

Hi Roshan and Julian,
  The har file system can be used as a input filesystem. You can just
provide the input to map reduce as har:///something/some.har , where
some.har is your har archive. This way amp reduce will use har filesystem a=
s
an input. The only problem being that maps cannot run across logical files
in har.=20

You can specify whatever input  format these files have/had before you
included them into har archives. The point being that har:/// can be used a=
s
a input filesystem for map reduce, which will give map reduce a view of
logical files inside of har.

Hope this helps.
mahadev


On 6/26/09 2:37 AM, "jchernandez" <jchernandez@agnitio.es> wrote:

>=20
> I also need help with this. I need to know how to handle a HAR file when =
it
> is the input to a MapReduce task. How do we read the HAR file so we can w=
ork
> on the individual logical files? I suppose we need to create our own
> InputFormat and RecordReader files, but I=B4m not sure how to proceed.
>=20
> Julian=20
>=20
>=20
> Roshan James-3 wrote:
>>=20
>> When I run map reduce task over a har file as the input, I see that the
>> input splits refer to 64mb byte boundaries inside the part file.
>>=20
>> My mappers only know how to process the contents of each logical file
>> inside
>> the har file. Is there some way by which I can take the offset range
>> specified by the input split and determine which logical files lie in th=
at
>> offset range? (How else would one do map reduce over a har file?)
>>=20
>> Roshan
>>=20
>>=20