Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com
 designates 209.85.160.50 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=cLEk6nwVMIbv4I0165zkzbokU4m8/7SX++U9pFFUKJnJD3RNwc7QQIgCiMQbJyHyZl
         rLqUtzJ0KPTnJh5/u8RXpDMiD0dQ8wYz/A0Nw7pzFTdWt/GfcWc4ZC55sKlfkxHnKYGV
         pWjNpo0ej921igzUDA6Fjqe2Dqkw4tuFbVSpA=
MIME-Version: 1.0
In-Reply-To: <4AEC4C01.2000500@swe-blog.net>
References: <958723.91055.qm@web112111.mail.gq1.yahoo.com>
	 <8211a1320910291742v4e6a61c6hda74f7b49edd12cc@mail.gmail.com>
	 <4AEC4C01.2000500@swe-blog.net>
Date: Sat, 31 Oct 2009 08:18:32 -0700
Message-ID: <314098690910310818q10d9277fpcc5340a60c32e0ff@mail.gmail.com>
Subject: Re: What if an XML file cross boundary of HDFS chunks?
From: Jason Venner <jason.hadoop@gmail.com>
To: common-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2e35456b62c04773ca890

--000e0cd2e35456b62c04773ca890
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I use the StreamXMLRecordReader out of the streaming contrib package, it
works very well. Your key becomes the stanza you are looking for.

On Sat, Oct 31, 2009 at 7:38 AM, Oliver B. Fischer <o.b.fischer@swe-blog.ne=
t
> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello Jeff,
>
> does it means, that there is no programmatic possibility to define where
> a logical file will be splitted independent of the distribution of it
> blocks in the HDFS?
>
> Regards
>
> Oliver
>
> Jeff Zhang schrieb:
> > Hi Steve,
> >
> > When you want to read xml, you should provide your custom InputFormat
> which
> > extends FileInputFormat.
> >
> > and override the method isSplitable to not split a file , that means on=
e
> xml
> > file for one mapper.
> >
> >
> >   protected boolean isSplitable(FileSystem fs, Path filename) {
> >     return false;
> >   }
>
>
> - --
> Oliver B. Fischer, Sch=F6nhauser Allee 64, 10437 Berlin
> Tel. +49 30 44793251, Mobil: +49 178 7903538
> Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJK7EwBAAoJELeiwuwqd1DGO/wIAJl8wwf6uNgm/ZwsGh8M1xvz
> wSEH9sD2cfjUSV3rmpHndKEfSTEOeHvvaJmJn24K9HhB9w8QyDogAgHawCdBY2TE
> K27n4wqSGlbLpQz4XmKUOVtFSooeEPUT58Jn2aMAno+nrWHM7oq9tuCJAAYkBexV
> wCrc7eE+o55TlAlx+LDWWS9mJrdTNBYqzoHh0gnWsEGm98CWvzn08tNA/L2moJbQ
> HZwnWzfgEBKBwAZUOYLFt2GigIYN3GE0pMp33BgjWi91zPwGSk7Bcq7XhObLK7o/
> uYxS+s3BTkLy+R6ngjOW1NLvg6STX37FpFNZowDmPt8Bzd8GxAefnqcxkVcnb90=3D
> =3Dt6vV
> -----END PGP SIGNATURE-----
>
>


--=20
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--000e0cd2e35456b62c04773ca890--