Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of sandy.ryza@cloudera.com
 designates 209.85.215.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAG+ckK9CS3=6_grp_5VX1RA7WFchAb4yefewv0WWV9--4SoLsg@mail.gmail.com>
References: 
 <CAG+ckK9CS3=6_grp_5VX1RA7WFchAb4yefewv0WWV9--4SoLsg@mail.gmail.com>
Date: Sun, 31 Mar 2013 11:10:23 -0700
Message-ID: 
 <CACBYxKJ_4heLf3Hgh7ndKALDR-VB18+NDhsjFXCrZ1CTMM-aJQ@mail.gmail.com>
Subject: Re: Streaming value of (200MB) from a SequenceFile
From: Sandy Ryza <sandy.ryza@cloudera.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e0160c21c03128d04d93c6b96

--089e0160c21c03128d04d93c6b96
Content-Type: text/plain; charset=ISO-8859-1

Hi Jerry,

I assume you're providing your own Writable implementation? The Writable
readFields method is given a stream.  Are you able to perform you able to
perform your processing while reading the it there?

-Sandy

On Sat, Mar 30, 2013 at 10:52 AM, Jerry Lam <chilinglam@gmail.com> wrote:

> Hi everyone,
>
> I'm having a problem to stream individual key-value pair of 200MB to 1GB
> from a MapFile.
> I need to stream the large value to an outputstream instead of reading the
> entire value before processing because it potentially uses too much memory.
>
> I read the API for MapFile, the next(WritableComparable key, Writable val)
> does not return an input stream.
>
> How can I accomplish this?
>
> Thanks,
>
> Jerry
>

--089e0160c21c03128d04d93c6b96
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Jerry,<div><br></div><div>I assume you&#39;re providing your own Writabl=
e implementation? The Writable readFields method is given a stream. =A0Are =
you able to perform you able to perform your processing while reading the i=
t there?</div>
<div><br></div><div>-Sandy</div><div><br><div class=3D"gmail_quote">On Sat,=
 Mar 30, 2013 at 10:52 AM, Jerry Lam <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:chilinglam@gmail.com" target=3D"_blank">chilinglam@gmail.com</a>&gt;</spa=
n> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi everyone,<div><br></div>=
<div>I&#39;m having a problem to stream individual key-value pair of 200MB =
to 1GB from a MapFile.</div>
<div>I need to stream the large value to an outputstream instead of reading=
 the entire value before processing because it potentially uses too much me=
mory.</div>
<div><br></div><div>I read the API for MapFile, the=A0next(WritableComparab=
le key, Writable val) does not return an input stream.</div><div><br></div>=
<div>How can I accomplish this?=A0</div><div><br></div>
<div>Thanks,</div><div><br></div><div>Jerry</div></div>
</blockquote></div><br></div>

--089e0160c21c03128d04d93c6b96--