Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jthievre@gmail.com designates
 209.85.216.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:from:date
         :x-google-sender-auth:message-id:subject:to:content-type;
        b=oUTvEGZ+5/St/TrMNNIYByS20kq6Cc9txZIB7Mu9pGgU5mM7zMYPe+oDQozER+is1w
         jKzLRXau5A0GczLy9+GaAPFukKRkwIgRQUx16koCw31kp9qogSxYom1RfOfDLRMFeVtx
         8upYYESx3ecFkXkWDVjlroBzMRFQjZgJ+OZXw=
MIME-Version: 1.0
Sender: jthievre@gmail.com
In-Reply-To: <AANLkTikKpwNxCSAgfzUfySM-D4thDehgYv=FEGQumCvT@mail.gmail.com>
References: <AANLkTi=eoqJrJ3QjqOWzSx91tx-y7UCcrZj+WV+i2uQO@mail.gmail.com>
 <AANLkTikKpwNxCSAgfzUfySM-D4thDehgYv=FEGQumCvT@mail.gmail.com>
From: =?ISO-8859-1?B?Suly9G1lIFRoaeh2cmUgSU5B?= <jthievre@ina.fr>
Date: Fri, 7 Jan 2011 14:16:11 +0100
Message-ID: <AANLkTi=6EqRtncubu0L5SejXXF8KsffZ=bAS9epHB5Rv@mail.gmail.com>
Subject: Re: How to manage large record in MapReduce
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636284a343ef6240499416d23

--001636284a343ef6240499416d23
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Sonal,

thank you, I have just implemented a solution similar to yours (without
copying to a temp file as suggested in my inital post), and it seems to
work.
Best Regards,

J=E9r=F4me

2011/1/7 Sonal Goyal <sonalgoyal4@gmail.com>

> Jerome,
>
> You can take a look at FileStreamInputFormat at
>
> https://github.com/sonalgoyal/hiho/tree/hihoApache0.20/src/co/nubetech/hi=
ho/mapreduce/lib/input
>
> This provides an input stream per file. In our case, we are using the inp=
ut
> stream to load data into the database directly. Maybe you can use this or=
 a
> similar approach for working with your videos.
>
> HTH
>
> Thanks and Regards,
> Sonal
> <https://github.com/sonalgoyal/hiho>Connect Hadoop with databases,
> Salesforce, FTP servers and others <https://github.com/sonalgoyal/hiho>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Thu, Jan 6, 2011 at 4:23 PM, J=E9r=F4me Thi=E8vre <jthievre@gmail.com>=
 wrote:
>
> > Hi,
> >
> > we are currently using Hadoop (version 0.20.2) to manage some web
> archiving
> > processes like fulltext indexing, and it works very well with small
> records
> > that contains html.
> > Now, we would like to work with other type of web data like videos. The=
se
> > kind of data could be really large and of course these records doesn't
> fit
> > in memory.
> >
> > Is it possible to manage record which content doesn't reside in memory
> but
> > on disk.
> > A possibility would be to implements a Writable that read its content
> from
> > a
> > DataInput but doesn't load it in memory, instead it would copy that
> content
> > to a temporary file in the local file system and allows to stream its
> > content using an InputStream (an InputStreamWritable).
> >
> > Has somebody tested a similar approach, and if not do you think some bi=
g
> > problems could happen (that impacts performance) with this method ?
> >
> > Thanks,
> >
> > J=E9r=F4me Thi=E8vre
> >
>

--001636284a343ef6240499416d23--