Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of hemanty@thoughtworks.com
 designates 64.18.0.188 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <527D1AEF4F04D445A1055B58C609A48523FF573E@FMSMSX101.amr.corp.intel.com>
References: 
 <527D1AEF4F04D445A1055B58C609A48523FF573E@FMSMSX101.amr.corp.intel.com>
Date: Tue, 4 Dec 2012 14:06:49 +0530
Message-ID: 
 <CAEAKFL8fH5Ly_Yn22aOpH7qvrpajWUuO-HQPhCDijrzJZB5b6A@mail.gmail.com>
Subject: Re: Using Hadoop infrastructure with input streams instead of
 key/value input
From: Hemanth Yamijala <yhemanth@thoughtworks.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d04339cde59bdd704d002c4b9

--f46d04339cde59bdd704d002c4b9
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi,

I have not tried this myself before, but would libhdfs help ?

http://hadoop.apache.org/docs/stable/libhdfs.html

Thanks
Hemanth


On Mon, Dec 3, 2012 at 9:52 PM, Wheeler, Bill NPO <
bill.npo.wheeler@intel.com> wrote:

>  I am trying to use Hadoop=92s partitioning/scheduling/storage
> infrastructure to process many HDFS files of data in parallel (1 HDFS fil=
e
> per map task), but in a way that does not naturally fit into the key/valu=
e
> pair input framework.  Specifically my application=92s =93map=94 function
> equivalent does not want to receive formatted data as key/value
> pairs=97instead, I=92d like to receive a Hadoop input stream object for m=
y map
> processing so that I can read bytes out in many different ways with much
> greater flexibility and efficiency than what I=92d get with the key/value
> pair input constraint.  The input stream would handle the complexity of
> fetching local and remote HDFS data blocks as needed on my behalf.  The
> result of the map processing would then conform to key/value pair map
> outputs and be subsequently processed by traditional reduce code.****
>
> ** **
>
> I=92m guessing that I am not the only person who would like to read HDFS
> file input directly as this capability could open up a new type of Hadoop
> use models.  Is there any support for acquiring input streams directly in=
to
> java map code?  And is there any support for doing the same into C++ map
> code ala Pipes?****
>
> ** **
>
> For added context, my application is in the video analytic space,
> requiring me to read video files .  I have implemented a solution, but it
> is a hack with less than ideal characteristics:  I have RecordReader code
> which simply passes the HDFS filename thru in the key field of my key/val=
ue
> input.  I=92m using Pipes to implement the map function in C++ code.  The=
 C++
> map code then performs a system call, =93hadoop fs =96copyToLocal hdfs_fi=
lename
> local_filename=94 to put the entire HDFS file on the datanode=92s local f=
ile
> system where it is readable by C++ IO calls.  I then simply open up this
> file and process it.  It would be much better to avoid having to do all t=
he
> extra IO associated with =93copyToLocal=94 and instead somehow receive an=
 input
> stream object from which to directly read from HDFS.****
>
> ** **
>
> Any way of doing this in a more elegant fashion?****
>
> ** **
>
> Thanks,****
>
> Bill****
>

--f46d04339cde59bdd704d002c4b9
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi,<div><br></div><div>I have not tried this myself before, but would libhd=
fs help ?=A0</div><div><br></div><div><a href=3D"http://hadoop.apache.org/d=
ocs/stable/libhdfs.html">http://hadoop.apache.org/docs/stable/libhdfs.html<=
/a><br>
</div><div><br></div><div>Thanks</div><div>Hemanth</div><div class=3D"gmail=
_extra"><br><br><div class=3D"gmail_quote">On Mon, Dec 3, 2012 at 9:52 PM, =
Wheeler, Bill NPO <span dir=3D"ltr">&lt;<a href=3D"mailto:bill.npo.wheeler@=
intel.com" target=3D"_blank">bill.npo.wheeler@intel.com</a>&gt;</span> wrot=
e:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">


<div lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div>
<p class=3D"MsoNormal">I am trying to use Hadoop=92s partitioning/schedulin=
g/storage infrastructure to process many HDFS files of data in parallel (1 =
HDFS file per map task), but in a way that does not naturally fit into the =
key/value pair input framework.=A0 Specifically
 my application=92s =93map=94 function equivalent does not want to receive =
formatted data as key/value pairs=97instead, I=92d like to receive a Hadoop=
 input stream object for my map processing so that I can read bytes out in =
many different ways with much greater flexibility
 and efficiency than what I=92d get with the key/value pair input constrain=
t.=A0 The input stream would handle the complexity of fetching local and re=
mote HDFS data blocks as needed on my behalf.=A0 The result of the map proc=
essing would then conform to key/value
 pair map outputs and be subsequently processed by traditional reduce code.=
<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal">I=92m guessing that I am not the only person who wou=
ld like to read HDFS file input directly as this capability could open up a=
 new type of Hadoop use models.=A0 Is there any support for acquiring input=
 streams directly into java map code?=A0
 And is there any support for doing the same into C++ map code ala Pipes?<u=
></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal">For added context, my application is in the video an=
alytic space, requiring me to read video files .=A0 I have implemented a so=
lution, but it is a hack with less than ideal characteristics:=A0 I have Re=
cordReader code which simply passes the
 HDFS filename thru in the key field of my key/value input.=A0 I=92m using =
Pipes to implement the map function in C++ code.=A0 The C++ map code then p=
erforms a system call, =93hadoop fs =96copyToLocal hdfs_filename local_file=
name=94 to put the entire HDFS file on the datanode=92s
 local file system where it is readable by C++ IO calls.=A0 I then simply o=
pen up this file and process it.=A0 It would be much better to avoid having=
 to do all the extra IO associated with =93copyToLocal=94 and instead someh=
ow receive an input stream object from which
 to directly read from HDFS.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal">Any way of doing this in a more elegant fashion?<u><=
/u><u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal">Thanks,<u></u><u></u></p>
<p class=3D"MsoNormal">Bill<u></u><u></u></p>
</div>
</div>

</blockquote></div><br></div>

--f46d04339cde59bdd704d002c4b9--