Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: how to pass a hdfs file to a c++ process
Date: Tue, 23 Aug 2011 09:59:28 -0500
Message-ID: <8953B8331AA98041B0C11DBC678AFC08AEC748@srvemail1.calpont.com>
In-Reply-To: <2224A47D-F80E-4994-9279-A6802D188CD4@hortonworks.com>
Thread-Topic: how to pass a hdfs file to a c++ process
Thread-Index: AcxhpDSWltJqBqIVS1uzK2abpfnfqwAADkQw
References: <8953B8331AA98041B0C11DBC678AFC08AEC745@srvemail1.calpont.com>
 <2224A47D-F80E-4994-9279-A6802D188CD4@hortonworks.com>
From: "Zhixuan Zhu" <zzhu@calpont.com>
To: <common-dev@hadoop.apache.org>

I'll actually invoke one executable from each of my map. Because this
C++ program has been implemented and used in the past, I just want to
integrate it to our Hadoop map/reduce without having to re-implement the
process in java. So my map is going to be very simple with just calling
the process and pass the input files.

Thanks,
Grace

-----Original Message-----
From: Arun C Murthy [mailto:acm@hortonworks.com]=20
Sent: Tuesday, August 23, 2011 9:51 AM
To: common-dev@hadoop.apache.org
Subject: Re: how to pass a hdfs file to a c++ process


On Aug 22, 2011, at 12:57 PM, Zhixuan Zhu wrote:

> Hi All,
>=20
> I'm using hadoop-0.20.2 to try out some simple tasks. I asked a
question
> about FileInputFormat a few days ago and get some prompt replys from
> this forum and it helped a lot. Thanks again! Now I have another
> question. I'm trying to invoke a C++ process from my mapper for each
> hdfs file in the input directory to achieve some parallel processing.

That seems weird - why aren't you using more maps and one file per-map?

> But how do I pass the file to the program? I would want to do
something
> like the following in my mapper:

IAC, libhdfs is one way to do HDFS ops via c/c++.

Arun

>=20
> Process lChldProc =3D Runtime.getRuntime().exec("myprocess -file
> $filepath");
>=20
> How do I pass the hdfs filesystem to an outside process like that? Is
> HadoopStreaming the direction I should go?
>=20
> Thanks very much for any reply in advance.
>=20
> Best,
> Grace