Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of tucu@cloudera.com designates
 209.85.220.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <0005DA68C31EA7428873518FF86A94368E7E85@MAILSVR01.domino.softonic.com>
References: 
 <0005DA68C31EA7428873518FF86A94368E7E85@MAILSVR01.domino.softonic.com>
From: Alejandro Abdelnur <tucu@cloudera.com>
Date: Tue, 27 Nov 2012 09:10:48 -0800
Message-ID: 
 <CAJs-t7M8dJAUKLFqQftCPn=vozVHgoCYBc7iyzcK9xmhfrsnPw@mail.gmail.com>
Subject: Re: Complex MapReduce applications with the streaming API
To: "common-user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=f46d043890937c4c3e04cf7d23fd

--f46d043890937c4c3e04cf7d23fd
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

> Using Oozie seems to be an overkilling for this application, besides, it
doesn't support "loops"
> so the recusrsion can't really be implemented.

Correct, Oozie does not support loops, this is a restriction by design
(early prototypes supported loops). The idea was that you didn't want never
ending workflows. To this end, Coordinator Jobs address the recurrent run
of workflow jobs.

Still, if you want to do recursion in Oozie, you certainly can, a workflow
invoking to itself as a sub-workflow. Just make sure you define properly
your exit condition.

If you have additional questions, please move this thread to the
user@oozie.apache.org alias.


Thx


On Tue, Nov 27, 2012 at 4:03 AM, Zolt=E1n T=F3th-Czifra <
zoltan.tothczifra@softonic.com> wrote:

>  Hi everyone,
>
>  Thanks in advance for the support. My problem is the following:
>
>  I'm trying to develop a fairly complex MapReduce application using the
> streaming API (for demonstation purposes, so unfortunately the "use Java"
> answer doesn't work :-( ). I can get one single MapReduce phase running
> from command line with no problem. The problem is when I want to add more
> MapReduce phases which use each others output, and I maybe even want to d=
o
> a recursion (feed the its output to the same phase again) conditioned by =
a
> counter.
>
>  The solution in Java MapReduce is trivial (i.e. creating multiple Job
> instances and monitoring counters) but with the streaming API not quite.
> What is the correct way to manage my application with its native code?
> (Python, PHP, Perl...) Calling shell commands from a "controller" script?
> How should I obtain counters?...
>
>  Using Oozie seems to be an overkilling for this application, besides, it
> doesn't support "loops" so the recusrsion can't really be implemented.
>
>  Thanks a lot!
> Zoltan
>


--=20
Alejandro

--f46d043890937c4c3e04cf7d23fd
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

&gt;=A0<span style=3D"font-family:Tahoma;font-size:13px">Using Oozie seems =
to be an overkilling for this application, besides, it doesn&#39;t support =
&quot;loops&quot;=A0</span><div><span style=3D"font-family:Tahoma;font-size=
:13px">&gt; so the recusrsion can&#39;t really be implemented.</span></div>

<div><span style=3D"font-family:Tahoma;font-size:13px"><br></span></div><di=
v><span style=3D"font-family:Tahoma;font-size:13px">Correct, Oozie does not=
 support loops, this is a restriction by design (early prototypes supported=
 loops). The idea was that you didn&#39;t want never ending workflows. To t=
his end, Coordinator Jobs address the recurrent run of workflow jobs.</span=
></div>

<div><span style=3D"font-family:Tahoma;font-size:13px"><br></span></div><di=
v><font face=3D"Tahoma">Still, if you want to do recursion in Oozie, you ce=
rtainly can, a workflow invoking to itself as a sub-workflow. Just make sur=
e you define properly your exit condition.</font></div>

<div><font face=3D"Tahoma"><br></font></div><div><font face=3D"Tahoma">If y=
ou have additional questions, please move this thread to the <a href=3D"mai=
lto:user@oozie.apache.org">user@oozie.apache.org</a> alias.</font></div><di=
v>

<font face=3D"Tahoma"><br></font></div><div><font face=3D"Tahoma"><br></fon=
t></div><div><font face=3D"Tahoma">Thx</font></div><div class=3D"gmail_extr=
a"><br><br><div class=3D"gmail_quote">On Tue, Nov 27, 2012 at 4:03 AM, Zolt=
=E1n T=F3th-Czifra <span dir=3D"ltr">&lt;<a href=3D"mailto:zoltan.tothczifr=
a@softonic.com" target=3D"_blank">zoltan.tothczifra@softonic.com</a>&gt;</s=
pan> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">


<div>
<div style=3D"direction:ltr;font-size:10pt;font-family:Tahoma">Hi everyone,
<div><br>
</div>
<div>Thanks in advance for the support. My problem is the following:</div>
<div><br>
</div>
<div>I&#39;m trying to develop a fairly complex MapReduce application using=
 the streaming API (for demonstation purposes, so unfortunately the &quot;u=
se Java&quot; answer doesn&#39;t work :-( ). I can get one single MapReduce=
 phase running from command line with no problem.
 The problem is when I want to add more MapReduce phases which use each oth=
ers output, and I maybe even want to do a recursion (feed the its output to=
 the same phase again) conditioned by a counter.</div>
<div><br>
</div>
<div>The solution in Java MapReduce is trivial (i.e. creating multiple=A0Jo=
b instances and monitoring counters) but with the streaming API not quite. =
What is the correct way to manage my application with its native code? (Pyt=
hon, PHP, Perl...) Calling shell commands
 from a &quot;controller&quot; script? How should I obtain counters?...</di=
v>
<div><br>
</div>
<div>Using Oozie seems to be an overkilling for this application, besides, =
it doesn&#39;t support &quot;loops&quot; so the recusrsion can&#39;t really=
 be implemented.</div>
<div><br>
</div>
<div>Thanks a lot!</div>
<div>Zoltan</div>
</div>
</div>

</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>Alejandro<br=
>
</div>

--f46d043890937c4c3e04cf7d23fd--