Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of rabmdu@gmail.com designates
 209.85.220.169 as permitted sender)
MIME-Version: 1.0
Date: Wed, 23 Jul 2014 20:34:38 +0530
Message-ID: 
 <CANXCz3TL5_tRVg1bFdCwurEVU00E0Yx6GyFdBUmUSFU6yu1sYw@mail.gmail.com>
Subject: Hadoop streaming - Class not found
From: rab ra <rabmdu@gmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11364a64b4708504fedda8ae

--001a11364a64b4708504fedda8ae
Content-Type: text/plain; charset=UTF-8

Hello,

I am trying to run an executable using hadoop streaming 2.4

My executable is my mapper which is a groovy script. This script uses a
class from a jar file which I am sending via -libjars argument.

The hadoop streaming is made to span maps via an input file, each line
feeds to one map.

The question is, though the hadoop successfully executes the use case, but,
I see that some maps failed and restarted later. The failure was due to
failing to locate the class. The script has some imports and they are not
found. However, they are all in jar file.

I am tempted to think that when hadoop executes the first few map tasks,
the jar file is not "prepared yet" to be made available to maps and hence
the initial maps failed to locate the class, and later, when they are
restarted, it is able to locate the class and executes smoothly.

Is this correct? If not, can someone tell me why this behavior? How can I
get around this issue? Because of this, the use case takes little more time
to execute. I fear, when I expand the use case, this will surely cause
performance delay.


with regards
rab

--001a11364a64b4708504fedda8ae
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello,<div><br></div><div>I am trying to run an executable=
 using hadoop streaming 2.4</div><div><br></div><div>My executable is my ma=
pper which is a groovy script. This script uses a class from a jar file whi=
ch I am sending via -libjars argument.</div>
<div><br></div><div>The hadoop streaming is made to span maps via an input =
file, each line feeds to one map.=C2=A0</div><div><br></div><div>The questi=
on is, though the hadoop successfully executes the use case, but, I see tha=
t some maps failed and restarted later. The failure was due to failing to l=
ocate the class. The script has some imports and they are not found. Howeve=
r, they are all in jar file.=C2=A0</div>
<div><br></div><div>I am tempted to think that when hadoop executes the fir=
st few map tasks, the jar file is not &quot;prepared yet&quot; to be made a=
vailable to maps and hence the initial maps failed to locate the class, and=
 later, when they are restarted, it is able to locate the class and execute=
s smoothly.</div>
<div><br></div><div>Is this correct? If not, can someone tell me why this b=
ehavior? How can I get around this issue? Because of this, the use case tak=
es little more time to execute. I fear, when I expand the use case, this wi=
ll surely cause performance delay.</div>
<div><br></div><div><br></div><div>with regards</div><div>rab</div><div><br=
></div></div>

--001a11364a64b4708504fedda8ae--