Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of mirko.kaempf@gmail.com
 designates 209.85.160.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAN_uYPji=oF3JAzg0mKeWVueAEBj3oNXUw0W-0gJ0aPwQstGVw@mail.gmail.com>
References: 
 <CAN_uYPji=oF3JAzg0mKeWVueAEBj3oNXUw0W-0gJ0aPwQstGVw@mail.gmail.com>
From: =?ISO-8859-1?Q?Mirko_K=E4mpf?= <mirko.kaempf@gmail.com>
Date: Tue, 10 Dec 2013 12:07:19 +0100
Message-ID: 
 <CALKHTG8B4h=v73y0vOnPbf6Yu=GHod7qgRz+X_QUvnow35G7cg@mail.gmail.com>
Subject: Re: Execute hadoop job remotely and programmatically
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7b160513e585b304ed2c1e34

--047d7b160513e585b304ed2c1e34
Content-Type: text/plain; charset=ISO-8859-1

Hi Yexi,

please have a look at the -libjars option of the hadoop cmd. It tells the
system what additional libs have to be sent to the cluster before the job
can start. Each time you submit the job, this kind of distribution happens
again. So its not a good idea for really large libs, those you should
deploy on all nodes and than you have to configure the classpath of the
JVMs running the tasks.

Best wishes.
Mirko


Best


2013/12/9 Yexi Jiang <yexijiang@gmail.com>

> Hi, All,
>
> I am working on a project that requires to execute a hadoop job remotely
> and the job requires some third-part libraries (jar files).
>
> Based on my understanding, I tried:
>
> 1. Copy these jar files to hdfs.
> 2. Copy them into the distributed cache using
> DistributedCache.addFileToClassPath so that hadoop can spread these jar
> files to each of the slave nodes.
>
> However, my program still throws ClassNotFoundException. Indicating that
> some of the classes cannot be found when the job is running.
>
> So I'm wondering:
> 1. What is the correct way to run a job remotely and programmatically
> while the job requires some third-party jar files.
> 2. I found DistributedCache is deprecated (I'm using hadoop 1.2.0), what
> is the alternative class?
>
> Regards,
> Yexi
>

--047d7b160513e585b304ed2c1e34
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Yexi,<div><br></div><div style>please have a look at th=
e -libjars option of the hadoop cmd. It tells the system what additional li=
bs have to be sent to the cluster before the job can start. Each time you s=
ubmit the job, this kind of distribution happens again. So its not a good i=
dea for really large libs, those you should deploy on all nodes and than yo=
u have to configure the classpath of the JVMs running the tasks.</div>

<div style><br></div><div style>Best wishes.</div><div style>Mirko</div><di=
v style><br></div><div style><br></div><div style>Best=A0</div></div><div c=
lass=3D"gmail_extra"><br><br><div class=3D"gmail_quote">2013/12/9 Yexi Jian=
g <span dir=3D"ltr">&lt;<a href=3D"mailto:yexijiang@gmail.com" target=3D"_b=
lank">yexijiang@gmail.com</a>&gt;</span><br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi, All,
<div><br></div><div>I am working on a project that requires to execute a ha=
doop job remotely and the job requires some third-part libraries (jar files=
).</div><div><br></div><div>Based on my understanding, I tried:</div><div>


<br></div><div>1. Copy these jar files to hdfs.</div><div>2. Copy them into=
 the distributed cache using DistributedCache.addFileToClassPath so that ha=
doop can spread these jar files to each of the slave nodes.=A0</div><div>


<br></div><div>However, my program still throws ClassNotFoundException. Ind=
icating that some of the classes cannot be found when the job is running.</=
div><div><br></div><div>So I&#39;m wondering:</div><div>1. What is the corr=
ect way to run a job remotely and programmatically while the job requires s=
ome third-party jar files.</div>


<div>2. I found DistributedCache is deprecated (I&#39;m using hadoop 1.2.0)=
, what is the alternative class?</div><div><br></div><div>Regards,</div><di=
v>Yexi</div></div>
</blockquote></div><br></div>

--047d7b160513e585b304ed2c1e34--