Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9664F2E3C for ; Fri, 29 Apr 2011 17:59:48 +0000 (UTC) Received: (qmail 95345 invoked by uid 500); 29 Apr 2011 17:59:47 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 95303 invoked by uid 500); 29 Apr 2011 17:59:47 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 95295 invoked by uid 99); 29 Apr 2011 17:59:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 17:59:47 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [98.138.91.75] (HELO nm10-vm1.bullet.mail.ne1.yahoo.com) (98.138.91.75) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 29 Apr 2011 17:59:41 +0000 Received: from [98.138.90.48] by nm10.bullet.mail.ne1.yahoo.com with NNFMP; 29 Apr 2011 17:59:20 -0000 Received: from [98.138.89.192] by tm1.bullet.mail.ne1.yahoo.com with NNFMP; 29 Apr 2011 17:59:20 -0000 Received: from [127.0.0.1] by omp1050.mail.ne1.yahoo.com with NNFMP; 29 Apr 2011 17:59:20 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 196716.21297.bm@omp1050.mail.ne1.yahoo.com Received: (qmail 87416 invoked by uid 60001); 29 Apr 2011 17:59:20 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304099960; bh=3HBD03b0RaCbgzQMtRYID/PxG7PnwVPUfGRAeL4WHKI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=dutDJunBZ8lNE9UFb/J/K2GKEVSRBMgXvoeD9Xvnn1Wqm2ZoscysnZg2Ql34cPYgHfih54UrnXteDMZ6uWbbJIyaxGFlvHS5JpmYVVPF6oJXP/dCrDiqSQLPs7ayoz4mznNaEuSzXEIqaFzvtSo/hYJuJ2/9trc+IiQr1Ikoke8= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=TA2f5/bZMQpZJdnzRSdkaiuoa09fu6PayWr0Z61xO1UN85ZLoWBJiYDVdWI1Oill7OD3W8VZgr9+S2+N3Zu6xf5JpZO4gMhP1ATEG8MDuuaYQIzVe4GHugK0tnIZreoroFuIM+/C8/tA9jRPz86yXd6e8MW8JH5KiC0JM9Hq+yQ=; Message-ID: <79218.84523.qm@web120420.mail.ne1.yahoo.com> X-YMail-OSG: yJawRFEVM1njDl9l3xGIIFx2rxwclqdjv_70JmxNH6LymBA Gr2uYJbhhu0JckUBMZ5RzF0yqlt9QXMZEqRFS2rsRAi6AYvLUlbRm1uPa_U5 qvCg1s0c7ZcXqgn2tIHHVdLy1b32MH7Tx4A74VWpITl0VSQKXDe_jFMfaKMh ElBuTkJ1EpwayeOQQzyYE7c_DHbjvNXSN7falgrkszPtBawO6UkzGzrogAqj jUza2PcLblUkHkQIcoIv5Y2tfC1ZJjFOgmd3SQ9L6Y64TahOO5W35E1Jweoe EMl9atQXgX9GlpHEA.nZDU5AUjEQFqnS0akhaDVKt2TGfMHjOXCOYQPSeYjD 2r6KvwzpBfG6M_f1XpBPVmj4k Received: from [151.100.59.194] by web120420.mail.ne1.yahoo.com via HTTP; Fri, 29 Apr 2011 10:59:19 PDT X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.110.299900 References: <431050.21183.qm@web120403.mail.ne1.yahoo.com> <280940.18276.qm@web120420.mail.ne1.yahoo.com> <645665.12673.qm@web120414.mail.ne1.yahoo.com> Date: Fri, 29 Apr 2011 10:59:19 -0700 (PDT) From: Donatella Firmani Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH To: mapreduce-user@hadoop.apache.org In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-1378914708-1304099959=:84523" --0-1378914708-1304099959=:84523 Content-Type: text/plain; charset=us-ascii Dear Alex, that's exactly the point. I made my mapper process dump on log files the result of System.getenv("LD_LIBRARY_PATH") System.getProperty("java.library.path") and none of the values seem to be affected neither by the setting of mapred.child.java.opts or of mapred.child.env. in the mapred-site.xml file. Maybe is there something else that I have to do to make LD_LIBRARY_PATH in the JVM environment be correctly set? There are some restrictions on the values that it can assume (i.e. under HDFS an non in the FS of the node)? Cheers, DF ________________________________ From: Alex Kozlov To: mapreduce-user@hadoop.apache.org Sent: Fri, April 29, 2011 7:52:19 PM Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH The option should be passed to the child JVM environment when it is started. You can set most of the environment variables to garbage with no side-effect. A more important question what is the LD_LIBRARY_PATH in your JVM environment. Once again, check the job.xml file in the mapred.local.dir (should be /tmp/cache/${user.name}/... or something like this in the pseudo-config environment) or try to print out the environment variables directly in your map/reduce task. Alex K On Fri, Apr 29, 2011 at 10:37 AM, Donatella Firmani wrote: I just tried giving the option -Dmapred.child.env="LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3" writing no-sense environment variables like -Dmapred.child.env="blahblablah". > >It continues working... so I think that the option is completely ignored by the >bin/hadoop script. > >Do you think it is an expected behavior? > >Cheers, >DF > > > > ________________________________ From: Alex Kozlov > >To: mapreduce-user@hadoop.apache.org >Sent: Fri, April 29, 2011 7:03:50 PM > >Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH > > >You need only to edit the config files on the client or give the option with a >-Dmapred.child.env="LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3" > flag (if you implement Tool). You can check the job.xml file via JT UI to >verify that the parameters have the correct values for the job. > > >On Fri, Apr 29, 2011 at 9:05 AM, Donatella Firmani >wrote: > >Dear Yin, >> >>Good point: I can try to install 0.19 and reproduce the problem. I'll let you >>know ASAP. >> >>Thanks, >>DF >> >> >> >> >> >> ________________________________ From: Yin Lou >> >>To: mapreduce-user@hadoop.apache.org >>Sent: Fri, April 29, 2011 5:59:14 PM >>Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH >> >> >>Just curious, can we do this in 0.19? >> >>Thanks, >>Yin >> >> >>On Fri, Apr 29, 2011 at 10:29 AM, Robert Evans wrote: >> >>DF, >>> >>>You can set mapred.child.java.opts to set java options, but you can also set >>>mapred.child.env to set environment variables, be careful because they are space >>>separated with an = in between them. >>> >>> >>> >>> mapred.child.env >>>> LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3 >>>> >>>> >> >>--Bobby >> >> >>On 4/29/11 5:58 AM, "Donatella Firmani" wrote: >> >> >>To solve the issue addressed in my previous message, i tried setting property >>>mapred.child.java.opts in mapred-site.xml. But - even if it seems the right >>>approach in relation to what said in blogs & forums - there is a big problem >>>with it. >>> >>>Following the tutorial (hadoop website) as section Task Execution & >>Environment, >>> >>>my xml looks like: >>> >>> >>> >>> mapred.job.tracker >>> localhost:9001 >>> >>> >>> mapred.child.java.opts >>> >>>-Djava.library.path=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3 >>> >>> >>> >>> >>> >>> >>> >>>The problem arises when executing the job, because it is thrown an exception: >>> >>>Exception in thread "main" java.lang.NoClassDefFoundError: >>>-Djava/library/path=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3 >>> >>> >>> >>> >>>Any help would be appreciated. >>>Thanks in advance, >>> >>>DF >>> >>> >>> >>>----- Original Message ---- >>>From: Donatella Firmani >>>To: mapreduce-user@hadoop.apache.org >>>Sent: Fri, April 29, 2011 12:57:52 PM >>>Subject: Hadoop Mapreduce jobs and LD_LIBRARY_PATH >>> >>> >>> >>>Hi to all, >>> >>>I just subscribed to this mailing list and I'd like to ask you if anyone knows >>>how to deal with LD_LIBRARY_PATH. >>>I have a Java application that needs a proper setting of this environment >>>variable to work under Linux-Ubuntu. >>>I want to use tis application from a mapreduce job, unfortunately I could not >>>find a way to make things work against the LD_LIBRARY_PATH environment >>variable. >>> >>>I tried so many different strategies and I am stuck. Maybe someone of you can >>>help. >>> >>>Thanks in advance, >>>Cheers. >>> >>>DF >>> >>>PS: I use hadoop-0-20-2 >>> >>> >> > --0-1378914708-1304099959=:84523 Content-Type: text/html; charset=us-ascii
Dear Alex,

that's exactly the point. I made my mapper process dump on log files the result of

System.getenv("LD_LIBRARY_PATH")
System.getProperty("java.library.path")

and none of the values seem to be affected neither by the setting of mapred.child.java.opts or of mapred.child.env. in the mapred-site.xml file.

Maybe is there something else that I have to do to make LD_LIBRARY_PATH in the JVM environment be correctly set? There are some restrictions on the values that it can assume (i.e. under HDFS an non in the FS of the node)?

Cheers,
DF

From: Alex Kozlov <alexvk@cloudera.com>
To: mapreduce-user@hadoop.apache.org
Sent: Fri, April 29, 2011 7:52:19 PM
Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH

The option should be passed to the child JVM environment when it is started.  You can set most of the environment variables to garbage with no side-effect.  A more important question what is the LD_LIBRARY_PATH in your JVM environment.

Once again, check the job.xml file in the mapred.local.dir (should be /tmp/cache/${user.name}/... or something like this in the pseudo-config environment) or try to print out the environment variables directly in your map/reduce task.

Alex K

On Fri, Apr 29, 2011 at 10:37 AM, Donatella Firmani <donatellafirmani@yahoo.com> wrote:
I just tried giving the option -Dmapred.child.env="LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3" writing no-sense environment variables like  -Dmapred.child.env="blahblablah".

It continues working... so I think that the option is completely ignored by the bin/hadoop script.

Do you think it is an expected behavior?

Cheers,
DF

From: Alex Kozlov <alexvk@cloudera.com>

To: mapreduce-user@hadoop.apache.org
Sent: Fri, April 29, 2011 7:03:50 PM

Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH

You need only to edit the config files on the client or give the option with a -Dmapred.child.env="LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3" flag (if you implement Tool).  You can check the job.xml file via JT UI to verify that the parameters have the correct values for the job.

On Fri, Apr 29, 2011 at 9:05 AM, Donatella Firmani <donatellafirmani@yahoo.com> wrote:
Dear Yin,

Good point: I can try to install 0.19 and reproduce the problem. I'll let you know ASAP.

Thanks,
DF


From: Yin Lou <yin.lou.07@gmail.com>Sent: Fri, April 29, 2011 5:59:14 PM
Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH

Just curious, can we do this in 0.19?

Thanks,
Yin

On Fri, Apr 29, 2011 at 10:29 AM, Robert Evans <evans@yahoo-inc.com> wrote:
DF,

You can set mapred.child.java.opts to set java options, but you can also set mapred.child.env to set environment variables, be careful because they are space separated with an = in between them.

     <property>
   <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3</value>
    </property>

--Bobby


On 4/29/11 5:58 AM, "Donatella Firmani" <donatellafirmani@yahoo.com> wrote:

To solve the issue addressed in my previous message, i tried setting property
mapred.child.java.opts in mapred-site.xml. But - even if it seems the right
approach in relation to what said in blogs & forums - there is a big problem
with it.

Following the tutorial (hadoop website) as section Task Execution & Environment,

my xml looks like:

<configuration>
     <property>
         <name>mapred.job.tracker</name>
         <value>localhost:9001</value>
     </property>
     <property>
         <name>mapred.child.java.opts</name>
         <value>
-Djava.library.path=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3


         </value>
     </property>
</configuration>

The problem arises when executing the job, because it is thrown an exception:

Exception in thread "main" java.lang.NoClassDefFoundError:
-Djava/library/path=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3



Any help would be appreciated.
Thanks in advance,

DF



----- Original Message ----
From: Donatella Firmani <donatellafirmani@yahoo.com>
To: mapreduce-user@hadoop.apache.org
Sent: Fri, April 29, 2011 12:57:52 PM
Subject: Hadoop Mapreduce jobs and LD_LIBRARY_PATH



Hi to all,

I just subscribed to this mailing list and I'd like to ask you if anyone knows
how to deal with LD_LIBRARY_PATH.
I have a Java application that needs a proper setting of this environment
variable to work under Linux-Ubuntu.
I want to use tis application from a mapreduce job, unfortunately I could not
find a way to make things work against the LD_LIBRARY_PATH environment variable.

I tried so many different strategies and I am stuck. Maybe someone of you can
help.

Thanks in advance,
Cheers.

DF

PS: I use hadoop-0-20-2




--0-1378914708-1304099959=:84523--