hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdelrahman Shettia <ashet...@hortonworks.com>
Subject Re: using "-libjars" in Hadoop 2.2.1
Date Wed, 16 Apr 2014 20:30:55 GMT
Hi Kim,

You can try to grep on the RM java process by running the following
command:

ps aux | grep




On Wed, Apr 16, 2014 at 10:31 AM, Kim Chew <kchew534@gmail.com> wrote:

> Thanks Rahman, I have mixed things up a little bit in my mapred-site.xml
> so it tried to run the job locally. Now I am running into the problem that
> Rahul has, I am unable to to connect to the ResourceManager.
>
> The setup of my targeted cluster runs MR1 instead of YARN, hence the "
> mapreduce.framework.name" is set to "classic".
>
> Here are my settings in my mapred-site.xml on the client side.
>
> <property>
>     <!-- Pointed to the remote JobTracker -->
>         <name>mapreduce.job.tracker.address</name>
>         <value>172.31.3.150:8021</value>
>     </property>
>     <property>
>         <name>mapreduce.framework.name</name>
>         <value>yarn</value>
>     </property>
>
> and my yarn-site.xml
>
>        <property>
>             <description>The hostname of the RM.</description>
>             <name>yarn.resourcemanager.hostname</name>
>             <value>172.31.3.150</value>
>         </property>
>
>         <property>
>             <description>The address of the applications manager interface
> in the RM.</description>
>             <name>yarn.resourcemanager.address</name>
>             <value>${yarn.resourcemanager.hostname}:8032</value>
>         </property>
>
> 14/04/16 10:23:02 INFO client.RMProxy: Connecting to ResourceManager at /
> 172.31.3.150:8032
> 14/04/16 10:23:10 INFO ipc.Client: Retrying connect to server:
> hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1 SECONDS)
>
> Therefore, the question is how do I figure out where the ResourceManager
> is running?
>
> TIA
>
> Kim
>
>
>
> On Wed, Apr 16, 2014 at 8:43 AM, Abdelrahman Shettia <
> ashettia@hortonworks.com> wrote:
>
>> Hi Kim,
>>
>> It looks like it is pointing to hdfs location. Can you create the hdfs
>> dir and put the jar there? Hope this helps
>> Thanks,
>> Rahman
>>
>> On Apr 16, 2014, at 8:39 AM, Rahul Singh <smart.rahul.iiit@gmail.com>
>> wrote:
>>
>> any help...all are welcome?
>>
>>
>> On Wed, Apr 16, 2014 at 1:13 PM, Rahul Singh <smart.rahul.iiit@gmail.com>wrote:
>>
>>> Hi,
>>>  I am running with the following command but still, jar is not available
>>> to mapper and reducers.
>>>
>>> hadoop jar /home/hduser/workspace/Minerva.jar my.search.Minerva
>>> /user/hduser/input_minerva_actual /user/hduser/output_merva_actual3
>>> -libjars /home/hduser/Documents/Lib/json-simple-1.1.1.jar
>>> -Dmapreduce.user.classpath.first=true
>>>
>>>
>>> Error Log
>>>
>>> 14/04/16 13:08:37 INFO client.RMProxy: Connecting to ResourceManager at /
>>> 0.0.0.0:8032
>>> 14/04/16 13:08:37 INFO client.RMProxy: Connecting to ResourceManager at /
>>> 0.0.0.0:8032
>>> 14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line
>>> option parsing not performed. Implement the Tool interface and execute your
>>> application with ToolRunner to remedy this.
>>> 14/04/16 13:08:37 INFO mapred.FileInputFormat: Total input paths to
>>> process : 1
>>> 14/04/16 13:08:37 INFO mapreduce.JobSubmitter: number of splits:10
>>> 14/04/16 13:08:37 INFO mapreduce.JobSubmitter: Submitting tokens for
>>> job: job_1397534064728_0028
>>> 14/04/16 13:08:38 INFO impl.YarnClientImpl: Submitted application
>>> application_1397534064728_0028
>>> 14/04/16 13:08:38 INFO mapreduce.Job: The url to track the job:
>>> http://L-Rahul-Tech:8088/proxy/application_1397534064728_0028/<http://l-rahul-tech:8088/proxy/application_1397534064728_0028/>
>>> 14/04/16 13:08:38 INFO mapreduce.Job: Running job: job_1397534064728_0028
>>> 14/04/16 13:08:47 INFO mapreduce.Job: Job job_1397534064728_0028 running
>>> in uber mode : false
>>> 14/04/16 13:08:47 INFO mapreduce.Job:  map 0% reduce 0%
>>> 14/04/16 13:08:58 INFO mapreduce.Job: Task Id :
>>> attempt_1397534064728_0028_m_000005_0, Status : FAILED
>>> Error: java.lang.RuntimeException: Error in configuring object
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>>>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at javax.security.auth.Subject.doAs(Subject.java:416)
>>>     at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>>> Caused by: java.lang.reflect.InvocationTargetException
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>     at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>     at java.lang.reflect.Method.invoke(Method.java:622)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
>>>     ... 9 more
>>> Caused by: java.lang.NoClassDefFoundError:
>>> org/json/simple/parser/ParseException
>>>     at java.lang.Class.forName0(Native Method)
>>>     at java.lang.Class.forName(Class.java:270)
>>>     at
>>> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1821)
>>>     at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1786)
>>>     at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1880)
>>>     at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1906)
>>>     at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1107)
>>>     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
>>>     ... 14 more
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.json.simple.parser.ParseException
>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
>>>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
>>>     ... 22 more
>>>
>>> When i analyzed the logs it says
>>> "14/04/16 13:08:37 WARN mapreduce.JobSubmitter: Hadoop command-line
>>> option parsing not performed. Implement the Tool interface and execute your
>>> application with ToolRunner to remedy this."
>>>
>>> But i have implemented the tool class as described below:
>>>
>>> package my.search;
>>>
>>> import org.apache.hadoop.conf.Configured;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.FileInputFormat;
>>> import org.apache.hadoop.mapred.FileOutputFormat;
>>> import org.apache.hadoop.mapred.JobClient;
>>> import org.apache.hadoop.mapred.JobConf;
>>> import org.apache.hadoop.mapred.TextInputFormat;
>>> import org.apache.hadoop.mapred.TextOutputFormat;
>>> import org.apache.hadoop.util.Tool;
>>> import org.apache.hadoop.util.ToolRunner;
>>>
>>> public class Minerva extends Configured implements Tool
>>> {
>>>     public int run(String[] args) throws Exception {
>>>         JobConf conf = new JobConf(Minerva.class);
>>>         conf.setJobName("minerva sample job");
>>>
>>>         conf.setMapOutputKeyClass(Text.class);
>>>         conf.setMapOutputValueClass(TextArrayWritable.class);
>>>
>>>         conf.setOutputKeyClass(Text.class);
>>>         conf.setOutputValueClass(Text.class);
>>>
>>>         conf.setMapperClass(Map.class);
>>>         // conf.setCombinerClass(Reduce.class);
>>>         conf.setReducerClass(Reduce.class);
>>>
>>>         conf.setInputFormat(TextInputFormat.class);
>>>         conf.setOutputFormat(TextOutputFormat.class);
>>>
>>>         FileInputFormat.setInputPaths(conf, new Path(args[0]));
>>>         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>>>
>>>         JobClient.runJob(conf);
>>>
>>>         return 0;
>>>     }
>>>
>>>     public static void main(String[] args) throws Exception {
>>>         int res = ToolRunner.run(new Minerva(), args);
>>>         System.exit(res);
>>>     }
>>> }
>>>
>>>
>>> Please let me know if you see any issues?
>>>
>>>
>>>
>>> On Thu, Apr 10, 2014 at 9:29 AM, Shengjun Xin <sxin@gopivotal.com>wrote:
>>>
>>>> add '-Dmapreduce.user.classpath.first=true' to your command and try
>>>> again
>>>>
>>>>
>>>>
>>>> On Wed, Apr 9, 2014 at 6:27 AM, Kim Chew <kchew534@gmail.com> wrote:
>>>>
>>>>> It seems to me that in Hadoop 2.2.1, using the "libjars" option does
>>>>> not search the jars located in the the local file system but HDFS. For
>>>>> example,
>>>>>
>>>>> hadoop jar target/myJar.jar Foo -libjars
>>>>> /home/kchew/test-libs/testJar.jar /user/kchew/inputs/raw.vector
>>>>> /user/kchew/outputs hdfs://remoteNN:8020 remoteJT:8021
>>>>>
>>>>> 14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>>>>> processName=JobTracker, sessionId=
>>>>> 14/04/08 15:11:02 INFO mapreduce.JobSubmitter: Cleaning up the staging
>>>>> area
>>>>> file:/tmp/hadoop-kchew/mapred/staging/kchew202924688/.staging/job_local202924688_0001
>>>>> 14/04/08 15:11:02 ERROR security.UserGroupInformation:
>>>>> PriviledgedActionException as:kchew (auth:SIMPLE)
>>>>> cause:java.io.FileNotFoundException: File does not exist:
>>>>> hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar
>>>>> java.io.FileNotFoundException: File does not exist:
>>>>> hdfs:/remoteNN:8020/home/kchew/test-libs/testJar.jar
>>>>>     at
>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>>>>>     at
>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>>>>>     at
>>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>>>>     at
>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>>>>>     at
>>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
>>>>>     at
>>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
>>>>>     at
>>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
>>>>>     at
>>>>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
>>>>>     at
>>>>> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
>>>>>
>>>>> So under Hadoop 2.2.1, do I have to explicitly set some configurations
>>>>> so when using the "libjars" option it will copy the file to hdfs from
local
>>>>> fs?
>>>>>
>>>>> TIA
>>>>>
>>>>> Kim
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards
>>>> Shengjun
>>>>
>>>
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message