hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bibin A Chundatt (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6779) Mapreduce job failure on submission
Date Thu, 15 Sep 2016 16:53:21 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15493913#comment-15493913
] 

Bibin A Chundatt commented on MAPREDUCE-6779:
---------------------------------------------

[~sjlee0]

>From ClassLoader  file is removed as per new implementation.Since the jar is not available
in applcation class path all map execution will fail.

{noformat}
-      classPath.add(file.toURI().toURL());
{noformat}
When {{JobConf#setJarByClass}} is set tries to get the jar name from classloader
{noformat}
  public void setJarByClass(Class cls) {
    String jar = ClassUtil.findContainingJar(cls);
    if (jar != null) {
      setJar(jar);
    }   
  }
{noformat}
{{ClassUtil#findContainingJar}}
{noformat}
    for(final Enumeration<URL> itr = loader.getResources(classFile);
          itr.hasMoreElements();) {
        final URL url = itr.nextElement();
        if ("jar".equals(url.getProtocol())) {
          String toReturn = url.getPath();
          if (toReturn.startsWith("file:")) {
            toReturn = toReturn.substring("file:".length());
          }
          toReturn = URLDecoder.decode(toReturn, "UTF-8");
          return toReturn.replaceAll("!.*$", "");
        }
      }
{noformat}
which returns null. The job jar will not get loaded to staging dir in {{JobResourceUploader#uploadResources}}
since jobjar is {{null}}.
{noformat}
  private void uploadJobJar(Job job, String jobJar, Path submitJobDir,
      short submitReplication) throws IOException {
    if (jobJar != null) { 
{noformat}


> Mapreduce job failure on submission
> -----------------------------------
>
>                 Key: MAPREDUCE-6779
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6779
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Bibin A Chundatt
>            Priority: Blocker
>
> Configure Hibench
> Try running Enhanced TestDFSIO
> {noformat}
> 2016-09-15 18:20:24,849 INFO mapreduce.Job: Task Id : attempt_1473943118844_0001_m_000001_0,
Status : FAILED
> Error: java.lang.RuntimeException: Error in configuring object
>         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
>         at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
>         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1806)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
>         ... 9 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh not found
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2330)
>         at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1108)
>         at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
>         ... 14 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh
not found
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2298)
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2322)
>         ... 16 more
> Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh
not found
>         at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2202)
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2296)
>         ... 17 more
> {noformat}
> *mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found.
See Job or Job#setJar(String).*
> Job jar is not getting set and not uploaded to staging dir



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message