hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Sankar Sivarama Subramaniyan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-5796) percentjobcomplete returned by webhcat is null
Date Wed, 20 Nov 2013 00:57:21 GMT

     [ https://issues.apache.org/jira/browse/HIVE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hari Sankar Sivarama Subramaniyan updated HIVE-5796:
----------------------------------------------------

    Affects Version/s: 0.12.0

> percentjobcomplete returned by webhcat is null
> ----------------------------------------------
>
>                 Key: HIVE-5796
>                 URL: https://issues.apache.org/jira/browse/HIVE-5796
>             Project: Hive
>          Issue Type: Bug
>          Components: WebHCat
>    Affects Versions: 0.12.0
>            Reporter: Hari Sankar Sivarama Subramaniyan
>            Assignee: Hari Sankar Sivarama Subramaniyan
>             Fix For: 0.13.0
>
>         Attachments: HIVE-5796.1.patch.txt
>
>
> Currently the WebHCat E2E test TestMapReduce_1 fails when comparing the job status field
percentComplete which is returned as null, the expected value is "map 100% reduce 100%".
> In the templeton.log I see the following message:
> {noformat}
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Using Hadoop Version:
0.23
> INFO  | 07 Nov 2013 17:14:08,106 | org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob
| WEBHCAT_CONF_DIR=null
> WARN  | 07 Nov 2013 17:14:08,106 | org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob
| Could not find D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\bin\null\override-container-log4j.properties.
Monitoring of Hadoop jobs submitted through WebHCat may be affected.
> {noformat}
> To get past this problem I explicitly set the WEBHCAT_CONF_DIR environment variable to
HCATALOG_HOME/etc/webhcat where override-container-log4j.properties is present.
> After fixing the above and rerunning, I still see the following error:
> {noformat}
> INFO  | 07 Nov 2013 18:29:39,517 | org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob
| WEBHCAT_CONF_DIR=D:\hdp\\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat
> INFO  | 07 Nov 2013 18:29:39,517 | org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob
| Found D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat\override-container-log4j.properties
to use for job submission.
> ERROR | 07 Nov 2013 18:29:39,519 | org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob
| org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob is not properly initialized.
Monitoring of Hadoop jobs submitted through WebHCat may be affected.
> java.lang.IllegalArgumentException: Pathname /d:/hadoop/hdfs/tmp from d:/hadoop/hdfs/tmp
is not a valid DFS filename.
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> 	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
> 	at org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:93)
> 	at org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:82)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Unknown Source)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> 	at org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.copyLog4JtoFileSystem(TempletonControllerJob.java:82)
> 	at org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.<clinit>(TempletonControllerJob.java:126)
> 	at org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:104)
> 	at org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:101)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Unknown Source)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> 	at org.apache.hive.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:101)
> 	at org.apache.hive.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:82)
> 	at org.apache.hive.hcatalog.templeton.JarDelegator.run(JarDelegator.java:55)
> 	at org.apache.hive.hcatalog.templeton.Server.mapReduceJar(Server.java:690)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> 	at java.lang.reflect.Method.invoke(Unknown Source)
> 	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> 	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> 	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> 	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> 	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> 	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350)
> 	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> 	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
> 	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
> 	at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> 	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> 	at org.eclipse.jetty.server.Server.handle(Server.java:349)
> 	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> 	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
> 	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> 	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
> 	at java.lang.Thread.run(Unknown Source)
> {noformat}
> I think the problem is due to the reliance on the value of hadoop.tmp.dir as a temporary
HDFS directory which is a temporary file location and applies to both local file system and
HDFS. There are some derived parameters that build on top of this location assuming its local
(eg. mapred.local.dir) and sometimes assuming its on hdfs (eg. mapred.system.dir), in short
it is very hairy to rely on hadoop.tmp.dir.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message