Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 511B111C38 for ; Mon, 5 May 2014 15:16:47 +0000 (UTC) Received: (qmail 93986 invoked by uid 500); 5 May 2014 15:16:23 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 93887 invoked by uid 500); 5 May 2014 15:16:20 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 93751 invoked by uid 500); 5 May 2014 15:16:18 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 93706 invoked by uid 99); 5 May 2014 15:16:17 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 May 2014 15:16:17 +0000 Date: Mon, 5 May 2014 15:16:17 +0000 (UTC) From: "Ashutosh Chauhan (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-7009) HIVE_USER_INSTALL_DIR could not bet set to non-HDFS filesystem MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7009: ----------------------------------- Assignee: Chuan Liu > HIVE_USER_INSTALL_DIR could not bet set to non-HDFS filesystem > -------------------------------------------------------------- > > Key: HIVE-7009 > URL: https://issues.apache.org/jira/browse/HIVE-7009 > Project: Hive > Issue Type: Bug > Components: Tez > Affects Versions: 0.13.0 > Reporter: Chuan Liu > Assignee: Chuan Liu > Attachments: HIVE-7009.patch > > > In {{hive/ql/exec/tez/DagUtils.java}}, we enforce the user path get from {{HIVE_USER_INSTALL_DIR}} to be HDFS. This makes it impossible to run Hive+Tez jobs on non-HDFS filesystem, e.g. WASB. Relevant code are as follows: > {noformat} > public Path getDefaultDestDir(Configuration conf) throws LoginException, IOException { > UserGroupInformation ugi = ShimLoader.getHadoopShims().getUGIForConf(conf); > String userName = ShimLoader.getHadoopShims().getShortUserName(ugi); > String userPathStr = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_USER_INSTALL_DIR); > Path userPath = new Path(userPathStr); > FileSystem fs = userPath.getFileSystem(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IOException(ErrorMsg.INVALID_HDFS_URI.format(userPathStr)); > } > {noformat} > Exceptions running jobs with defaultFs configured to WASB. > {noformat} > 2014-05-01 00:21:39,847 ERROR exec.Task (TezTask.java:execute(192)) - Failed to execute tez graph. > java.io.IOException: wasb://hdi31-chuanliu@clhdistorage.blob.core.windows.net/user is not a hdfs uri > at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:662) > at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:759) > at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:321) > at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:159) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:154) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)