hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14167) Use work directories provided by Tez instead of directly using YARN local dirs
Date Thu, 14 Jul 2016 21:47:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378453#comment-15378453
] 

Siddharth Seth commented on HIVE-14167:
---------------------------------------

{code}
+    if (LlapProxy.isDaemon()) {
+      localDirList = HiveConf.getVar(conf, HiveConf.ConfVars.LLAP_DAEMON_WORK_DIRS);
+      if (localDirList != null && !localDirList.isEmpty()) {
+        return localDirList;
+      } // otherwise, fall back to use tez work dirs
+    }
{code}
This can be improved further for LLAP. WIll create a follow up jira.

{code}
  if (conf.get("hive.execution.engine").equals("tez")) {
{code}
HiveConf.get( ... instead of conf.get ?

{code}
 return null;
{code}
I suspect this will not work too well with HiveOnSpark.

> Use work directories provided by Tez instead of directly using YARN local dirs
> ------------------------------------------------------------------------------
>
>                 Key: HIVE-14167
>                 URL: https://issues.apache.org/jira/browse/HIVE-14167
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 2.1.0
>            Reporter: Siddharth Seth
>            Assignee: Wei Zheng
>         Attachments: HIVE-14167.1.patch
>
>
> HIVE-13303 fixed things to use multiple directories instead of a single tmp directory.
However it's using yarn-local-dirs directly.
> I'm not sure how well using the yarn-local-dir will work on a secure cluster.
> Would be better to use Tez*Context.getWorkDirs. This provides an app specific directory
- writable by the user.
> cc [~sershe]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message