hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayush Saxena (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-24339) REPL LOAD command ignores config properties set by WITH clause
Date Tue, 12 Jan 2021 14:12:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263354#comment-17263354
] 

Ayush Saxena commented on HIVE-24339:
-------------------------------------

Hi [~abstractdog]
 {{hive.exec.parallel.thread.number}} doesn't take affect as part of the {{With}} clause
from repl load. but hive.exec.parallel does; because whether the task can be run in parallel,
is figured out from task conf only :
{code:java}
 private TaskRunner launchTask(Task<?> task, boolean noName, String jobName, int jobCount)
throws HiveException {
.
.
.
    if (HiveConf.getBoolVar(task.getConf(), HiveConf.ConfVars.EXECPARALLEL) && task.canExecuteInParallel())
{
{code}
But the number of tasks to launch is from the driver :
{code:java}
  private void launchTasks(boolean noName, int jobCount, String jobName) throws HiveException
{
    // Launch upto maxthreads tasks
    Task<?> task;
    int maxthreads = HiveConf.getIntVar(driverContext.getConf(), HiveConf.ConfVars.EXECPARALLETHREADNUMBER);
{code}
So, I think this is a correct behaviour, you need to set that value at session level only.

> REPL LOAD command ignores config properties set by WITH clause
> --------------------------------------------------------------
>
>                 Key: HIVE-24339
>                 URL: https://issues.apache.org/jira/browse/HIVE-24339
>             Project: Hive
>          Issue Type: Bug
>            Reporter: László Bodor
>            Priority: Major
>
> By debug messages we confirmed that REPL LOAD command ignored some config properties
when they were provided in WITH clause, e.g.:
> {code}
> REPL LOAD bdpp01pub FROM 'hdfs://prdpdp01//apps/hive/repl/8237c7bd-ba26-4425-8659-3a0d32ab312c'
WITH ('mapreduce.job.queuename'='default','hive.exec.parallel'='true','hive.exec.parallel.thread.number'='128',
> ...
> {code}
> We found that it was working on 16 threads, ignoring 'hive.exec.parallel.thread.number'='128'.
Setting this property on session level worked.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message