hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayush Saxena (Jira)" <>
Subject [jira] [Commented] (HIVE-24339) REPL LOAD command ignores config properties set by WITH clause
Date Tue, 12 Jan 2021 14:12:00 GMT


Ayush Saxena commented on HIVE-24339:

Hi [~abstractdog]
 {{hive.exec.parallel.thread.number}} doesn't take affect as part of the {{With}} clause
from repl load. but hive.exec.parallel does; because whether the task can be run in parallel,
is figured out from task conf only :
 private TaskRunner launchTask(Task<?> task, boolean noName, String jobName, int jobCount)
throws HiveException {
    if (HiveConf.getBoolVar(task.getConf(), HiveConf.ConfVars.EXECPARALLEL) && task.canExecuteInParallel())
But the number of tasks to launch is from the driver :
  private void launchTasks(boolean noName, int jobCount, String jobName) throws HiveException
    // Launch upto maxthreads tasks
    Task<?> task;
    int maxthreads = HiveConf.getIntVar(driverContext.getConf(), HiveConf.ConfVars.EXECPARALLETHREADNUMBER);
So, I think this is a correct behaviour, you need to set that value at session level only.

> REPL LOAD command ignores config properties set by WITH clause
> --------------------------------------------------------------
>                 Key: HIVE-24339
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: László Bodor
>            Priority: Major
> By debug messages we confirmed that REPL LOAD command ignored some config properties
when they were provided in WITH clause, e.g.:
> {code}
> REPL LOAD bdpp01pub FROM 'hdfs://prdpdp01//apps/hive/repl/8237c7bd-ba26-4425-8659-3a0d32ab312c'
WITH ('mapreduce.job.queuename'='default','hive.exec.parallel'='true','hive.exec.parallel.thread.number'='128',
> ...
> {code}
> We found that it was working on 16 threads, ignoring 'hive.exec.parallel.thread.number'='128'.
Setting this property on session level worked.

This message was sent by Atlassian Jira

View raw message