hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-8294) Hive-on-Tez is going into MapOperator::cleanUpInputFileChangedOp()
Date Mon, 29 Sep 2014 20:40:33 GMT
Gopal V created HIVE-8294:
-----------------------------

             Summary: Hive-on-Tez is going into MapOperator::cleanUpInputFileChangedOp()
                 Key: HIVE-8294
                 URL: https://issues.apache.org/jira/browse/HIVE-8294
             Project: Hive
          Issue Type: Bug
          Components: Query Processor
    Affects Versions: 0.14.0
            Reporter: Gopal V
            Assignee: Gopal V


Hive-on-Tez produces a significant amount of time in small queries going over the list of
files in the plan to check for input context.

{code}
for (String onefile : conf.getPathToAliases().keySet()) {
      Path onepath = normalizePath(onefile);
....
   for (String onealias : conf.getPathToAliases().get(onefile)) {
        Operator<? extends OperatorDesc> op = conf.getAliasToWork().get(onealias);
        MapInputPath inp = new MapInputPath(onefile, onealias, op, partDesc);
        MapOpCtx context = opCtxMap.get(inp);
        if (context != null) {
          current = context;
          LOG.info("Processing alias " + onealias + " for file " + onefile);
          return;
        }
      }
{code}

For a simple test like {{select distinct l_shipdate from lineitem;}} it produced 5861039 log
lines with "Processing alias ".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message