hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <>
Subject [jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly
Date Tue, 04 Mar 2014 01:47:30 GMT


Sergey Shelukhin updated HIVE-6537:

    Attachment: HIVE-6537.01.patch

attempt #2... presumably not only TableScans can be valid parents, because if I remove all
other operators (as in the initial version) the tests fail. The input from someone with better
knowledge of the original path would be helpful

> NullPointerException when loading hashtable for MapJoin directly
> ----------------------------------------------------------------
>                 Key: HIVE-6537
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-6537.01.patch, HIVE-6537.patch
> We see the following error:
> {noformat}
> 2014-02-20 23:33:15,743 FATAL [main] org.apache.hadoop.hive.ql.metadata.HiveException:
>         at
>         at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(
>         at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(
>         at org.apache.hadoop.hive.ql.exec.MapOperator.process(
>         at
>         at
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(
>         at
>         at org.apache.hadoop.mapred.YarnChild$
>         at Method)
>         at
>         at
>         at org.apache.hadoop.mapred.YarnChild.main(
> Caused by: java.lang.NullPointerException
>         at java.util.Arrays.fill(
>         at
>         at
>         ... 15 more
> {noformat}
> It appears that the tables in Arrays.fill call is nulls. I don't really have full understanding
of this path, but what I gleaned so far is this...
> From what I see, tables would be set unconditionally in initializeOp of the sink, and
in no other place, so I assume for this code to ever  work that startForward calls it at least
some time.
> Here, it doesn't call it, so it's null. 
> Previous loop also uses tables, and should have NPE-d before fill was ever called; it
didn't, so I'd assume it never executed. 
> There's a little bit of inconsistency in the above code where directWorks are added to
parents unconditionally but sink is only added as child conditionally. I think it may be that
some of the direct works are not table scans; in fact given that loop never executes they
may be null (which is rather strange). 
> Regardless, it seems that the logic should be fixed, it may be the root cause

This message was sent by Atlassian JIRA

View raw message