hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
Date Thu, 18 Jul 2013 05:06:50 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13712016#comment-13712016
] 

Phabricator commented on HIVE-4502:
-----------------------------------

navis has commented on the revision "HIVE-4502 [jira] NPE - subquery smb joins fails".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:619 Ah sorry. This was throwing NPE
when debugging.
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 Ok, sure.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink1.java:70 I think it's not possible
to have 2 or more children for RS, but might be possible for Tez. I'll do that.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:261 ok
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:281 No. Will be changed
to package local.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:310 MapRedTask in fact.
If any of MapRedTask need BucketizedInputFormat, use BucketizedInputFormat. It's fix for https://issues.apache.org/jira/browse/HIVE-4502?focusedCommentId=13679262&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13679262
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:851 ok.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:942 done
  ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out:165 If Task-A is merged into
Task-B, task-A will not be visible anywhere. I've changed not to walk further when all parents
are not walked, which changed stage-ids in result.

  TS1  TS2
      SMB
       RS
      GBY
       RS
      GBY
       FS

  before :
  TS1(stage-1)-SMB-RS-GBY[split]RS(stage-2)-GBY-FS
  TS2(stage-3)-SMB[merge into stage-1]-RS-GBY-RS-GBY-RS-FS

  after :
  TS1(stage-1)-SMB
  TS2(stage-2)-SMB[merged into stage-1]-RS-GBY[split]RS(stage-3)-GBY-FS

  I'll make a issue for rearranging Stage-IDs. (and order of explain output, too)
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinFactory.java:228 It's just a refactoring.
setupBucketMapJoinInfo() was called both in initMapJoinPlan() and joinMapJoinPlan().
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:318 ok
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:949 It's called in join-checked
block.
  if (reducer.getClass() == JoinOperator.class) {
    ....
    cplan.setNeedsTagging(true);
  }

REVISION DETAIL
  https://reviews.facebook.net/D10695

BRANCH
  HIVE-4502

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, navis
Cc: brock, vikram

                
> NPE - subquery smb joins fails
> ------------------------------
>
>                 Key: HIVE-4502
>                 URL: https://issues.apache.org/jira/browse/HIVE-4502
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.11.0
>            Reporter: Vikram Dixit K
>            Assignee: Navis
>         Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch,
smb_mapjoin_25.q, smb_mapjoin_25.q
>
>
> Found this issue while running some SMB joins. Attaching test case that causes this error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message