Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Thu, 18 Jul 2013 05:06:50 +0000 (UTC)
From: "Phabricator (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12646157.1367819315334.71124.1374124010470@arcas>
In-Reply-To: <JIRA.12646157.1367819315334@arcas>
References: <JIRA.12646157.1367819315334@arcas>
Subject: [jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13712016#comment-13712016 ] 

Phabricator commented on HIVE-4502:
-----------------------------------

navis has commented on the revision "HIVE-4502 [jira] NPE - subquery smb joins fails".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:619 Ah sorry. This was throwing NPE when debugging.
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 Ok, sure.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink1.java:70 I think it's not possible to have 2 or more children for RS, but might be possible for Tez. I'll do that.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:261 ok
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:281 No. Will be changed to package local.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:310 MapRedTask in fact. If any of MapRedTask need BucketizedInputFormat, use BucketizedInputFormat. It's fix for https://issues.apache.org/jira/browse/HIVE-4502?focusedCommentId=13679262&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13679262
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:851 ok.
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:942 done
  ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out:165 If Task-A is merged into Task-B, task-A will not be visible anywhere. I've changed not to walk further when all parents are not walked, which changed stage-ids in result.

  TS1  TS2
      SMB
       RS
      GBY
       RS
      GBY
       FS

  before :
  TS1(stage-1)-SMB-RS-GBY[split]RS(stage-2)-GBY-FS
  TS2(stage-3)-SMB[merge into stage-1]-RS-GBY-RS-GBY-RS-FS

  after :
  TS1(stage-1)-SMB
  TS2(stage-2)-SMB[merged into stage-1]-RS-GBY[split]RS(stage-3)-GBY-FS

  I'll make a issue for rearranging Stage-IDs. (and order of explain output, too)
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinFactory.java:228 It's just a refactoring. setupBucketMapJoinInfo() was called both in initMapJoinPlan() and joinMapJoinPlan().
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:318 ok
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:949 It's called in join-checked block.
  if (reducer.getClass() == JoinOperator.class) {
    ....
    cplan.setNeedsTagging(true);
  }

REVISION DETAIL
  https://reviews.facebook.net/D10695

BRANCH
  HIVE-4502

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, navis
Cc: brock, vikram

                
> NPE - subquery smb joins fails
> ------------------------------
>
>                 Key: HIVE-4502
>                 URL: https://issues.apache.org/jira/browse/HIVE-4502
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.11.0
>            Reporter: Vikram Dixit K
>            Assignee: Navis
>         Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q
>
>
> Found this issue while running some SMB joins. Attaching test case that causes this error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira