Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (nike.apache.org: domain of muralikrishna.parimi@icloud.com
 designates 17.164.88.161 as permitted sender)
Content-type: text/plain; charset=us-ascii
MIME-version: 1.0 (1.0)
Subject: Re: Partitioned table and Bucket Map Join
From: murali parimi <muralikrishna.parimi@icloud.com>
In-reply-to: 
 <CAONr5=uXSEWYDwM9zOi2Ts0u6TRmXJc2kDoLmbC8ygUxXmjVKQ@mail.gmail.com>
Date: Thu, 29 Jan 2015 07:51:31 -0600
Content-transfer-encoding: quoted-printable
Message-id: <2B3C195A-7622-4309-879C-3079869E0346@icloud.com>
References: 
 <CAONr5=uXSEWYDwM9zOi2Ts0u6TRmXJc2kDoLmbC8ygUxXmjVKQ@mail.gmail.com>
To: "user@hive.apache.org" <user@hive.apache.org>

I faced the same situation where two tables with 3 billion records on each s=
ide and partitioned, sorted on same key. Set the following parameters in the=
 hive query assuming the join will happen in the map phase.=20

set hive.input.format=3Dorg.apache.hadoop.hive.ql.io.BucketizedHiveInputForm=
at;
set hive.optimize.bucketmapjoin=3Dtrue;
set hive.optimize.bucketmapjoin.sortedmerge=3Dtrue;
set hive.enforce.sorting=3Dtrue;

I am using hive version 13 and the storage format is Orc. One of the table i=
s small in size but I haven't checked whether irfan fit in the cache as we h=
ave huge memory. But the map sided join didn't happen. What could be the rea=
son?

Sent from my iPhone

> On Jan 29, 2015, at 7:38 AM, matshyeq <matshyeq@gmail.com> wrote:
>=20
> I do have two tables partitioned on the same criteria.
> Could I still take advantage of Bucket Map Join or better, Sort Merge Buck=
et Map Join?
> How?
>=20
> ~Maciek