hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal Vijayaraghavan <gop...@apache.org>
Subject Re: Cross join/cartesian product explanation
Date Fri, 06 Nov 2015 22:09:56 GMT

> Over the last few week I¹ve been trying to use cross joins/cartesian
>products and was wondering why, exactly, this all gets sent to one
>reducer. All I¹ve heard or read is that Hive can¹t/doesn¹t parallelize
>the job. 

The hashcode of the shuffle key is 0, since you need to process every row
against every key - there's no possibility of dividing up the work.

Tez will actually have a cross-product edge (TEZ-2104), which is a
distributed cross-product proposal but wasn't picked up in the last Google
Summer of Code.

The real trouble is that MapReduce cannot re-direct data at all (there's
only shuffle edges).

> Does anyone have a workaround?

I use a staged partitioned table as a workaround for this, hashed on a
high nDV key - the goal of the Tez edge is to shuffle the data similarly
at runtime.

For instance, this python script makes a query with a 19x improvement in
distribution for a cross-product which generates 50+Gb of data from a
~10Mb input.

https://gist.github.com/t3rmin4t0r/cfb5bb4f7094d595c1e8


It is possible for Hive-Tez to actually generate UNION VertexGroups, but
it's much more efficient to do this as a edge with a custom EdgeManager,
since that opens up potentially implementing ThetaJoins in hive using that.

Cheers,
Gopal



Mime
View raw message