hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gautam <gautamkows...@gmail.com>
Subject Tez reducer parallelism ..
Date Wed, 16 Mar 2016 00:37:11 GMT
Hello,


I'm trying to optimize some queries in Hive that were recently switched to
Tez.. had a general question regarding reducer parallelism ..

A lot of our queries do the following style of simultaneous windowing ..


 SELECT

    row_number() OVER( PARTITION BY app, user, type ORDER BY ts ) as
a_number,

    row_number() OVER( PARTITION BY day, app, user, type ORDER BY ts ) as
type_rank,

    row_number() OVER( PARTITION BY day, app, user   ORDER BY ts ) as
dau_rank,

 FROM messages

 WHERE ...


Since each OVER / PARTITION-By clause is independent they can the put into
parallelized Reducer phases. But what I see is that these get serialized
into M1 -> R1 -> R2 -> R3 .. instead of M1 -> [ R1, R2, R3 ]

Is this something that Tez tries to do at all or an optimization that I can
use to my benefit ?

Cheers,
-Gautam.

Mime
View raw message