hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Furcy Pin <furcy....@flaminem.com>
Subject about collect_set and ordering
Date Wed, 30 Jul 2014 09:44:01 GMT
Hi all,

I just wanted to point out a little gotcha we got while using the
collect_set UDF :

You should not perform directly a GROUP BY over a collect_set(...), because
the
set is cast as an array and is not necessarilly sorted.

For example, we ran a query looking like this;

SELECT
set,
COUNT(1) as nb
GROUP BY set
FROM
(
SELECT
colA
collect_set(colB) as set
FROM db.table
GROUP BY colA
) T
;

and got :

[A] 10205
[B] 93856
[A,B] 34865
[B,A] 48324

We had to replace it with a sort_array(collect_set(...)).

I just wanted to point out that perhaps this subtelty should be mentionned
in the doc of the collect_set UDF...

As a corollary, do you guys think a collect_sorted_set using a TreeSet
would be useful (and/or
more efficient than using sort_array(collect_set)) ?

Regards,

Furcy

Mime
View raw message