hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Stewart <robstewar...@googlemail.com>
Subject Hive Performance
Date Sat, 07 Nov 2009 18:03:20 GMT
Hi there. I'm in the process of writing a paper, and part of it I aim to
write (yet another) comparative study on various interfaces with Hadoop.

This will almost certainly include Pig and Hive, probably MapReduce, and
maybe JAQL.

I have read the papers published on the Hive JIRA (pig vs hive vs MapReduce
for 2 queries, an aggregation, and a join). I am, however, wanting to know a
bit from the Hive community.

1. Do you guys (the Hive developers) have a standardized benchmarking tool
to use prior to each Hive release? I am thinking of something similar to
PigMix, used by the Pig developers. In case you don't know, PigMix is a set
of 12 designed queries, implemented in Pig and Java Hadoop, and comparisons
are made on execution time. Does the Hive community have something similar?

2. The Pig wiki point out some unique features of Pig that allow optimal
execution performance. For instance, they have a methods to optimize queries
on skewed data (by taking samples of the data for reduce key allocations. Is
there something about the implementation of Hive that gives it some
functionality not found in other interfaces. And better still, would there
some Hive implementation that could work as a proof of concept to show any
optimized features of Hive?

3. One section suggested for investigation within the Pig development team
is to create a SQL like language that could be compiled down through Pig to
MR jobs. If such a project was to achieve parity with Hive's SQL like
interface, where would be the distinction be between Pig and Hive.
Certainly, from a users perspective, there would be very little difference.
If the only difference turns out to be the execution performance achieved by
one interface over another, where would this put the inferior interface (be
that either Pig or Hive) in terms of its relevance in the Hadoop software
stack?


Many thanks,


Rob Stewart

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message