hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Hive and Impala
Date Wed, 02 Mar 2016 14:37:41 GMT
I think you can always make a benchmark that has this and this result. You always have to see
what is evaluated and generally I recommend to always try yourself for your data and your
queries.

There is also a lot of change within the projects. Impala may have Kudo, but Hive has ORC,
Tez and Spark in combination with LLAP. 

As I said I always recommend to understand and try out the different technologies. 

> On 02 Mar 2016, at 14:52, Dayong <willddy@gmail.com> wrote:
> 
> As I remember of few weeks before in Hadoop weekly news feed, cloudera has a benchmark
showing implala is a little better than spark SQL and hive with tez. You can check that. From
my experience, hive is still leading tool for regular ETL job since it is stable. The other
tool are better for adhoc and interactive query use case. Cloudera bet on implala especially
with its new kudo project. 
> 
> Thanks,
> Dayong
> 
>> On Mar 1, 2016, at 5:14 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
>> 
>> My nocks on impala. (not intended to be a post knocking impala)
>> 
>> Impala really has not delivered on the complex types that hive has (after promising
it for quite a while), also it only works with the 'blessed' input formats, parquet, avro,
text.
>> 
>> It is very annoying to work with impala, In my version if you create a partition
in hive impala does not see it. You have to run "refresh". 
>> 
>> In impala I do not have all the UDFS that hive has like percentile, etc. 
>> 
>> Impala is fast. Many data-analysts / data-scientist types that can't wait 10 seconds
for a query so when I need top produce something for them I make sure the data has no complex
types and uses a table type that impala understands. 
>> 
>> But for my work I still work primarily in hive, because I do not want to deal with
all the things that impala does not have/might have/ and when I need something special like
my own UDFs it is easier to whip up the solution in hive. 
>> 
>> Having worked with M$ SQL server, and vertica, Impala is on par with them but I don'think
of it like i think of hive. To me it just feels like a vertica that I can cheat loading sometimes
because it is backed by hdfs. 
>> 
>> Hive is something different, I am making pipelines, I am transforming data, doing
streaming, writing custom udfs, querying JSON directly. Its not != impala.
>> 
>> ::random message of the day::
>> 
>> 
>>  
>> 
>>> On Tue, Mar 1, 2016 at 4:38 PM, Ashok Kumar <ashok34668@yahoo.com> wrote:
>>> 
>>> Dr Mitch,
>>> 
>>> My two cents here.
>>> 
>>> I don't have direct experience of Impala but in my humble opinion I share your
views that Hive provides the best metastore of all Big Data systems. Looking around almost
every product in one form and shape use Hive code somewhere. My colleagues inform me that
Hive is one of the most stable Big Data products.
>>> 
>>> With the capabilities of Spark on Hive and Hive on Spark or Tez plus of course
MR, there is really little need for many other products in the same space. It is good to keep
things simple.
>>> 
>>> Warmest
>>> 
>>> 
>>> On Tuesday, 1 March 2016, 11:33, Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:
>>> 
>>> 
>>> I have not heard of Impala anymore. I saw an article in LinkedIn titled
>>> 
>>> "Apache Hive Or Cloudera Impala? What is Best for me?"
>>> 
>>> "We can access all objects from Hive data warehouse with HiveQL which leverages
the map-reduce architecture in background for data retrieval and transformation and this results
in latency."
>>> 
>>> My response was
>>> 
>>> This statement is no longer valid as you have choices of three engines now with
MR, Spark and Tez. I have not used Impala myself as I don't think there is a need for it with
Hive on Spark or Spark using Hive metastore providing whatever needed. Hive is for Data Warehouse
and provides what is says on the tin. Please also bear in mind that Hive offers ORC storage
files that provide store Index capabilities further optimizing the queries with additional
stats at file, stripe and row group levels. 
>>> 
>>> Anyway the question is with Hive on Spark or Spark using Hive metastore what
we cannot achieve that we can achieve with Impala?
>>> 
>>> 
>>> Dr Mich Talebzadeh
>>>  
>>> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>  
>>> http://talebzadehmich.wordpress.com
>> 

Mime
View raw message