hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dayong <will...@gmail.com>
Subject Re: Hive and Impala
Date Wed, 02 Mar 2016 13:52:20 GMT
As I remember of few weeks before in Hadoop weekly news feed, cloudera has a benchmark showing
implala is a little better than spark SQL and hive with tez. You can check that. From my experience,
hive is still leading tool for regular ETL job since it is stable. The other tool are better
for adhoc and interactive query use case. Cloudera bet on implala especially with its new
kudo project. 

Thanks,
Dayong

> On Mar 1, 2016, at 5:14 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
> 
> My nocks on impala. (not intended to be a post knocking impala)
> 
> Impala really has not delivered on the complex types that hive has (after promising it
for quite a while), also it only works with the 'blessed' input formats, parquet, avro, text.
> 
> It is very annoying to work with impala, In my version if you create a partition in hive
impala does not see it. You have to run "refresh". 
> 
> In impala I do not have all the UDFS that hive has like percentile, etc. 
> 
> Impala is fast. Many data-analysts / data-scientist types that can't wait 10 seconds
for a query so when I need top produce something for them I make sure the data has no complex
types and uses a table type that impala understands. 
> 
> But for my work I still work primarily in hive, because I do not want to deal with all
the things that impala does not have/might have/ and when I need something special like my
own UDFs it is easier to whip up the solution in hive. 
> 
> Having worked with M$ SQL server, and vertica, Impala is on par with them but I don'think
of it like i think of hive. To me it just feels like a vertica that I can cheat loading sometimes
because it is backed by hdfs. 
> 
> Hive is something different, I am making pipelines, I am transforming data, doing streaming,
writing custom udfs, querying JSON directly. Its not != impala.
> 
> ::random message of the day::
> 
> 
>  
> 
>> On Tue, Mar 1, 2016 at 4:38 PM, Ashok Kumar <ashok34668@yahoo.com> wrote:
>> 
>> Dr Mitch,
>> 
>> My two cents here.
>> 
>> I don't have direct experience of Impala but in my humble opinion I share your views
that Hive provides the best metastore of all Big Data systems. Looking around almost every
product in one form and shape use Hive code somewhere. My colleagues inform me that Hive is
one of the most stable Big Data products.
>> 
>> With the capabilities of Spark on Hive and Hive on Spark or Tez plus of course MR,
there is really little need for many other products in the same space. It is good to keep
things simple.
>> 
>> Warmest
>> 
>> 
>> On Tuesday, 1 March 2016, 11:33, Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:
>> 
>> 
>> I have not heard of Impala anymore. I saw an article in LinkedIn titled
>> 
>> "Apache Hive Or Cloudera Impala? What is Best for me?"
>> 
>> "We can access all objects from Hive data warehouse with HiveQL which leverages the
map-reduce architecture in background for data retrieval and transformation and this results
in latency."
>> 
>> My response was
>> 
>> This statement is no longer valid as you have choices of three engines now with MR,
Spark and Tez. I have not used Impala myself as I don't think there is a need for it with
Hive on Spark or Spark using Hive metastore providing whatever needed. Hive is for Data Warehouse
and provides what is says on the tin. Please also bear in mind that Hive offers ORC storage
files that provide store Index capabilities further optimizing the queries with additional
stats at file, stripe and row group levels. 
>> 
>> Anyway the question is with Hive on Spark or Spark using Hive metastore what we cannot
achieve that we can achieve with Impala?
>> 
>> 
>> Dr Mich Talebzadeh
>>  
>> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>  
>> http://talebzadehmich.wordpress.com
> 

Mime
View raw message