hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Hive in-memory offerings in forthcoming releases
Date Mon, 10 Oct 2016 16:35:18 GMT
thanks.

Certainly it will be icing on the cake if Hive gets this add on.

Now I know Oracle 12c,  so perhaps the following will help. The Oracle
Database In-Memory Option is not a separate database. What it does is allow
the user to store a *copy* of selected tables, or partitions, in*columnar*
format in-memory within the Oracle Database memory space. All tables are
still present in row format and all copies on storage are in row format.
For tables / partitions that the user elects to store ‘in memory’ an
*additional* copy is stored, in columnar format, purely in memory. *These
columnar copies are not logged nor are they ever persisted to disk.*

The columnar copies may be loaded into memory either at database startup or
on first access. There are mechanisms (I believe Materialized Views
snapshot) that keep the in-memory copy in sync with the underlying row
format 'master’ copy. The  optimizer is aware of the presence and currency
of the in-memory copies and transparently uses them for any analytical
style queries that can benefit from the vastly faster processing speed.
This is all completely transparent to applications (which what should be
doing in Hive as well).



This part is interesting. The primary use case for this capability is to
accelerate the analytics part of mixed OLTP and Analytical workloads by
eliminating the need* for most of the Analytics indexes. *This speed up the
analytical queries by a huge amount.



HTH




Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 10 October 2016 at 17:00, Alan Gates <alanfgates@gmail.com> wrote:

> Hive doesn’t usually publish long term roadmaps.
>
> I am not familiar with either SAP ASE or Oracle 12c so I can’t say whether
> Hive is headed in that direction or not.
>
> We see LLAP as very important for speeding up Hive processing, especially
> in the cloud where fetches from blob storage are very expensive.  As an
> example, see how HDInsight on Microsoft’s Azure cloud is already using
> LLAP.  At the moment LLAP is read only, so an obvious next step here is
> adding write capabilities (see https://issues.apache.org/
> jira/browse/HIVE-14535 for some thoughts on how this might work).
>
> I don’t know if this answers your question or not.
>
> Alan.
>
> > On Oct 8, 2016, at 12:48, Mich Talebzadeh <mich.talebzadeh@gmail.com>
> wrote:
> >
> > Hi,
> >
> > Is there any documentation on Apache Hive proposed new releases which is
> going to offer an in-memory database (IMDB) in the form of LLAP or built on
> LLAP.
> >
> > Love to see something like SAP ASE IMDB or Oracle 12c in-memory
> offerings with Hive as well.
> >
> > Regards,
> >
> > Dr Mich Talebzadeh
> >
> > LinkedIn  https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >
> > http://talebzadehmich.wordpress.com
> >
> > Disclaimer: Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
> >
>
>

Mime
View raw message