Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C45A6200BA8 for ; Mon, 24 Oct 2016 09:29:17 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C2E75160AEB; Mon, 24 Oct 2016 07:29:17 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 94226160AE1 for ; Mon, 24 Oct 2016 09:29:16 +0200 (CEST) Received: (qmail 94077 invoked by uid 500); 24 Oct 2016 07:29:15 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 94066 invoked by uid 99); 24 Oct 2016 07:29:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Oct 2016 07:29:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C12991806F7 for ; Mon, 24 Oct 2016 07:29:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.179 X-Spam-Level: *** X-Spam-Status: No, score=3.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, KAM_LINEPADDING=1.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=klarna-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id ZJwLK1h8AIAw for ; Mon, 24 Oct 2016 07:29:12 +0000 (UTC) Received: from mail-yw0-f177.google.com (mail-yw0-f177.google.com [209.85.161.177]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 7F2865FC27 for ; Mon, 24 Oct 2016 07:29:11 +0000 (UTC) Received: by mail-yw0-f177.google.com with SMTP id w3so173116378ywg.1 for ; Mon, 24 Oct 2016 00:29:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klarna-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Y7KSwtxzBXvQRUGoCdO9gbfkG0DDUUtRspsixlf4d+Q=; b=dGU7tNVEVpmGxnPKEPl2XAqXWTX2wYA6fyUVgpkEzIgzla3ffa98FKaBuIvh+gc37V cUsptz6cbw27wnBfS0lCF8sq0Nu95MiLfkUXOF4B8OMBHfwpScfKANJxeEShu03cvq/9 wfDse/w7GT9fNJ8mDz87dKkxrbQtmvr7VUmXbEa9jKnhurdE51VXmu1L1f9zP3Vijyhr H+NEsAoGhY3UJlSpxIMpz7BB7msjUBu1QPOH7V9K3hZx1Df64uctOb7fzb27XltfdhmI 7N6F1Vtm70nW3y4DwSzutasmXy+OIA0n603QbsD9d7P75Tre2VUIGRxMdTcatbmF3Fim S/Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Y7KSwtxzBXvQRUGoCdO9gbfkG0DDUUtRspsixlf4d+Q=; b=LoQSoO/GoTwIf++uhIJo2uEjHWwKz3OO4bRkf3cLy7Siq7I7zEkxS+eOZDzp9eJo9R zmzCvvITo9MKXvxIEsDwl5Hjr7e1w0bm26avgKKKEoJUJbgNsUtTOL+4xUxPDh5I0NpI ORlqBHsixVNw0SowvHSWyWb8ubXIt8u2yRHtovjCc6lRM8ADx9sAHXqr5YuLDpsg35Nn rs9vShacnQ8qVYknzdN/DN1z0N4XM4x4Tv9mE3481z5bwc2n+OvNb4bsAKll8A7aHOKD hdUwdhtpMm8Vk5INcAvfIJFJi1wHZXMgy4LevyuJUfq6xYd6ExpNYxY5jjtu55G/8Cyn IoJg== X-Gm-Message-State: ABUngvcSqW5FYscMcClu/241QtwkfvokLGFH82pAZNwgoktslzeug1c5nyaSs0ZbMtGz47tLBVG1MuMKRicSvo/7 X-Received: by 10.129.98.212 with SMTP id w203mr13267835ywb.345.1477294150308; Mon, 24 Oct 2016 00:29:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.170.177 with HTTP; Mon, 24 Oct 2016 00:29:09 -0700 (PDT) In-Reply-To: References: <72022890-7998-449D-A472-6EDA0C2E86E6@gmail.com> From: Per Ullberg Date: Mon, 24 Oct 2016 09:29:09 +0200 Message-ID: Subject: Re: Hive metadata on Hbase To: "user@hive.apache.org" Content-Type: multipart/alternative; boundary=001a1146c91a0f0e6f053f97591e archived-at: Mon, 24 Oct 2016 07:29:17 -0000 --001a1146c91a0f0e6f053f97591e Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable What version of hive are you running? /Pelle On Monday, October 24, 2016, Mich Talebzadeh wrote: > @Per > > We run full transactional enabled Hive metadb on an Oracle DB. > > I don't have statistics now but will collect from AWR reports no problem. > > @Jorn, > > The primary reason Oracle was chosen is because the company has global > licenses for Oracle + MSSQL + SAP and they are classified as Enterprise > Grade databases. > > None of MySQL and others are classified as such so they cannot be deploye= d > in production. > > Besides, for us to have Hive metadata on Oracle makes sense as our > infrastructure does all the support, HA etc for it and they have trained > DBAs to look after it 24x7. > > Admittedly we are now relying on HDFS itself plus Hbase as well for > persistent storage. So the situation might change. > > HTH > > > > > > > > Dr Mich Talebzadeh > > > > LinkedIn * https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianrbJ= d6zP6AcPCCdOABUrV8Pw > * > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising fro= m > such loss, damage or destruction. > > > > On 24 October 2016 at 06:46, Per Ullberg > wrote: > >> I thought the main gain was to get ACID on Hive performant enough. >> >> @Mich: Do you run with ACID-enabled tables? How many >> Create/Update/Deletes do you do per second? >> >> best regards >> /Pelle >> >> On Mon, Oct 24, 2016 at 7:39 AM, J=C3=B6rn Franke > > wrote: >> >>> I think the main gain is more about getting rid of a dedicated database >>> including maintenance and potential license cost. >>> For really large clusters and a lot of users this might be even more >>> beneficial. You can avoid clustering the database etc. >>> >>> On 24 Oct 2016, at 00:46, Mich Talebzadeh >> > wrote: >>> >>> >>> A while back there was some notes on having Hive metastore on Hbase as >>> opposed to conventional RDBMSs >>> >>> I am currently involved with some hefty work with Hbase and Phoenix for >>> batch ingestion of trade data. As long as you define your Hbase table >>> through Phoenix and with secondary Phoenix indexes on Hbase, the speed = is >>> impressive. >>> >>> I am not sure how much having Hbase as Hive metastore is going to add t= o >>> Hive performance. We use Oracle 12c as Hive metastore and the Hive >>> database/schema is built on solid state disks. Never had any issues wit= h >>> lock and concurrency. >>> >>> Therefore I am not sure what one is going to gain by having Hbase as th= e >>> Hive metastore? I trust that we can still use our existing schemas on >>> Oracle. >>> >>> HTH >>> >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianr= bJd6zP6AcPCCdOABUrV8Pw >>> * >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damag= es >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >> >> >> -- >> >> *Per Ullberg* >> Data Vault Tech Lead >> Odin Uppsala >> +46 701612693 <+46+701612693> >> >> Klarna AB (publ) >> Sveav=C3=A4gen 46, 111 34 Stockholm >> Tel: +46 8 120 120 00 <+46812012000> >> Reg no: 556737-0431 >> klarna.com >> >> > --=20 *Per Ullberg* Data Vault Tech Lead Odin Uppsala +46 701612693 <+46+701612693> Klarna AB (publ) Sveav=C3=A4gen 46, 111 34 Stockholm Tel: +46 8 120 120 00 <+46812012000> Reg no: 556737-0431 klarna.com --001a1146c91a0f0e6f053f97591e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable What version of hive are you running?

/Pelle=C2=A0=

On Monday, October 24, 2016, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:<= br>
@Per

=
We run full transactional enabled Hive metadb on an Oracle DB. <= /div>

I don't have statistics now but will collect f= rom AWR reports no problem.

@Jorn,

<= /div>
The primary reason Oracle was chosen is because the company has g= lobal licenses for Oracle + MSSQL + SAP and they are classified as Enterpri= se Grade databases.

None of MySQL and others are c= lassified as such so they cannot be deployed in production.

<= /div>
Besides, for us to have Hive metadata on Oracle makes sense as ou= r infrastructure does all the support, HA etc for it and they have trained = DBAs to look after it 24x7.

Admittedly we are now = relying on HDFS itself plus Hbase as well for persistent storage. So the si= tuation might change.

HTH







Dr Mich Talebzadeh

=C2=A0

LinkedIn =C2=A0https://www.linkedin.com/profile/view?id= =3DAAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw<= /p>

=C2=A0

http:= //talebzadehmich.wordpress.com


Disclaimer:=C2=A0= Use it=C2=A0at your own risk. Any and all responsib= ility for any loss, damage or destruction of data or any other property which may arise from relying on this email= 9;s=C2=A0technical=C2=A0content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from = such loss, damage or destruction.

=C2=A0

<= font color=3D"#000000" face=3D"Times New Roman" size=3D"3">

On 24 October 2016 at 06:46, Per Ullberg <per.ullberg@klarna.com>= wrote:
I thought= the main gain was to get ACID on Hive performant enough.

@Mich: Do you run with ACID-enabled tables? How many Create/Update/Delete= s do you do per second?

best regards
/Pe= lle

On Mon, Oct 24, 2016 at 7:39 AM, J=C3=B6rn Franke <jornfranke@gmail.com> wrote:
=
I think the main gain is= more about getting rid of a dedicated database including maintenance and p= otential license cost.=C2=A0
For really large clusters and a lot = of users this might be even more beneficial. You can avoid clustering the d= atabase etc.

On 24 Oct 2016, at 00:46, Mich Talebza= deh <mich.talebzadeh@gmail.com> wrot= e:


A while back there was some notes on having Hive metastore on Hbas= e as opposed to conventional RDBMSs

I am currently= involved with some hefty work with Hbase and Phoenix for batch ingestion o= f trade data. As long as you define your Hbase table through Phoenix and wi= th secondary Phoenix indexes on Hbase, the speed is impressive.
<= br>
I am not sure how much having Hbase as Hive metastore is goin= g to add to Hive performance. We use Oracle 12c=C2=A0as Hive metastore and = the Hive database/schema is built on solid state disks. Never had any issue= s with lock and concurrency.

Therefore I am not su= re what one is going to gain by having Hbase as the Hive metastore? I trust= that we can still use our existing schemas on Oracle.

=
HTH



Dr Mich Talebzadeh

=C2=A0

LinkedIn =C2=A0https://www.linkedin.com/profile/view?id= =3DAAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

=C2=A0

http:= //talebzadehmich.wordpress.com


Disclaimer:=C2=A0= Use it=C2=A0at your own risk. Any and all responsib= ility for any loss, damage or destruction of data or any other property which may arise from relying on this email= 9;s=C2=A0technical=C2=A0content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from = such loss, damage or destruction.

=C2=A0

<= font color=3D"#000000" face=3D"Times New Roman" size=3D"3">



--

Per Ullberg
Data Vault Tech Lead
Odin Uppsala
+46 701612693

Klarna AB=C2=A0(publ)
Sveav=C3=A4gen 46, 111 34 S= tockholm
Tel:=C2=A0+46 8 120 120 00
Reg= no: 556737-0431
klarna.com




--
Per Ullberg
Data Vault Tech Lead=
Odin Uppsala
+46 701612693

Klar= na AB=C2=A0(publ)=
Sveav=C3=A4gen 46, 111 34 Stockholm
Tel:=C2=A0+46 8 120 120 00
Reg no: 556737-0431
klarna.com


--001a1146c91a0f0e6f053f97591e--