Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9B6192009F4 for ; Thu, 26 May 2016 11:02:00 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 99A3C160A10; Thu, 26 May 2016 09:02:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id BA4EB160939 for ; Thu, 26 May 2016 11:01:59 +0200 (CEST) Received: (qmail 28918 invoked by uid 500); 26 May 2016 09:01:58 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 28908 invoked by uid 99); 26 May 2016 09:01:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2016 09:01:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 301331A1512 for ; Thu, 26 May 2016 09:01:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id vyG9MKf-un59 for ; Thu, 26 May 2016 09:01:56 +0000 (UTC) Received: from mail-qk0-f178.google.com (mail-qk0-f178.google.com [209.85.220.178]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 627045F36C for ; Thu, 26 May 2016 09:01:46 +0000 (UTC) Received: by mail-qk0-f178.google.com with SMTP id n63so53847005qkf.0 for ; Thu, 26 May 2016 02:01:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=2/oVWzAFJMQs05Gz8HPwCVUPCq1Fi7EdNjaXTTlgHLA=; b=FP4j2yhOCl4nx34BC1umwWCT5A2q1HXP5ATjD9u1fz4QPmb+V+cr6ge+LNBdYJD1fG VnTEJfsLg1ceVoSbMJcSHCkdQts7VhzVxfYI+9rgukPRhaa5ZpQuSBbuvnig0KzSYjif bzfdPSoRLCFvPJJMwg2//EUkqkjTQfHd8aKO3iphaDQQ1w5PF798VpIl+1/9gXMVIPIf cC5JVPCGrjxh3k5ZynyRdFeIQqGDDehjri7MjV7iGUQ1to9Ds7KH2VH5sAe8RnlsXlJV yMlR+Z9WjxhH7Mf5Z/AKnP6aKj4NGbyb8neoihQv0Iot9/ANPrrjFpA+SXu/xKIqRVPk D4Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=2/oVWzAFJMQs05Gz8HPwCVUPCq1Fi7EdNjaXTTlgHLA=; b=AHvC8aREuATQgv0wRYoSXNLQ1d2wZmiJnhsj3z1gNuhlvs9UhHbnACrzz0yjmbn0lU +yBJyMf/NozeB7K/OdqTkfzByqzhqnmCgWAIXZ0rrQO4il/BzIiCBlxTQ1G3LU0IRb00 MW6YosFY6eBQEvxMTpZYWV8mdqkI40ldkCrKL1p0DhLsoC2Cq5W+C5RtQV/HNYRfAN4r ZkqPnzn+s9BZ8+Ri7WrNcY+7TSkQwex3tFrlUKOCXieSDrS69AdVhTGFwrVHOh1YiDy+ /L6NpkUJq0yCBTvmR5xp4Rz7ZntPqm5YPneyFDyyrdFoK0hvv6EjUMwQZn9+40PMIgqM gsdw== X-Gm-Message-State: ALyK8tLY3yBUok2tlW3gAA+SGEXKVsBwau70Musym0reHzj4PzeQ5msoDdClgd7ZWjJvCdKayf2xUcuVgIBssw== MIME-Version: 1.0 X-Received: by 10.55.39.140 with SMTP id n134mr8118857qkn.10.1464253305784; Thu, 26 May 2016 02:01:45 -0700 (PDT) Received: by 10.55.126.131 with HTTP; Thu, 26 May 2016 02:01:45 -0700 (PDT) In-Reply-To: References: <2025950601.3106319.1460154291292.JavaMail.yahoo.ref@mail.yahoo.com> <2025950601.3106319.1460154291292.JavaMail.yahoo@mail.yahoo.com> Date: Thu, 26 May 2016 10:01:45 +0100 Message-ID: Subject: Re: Copying all Hive tables from Prod to UAT From: Mich Talebzadeh To: user Cc: Ashok Kumar Content-Type: multipart/alternative; boundary=94eb2c095a7c27128e0533bb0a5c archived-at: Thu, 26 May 2016 09:02:00 -0000 --94eb2c095a7c27128e0533bb0a5c Content-Type: text/plain; charset=UTF-8 That is a good point Jorn with regard to JDBC and Hive data I believe you can use JDBC to get a compressed data from an Oraclle or Sybase database cause decompression happens at the time of data access much like using a sqlplus or isql tool. However, it is worth trying what happens when one accesses Hive data through JDBC where the underlying table is compress using bzip2 or snappy etc. If this is oone-off request say copy all table from certain DB in Hive in Prod to UAT, I am not sure replication will be suitable as the request is for a snapshot. EXPORT/IMPORT through NAS or scp should be an option. NAS is better as it saves scp and copy across with taget having enough external space to get the files in. More useful tool would be to export the full Hive database in binary format and import it in target. Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 26 May 2016 at 07:28, Elliot West wrote: > Hello, > > I've been looking at this recently for moving Hive tables from on-premise > clusters to the cloud, but the principle should be the same for your > use-case. If you wish to do this in an automated way, some tools worth > considering are: > > - Hive's built in replication framework: > https://cwiki.apache.org/confluence/display/Hive/Replication > - Hive's IMPORT/EXPORT primitives: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport > - AirBnB's ReAir replication tool: > https://medium.com/airbnb-engineering/reair-easy-to-use-tools-for-migrating-and-replicating-petabyte-scale-data-warehouses-5153f8a433da > > Elliot. > > On 8 April 2016 at 23:24, Ashok Kumar wrote: > >> Hi, >> >> Anyone has suggestions how to create and copy Hive and Spark tables from >> Production to UAT. >> >> One way would be to copy table data to external files and then move the >> external files to a local target directory and populate the tables in >> target Hive with data. >> >> Is there an easier way of doing so? >> >> thanks >> >> >> > --94eb2c095a7c27128e0533bb0a5c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
That is a good point Jorn with regard to JDBC and Hiv= e data

I believe you can use JDBC to get a compres= sed data from an Oraclle or Sybase database cause decompression happens at = the time of data access much like using a sqlplus or isql tool.
<= br>
However, it is worth trying what happens when one accesses Hi= ve data through JDBC where the underlying table is compress using bzip2 or = snappy etc.

If this is oone-off request say copy a= ll table from certain DB in Hive in Prod to UAT, I am not sure replication = will be suitable as the request is for a snapshot.

EXPORT/IMPORT through NAS or scp should be an option. NAS is better as it = saves scp and copy across with taget having enough external space to get th= e files in.

More useful tool would be to export th= e full Hive database in binary format and import it in target.
<= br>
Cheers


On 26 May 2016 at 07:28, Elliot West <teabot= @gmail.com> wrote:
Hello,

I've been looking at this recently f= or moving Hive tables from on-premise clusters to the cloud, but the princi= ple should be the same for your use-case. If you wish to do this in an auto= mated way, some tools worth considering are:

On 8 April 2016 at 23:24, Ashok Kumar <ashok34668@yahoo.com> wrote:
Hi,

Anyo= ne has suggestions how to create and copy Hive and Spark tables from Produc= tion to UAT.

One way would= be to copy table data to external files and then move the external files t= o a local target directory and populate the tables in target Hive with data= .

Is there an easier way o= f doing so?

thanks




--94eb2c095a7c27128e0533bb0a5c--