Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2487E18860 for ; Wed, 20 Jan 2016 09:27:42 +0000 (UTC) Received: (qmail 72298 invoked by uid 500); 20 Jan 2016 09:27:39 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 72233 invoked by uid 500); 20 Jan 2016 09:27:39 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 72223 invoked by uid 99); 20 Jan 2016 09:27:39 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2016 09:27:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 44D2AC01A8 for ; Wed, 20 Jan 2016 09:27:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.974 X-Spam-Level: *** X-Spam-Status: No, score=3.974 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, LOTS_OF_MONEY=0.001, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=liulishuo-com.20150623.gappssmtp.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id OiBHvJ6P0S76 for ; Wed, 20 Jan 2016 09:27:28 +0000 (UTC) Received: from mail-io0-f178.google.com (mail-io0-f178.google.com [209.85.223.178]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id AAB642054B for ; Wed, 20 Jan 2016 09:27:27 +0000 (UTC) Received: by mail-io0-f178.google.com with SMTP id 1so13732617ion.1 for ; Wed, 20 Jan 2016 01:27:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=liulishuo-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=w9MQhRM7SxxjEvGJihppsMhVhjwO3Q1hKYwlQbFy8fk=; b=knu6NGyh/XqR+fWqnNRtgD4KtplG8pUxUqQleePtuHpHm1jCVPQsS+E1PqB/Kl7nkr gyTqJehDKYL76pZlS145++pvWqfKFUM/MH5BU8mC79CKIGWzkmVXgzIqPTYO4kw51MBh dDj8AwENfuwAd+PWFOpY+uwzScw2KXANVFUi3ENWYSdvJVddDtC+PBc2qMMKOd6nIf40 3FM6pH7yyfF5i+FwgM+mM/g+nQU48qmYDKeVam+9GqKTuaRaWYkcS05fcDwe0anmZj2G GSvLBPyJPYg8wjL/imPAQjjbE0INOCihGXNWsquFJLSLenVi0ZikporYWwvyUJF/EyY+ 2PkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=w9MQhRM7SxxjEvGJihppsMhVhjwO3Q1hKYwlQbFy8fk=; b=lNs4Z2mL55Rj97ZcTbRojxtrtWd66S78X5L0OiPRZWNBb5bbO1Zr9CAjoTzIr/pC7E LZn1D4NSGXoT5peHHAWTp0on07yJXvSOOGWFwEQAAeds1xSo0zwlySrgF00S9M+pXUOH lnHyi76fRj2ZAF30O5hzc30dRihkZmVgiHezchtSUyoldSLbzoZ9IWGX1kxOxRhBXJte TPV2iCvpS2dn+RSUGIbS+ELKfC1u/aLk5A5A599YvBNAIb2kVDL+RxNEavWDyFqv9+Gj NgdobPMRs5PjyaIMCqoAMqVoNHVcb3Chg6dICr5Eskt7tFKdjvPULaragRp5ppKEBGq0 M4VQ== X-Gm-Message-State: ALoCoQk2/fIT4/F0xsXiCVEKg4c8RQ+aPutQvp90ua/O0Xbm+p9T7KSEwfm9lZUL5rYr6EcbJl068V3Ixy0n587Pqce4uAxl7JvqxI1sqEKQgGHJElzf7g0= MIME-Version: 1.0 X-Received: by 10.107.185.214 with SMTP id j205mr36829263iof.175.1453282046939; Wed, 20 Jan 2016 01:27:26 -0800 (PST) Received: by 10.36.101.5 with HTTP; Wed, 20 Jan 2016 01:27:26 -0800 (PST) In-Reply-To: References: Date: Wed, 20 Jan 2016 17:27:26 +0800 Message-ID: Subject: Re: the `use database` command will change the scheme of target table? From: =?UTF-8?B?6JGj5Lqa5Yab?= To: user@hive.apache.org Content-Type: multipart/alternative; boundary=94eb2c0769c42aad260529c098dd --94eb2c0769c42aad260529c098dd Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable hi all, I read the wiki of section Temporary Folders: https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#= AdminManualConfiguration-HiveMetastoreConfigurationVariables My target table's filesystem is HDFS, but hive write the temporary data on S3 after I use an S3 database. Hive uses temporary folders both on the machine running the Hive client and the default HDFS instance. These folders are used to store per-query temporary/intermediate data sets and are normally cleaned up by the hive client when the query is finished. However, in cases of abnormal hive client termination, some data may be left behind. The configuration details are as follows: - On the HDFS cluster this is set to */tmp/hive-* by default and is controlled by the configuration variable *hive.exec.scratchdir* - On the client machine, this is hardcoded to */tmp/* Note that when *writing data to a table/partition, Hive will first write to a temporary location on the target table's filesystem* (using hive.exec.scratchdir as the temporary location) and then move the data to the target table. This applies in all cases - whether tables are stored in HDFS (normal case) or in file systems like S3 or even NFS. On Wed, Jan 20, 2016 at 12:06 PM, =E8=91=A3=E4=BA=9A=E5=86=9B wrote: > thanks Marcin, > > t1 created within the temp database right? which is point to HDFS. so th= e > output directory of m/r job should be in HDFS? > > my problem is why the output directory was host in s3 filesystem after I = *use > prd* database. > > > > On Wed, Jan 20, 2016 at 11:52 AM, Marcin Tustin > wrote: > >> That is the expected behaviour. Managed tables are created within the >> directory of their host database. >> >> >> On Tuesday, 19 January 2016, =E8=91=A3=E4=BA=9A=E5=86=9B wrote: >> >>> hi list, >>> >>> we use the HDFS and S3 as the Hive Filesystem at the same time. here >>> has an issue: >>> >>> >>> *scenario* 1: >>> >>> hive command: >>> >>> use default; >>> >>> create table temp.t1 // the database of temp which points to HDFS >>> as >>> select c1 from prd.t2; // the database of prd and the table t2 are >>> all points to S3 >>> >>> it works well. >>> >>> >>> *scenario* 2: >>> >>> hive command: >>> >>> *use prd; * >>> >>> create table temp.t1 // the database of temp which points to HDFS >>> as >>> select c1 from prd.t2; // the database of prd and the table t2 are >>> all point to S3 >>> >>> the exception occurred with: >>> >>> Failed with exception Unable to move source >>> s3a://warehouse-tmp/tmp/hive-ubuntu/hive_2016-01-20_xxxxxx/-ext-10001 t= o >>> destination hdfs://hadoop-0/warehouse/temp.db/t1/ >>> >>> and then, I try to change the Scratch space by the configuration key: >>> hive.exec.scratchdir, and set the value to hdfs://hadoop-0/*tmp-foo*/..= . >>> , but also failed with: >>> >>> Unable to move source s3a://warehouse-tmp*/tmp-foo* ... to >>> >>> it seems to the *use database* command change the scheme of the path >>> for target table? >>> >>> hive version: 0.13.1 >>> >>> >>> thanks. >>> >> >> Want to work at Handy? Check out our culture deck and open roles >> >> Latest news at Handy >> Handy just raised $50m >> led >> by Fidelity >> >> > --94eb2c0769c42aad260529c098dd Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
hi all,=C2=A0

My target table's= filesystem is HDFS, but hive write the temporary data on S3 after I use an= S3 database. =C2=A0

Hive uses temporary folders both on the machine runni= ng the Hive client and the default HDFS instance. These folders are used to= store per-query temporary/intermediate data sets and are normally cleaned = up by the hive client when the query is finished. However, in cases of abno= rmal hive client termination, some data may be left behind. The configurati= on details are as follows:

  • O= n the HDFS cluster this is set to=C2=A0/tmp/hive-<username>= =C2=A0by default and is controlled by the configuration variable=C2=A0h= ive.exec.scratchdir
  • On the client machine, this is hardcoded t= o=C2=A0/tmp/<username>

Note that when writing data to a table/partition,= Hive will first write to a temporary location on the target table's filesystem (using hive.exec.scratchd= ir as the temporary location) and then move the data to the target table. T= his applies in all cases - whether tables are stored in HDFS (normal case) = or in file systems like S3 or even NFS.


On Wed, Jan 20, 2016 at 12:06 PM, =E8= =91=A3=E4=BA=9A=E5=86=9B <ric.dong@liulishuo.com> wrote= :
thanks Marcin,=C2=A0

t1 created wi= thin the temp database right? which is point to HDFS. =C2=A0so the output d= irectory of m/r job should be in HDFS?=C2=A0

my problem is why the output directory was host in s3=C2=A0filesystem a= fter I use prd database.=C2=A0


On Wed, Jan 20, 2016 at 11:52 AM= , Marcin Tustin <mtustin@handybook.com> wrote:
That is the expected behaviour. Managed tables are= created within the directory of their host database.=C2=A0
=

On Tuesday, 19 January 2016, =E8=91=A3=E4=BA=9A=E5=86=9B <= ;ric.dong@liuli= shuo.com> wrote:
= hi list,=C2=A0

we use the HDFS and S3 as the Hive Filesys= tem at the same time. =C2=A0 here has an issue:=C2=A0


scenari= o 1:

hive command:
=C2=A0
use default;=C2=A0

creat= e table temp.t1 =C2=A0 =C2=A0 =C2=A0 // the database of temp which points t= o HDFS=C2=A0
as=C2=A0
select c1 from prd.t2; =C2=A0 =C2=A0 // the dat= abase of prd and the table t2 are all points to S3

it works well.=C2= =A0


scenario 2:=C2=A0

hive command:=C2=A0

<= b>use prd;=C2=A0

create table temp.t1 =C2=A0 =C2=A0 =C2=A0 // th= e database of temp which points to HDFS=C2=A0
as=C2=A0
select c1 from= prd.t2; =C2=A0 =C2=A0 // the database of prd and the table t2 are all poin= t to S3

the exception occurred with:=C2=A0

Failed with except= ion Unable to move source s3a://warehouse-tmp/tmp/hive-ubuntu/hive_2016-01-= 20_xxxxxx/-ext-10001 to destination hdfs://hadoop-0/warehouse/temp.db/t1/
and then, I try to change the Scratch space by the configuration key:= hive.exec.scratchdir, and set the value to hdfs://hadoop-0/tmp-foo/= ... , but also failed with:

Unable to move source s3a://warehouse-tm= p/tmp-foo ... to=C2=A0

it seems to the use database= command change the scheme of the path for target table?=C2=A0

h= ive version: 0.13.1=C2=A0

thanks.=C2=A0

Want to= work at Handy? Check out our=C2=A0culture deck = and open roles
=
Latest= =C2=A0news=C2=A0at Handy
Handy=C2=A0just raised $5= 0m=C2=A0led by Fidelity



--94eb2c0769c42aad260529c098dd--