Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 44A0FE863 for ; Mon, 27 May 2013 08:39:14 +0000 (UTC) Received: (qmail 85805 invoked by uid 500); 27 May 2013 08:39:12 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 85739 invoked by uid 500); 27 May 2013 08:39:11 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 85710 invoked by uid 99); 27 May 2013 08:39:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 May 2013 08:39:10 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nitinpawar432@gmail.com designates 209.85.128.46 as permitted sender) Received: from [209.85.128.46] (HELO mail-qe0-f46.google.com) (209.85.128.46) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 May 2013 08:39:04 +0000 Received: by mail-qe0-f46.google.com with SMTP id 1so3487906qee.19 for ; Mon, 27 May 2013 01:38:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=fZJ7EKo7EzSRBmvfnQ6qZdCvTjHDFFZ+zKQ/62PTEvI=; b=uv4caBQwlw4Pdy6vjBxQCfefpIPAe2AoypGoo4lmp/pQW4NLnDpsQRFN9sS3VJDM/5 iAU1A633p5pasZ4JiaGAESL+iXX8XjzLBJJpu/VJrdrqYMocjCC0fKO5DePSXk/Ydd8H HXeh3Gf0h9FDxghy4e2XYxx0JNZdfsip1ZaBdCK61V56ubNDuY5L5gfewXptx8mSDlqs c/USdSIkzWt6NDV5GRT+eqcpQZcW1YAq8yf3uYP2udpxTzgssjh66WowTQE/COUqNaIB zwxAIWy7/1YIbLhQXi+Bgs1Lo0eTUCy2W8JVkRsGq+ETYFx9I2Xnz0NCS9vvYRDSLwMg jALg== MIME-Version: 1.0 X-Received: by 10.224.7.195 with SMTP id e3mr26788601qae.5.1369643923561; Mon, 27 May 2013 01:38:43 -0700 (PDT) Received: by 10.224.40.71 with HTTP; Mon, 27 May 2013 01:38:43 -0700 (PDT) In-Reply-To: <1369640291.98157.YahooMailNeo@web190706.mail.sg3.yahoo.com> References: <1362385827.52205.YahooMailNeo@web194702.mail.sg3.yahoo.com> <1362386275.5150.YahooMailNeo@web194704.mail.sg3.yahoo.com> <1362392680.58000.YahooMailNeo@web194702.mail.sg3.yahoo.com> <1362393994.40284.YahooMailNeo@web194706.mail.sg3.yahoo.com> <1362484093.62705.YahooMailNeo@web194701.mail.sg3.yahoo.com> <1362485373.52473.YahooMailNeo@web194706.mail.sg3.yahoo.com> <1362485890.70318.YahooMailNeo@web194701.mail.sg3.yahoo.com> <1362502593.11290.YahooMailNeo@web194706.mail.sg3.yahoo.com> <1362502925.61539.YahooMailNeo@web194702.mail.sg3.yahoo.com> <1362564010.31506.YahooMailNeo@web194703.mail.sg3.yahoo.com> <1362585758.24571.YahooMailNeo@web194706.mail.sg3.yahoo.com> <1362655630.59684.YahooMailNeo@web194705.mail.sg3.yahoo.com> <1362721390.11663.YahooMailNeo@web194705.mail.sg3.yahoo.com> <1362740013.16235.YahooMailNeo@web194704.mail.sg3.yahoo.com> <1362903547.16452.YahooMailNeo@web194703.mail.sg3.yahoo.com> <1362918367.33445.YahooMailNeo@web194705.mail.sg3.yahoo.com> <1947113031-1362942406-cardhu_decombobulator_blackberry.rim.net-166473606-@b1.c16.bise7.blackberry> <1362972313.67427.YahooMailNeo@web194705.mail.sg3.yahoo.com> <1369393187.8522.YahooMailNeo@web190703.mail.sg3.yahoo.com> <1369394495.72604.YahooMailNeo@web190706.mail.sg3.yahoo.com> <1369395022.77168.YahooMailNeo@web190706.mail.sg3.yahoo.com> <1369635749.89783.YahooMailNeo@web190701.mail.sg3.yahoo.com> <1369639292.34547.YahooMailNeo@web190702.mail.sg3.yahoo.com> <1369640291.98157.YahooMailNeo@web190706.mail.sg3.yahoo.com> Date: Mon, 27 May 2013 14:08:43 +0530 Message-ID: Subject: Re: Partitioning confusion From: Nitin Pawar To: user@hive.apache.org, Sai Sai Content-Type: multipart/alternative; boundary=e89a8f923b368861a304ddaf13d5 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f923b368861a304ddaf13d5 Content-Type: text/plain; charset=ISO-8859-1 when you specify the load data query with specific partition, it will put the entire data into that partition. On Mon, May 27, 2013 at 1:08 PM, Sai Sai wrote: > > After creating a partition for a country (USA) and state (IL) and when we > go to the the hdfs site to look at the partition in the browser we r seeing > all the records for all the countries and states rather than just for the > partition created for US and IL given below, is this correct behavior: > ******************** > Here is my commands: > ******************** > > CREATE TABLE employees (name STRING, salary FLOAT, subordinates > ARRAY, deductions MAP, address STRUCT city:STRING, state:STRING, zip:INT, country:STRING> ) PARTITIONED BY > (country STRING, state STRING); > > LOAD DATA LOCAL INPATH > '/home/satish/data/employees/input/employees-country.txt' INTO TABLE > employees PARTITION (country='USA',state='IL'); > > ******************** > Here is my original data file, where i have a few countries data such as > USA, INDIA, UK, AUS: > ******************** > > John Doe100000.0Mary SmithTodd JonesFederal Taxes.2State > Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600USA > Mary Smith80000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100 > Ontario St.ChicagoIL60601USA > Todd Jones70000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago > Ave.Oak ParkIL60700USA > Bill King60000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure > Dr.ObscuriaIL60100USA > Boss Man200000.0John DoeFred FinanceFederal Taxes.3State > Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500USA > Fred Finance150000.0Stacy AccountantFederal Taxes.3State > Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500USA > Stacy Accountant60000.0Federal Taxes.15State Taxes.03Insurance.1300 Main > St.NapervilleIL60563USA > John Doe 2100000.0Mary SmithTodd JonesFederal Taxes.2State > Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600INDIA > Mary Smith 280000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100 > Ontario St.ChicagoIL60601INDIA > Todd Jones 270000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago > Ave.Oak ParkIL60700AUSTRALIA > Bill King 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure > Dr.ObscuriaIL60100AUSTRALIA > Boss Man2 200000.0John DoeFred FinanceFederal Taxes.3State > Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500UK > Fred Finance 2150000.0Stacy AccountantFederal Taxes.3State > Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500UK > Stacy Accountant 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Main > St.NapervilleIL60563UK > ******************** > Now when i navigate to: > Contents of directory > /user/hive/warehouse/db1.db/employees/country=USA/state=IL > ******************** > I see all the records and was wondering if it should have only USA & IL > records. > Please help. > -- Nitin Pawar --e89a8f923b368861a304ddaf13d5 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
when you specify the load data query with specific partiti= on, it will put the entire data into that partition.=A0



On Mon, May = 27, 2013 at 1:08 PM, Sai Sai <saigraph@yahoo.in> wrote:

After creating a partition for a country (USA) and state (IL) and whe= n we go to the the hdfs site to look at the partition in the browser we r s= eeing =A0all the records for all the countries and states rather than just = for the partition created for US and IL given below, is this correct behavi= or:
****= ****************
Here is my commands:
********************


LOAD DATA LOCAL INPATH '/home/satish/data/employees/input/employe= es-country.txt' INTO TABLE employees PARTITION (country=3D'USA'= ,state=3D'IL');

********************
Her= e is my original data file, where i have a few countries data such as USA, = INDIA, UK, AUS:
********************

John Doe100000.0Ma= ry SmithTodd JonesFederal Taxes.2State Taxes.05Insurance.11 Michigan Ave.Ch= icagoIL60600USA
Mary Smith80000.0Bill KingFederal= Taxes.2State Taxes.05Insurance.1100 Ontario St.ChicagoIL60601USA
Todd Jones70000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago Ave.O= ak ParkIL60700USA
Bill King6= 0000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure Dr.ObscuriaIL601= 00USA
Boss Man200000.0John DoeFred Fina= nceFederal Taxes.3State Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60= 500USA
Fred Finance150000.0Stacy Account= antFederal Taxes.3State Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60= 500USA
Stacy Accountant60000.0Federal Taxes.15State Taxes.03Insurance.1300 Ma= in St.NapervilleIL60563USA
John Doe 2100000.0Mary SmithTodd JonesFederal Taxes.2State Taxes.05Insuranc= e.11 Michigan Ave.ChicagoIL60600INDIA
Mary Smith 280000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100 Ont= ario St.ChicagoIL60601INDIA
Todd Jones 270000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago Ave= .Oak ParkIL60700AUSTRALIA
Bill King 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure Dr.O= bscuriaIL60100AUSTRALIA
Boss= Man2 200000.0John DoeFred FinanceFederal Taxes.3State Taxes.07Insurance.05= 1 Pretentious Drive.ChicagoIL60500UK
Fred Finance 2150000.0Stacy Accou= ntantFederal Taxes.3State Taxes.07Insurance.052 Pretentious Drive.ChicagoIL= 60500UK
Stacy Accountant 260000.0Federal = Taxes.15State Taxes.03Insurance.1300 Main St.NapervilleIL60563UK
********************
Now= when i navigate to:
Contents of directory /user/hive/warehouse/d= b1.db/employees/country=3DUSA/state=3DIL
********************
I see all the records and was= wondering if it should have only USA & IL records.
Please he= lp.



--
Nitin Pawar
--e89a8f923b368861a304ddaf13d5--