Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
MIME-Version: 1.0
Date: Thu, 5 May 2016 13:19:03 -0700
Message-ID: <CAJAqy=U1JrmSoSG8dSLc3XN33qjJFwgYLPn6-CndGWtF0aOVnQ@mail.gmail.com>
Subject: ListBucketing feature does not support uppercase string.
From: Jim Green <openkbinfo@gmail.com>
To: dev@hive.apache.org, user@hive.apache.org
Content-Type: multipart/alternative; boundary=001a113ac280b304af05321e0df5
archived-at: Thu, 05 May 2016 20:19:10 -0000

--001a113ac280b304af05321e0df5
Content-Type: text/plain; charset=UTF-8

Hi Team,

I found when there is uppercase string as the skew value, ListBucketing is
not working.
https://issues.apache.org/jira/browse/HIVE-13697 is filed:

For example:
1. This is good:
CREATE TABLE testskew (id INT, a STRING)
SKEWED BY (a) ON ('abc', 'xyz') STORED AS DIRECTORIES;

set hive.mapred.supports.subdirectories=true;
set mapred.input.dir.recursive=true;

 INSERT OVERWRITE TABLE testskew
 SELECT 123,'abc' FROM dual
 union all
 SELECT 123,'xyz' FROM dual
 union all
 SELECT 123,'others' FROM dual;

# hadoop fs -ls /user/hive/warehouse/testskew
Found 3 items
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/a=abc
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/a=xyz

This is good, because both "abc" and "xyz" directories got created.

2. This is bad:
Drop table testskew2;
CREATE TABLE testskew2 (id INT, a STRING)
SKEWED BY (a) ON ('aus', 'US') STORED AS DIRECTORIES;

set hive.mapred.supports.subdirectories=true;
set mapred.input.dir.recursive=true;

 INSERT OVERWRITE TABLE testskew2
 SELECT 123, 'aus' FROM dual
 union all
 SELECT 123, 'US' FROM dual
 union all
 SELECT 123, 'others' FROM dual;

# hadoop fs -ls /user/hive/warehouse/testskew2
Found 2 items
drwxrwxrwx   - mapr mapr          1 2016-05-05 15:11
/user/hive/warehouse/testskew2/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
drwxrwxrwx   - mapr mapr          1 2016-05-05 15:11
/user/hive/warehouse/testskew2/a=aus

You can see, only "aus" directory got created...


-- 
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)

--001a113ac280b304af05321e0df5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Team,<div><br></div><div>I found when there is uppercas=
e string as the skew value,=C2=A0ListBucketing is not working.</div><div><a=
 href=3D"https://issues.apache.org/jira/browse/HIVE-13697">https://issues.a=
pache.org/jira/browse/HIVE-13697</a> is filed:<br></div><div><br></div><div=
><pre class=3D"" id=3D"comment_text_1" style=3D"font-size:medium;white-spac=
e:pre-wrap;width:50em;color:rgb(0,0,0);line-height:normal">For example:
1. This is good:
CREATE TABLE testskew (id INT, a STRING)
SKEWED BY (a) ON (&#39;abc&#39;, &#39;xyz&#39;) STORED AS DIRECTORIES;

set hive.mapred.supports.subdirectories=3Dtrue;
set mapred.input.dir.recursive=3Dtrue;

 INSERT OVERWRITE TABLE testskew=20
 SELECT 123,&#39;abc&#39; FROM dual
 union all
 SELECT 123,&#39;xyz&#39; FROM dual
 union all
 SELECT 123,&#39;others&#39; FROM dual;

# hadoop fs -ls /user/hive/warehouse/testskew
Found 3 items
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/a=3Dabc
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/a=3Dxyz

This is good, because both &quot;abc&quot; and &quot;xyz&quot; directories =
got created.

2. This is bad:
Drop table testskew2;
CREATE TABLE testskew2 (id INT, a STRING)
SKEWED BY (a) ON (&#39;aus&#39;, &#39;US&#39;) STORED AS DIRECTORIES;

set hive.mapred.supports.subdirectories=3Dtrue;
set mapred.input.dir.recursive=3Dtrue;

 INSERT OVERWRITE TABLE testskew2=20
 SELECT 123, &#39;aus&#39; FROM dual
 union all
 SELECT 123, &#39;US&#39; FROM dual
 union all
 SELECT 123, &#39;others&#39; FROM dual;

# hadoop fs -ls /user/hive/warehouse/testskew2
Found 2 items
drwxrwxrwx   - mapr mapr          1 2016-05-05 15:11
/user/hive/warehouse/testskew2/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
drwxrwxrwx   - mapr mapr          1 2016-05-05 15:11
/user/hive/warehouse/testskew2/a=3Daus

You can see, only &quot;aus&quot; directory got created...</pre></div><div>=
<br></div><div><br>-- <br><div class=3D"gmail_signature"><div dir=3D"ltr">T=
hanks,<div><a href=3D"http://www.openkb.info" target=3D"_blank">www.openkb.=
info</a>=C2=A0</div><div>(Open KnowledgeBase for Hadoop/Database/OS/Network=
/Tool)</div></div></div>
</div></div>

--001a113ac280b304af05321e0df5--