hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Green <openkbi...@gmail.com>
Subject ListBucketing feature does not support uppercase string.
Date Thu, 05 May 2016 20:19:03 GMT
Hi Team,

I found when there is uppercase string as the skew value, ListBucketing is
not working.
https://issues.apache.org/jira/browse/HIVE-13697 is filed:

For example:
1. This is good:
CREATE TABLE testskew (id INT, a STRING)
SKEWED BY (a) ON ('abc', 'xyz') STORED AS DIRECTORIES;

set hive.mapred.supports.subdirectories=true;
set mapred.input.dir.recursive=true;

 INSERT OVERWRITE TABLE testskew
 SELECT 123,'abc' FROM dual
 union all
 SELECT 123,'xyz' FROM dual
 union all
 SELECT 123,'others' FROM dual;

# hadoop fs -ls /user/hive/warehouse/testskew
Found 3 items
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/a=abc
drwxrwxrwx   - mapr mapr          1 2016-05-05 14:56
/user/hive/warehouse/testskew/a=xyz

This is good, because both "abc" and "xyz" directories got created.

2. This is bad:
Drop table testskew2;
CREATE TABLE testskew2 (id INT, a STRING)
SKEWED BY (a) ON ('aus', 'US') STORED AS DIRECTORIES;

set hive.mapred.supports.subdirectories=true;
set mapred.input.dir.recursive=true;

 INSERT OVERWRITE TABLE testskew2
 SELECT 123, 'aus' FROM dual
 union all
 SELECT 123, 'US' FROM dual
 union all
 SELECT 123, 'others' FROM dual;

# hadoop fs -ls /user/hive/warehouse/testskew2
Found 2 items
drwxrwxrwx   - mapr mapr          1 2016-05-05 15:11
/user/hive/warehouse/testskew2/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
drwxrwxrwx   - mapr mapr          1 2016-05-05 15:11
/user/hive/warehouse/testskew2/a=aus

You can see, only "aus" directory got created...



-- 
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message