spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Song Jun (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-19832) DynamicPartitionWriteTask should escape the partition name
Date Mon, 06 Mar 2017 06:53:32 GMT
Song Jun created SPARK-19832:
--------------------------------

             Summary: DynamicPartitionWriteTask should escape the partition name 
                 Key: SPARK-19832
                 URL: https://issues.apache.org/jira/browse/SPARK-19832
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Song Jun


Currently in DynamicPartitionWriteTask, when we get the paritionPath of a parition, we just
escape the partition value, not escape the partition name.

this will cause some problems for some  special partition name situation, for example :
1) if the partition name contains '%' etc,  there will be two partition path created in the
filesytem, one is for escaped path like '/path/a%25b=1', another is for unescaped path like
'/path/a%b=1'.
and the data inserted stored in unescaped path, while the show partitions table will return
'a%25b=1' which the partition name is escaped. So here it is not consist. And I think the
data should be stored in the escaped path in filesystem, which Hive2.0.0 also have the same
action.

2) if the partition name contains ':', there will throw exception that new Path("/path","a:b"),
this is illegal which has a colon in the relative path.

{code}
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
URI: a:b
  at org.apache.hadoop.fs.Path.initialize(Path.java:205)
  at org.apache.hadoop.fs.Path.<init>(Path.java:171)
  at org.apache.hadoop.fs.Path.<init>(Path.java:88)
  ... 48 elided
Caused by: java.net.URISyntaxException: Relative path in absolute URI: a:b
  at java.net.URI.checkPath(URI.java:1823)
  at java.net.URI.<init>(URI.java:745)
  at org.apache.hadoop.fs.Path.initialize(Path.java:202)
  ... 50 more
{code}






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message