spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Supreeth Sharma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24400) Issue with spark while accessing managed table with partitions across multiple namespaces - HDFS Federation
Date Mon, 28 May 2018 06:00:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16492291#comment-16492291
] 

Supreeth Sharma commented on SPARK-24400:
-----------------------------------------

cc [~hyukjin.kwon]

> Issue with spark while accessing managed table with partitions across multiple namespaces
- HDFS Federation
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-24400
>                 URL: https://issues.apache.org/jira/browse/SPARK-24400
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.3.0
>            Reporter: Supreeth Sharma
>            Priority: Critical
>         Attachments: federation_managed_table.py
>
>
> Facing Issue with spark while accessing managed table with partitions across multiple
namespaces
> Test steps :
> 1) Create HDFS federated cluster with two namespaces.
> 2) Create a managed table whose location is in default namespace (CREATE TABLE test_managed_tbl
(id int, name string, dept string) PARTITIONED BY (year int))
> 3) Insert a row into table and check that action is going through fine.
> 4) Try to alter the table and set the new location which is in Namespace2. (ALTER TABLE
test_managed_tbl SET LOCATION 'hdfs://ns2/apps/hive/warehouse/test_managed_tbl')
> 5) Try to insert new value into the table (INSERT INTO test_managed_tbl PARTITION (year=2017)
VALUES (9,'Harris','CSE'))
> This action is failing with below error :
> {code:java}
> 18/05/23 02:50:59 INFO FileUtils: Creating directory if it doesn't exist: hdfs://ns2/apps/hive/warehouse/test_managed_tbl/year=2017
> Traceback (most recent call last):
>   File "/tmp/federation_managed.py", line 17, in <module>
>     spark.sql("INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES (9,'Harris','CSE')")
>   File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/session.py",
line 714, in sql
>   File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py",
line 1257, in __call__
>   File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/utils.py",
line 69, in deco
> pyspark.sql.utils.AnalysisException: u'java.lang.IllegalArgumentException: Wrong FS:
hdfs://ns2/apps/hive/warehouse/test_managed_tbl/.hive-staging_hive_2018-05-23_02-50-56_484_3662347267719413000-1/-ext-10000/part-00000-5ee3003b-d41f-41d8-adaa-8937919f896d-c000,
expected: hdfs://ns1;'
> 18/05/23 02:50:59 INFO SparkContext: Invoking stop() from shutdown hook 
> {code}
> Spark-submit command :
> {code:java}
> spark-submit --master yarn-client --conf spark.sql.catalogImplementation=hive /tmp/federation_managed_table.py
> {code}
> Attaching federation_managed_table.py .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message