spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajkumar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-19983) Getting ValidationFailureSemanticException on 'INSERT OVEWRITE'
Date Mon, 20 Mar 2017 18:52:41 GMT

     [ https://issues.apache.org/jira/browse/SPARK-19983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rajkumar updated SPARK-19983:
-----------------------------
    Description: 
Hi, I am creating a DataFrame and registering that DataFrame as temp table using df.createOrReplaceTempView('mytable').
After that I try to write the content from 'mytable' into Hive table(It has partition) using
the following query

insert overwrite table
   myhivedb.myhivetable
partition(testdate) // ( 1) : Note here : I have a partition named 'testdate'
select
  Field1, 
  Field2,
  ...
  TestDate //(2) : Note here : I have a field named 'TestDate' ; Both (1) & (2) have the
same name
from
  mytable

when I execute this query, I am getting the following error

Exception in thread "main" org.apache.hadoop.hive.ql.metadata.Table$ValidationFailureSemanticException:
Partition spec {testdate=, TestDate=2013-01-01}

Looks like I am getting this error because of the same field names ; ie testdate(the partition
in Hive) & TestDate (The field in temp table 'mytable')

Whereas if my partition name testdate is different from the fieldname(ie TestDate), the query
executes successuflly. Example...

insert overwrite table
  myhivedb.myhivetable
partition(my_partition)  //Note here the partition name is not 'testdate'
select
  Field1, 
  Field2,
  ...
  TestDate 
from
  mytable

  was:
Hi, I am creating a DataFrame and registering that DataFrame as temp table using df.createOrReplaceTempView('mytable').
After that I try to write the content from 'mytable' into Hive table(It has partition) using
the following query

insert overwrite table
   myhivedb.myhivetable
partition(testdate) // ( 1) : Note here : I have a partition named 'testdate'
select
  Field1, 
  Field2,
  ...
  TestDate //(2) : Note here : I have a field named 'TestDate' ; Both (1) & (2) have the
same name
from
  mytable

when I execute this query, I am getting the following error

Exception in thread "main" org.apache.hadoop.hive.ql.metadata.Table$ValidationFailureSemanticException:
Partition spec {testdate=, TestDate=2013-01-01}

Looks like I am getting this error because of the same field names ; ie testdate(the partition
in Hive) & TestDate (The field in temp table 'mytable')

Whereas if my fieldname 'TestDate' is different, the query executes successuflly. Example...

insert overwrite table
   myhivedb.myhivetable
partition(testdate) 
select
  Field1, 
  Field2,
  ...
  myDate //Note here : The field name is 'myDate' & not 'TestDate'
from
  mytable




> Getting ValidationFailureSemanticException on 'INSERT OVEWRITE'
> ---------------------------------------------------------------
>
>                 Key: SPARK-19983
>                 URL: https://issues.apache.org/jira/browse/SPARK-19983
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.1.0
>            Reporter: Rajkumar
>            Priority: Blocker
>              Labels: sparkSQL
>
> Hi, I am creating a DataFrame and registering that DataFrame as temp table using df.createOrReplaceTempView('mytable').
After that I try to write the content from 'mytable' into Hive table(It has partition) using
the following query
> insert overwrite table
>    myhivedb.myhivetable
> partition(testdate) // ( 1) : Note here : I have a partition named 'testdate'
> select
>   Field1, 
>   Field2,
>   ...
>   TestDate //(2) : Note here : I have a field named 'TestDate' ; Both (1) & (2) have
the same name
> from
>   mytable
> when I execute this query, I am getting the following error
> Exception in thread "main" org.apache.hadoop.hive.ql.metadata.Table$ValidationFailureSemanticException:
Partition spec {testdate=, TestDate=2013-01-01}
> Looks like I am getting this error because of the same field names ; ie testdate(the
partition in Hive) & TestDate (The field in temp table 'mytable')
> Whereas if my partition name testdate is different from the fieldname(ie TestDate), the
query executes successuflly. Example...
> insert overwrite table
>   myhivedb.myhivetable
> partition(my_partition)  //Note here the partition name is not 'testdate'
> select
>   Field1, 
>   Field2,
>   ...
>   TestDate 
> from
>   mytable



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message