Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 7 Jan 2017 23:58:58 +0000 (UTC)
From: "Xiao Li (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13032829.1483821088000.678127.1483833538338@Atlassian.JIRA>
In-Reply-To: <JIRA.13032829.1483821088000@Atlassian.JIRA>
References: <JIRA.13032829.1483821088000@Atlassian.JIRA> <JIRA.13032829.1483821088184@arcas>
Subject: [jira] [Updated] (SPARK-19120) Returned an Empty Result after
 Loading a Hive Table
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 07 Jan 2017 23:59:00 -0000


     [ https://issues.apache.org/jira/browse/SPARK-19120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Li updated SPARK-19120:
----------------------------
    Description: 
{noformat}
        sql(
          """
            |CREATE TABLE test (a STRING)
            |STORED AS PARQUET
          """.stripMargin)

       spark.table("test").show()

        sql(
          s"""
             |LOAD DATA LOCAL INPATH '$newPartitionDir' OVERWRITE
             |INTO TABLE test
           """.stripMargin)

        spark.table("test").show()
{noformat}

The returned result is empty after table loading. We should refresh the metadata cache after loading the data to the table. 

  was:
{noformat}
        sql(
          """
            |CREATE TABLE test_added_partitions (a STRING)
            |PARTITIONED BY (b INT)
            |STORED AS PARQUET
          """.stripMargin)

        // Create partition without data files and check whether it can be read
        sql(s"ALTER TABLE test_added_partitions ADD PARTITION (b='1')")
        spark.table("test_added_partitions").show()

        sql(
          s"""
             |LOAD DATA LOCAL INPATH '$newPartitionDir' OVERWRITE
             |INTO TABLE test_added_partitions PARTITION(b='1')
           """.stripMargin)

        spark.table("test_added_partitions").show()
{noformat}

The returned result is empty after table loading. We should refresh the metadata cache after loading the data to the table. 

This only happens if users manually add the partition by {{ALTER TABLE ADD PARTITION}}


> Returned an Empty Result after Loading a Hive Table
> ---------------------------------------------------
>
>                 Key: SPARK-19120
>                 URL: https://issues.apache.org/jira/browse/SPARK-19120
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.1.0
>            Reporter: Xiao Li
>            Assignee: Xiao Li
>            Priority: Critical
>              Labels: correctness
>
> {noformat}
>         sql(
>           """
>             |CREATE TABLE test (a STRING)
>             |STORED AS PARQUET
>           """.stripMargin)
>        spark.table("test").show()
>         sql(
>           s"""
>              |LOAD DATA LOCAL INPATH '$newPartitionDir' OVERWRITE
>              |INTO TABLE test
>            """.stripMargin)
>         spark.table("test").show()
> {noformat}
> The returned result is empty after table loading. We should refresh the metadata cache after loading the data to the table. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org