spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiao Li <gatorsm...@gmail.com>
Subject Re: A question about creating persistent table when in-memory catalog is used
Date Sun, 22 Jan 2017 19:21:49 GMT
Agree. : )

2017-01-22 11:20 GMT-08:00 Reynold Xin <rxin@databricks.com>:

> To be clear there are two separate "hive" we are talking about here. One
> is the catalog, and the other is the Hive serde and UDF support. We want to
> get to a point that the choice of catalog does not impact the functionality
> in Spark other than where the catalog is stored.
>
>
> On Sun, Jan 22, 2017 at 11:18 AM Xiao Li <gatorsmile@gmail.com> wrote:
>
>> We have a pending PR to block users to create the Hive serde table when
>> using InMemroyCatalog. See: https://github.com/apache/spark/pull/16587 I
>> believe it answers your question.
>>
>> BTW, we still can create the regular data source tables and insert the
>> data into the tables. The major difference is whether the metadata is
>> persistently stored or not.
>>
>> Thanks,
>>
>> Xiao Li
>>
>> 2017-01-22 11:14 GMT-08:00 Reynold Xin <rxin@databricks.com>:
>>
>> I think this is something we are going to change to completely decouple
>> the Hive support and catalog.
>>
>>
>> On Sun, Jan 22, 2017 at 4:51 AM Shuai Lin <linshuai2012@gmail.com> wrote:
>>
>> Hi all,
>>
>> Currently when the in-memory catalog is used, e.g. through `--conf
>> spark.sql.catalogImplementation=in-memory`, we can create a persistent
>> table, but inserting into this table would fail with error message "Hive
>> support is required to insert into the following tables..".
>>
>>     sql("create table t1 (id int, name string, dept string)") // OK
>>     sql("insert into t1 values (1, 'name1', 'dept1')")  // ERROR
>>
>>
>> This doesn't make sense for me, because this table would always be empty
>> if we can't insert into it, thus would be of no use. But I wonder if there
>> are other good reasons for the current logic. If not, I would propose to
>> raise an error when creating the table in the first place.
>>
>> Thanks!
>>
>> Regards,
>> Shuai Lin (@lins05)
>>
>>
>>
>>
>>
>>
>>
>>

Mime
View raw message