Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 17 Oct 2016 12:09:59 +0000 (UTC)
From: "Sean Owen (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12849097.1437836821000.3423.1476706199675@Atlassian.JIRA>
In-Reply-To: <JIRA.12849097.1437836821000@Atlassian.JIRA>
References: <JIRA.12849097.1437836821000@Atlassian.JIRA> <JIRA.12849097.1437836821558@arcas>
Subject: [jira] [Updated] (SPARK-9342) Spark SQL views don't work
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 17 Oct 2016 12:10:01 -0000


     [ https://issues.apache.org/jira/browse/SPARK-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-9342:
-----------------------------
    Assignee: Xiao Li

> Spark SQL views don't work
> --------------------------
>
>                 Key: SPARK-9342
>                 URL: https://issues.apache.org/jira/browse/SPARK-9342
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.1
>         Environment: Ubuntu on AWS
>            Reporter: Simeon Simeonov
>            Assignee: Xiao Li
>              Labels: sql, views
>             Fix For: 2.0.0
>
>
> The Spark SQL documentation's section on Hive support claims that views are supported. However, even basic view operations fail with exceptions related to column resolution. 
> For example,
> {code}
> // The test table has columns category & num
> ctx.sql("create view view1 as select * from test")
> ctx.table("view1").printSchema
> {code}
> generates
> {code}
> org.apache.spark.sql.AnalysisException: cannot resolve 'test.col' given input columns category, num; line 1 pos 7
> 	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
>         ...
> {code}
> You can see a standalone reproducible example with full spark-shell output demonstrating the problem at [https://gist.github.com/ssimeonov/57164f9d6b928ba0cfde]
> The problem is that {{ctx.sql("create view view1 as select * from test")}} puts the following in the metastore including {{cols:[FieldSchema(name:col, type:string, comment:null)]}} even though the {{test}} table has {{category}} and {{num}} columns:
> {code}
> 15/07/26 15:47:28 INFO HiveMetaStore: 0: create_table: Table(tableName:view1, dbName:default, owner:ubuntu, createTime:1437925648, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:null, parameters:{}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{}, viewOriginalText:select * from test, viewExpandedText:select `test`.`col` from `default`.`test`, tableType:VIRTUAL_VIEW)
> 15/07/26 15:47:28 INFO audit: ugi=ubuntu	ip=unknown-ip-addr	cmd=create_table: Table(tableName:view1, dbName:default, owner:ubuntu, createTime:1437925648, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:null, parameters:{}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{}, viewOriginalText:select * from test, viewExpandedText:select `test`.`col` from `default`.`test`, tableType:VIRTUAL_VIEW)
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org