spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Li (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-9342) Spark SQL views don't work
Date Sat, 08 Oct 2016 04:38:20 GMT

     [ https://issues.apache.org/jira/browse/SPARK-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xiao Li resolved SPARK-9342.
----------------------------
    Resolution: Fixed

> Spark SQL views don't work
> --------------------------
>
>                 Key: SPARK-9342
>                 URL: https://issues.apache.org/jira/browse/SPARK-9342
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.1
>         Environment: Ubuntu on AWS
>            Reporter: Simeon Simeonov
>              Labels: sql, views
>
> The Spark SQL documentation's section on Hive support claims that views are supported.
However, even basic view operations fail with exceptions related to column resolution. 
> For example,
> {code}
> // The test table has columns category & num
> ctx.sql("create view view1 as select * from test")
> ctx.table("view1").printSchema
> {code}
> generates
> {code}
> org.apache.spark.sql.AnalysisException: cannot resolve 'test.col' given input columns
category, num; line 1 pos 7
> 	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
>         ...
> {code}
> You can see a standalone reproducible example with full spark-shell output demonstrating
the problem at [https://gist.github.com/ssimeonov/57164f9d6b928ba0cfde]
> The problem is that {{ctx.sql("create view view1 as select * from test")}} puts the following
in the metastore including {{cols:[FieldSchema(name:col, type:string, comment:null)]}} even
though the {{test}} table has {{category}} and {{num}} columns:
> {code}
> 15/07/26 15:47:28 INFO HiveMetaStore: 0: create_table: Table(tableName:view1, dbName:default,
owner:ubuntu, createTime:1437925648, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col,
type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, compressed:false,
numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:null, parameters:{}), bucketCols:[],
sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})),
partitionKeys:[], parameters:{}, viewOriginalText:select * from test, viewExpandedText:select
`test`.`col` from `default`.`test`, tableType:VIRTUAL_VIEW)
> 15/07/26 15:47:28 INFO audit: ugi=ubuntu	ip=unknown-ip-addr	cmd=create_table: Table(tableName:view1,
dbName:default, owner:ubuntu, createTime:1437925648, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col,
type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, compressed:false,
numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:null, parameters:{}), bucketCols:[],
sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})),
partitionKeys:[], parameters:{}, viewOriginalText:select * from test, viewExpandedText:select
`test`.`col` from `default`.`test`, tableType:VIRTUAL_VIEW)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message