spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloud-fan <...@git.apache.org>
Subject [GitHub] spark pull request #20521: [SPARK-22977][SQL] fix web UI SQL tab for CTAS
Date Wed, 07 Feb 2018 04:38:59 GMT
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20521#discussion_r166517178
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
    @@ -493,9 +510,23 @@ case class DataSource(
             dataSource.createRelation(
               sparkSession.sqlContext, mode, caseInsensitiveOptions, Dataset.ofRows(sparkSession,
data))
           case format: FileFormat =>
    -        sparkSession.sessionState.executePlan(planForWritingFileFormat(format, mode,
data)).toRdd
    +        val cmd = planForWritingFileFormat(format, mode, data)
    +        val resolvedPartCols = cmd.partitionColumns.map { col =>
    +          // The partition columns created in `planForWritingFileFormat` should always
be
    +          // `UnresolvedAttribute` with a single name part.
    +          assert(col.isInstanceOf[UnresolvedAttribute])
    +          val unresolved = col.asInstanceOf[UnresolvedAttribute]
    +          assert(unresolved.nameParts.length == 1)
    +          val name = unresolved.nameParts.head
    +          outputColumns.find(a => equality(a.name, name)).getOrElse {
    +            throw new AnalysisException(
    +              s"Unable to resolve $name given [${data.output.map(_.name).mkString(",
")}]")
    +          }
    +        }
    +        val resolved = cmd.copy(partitionColumns = resolvedPartCols, outputColumns =
outputColumns)
    --- End diff --
    
    Because the CTAS node itself doesn't have a `QueryExecution`. It only has the physical
plan that was produced by the planner. `QueryExecution` only exists in the places that drive
the  analze/.../exeucte, e.g. `Dataset`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message