spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rdblue <...@git.apache.org>
Subject [GitHub] spark pull request #20521: [SPARK-22977][SQL] fix web UI SQL tab for CTAS
Date Tue, 06 Feb 2018 19:59:19 GMT
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20521#discussion_r166425787
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
    @@ -493,9 +510,23 @@ case class DataSource(
             dataSource.createRelation(
               sparkSession.sqlContext, mode, caseInsensitiveOptions, Dataset.ofRows(sparkSession,
data))
           case format: FileFormat =>
    -        sparkSession.sessionState.executePlan(planForWritingFileFormat(format, mode,
data)).toRdd
    +        val cmd = planForWritingFileFormat(format, mode, data)
    +        val resolvedPartCols = cmd.partitionColumns.map { col =>
    +          // The partition columns created in `planForWritingFileFormat` should always
be
    +          // `UnresolvedAttribute` with a single name part.
    +          assert(col.isInstanceOf[UnresolvedAttribute])
    +          val unresolved = col.asInstanceOf[UnresolvedAttribute]
    +          assert(unresolved.nameParts.length == 1)
    +          val name = unresolved.nameParts.head
    +          outputColumns.find(a => equality(a.name, name)).getOrElse {
    +            throw new AnalysisException(
    +              s"Unable to resolve $name given [${data.output.map(_.name).mkString(",
")}]")
    +          }
    +        }
    +        val resolved = cmd.copy(partitionColumns = resolvedPartCols, outputColumns =
outputColumns)
    --- End diff --
    
    Ok, I think I get it now. Thanks for explaining.
    
    Why not pass the `QueryExecution` so you have access to the resolved plan without copying
resolution rules here? I'm just curious here, I get that this is intended as a quick fix for
the release, so don't let my comments block you.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message