spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mallman <...@git.apache.org>
Subject [GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...
Date Thu, 23 Aug 2018 17:40:21 GMT
Github user mallman commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21320#discussion_r212396370
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
---
    @@ -202,11 +204,15 @@ private[parquet] class ParquetRowConverter(
     
       override def start(): Unit = {
         var i = 0
    -    while (i < currentRow.numFields) {
    +    while (i < fieldConverters.length) {
           fieldConverters(i).updater.start()
           currentRow.setNullAt(i)
           i += 1
         }
    +    while (i < currentRow.numFields) {
    --- End diff --
    
    These changes are related to my fix for the ignored unit test. If I apply my fix but keep
the master version of this file, 24 unit tests fail. If I apply my fix along with this file
diff then all tests pass, including the test that is currently ignored.
    
    I'm not sure I can develop a unit test for this current commit that should pass but will
fail without this file's changes. I haven't spent any time thinking about it, and I really
need to work on other things right now.
    
    If you want I will back out this change. However, I will re-incorporate it in a follow-on
PR.
    
    Thanks.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message