spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mallman <>
Subject [GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...
Date Thu, 23 Aug 2018 17:40:21 GMT
Github user mallman commented on a diff in the pull request:
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
    @@ -202,11 +204,15 @@ private[parquet] class ParquetRowConverter(
       override def start(): Unit = {
         var i = 0
    -    while (i < currentRow.numFields) {
    +    while (i < fieldConverters.length) {
           i += 1
    +    while (i < currentRow.numFields) {
    --- End diff --
    These changes are related to my fix for the ignored unit test. If I apply my fix but keep
the master version of this file, 24 unit tests fail. If I apply my fix along with this file
diff then all tests pass, including the test that is currently ignored.
    I'm not sure I can develop a unit test for this current commit that should pass but will
fail without this file's changes. I haven't spent any time thinking about it, and I really
need to work on other things right now.
    If you want I will back out this change. However, I will re-incorporate it in a follow-on


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message