spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloud-fan <>
Subject [GitHub] spark pull request #19148: [SPARK-21936][SQL][WIP] backward compatibility te...
Date Wed, 06 Sep 2017 16:51:53 GMT
GitHub user cloud-fan opened a pull request:

    [SPARK-21936][SQL][WIP] backward compatibility test framework for HiveExternalCatalog

    ## What changes were proposed in this pull request?
    `HiveExternalCatalog` is a semi-public interface. When creating tables, `HiveExternalCatalog`
converts the table metadata to hive table format and save into hive metastore. It's very import
to guarantee backward compatibility here, i.e., tables created by previous Spark versions
should still be readable in newer Spark versions.
    Previously we find backward compatibility issues manually, which is really easy to miss
bugs. This PR introduces a test framework to automatically test `HiveExternalCatalog` backward
compatibility, by downloading Spark binaries with different versions, and create tables with
these Spark versions, and read these tables with current Spark version.
    add more test cases
    make sure it works on jenkins
    ## How was this patch tested?
    test-only change

You can merge this pull request into a Git repository by running:

    $ git pull test

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19148
commit 283bc45d5b8ae63ba00d74c0322469c10b4f88eb
Author: Wenchen Fan <>
Date:   2017-09-06T05:39:53Z

    backward compatibility test framework for HiveExternalCatalog



To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message