Mailing-List: contact reviews-help@bahir.apache.org; run by ezmlm
Precedence: bulk
Reply-To: reviews@bahir.apache.org
From: emlaver <git@git.apache.org>
To: reviews@bahir.apache.org
Reply-To: reviews@bahir.apache.org
Message-ID: <git-pr-45-bahir@git.apache.org>
Subject: [GitHub] bahir pull request #45: [BAHIR-110] Implement _changes API for non-streaming...
Content-Type: text/plain
Date: Mon, 19 Jun 2017 18:30:53 +0000 (UTC)
archived-at: Mon, 19 Jun 2017 18:30:57 -0000

GitHub user emlaver opened a pull request:

    https://github.com/apache/bahir/pull/45

    [BAHIR-110] Implement _changes API for non-streaming receiver

    See [JIRA-110](https://issues.apache.org/jira/browse/BAHIR-110)
    
    _What_
    
    Add support for _changes API for non-streaming (data frames and SQL temp. views) receiver.
    
    _How_
    - New CloudantConfig option `apiReceiver` for selecting _all_docs and _changes endpoint in Cloudant to Spark data frames and SQL temp tables
    - Default is `_all_docs` endpoint for non-streaming receiver
    - Base abstract config class that's extended by an all_docs class and _changes class
    - JsonStoreConfigManager includes new 'cloudant.apiReceiver' config option for selecting _all_docs and _changes endpoint in Cloudant to Spark data frames and SQL temp tables
    - Updated README with details for 'cloudant.apiReceiver' option
    
    _Testing_
    
    - Added base class ClientSparkFunSuite for setting up, creating, and loading sample data from flat files to test databases.
    - CloudantAllDocsDFSuite to test Spark data frames using the _all_docs endpoint.
    - CloudantChangesDFSuite to test Spark data frames using the _changes endpoint.
    - CloudantOptionSuite to verify Cloudant config options.
    - CloudantSparkSQLSuite to test Spark SQL temp views.
    
    Note: 27,378 lines added for the JSON files used in the testing suite.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/emlaver/bahir 110-implement-changes-api-in-receiver

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/bahir/pull/45.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #45
    
----
commit 065752978e7d826cd311df4c517044183db0c372
Author: Esteban Laver <emlaver@us.ibm.com>
Date:   2017-06-16T18:17:30Z

    Excluded scala flat files for testing from build

commit 751a5c7b876eca822e3047a609165505b5eadc4c
Author: Esteban Laver <emlaver@us.ibm.com>
Date:   2017-06-16T18:23:39Z

    Added MapReduce example, removed unused imports, and replaced SQL TEMP TABLE with TEMP VIEW

commit 39be19029a5b02820749ea7a5e19f345a4d74de0
Author: Esteban Laver <emlaver@us.ibm.com>
Date:   2017-06-19T15:22:41Z

    New CloudantConfig option `apiReceiver` for selecting _all_docs and _changes endpoint in Cloudant to Spark data frames and SQL temp tables
    - Default is `_all_docs` endpoint for non-streaming receiver
    - Base abstract config class that's extended by an all_docs class and _changes class
    - CloudantException thrown when required Spark Cloudant config option is empty or invalid
    - Updated scala style

commit b662611722d118800b1135ab69f02a979ebedb3c
Author: Esteban Laver <emlaver@us.ibm.com>
Date:   2017-06-19T15:23:28Z

    JsonStoreConfigManager: new 'cloudant.apiReceiver' config option for selecting _all_docs and _changes endpoint in Cloudant to Spark data frames and SQL temp tables
    - Throw CloudantException when spark config value is invalid or empty
    JsonStoreDataAccess: Added selector for use with _changes API and to filter out design docs
    JsonStoreRDD: Partition set to 1 for _changes API
    Updated Scala style in common classes:
    - Fixed ordering of imports
    - Added type notation
    - Removed redundant parenthesis

commit 4c8fc6bff81df034e789d2e782db11fca6e7cd84
Author: Esteban Laver <emlaver@us.ibm.com>
Date:   2017-06-19T15:25:28Z

    JSON files and logging properties for testing suite

commit a798f4cd1ef63f10a2f261f3c4460ef018d8d95d
Author: Esteban Laver <emlaver@us.ibm.com>
Date:   2017-06-19T15:28:02Z

    Testing suite:
    ClientSparkFunSuite for setting up, creating, and loading sample data from flat files to test databases.
    CloudantAllDocsDFSuite to test Spark data frames using the _all_docs endpoint.
    CloudantChangesDFSuite to test Spark data frames using the _changes endpoint.
    CloudantOptionSuite to verify Cloudant config options.
    CloudantSparkSQLSuite to test Spark SQL temp views.
    
    - Version 2.6.7 for jackson dependencies resolves "Incompatible Jackson version" during build
    - Cloudant set-up and database creation using cloudant-client library

commit c6ecb836ef0eb714398360b21ab95f1c18b762e1
Author: Esteban Laver <emlaver@us.ibm.com>
Date:   2017-06-19T15:28:23Z

    Updated README
    - New option 'cloudant.apiReceiver' for selecting _all_docs or _changes endpoint
    - Fixed links to source code files

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---