carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aniketadnaik <...@git.apache.org>
Subject [GitHub] carbondata pull request #1064: [CARBONDATA-<1173>] Stream ingestion - write ...
Date Mon, 19 Jun 2017 23:05:51 GMT
GitHub user aniketadnaik opened a pull request:

    https://github.com/apache/carbondata/pull/1064

    [CARBONDATA-<1173>] Stream ingestion - write path framework 

    Description:
    This is a over all streaming write path framework implementation. It is mainly targeted
for "**streaming_ingest**" branch.  This framework implementation will help other developers
to leverage a common code path for developing other pieces of the streaming ingestion feature.

    - Whether new unit test cases have been added or why no new tests are required?
        - This is a just framework implementation, unit tests will be added with individual
functionality commits        - What manual testing you have done?
       - mvn clean verify 
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO] 
        [INFO] Apache CarbonData :: Parent ........................ SUCCESS [  1.533 s]
        [INFO] Apache CarbonData :: Common ........................ SUCCESS [  1.335 s]
        [INFO] Apache CarbonData :: Core .......................... SUCCESS [02:34 min]
        [INFO] Apache CarbonData :: Processing .................... SUCCESS [  5.158 s]
        [INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [  5.208 s]
        [INFO] Apache CarbonData :: Spark Common .................. SUCCESS [ 13.989 s]
        [INFO] Apache CarbonData :: Spark ......................... SUCCESS [02:33 min]
        [INFO] Apache CarbonData :: Spark Common Test ............. SUCCESS [05:15 min]
        [INFO] Apache CarbonData :: Assembly ...................... SUCCESS [  1.683 s]
        [INFO] Apache CarbonData :: Spark Examples ................ SUCCESS [  5.123 s]
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESS
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 10:58 min
        [INFO] Finished at: 2017-06-16T17:24:23-07:00
        [INFO] Final Memory: 85M/1078M
        [INFO] ------------------------------------------------------------------------
    
       - Made sure write path class invocation happens correctly with with spark structured
streaming (2.1)
       - Made sure write path execution work flow with structured streaming(2.1) using both
socket and file 
         sources
    
    - Any additional information to help reviewers in testing this change.
       - structured streaming now accepts carbondata as a file format in addition to data
source 
       - temporary functionality writes out data in text format instead of carbondata format,
will be replaced with new functionality additions.
     

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aniketadnaik/carbondataStreamIngest streamIngest-1173

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1064.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1064
    
----
commit 5403972898dc0d62471296476d6d5603dccbda10
Author: Aniket Adnaik <aniket.adnaik@gmail.com>
Date:   2017-06-15T18:57:43Z

    [CARBONDATA-1173] Streaming Ingest - write path framework implementation

commit 5270952a43b5e373957682dd75edd15048cb5aa3
Author: Aniket Adnaik <aniket.adnaik@gmail.com>
Date:   2017-06-15T19:11:48Z

    [CARBONDATA-1173] Streaming Ingest - write path framework implementation

commit ac3f1bbbee9c33472726afe10bac73a17a607cce
Author: Aniket Adnaik <aniket.adnaik@gmail.com>
Date:   2017-06-16T18:04:19Z

    [CARBONDATA-1173] Streaming Ingest - write path framework implementation

commit 4fc68b6c7a986ceda2c02713884d9293d29961fc
Author: Aniket Adnaik <aniket.adnaik@gmail.com>
Date:   2017-06-19T22:19:21Z

    [CARBONDATA-1173] Streaming Ingest - write path framework implementation

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message