gobblin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (GOBBLIN-707) combine & standardize all gobblin scripts into one master script & restructure configs accordingly
Date Wed, 05 Jun 2019 22:18:00 GMT

     [ https://issues.apache.org/jira/browse/GOBBLIN-707?focusedWorklogId=254712&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-254712

ASF GitHub Bot logged work on GOBBLIN-707:

                Author: ASF GitHub Bot
            Created on: 05/Jun/19 22:17
            Start Date: 05/Jun/19 22:17
    Worklog Time Spent: 10m 
      Work Description: jhsenjaliya commented on pull request #2578: [GOBBLIN-707] rewrite
gobblin script to combine all modes and command
URL: https://github.com/apache/incubator-gobblin/pull/2578#discussion_r287553190

 File path: bin/gobblin-admin.sh
 @@ -1,142 +0,0 @@
 Review comment:
   I was targeting to cleanup and simplify those too many scripts Gobblin has, which could
be not only confusing for new users but also not a standard way of using it since all scripts
have different params. user should really move to new scripts to standardize, so not sure
how much the backward compatibility is required for scripts, as oppose to APIs, otherwise
we will never be able to clean up and provide better version. but I see ur point so may be
I will create wrapper as you suggested with note to remove those wrapper in future but i afraid
it wont be 100% backward compatible since loy of things (options and features) are getting
standardized here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 254712)
    Time Spent: 8.5h  (was: 8h 20m)

> combine & standardize all gobblin scripts into one master script & restructure
configs accordingly
> --------------------------------------------------------------------------------------------------
>                 Key: GOBBLIN-707
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-707
>             Project: Apache Gobblin
>          Issue Type: Improvement
>            Reporter: Jay Sen
>            Priority: Major
>          Time Spent: 8.5h
>  Remaining Estimate: 0h
> gobblin supports multiple modes of executions ( CLI, Standalone, cluster-master, cluster-worker,
AWS, YARN, MR ) and various command lines utility to run cli and admin commands. There is
a individual script for each of them.
> Having individual script introduces lot of issues
>  # all scripts handles gobblin variables, user parameters differently, and its highly
inconsistent among various different gobblin scripts
>  # functionality around start, stop, status checking and handling PID's among lot of
other things, varies vastly as per the implementation of the script.
>  # features like GC & JVM params, log4j file selection, classpath calculation, etc...
exists in some gobblin scripts but not all, adding to inconsistent user experience.
>  # maintaining total 13 script would be too much effort.
> Also all the gobblin scripts share lot of common code to handle params, start, stop services,
status checks, pid handling, etc... combining all the scripts into  1 not only makes maintenance
easier but also brings clarity and consistency.
> Solution:
> 1. there can be one gobblin.sh script to handle all gobblin commands and deployment options
as per following signature. NOTE: This
> {{gobblin.sh  <command> <params>}}
>  {{gobblin.sh  <execution-mode> <start|stop|status>}}
> {{commands values: admin, cli, statestore-check, statestore-clean, historystore-manager,
>  {{service values: standalone, cluster-master, cluster-worker, aws, yarn, mr, service}}
> with above change, following becomes valid command.
> {code:java}
> # all under GobblinCli class
> gobblin run listQuickApps  –> gobblin cli run listQuickApps
> gobblin run listQuickApps  –> gobblin cli run listQuickApps
> gobblin run <quick-app-name> -> gobblin cli run <quick-app-name>
> # class: JobStateToJsonConverter
> statestore-checker.sh <args> -> gobblin statestore-checker <args>
> # class: StateStoreCleaner
> statestore-clean.sh <args> -> gobblin statestore-clean <args>
> # class: DatabaseJobHistoryStoreSchemaManager
> historystore-manager.sh <args> -> gobblin historystore-manager <args>
> # class: Cli
> gobblin-admin.sh <args>   -> gobblin admin <args>
> # all gobblin deployment modes
> gobblin-cluster-master.sh   -> gobblin cluster-mater start|stop|status
> gobblin-cluster-worker.sh   -> gobblin cluster-mater start|stop|status
> gobblin-compaction.sh       -> gobblin cluster-mater start|stop|status
> gobblin-env.sh              -> gobblin cluster-mater start|stop|status
> gobblin-mapreduce.sh        -> gobblin cluster-mater start|stop|status
> gobblin-service.sh          -> gobblin cluster-mater start|stop|status
> gobblin-standalone.sh       -> gobblin cluster-mater start|stop|status
> gobblin-yarn.sh             -> gobblin cluster-mater start|stop|status
> {code}
> 2. Also configs needs to be structured and deduped accordingly to make it clear on which
config will be picked up for which execution mode.
>  {color:#ff0000}
>  NOTE: this refactoring adds all cli and service commands to gobblin.sh and hence changes
the syntax for all commands and services.{color}

This message was sent by Atlassian JIRA

View raw message