From dev-return-4378-archive-asf-public=cust-asf.ponee.io@gobblin.incubator.apache.org Wed Jun 5 22:18:06 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 03EF918065D for ; Thu, 6 Jun 2019 00:18:05 +0200 (CEST) Received: (qmail 25800 invoked by uid 500); 5 Jun 2019 22:18:05 -0000 Mailing-List: contact dev-help@gobblin.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@gobblin.incubator.apache.org Delivered-To: mailing list dev@gobblin.incubator.apache.org Received: (qmail 25776 invoked by uid 99); 5 Jun 2019 22:18:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jun 2019 22:18:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id C75831A4699 for ; Wed, 5 Jun 2019 22:18:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -112.199 X-Spam-Level: X-Spam-Status: No, score=-112.199 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id NGbdw-5981QS for ; Wed, 5 Jun 2019 22:18:03 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id BACF75F17E for ; Wed, 5 Jun 2019 22:18:02 +0000 (UTC) Received: (qmail 25765 invoked by uid 99); 5 Jun 2019 22:18:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jun 2019 22:18:02 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 71547E2831 for ; Wed, 5 Jun 2019 22:18:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id CF06D24599 for ; Wed, 5 Jun 2019 22:18:00 +0000 (UTC) Date: Wed, 5 Jun 2019 22:18:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: dev@gobblin.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Work logged] (GOBBLIN-707) combine & standardize all gobblin scripts into one master script & restructure configs accordingly MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/GOBBLIN-707?focusedWorklogId= =3D254712&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:worklog-ta= bpanel#worklog-254712 ] ASF GitHub Bot logged work on GOBBLIN-707: ------------------------------------------ Author: ASF GitHub Bot Created on: 05/Jun/19 22:17 Start Date: 05/Jun/19 22:17 Worklog Time Spent: 10m=20 Work Description: jhsenjaliya commented on pull request #2578: [GOBBL= IN-707] rewrite gobblin script to combine all modes and command URL: https://github.com/apache/incubator-gobblin/pull/2578#discussion_r2875= 53190 =20 =20 ########## File path: bin/gobblin-admin.sh ########## @@ -1,142 +0,0 @@ -#!/bin/bash =20 Review comment: I was targeting to cleanup and simplify those too many scripts Gobblin h= as, which could be not only confusing for new users but also not a standard= way of using it since all scripts have different params. user should reall= y move to new scripts to standardize, so not sure how much the backward com= patibility is required for scripts, as oppose to APIs, otherwise we will ne= ver be able to clean up and provide better version. but I see ur point so m= ay be I will create wrapper as you suggested with note to remove those wrap= per in future but i afraid it wont be 100% backward compatible since loy of= things (options and features) are getting standardized here. =20 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. =20 For queries about this service, please contact Infrastructure at: users@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 254712) Time Spent: 8.5h (was: 8h 20m) > combine & standardize all gobblin scripts into one master script & restru= cture configs accordingly > -------------------------------------------------------------------------= ------------------------- > > Key: GOBBLIN-707 > URL: https://issues.apache.org/jira/browse/GOBBLIN-707 > Project: Apache Gobblin > Issue Type: Improvement > Reporter: Jay Sen > Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > gobblin supports multiple modes of executions ( CLI, Standalone, cluster-= master, cluster-worker, AWS, YARN, MR ) and various command lines utility t= o run cli and admin commands. There is a individual script for each of them= . > Having individual script introduces lot of issues > # all scripts handles gobblin variables, user parameters differently, an= d its highly inconsistent among various different gobblin scripts > # functionality around start, stop, status checking and handling PID's a= mong lot of other things, varies vastly as per the implementation of the sc= ript. > # features like GC & JVM params, log4j file selection,=C2=A0classpath ca= lculation, etc... exists in some gobblin scripts but not all, adding to inc= onsistent user experience. > # maintaining total 13 script would be too much effort. > Also all the gobblin scripts share lot of common code to handle params, s= tart, stop services, status checks, pid handling, etc... combining all the = scripts into=C2=A0 1 not only makes maintenance easier but also brings clar= ity and consistency. > =C2=A0 > Solution: > 1. there can be one gobblin.sh script to handle all gobblin commands and = deployment options as per following signature. NOTE: This > {{gobblin.sh=C2=A0 }} > {{gobblin.sh=C2=A0 }} > {{commands values: admin, cli, statestore-check, statestore-clean, histor= ystore-manager, classpath}} > {{service values: standalone, cluster-master, cluster-worker, aws, yarn,= mr, service}} > with above change, following becomes valid command. > {code:java} > # all under GobblinCli class > gobblin run listQuickApps=C2=A0 =E2=80=93> gobblin cli run listQuickApps > gobblin run listQuickApps=C2=A0 =E2=80=93> gobblin cli run listQuickApps > gobblin run -> gobblin cli run > # class: JobStateToJsonConverter > statestore-checker.sh -> gobblin statestore-checker > # class: StateStoreCleaner > statestore-clean.sh -> gobblin statestore-clean > # class: DatabaseJobHistoryStoreSchemaManager > historystore-manager.sh -> gobblin historystore-manager > # class: Cli > gobblin-admin.sh -> gobblin admin > # all gobblin deployment modes > gobblin-cluster-master.sh -> gobblin cluster-mater start|stop|status > gobblin-cluster-worker.sh -> gobblin cluster-mater start|stop|status > gobblin-compaction.sh -> gobblin cluster-mater start|stop|status > gobblin-env.sh -> gobblin cluster-mater start|stop|status > gobblin-mapreduce.sh -> gobblin cluster-mater start|stop|status > gobblin-service.sh -> gobblin cluster-mater start|stop|status > gobblin-standalone.sh -> gobblin cluster-mater start|stop|status > gobblin-yarn.sh -> gobblin cluster-mater start|stop|status > {code} > =C2=A0 > 2. Also configs needs to be structured and deduped accordingly to make it= clear on which config will be picked up for which execution mode. > =C2=A0{color:#ff0000} > NOTE: this refactoring=C2=A0adds all cli and service commands to=C2=A0go= bblin.sh and hence changes the syntax for all commands and services.{color} -- This message was sent by Atlassian JIRA (v7.6.3#76005)