From commits-return-13152-archive-asf-public=cust-asf.ponee.io@dolphinscheduler.apache.org Tue Jul 14 00:46:07 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0721F18037A for ; Tue, 14 Jul 2020 02:46:06 +0200 (CEST) Received: (qmail 72270 invoked by uid 500); 14 Jul 2020 00:46:06 -0000 Mailing-List: contact commits-help@dolphinscheduler.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@dolphinscheduler.apache.org Delivered-To: mailing list commits@dolphinscheduler.apache.org Received: (qmail 72261 invoked by uid 99); 14 Jul 2020 00:46:06 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2020 00:46:06 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id CBFA9811C0; Tue, 14 Jul 2020 00:46:05 +0000 (UTC) Date: Tue, 14 Jul 2020 00:46:05 +0000 To: "commits@dolphinscheduler.apache.org" Subject: [incubator-dolphinscheduler-website] branch asf-site updated: Automated deployment: Tue Jul 14 00:45:54 UTC 2020 74350fb6e7c9875d843eea52f779515bd9f62383 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <159468756560.9601.7424457523254001085@gitbox.apache.org> From: github-bot@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: incubator-dolphinscheduler-website X-Git-Refname: refs/heads/asf-site X-Git-Reftype: branch X-Git-Oldrev: 381ea8c58aaccb43d2ca63cf63e4676c31443078 X-Git-Newrev: b6ea12e44f51f749de158e40084e4a4163afd9d8 X-Git-Rev: b6ea12e44f51f749de158e40084e4a4163afd9d8 X-Git-NotificationType: ref_changed_plus_diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-dolphinscheduler-website.git The following commit(s) were added to refs/heads/asf-site by this push: new b6ea12e Automated deployment: Tue Jul 14 00:45:54 UTC 2020 74350fb6e7c9875d843eea52f779515bd9f62383 b6ea12e is described below commit b6ea12e44f51f749de158e40084e4a4163afd9d8 Author: dailidong AuthorDate: Tue Jul 14 00:45:55 2020 +0000 Automated deployment: Tue Jul 14 00:45:54 UTC 2020 74350fb6e7c9875d843eea52f779515bd9f62383 --- .../development/development-environment-setup.html | 239 ++++++++++----------- .../development/development-environment-setup.json | 2 +- 2 files changed, 112 insertions(+), 129 deletions(-) diff --git a/zh-cn/docs/development/development-environment-setup.html b/zh-cn/docs/development/development-environment-setup.html index d352271..e90dee7 100644 --- a/zh-cn/docs/development/development-environment-setup.html +++ b/zh-cn/docs/development/development-environment-setup.html @@ -18,14 +18,14 @@

首先从远端仓库fork dolphinscheduler 一份代码到自己的仓库中

  • -

    安装好MySQL/PostgreSQL,Zookeeper,Nginx,Hadoop(可选)

    +

    在开发环境中安装好MySQL/PostgreSQL、JDK、MAVEN

  • 把自己仓库clone到本地

    git clone https://github.com/apache/incubator-dolphinscheduler.git

  • -

    git clone项目后，进入目录，执行以下命令。

    +

    git clone项目后，进入项目目录，执行以下命令。

    • 1. git branch -a    #查看分支
@@ -33,148 +33,131 @@
 3. git pull #同步分支
 4. mvn -U clean package -Prelease -Dmaven.test.skip=true   #由于项目使用了gRPC，所以需要先编译项目生成需要的类。
    +

    安装node

    +
      +
    1. 安装nvm
      +curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
      2. +
    2. 刷新环境变量
      +source ~/.bash_profile
      3. +
    3. 安装node
      +nvm install v12.12.0
      +备注:mac用户还可以通过brew安装npm:brew install npm
      4. +
    4. 验证node安装成功
      +node --version
      5. +
    +

    安装zookeeper

    +
      +
    1. 下载zookeeper
      +http://apache.mirrors.hoobly.com/zookeeper/zookeeper-3.6.1/apache-zookeeper-3.6.1-bin.tar.gz
      2. +
    2. 复制配置文件
      +cp conf/zoo_sample.cfg conf/zoo.cfg
      3. +
    3. 修改配置
      +vi conf/zoo.cfg
      +dataDir=./tmp/zookeeper
      4. +
    4. 启动/停止zookeeper
      +./bin/zkServer.sh start +./bin/zkServer.sh stop
      5. +
    +

    创建数据库

    +
      +
    1. 创建用户名为ds_user,密码为dolphinscheduler的用户
      2. +
    +
    mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
+mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'ds_user'@'%' IDENTIFIED BY 'dolphinscheduler';
+mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'ds_user'@'localhost' IDENTIFIED BY 'dolphinscheduler';
+mysql> flush privileges;
+
+
    +

    搭建前端

    +
      +
    1. 进入dolphinscheduler-ui的目录
      +cd dolphinscheduler-ui
      2. +
    2. 执行npm install
      3. +

    搭建后端

    1. -

      创建ds数据库CREATE DATABASE dolphinscheduler。

      +

      将项目导入到idea中
      +file-->open

    2. -

      创建表和初始化数据： -修改dao模块resource目录下application.properties文件中的数据库配置信息，如果你使用MySQL作为数据库，那么还需要配置pom.xml中mysql-connector-java依赖的scope为compile，然后执行org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler的类，运行完，刷新数据库，表和数据都有了。

      +

      修改dao模块resource目录下datasource.properties文件中的数据库配置信息

      +
          spring.datasource.driver-class-name=com.mysql.jdbc.Driver
+    spring.datasource.url=jdbc:mysql://localhost:3306/dolphinscheduler
+    spring.datasource.username=ds_user
+    spring.datasource.password=dolphinscheduler  
+
    3. -

      启动MasterServer -org.apache.dolphinscheduler.server.master.MasterServer类main函数增加如下代码：

      -
      System.setProperty("spring.profiles.active","master");
-
      -

      修改server模块resources目录下master_logback.xml文件，增加以下代码：

      -
      <root level="INFO">
- <appender-ref ref="MASTERLOGFILE"/>
- <!-- 增加日志到控制台-->
- <appender-ref ref="STDOUT"/>
-</root>
-
      -

      修改common模块，quartz.properties中数据库配置信息，zookeeper.properties中链接信息(zookeeper.quorum), 然后执行MasterServer即可。

      +

      修改根项目中pom.xml，将mysql-connector-java依赖的scope修改为compile

    4. -

      启动WorkerServer -org.apache.dolphinscheduler.server.worker.WorkerServer类main函数增加如下代码：

      -
      System.setProperty("spring.profiles.active","worker");
-
      -

      修改server模块resources目录下worker_logback.xml文件，增加以下代码：

      -
      <root level="INFO">
- <appender-ref ref="TASKLOGFILE"/>
- <appender-ref ref="WORKERLOGFILE"/>
- <!-- 增加日志到控制台-->
- <appender-ref ref="STDOUT"/>
-</root>
-
      -

      然后执行MasterServer即可。

      +

      刷新dao模块，运行org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler的main方法，自动插入项目所需的表和数据

      +
      5. +
    5. +

      修改service模块zookeeper.properties中链接信息(zookeeper.quorum)
      +zookeeper.quorum=localhost:2181

      +
      6. +
    6. +

      修改dolphinscheduler-ui模块的.env文件

    -

    搭建前端

    -

    进入dolphinscheduler-ui的目录，执行ui项目的编译，由于是webpack和vue，所以需要以下命令：

    -
    1. npm install    #没有npm的，mac用brew安装。brew install npm。
-2. npm run build  #执行完build命令后，会生成dist文件夹，这个文件夹一定要和nginx配置文件的40行所指的目录相同。
+
    API_BASE = http://localhost:12345
+DEV_HOST = localhost
 
    
-
    保存以下内容到dolphinscheduler.conf文件：
    
-
    
-#user  nobody;
-worker_processes  1;
-
-error_log  /usr/local/etc/nginx/logs/error.log;
-error_log  /usr/local/etc/nginx/logs/error.log  notice;
-error_log  /usr/local/etc/nginx/logs/error.log  info;
-
-pid        /usr/local/etc/nginx/logs/nginx.pid;
-
-
-events {
-    worker_connections  1024;
-}
-
-
-http {
-    include       mime.types;
-    default_type  application/octet-stream;
-
-    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
-    #                  '$status $body_bytes_sent "$http_referer" '
-    #                  '"$http_user_agent" "$http_x_forwarded_for"';
-
-    access_log  /usr/local/etc/nginx/logs/access.log;
-
-    sendfile        on;
-    #tcp_nopush     on;
-
-    #keepalive_timeout  0;
-    keepalive_timeout  65;
-
-    #gzip  on;
-    server {
-        listen       8888;# access port
-        server_name  localhost;
-        #charset koi8-r;
-        access_log  /usr/local/etc/nginx/logs/host.access.log;
-        location / {
-            root   /xxx/xxx/incubator-dolphinscheduler/dolphinscheduler-ui/dist; #这里为ui目录，需要修改
-            index  index.html index.html;
-        }
-        location /dolphinscheduler {
-            proxy_pass http://localhost:12345; # interface address
-            proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header x_real_ipP $remote_addr;
-            proxy_set_header remote_addr $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_http_version 1.1;
-            proxy_connect_timeout 300s;
-            proxy_read_timeout 300s;
-            proxy_send_timeout 300s;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection upgrade;
-        }
-        #error_page  404              /404.html;
-        # redirect server error pages to the static page /50x.html
-        #
-        error_page   500 502 503 504  /50x.html;
-        location = /50x.html {
-            root   /usr/share/nginx/html;
-        }
-    }
-    include servers/*;
-}
-
-
    
-
    修改上面root指向，然后将文件放到nginx配置同级目录,使用以下命令，启动nginx。
    
-
    ./nginx -c dolphinscheduler.conf
-
    
-
    启动ApiApplicationServer类：
-org.apache.dolphinscheduler.api.ApiApplicationServer类main函数增加如下代码：
    
-
    System.setProperty("spring.profiles.active", "api");
-
    
-
    修改api模块resources目录下apiserver_logback.xml文件，增加以下代码：
    
-
    <root level="INFO">
- <appender-ref ref="APISERVERLOGFILE" />
- <!-- 增加日志到控制台-->
- <appender-ref ref="STDOUT"/>
-</root>
-
    
-
    执行ApiApplicationServer, 然后访问localhost:8888，账号admin/dolphinscheduler123。
+
    启动项目
    
+
      
+
    1. 
+
      启动zookeeper
      
+./bin/zkServer.sh start
      
+
      2. 
+
    2. 
+
      启动MasterServer，执行org.apache.dolphinscheduler.server.master.MasterServer的main方法,需要设置VM Options:
      
+
          -Dlogging.config=classpath:logback-master.xml -Ddruid.mysql.usePingMethod=false
 
      
-
      资源中心
      
-
      ds对于资源的存储在本地文件系统/HDFS/S3/minio中，HDFS的mac安装请参考:https://www.jianshu.com/p/935b4c5e4c25
-hdfs下载:https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz
-安装好hdfs后，修改common.properties配置：
      
-
      res.upload.startup.type=HDFS
+
      3. 
+
    3. 
+
      启动WorkerServer，执行org.apache.dolphinscheduler.server.worker.WorkerServer的main方法,需要设置VM Options:
      
+
          -Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false
 
      
-
      修改hadoop.properties：
      
-
      fs.defaultFS=hdfs://localhost:9000
+
      4. 
+
    4. 
+
      启动ApiApplicationServer，执行org.apache.dolphinscheduler.api.ApiApplicationServer的main方法,需要设置VM Options:
      
+
          -Dlogging.config=classpath:logback-api.xml -Dspring.profiles.active=api
 
      
-
      然后重启ApiApplicationServer即可。
      
-
      *注意：上传的文件大小，超过1M下，需要添加nginx参数
      
-
      client_max_body_size    100m;
+
      5. 
+
    5. 
+
      ,这里暂时不启动其它模块，如果启动其它模块，那么去查询script/dolphinscheduler-daemon.sh文件,设置相应的VM Options
      
+
          if [ "$command" = "api-server" ]; then
+      LOG_FILE="-Dlogging.config=classpath:logback-api.xml -Dspring.profiles.active=api"
+      CLASS=org.apache.dolphinscheduler.api.ApiApplicationServer
+    elif [ "$command" = "master-server" ]; then
+      LOG_FILE="-Dlogging.config=classpath:logback-master.xml -Ddruid.mysql.usePingMethod=false"
+      CLASS=org.apache.dolphinscheduler.server.master.MasterServer
+    elif [ "$command" = "worker-server" ]; then
+      LOG_FILE="-Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false"
+      CLASS=org.apache.dolphinscheduler.server.worker.WorkerServer
+    elif [ "$command" = "alert-server" ]; then
+      LOG_FILE="-Dlogback.configurationFile=conf/logback-alert.xml"
+      CLASS=org.apache.dolphinscheduler.alert.AlertServer
+    elif [ "$command" = "logger-server" ]; then
+      CLASS=org.apache.dolphinscheduler.server.log.LoggerServer
+    else
+      echo "Error: No command named \`$command' was found."
+      exit 1
+    fi
 
      
+
      6. 
+
    6. 
+
      启动前端ui模块
      
+cd dolphinscheduler-ui目录，执行npm run start
      
+
      7. 
+
    
+
    访问项目
    
+
      
+
    1. 访问http://localhost:8888
      
+输入管理员账户admin,密码dolphinscheduler123进行登陆
      2. 
+