kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [kylin] sanjulian commented on a change in pull request #1182: KYLIN-4446 Kylin on kubernetes in quick-start and production env
Date Fri, 01 May 2020 04:03:39 GMT

sanjulian commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r417748474



##########
File path: kubernetes/docker/hadoop-client/README.md
##########
@@ -0,0 +1,18 @@
+## Background
+### What is hadoop-client docker image and why we need this? 
+As we all know, the node you want to deploy Kylin, should has provided Hadoop 
+dependency(jars and configuration files), these dependency let you have access
+ to Hadoop Service, such as HDFS, HBase, Hive, which are needed by Apache Kylin. 
+Unfortunately, each Hadoop distribution(CHD or HDP etc.) has its own specific jars. So, we

+can provided specific image for specific Hadoop distribution, which will make image management
task
+more easier. This will have following two benefits:
+1. Someone who has better knowledge on Hadoop can do this work, and let kylin 
+ user build their Kylin image base on provided Hadoop-Client image.
+2. Upgrade Kylin will be much easier.
+
+### Build Step for CDH5.7
+1. Place Spark binary(*spark-2.3.2-bin-hadoop2.7.tgz*) into dir `provided-binary`.
+2. Run `build-image.sh` to build image.
+
+> If you are using other hadoop distribution, please consider build image yourself.
+

Review comment:
       Could we add the some build step for apache hadoop client?
   Such as download hadoop client first, and update version in dockerfile.

##########
File path: kubernetes/example/deployment/kylin-job/kylin-job-service.yaml
##########
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+kind: Service
+metadata:
+  name: kylin-job-svc

Review comment:
       I suppose we do not need the services for job, since the job pod only for building
cube instead of query.

##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: kylin-query
+  namespace: kylin-example
+  labels:
+    app: kylin
+    role: query
+    environment: poc
+    version: 3.0.1
+    hadoop.version: cdh5.7
+spec:
+  serviceName: kylin-query-svc
+  replicas: 1
+  selector:
+    matchLabels:
+      app: kylin
+      role: query
+  template:
+    metadata:
+      labels:
+        app: kylin
+        role: query
+    spec:
+      hostAliases:
+        - ip: "10.1.3.90"
+          hostnames:
+            - "cdh-master"
+        - ip: "10.1.3.93"
+          hostnames:
+            - "cdh-worker-1"
+        - ip: "10.1.3.94"
+          hostnames:
+            - "cdh-worker-2"
+        - ip: "10.1.3.91"
+          hostnames:
+            - "cdh-client"
+      containers:
+      - name: kylin
+        image: kylin-client:3.0.1-cdh57
+        imagePullPolicy: IfNotPresent
+        command:
+        - sh
+        - -c
+        args:
+        - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;

Review comment:
       Remove the copy before the init.
   Please use subPath to mount configMap or secret files for tomcat and kylin configuration.
   Here is the reference: https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath

##########
File path: kubernetes/docker/hadoop-client/apache-hadoop2.7/Dockerfile
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM centos:7.3.1611
+
+MAINTAINER Apache Kylin
+
+WORKDIR /tmp
+
+# install jdk and other commands
+RUN set -x \
+    && yum install -y which \
+       java-1.8.0-openjdk \
+       java-1.8.0-openjdk-devel \
+       krb5-workstation \
+    && yum clean all
+
+
+# version variables
+ARG HADOOP_VERSION=2.7.4
+ARG HIVE_VERSION=1.2.2
+ARG HBASE_VERSION=1.3.1
+ARG SPARK_VERSION=2.2.1
+ARG ZK_VERSION=3.4.6
+
+ARG APACHE_HOME=/apache
+
+RUN set -x \
+    && mkdir -p $APACHE_HOME
+
+ENV JAVA_HOME /etc/alternatives/jre
+
+# install hadoop
+ENV HADOOP_HOME=$APACHE_HOME/hadoop
+ADD hdp-clients/hadoop-${HADOOP_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+    && ln -s $APACHE_HOME/hadoop-${HADOOP_VERSION} $HADOOP_HOME
+
+#install hive
+ENV HIVE_HOME=$APACHE_HOME/hive 
+ADD hdp-clients/apache-hive-${HIVE_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+    && ln -s $APACHE_HOME/apache-hive-${HIVE_VERSION} $HIVE_HOME
+
+#install hbase
+ENV HBASE_HOME=$APACHE_HOME/hbase
+ADD hdp-clients/hbase-${HBASE_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+    && ln -s $APACHE_HOME/hbase-${HBASE_VERSION} $HBASE_HOME
+
+#install spark
+ENV SPARK_HOME=$APACHE_HOME/spark
+ADD hdp-clients/spark-${SPARK_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+    && ln -s $APACHE_HOME/spark-${SPARK_VERSION} $SPARK_HOME
+
+#install zk
+ENV ZK_HOME=$APACHE_HOME/zookeeper
+ADD hdp-clients/zookeeper-${ZK_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+    && ln -s $APACHE_HOME/zookeeper-${ZK_VERSION} $ZK_HOME
+
+ENV PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$ZK_HOME/bin
+
+COPY conf/hadoop $HADOOP_HOME/etc/hadoop

Review comment:
       I suppose we should remove the hadoop config. There are hadoop config for configmap
in development files.

##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: kylin-query
+  namespace: kylin-example
+  labels:
+    app: kylin
+    role: query
+    environment: poc
+    version: 3.0.1
+    hadoop.version: cdh5.7
+spec:
+  serviceName: kylin-query-svc
+  replicas: 1
+  selector:
+    matchLabels:
+      app: kylin
+      role: query
+  template:
+    metadata:
+      labels:
+        app: kylin
+        role: query
+    spec:
+      hostAliases:
+        - ip: "10.1.3.90"
+          hostnames:
+            - "cdh-master"
+        - ip: "10.1.3.93"
+          hostnames:
+            - "cdh-worker-1"
+        - ip: "10.1.3.94"
+          hostnames:
+            - "cdh-worker-2"
+        - ip: "10.1.3.91"
+          hostnames:
+            - "cdh-client"
+      containers:
+      - name: kylin
+        image: kylin-client:3.0.1-cdh57
+        imagePullPolicy: IfNotPresent
+        command:
+        - sh
+        - -c
+        args:
+        - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+          cp $KYLIN_HOME/kylin-more-config $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+          $TOOL_HOME/bootstrap.sh server -d;
+        ports:
+        - containerPort: 7070
+        - containerPort: 7443
+        volumeMounts:
+          - name: kylin-query-config
+            mountPath: /home/apache_kylin/kylin/conf
+          - name: tomcat-config
+            mountPath: /home/apache_kylin/kylin/tomcat-config
+          - name: kylin-more-config
+            mountPath: /home/apache_kylin/kylin/kylin-more-config/
+          - name: hadoop-config
+            mountPath: /etc/hadoop/conf
+          - name: hive-config
+            mountPath: /etc/hive/conf
+          - name: hbase-config
+            mountPath: /etc/hbase/conf
+          - name: kylin-logs
+            mountPath: /home/apache_kylin/kylin/logs
+          - name: tomcat-logs
+            mountPath: /home/apache_kylin/kylin/tomcat/logs
+        resources:
+          requests:
+            memory: 2Gi
+            cpu: 700m
+          limits:
+            memory: 2Gi
+            cpu: 700m
+      - name: filebeat
+        image: docker.elastic.co/beats/filebeat:6.4.3
+        args:
+        - -c
+        - /usr/share/filebeat/config/filebeat.yml
+        - -e
+        volumeMounts:
+        - name: kylin-logs
+          mountPath: /home/apache_kylin/kylin/logs
+        - name: tomcat-logs
+          mountPath: /home/apache_kylin/kylin/tomcat/logs
+        - name: filebeat-config
+          mountPath: /usr/share/filebeat/config
+          readOnly: true
+        resources:
+          requests:
+            memory: 1Gi
+            cpu: 300m
+          limits:
+            memory: 1Gi
+            cpu: 300m
+      volumes:
+      - name: kylin-logs
+        emptyDir:
+          sizeLimit: 3Gi
+      - name: tomcat-logs
+        emptyDir:
+          sizeLimit: 2Gi
+      - configMap:
+          name: hadoop-config
+        name: hadoop-config
+      - configMap:
+          name: hive-config
+        name: hive-config
+      - configMap:
+          name: hbase-config
+        name: hbase-config
+      - configMap:

Review comment:
       As I know, there are some secret files in kylin.properties, such as kylin.job.notification-mail-password,
kylin.security.ldap.connection-password... Could we change the kylin-***-config as secrets
instead of configMap? 

##########
File path: kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: kylin-query
+  namespace: kylin-example
+  labels:
+    app: kylin
+    role: query
+    environment: poc
+    version: 3.0.1
+    hadoop.version: cdh5.7
+spec:
+  serviceName: kylin-query-svc
+  replicas: 1
+  selector:
+    matchLabels:
+      app: kylin
+      role: query
+  template:
+    metadata:
+      labels:
+        app: kylin
+        role: query
+    spec:
+      hostAliases:
+        - ip: "10.1.3.90"
+          hostnames:
+            - "cdh-master"
+        - ip: "10.1.3.93"
+          hostnames:
+            - "cdh-worker-1"
+        - ip: "10.1.3.94"
+          hostnames:
+            - "cdh-worker-2"
+        - ip: "10.1.3.91"
+          hostnames:
+            - "cdh-client"
+      containers:
+      - name: kylin
+        image: kylin-client:3.0.1-cdh57
+        imagePullPolicy: IfNotPresent
+        command:
+        - sh
+        - -c
+        args:
+        - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+          cp $KYLIN_HOME/kylin-more-config $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+          $TOOL_HOME/bootstrap.sh server -d;
+        ports:
+        - containerPort: 7070
+        - containerPort: 7443
+        volumeMounts:
+          - name: kylin-query-config
+            mountPath: /home/apache_kylin/kylin/conf
+          - name: tomcat-config
+            mountPath: /home/apache_kylin/kylin/tomcat-config
+          - name: kylin-more-config
+            mountPath: /home/apache_kylin/kylin/kylin-more-config/
+          - name: hadoop-config
+            mountPath: /etc/hadoop/conf
+          - name: hive-config
+            mountPath: /etc/hive/conf
+          - name: hbase-config
+            mountPath: /etc/hbase/conf
+          - name: kylin-logs
+            mountPath: /home/apache_kylin/kylin/logs
+          - name: tomcat-logs
+            mountPath: /home/apache_kylin/kylin/tomcat/logs
+        resources:
+          requests:
+            memory: 2Gi
+            cpu: 700m
+          limits:
+            memory: 2Gi
+            cpu: 700m
+      - name: filebeat
+        image: docker.elastic.co/beats/filebeat:6.4.3
+        args:
+        - -c
+        - /usr/share/filebeat/config/filebeat.yml
+        - -e
+        volumeMounts:
+        - name: kylin-logs
+          mountPath: /home/apache_kylin/kylin/logs
+        - name: tomcat-logs
+          mountPath: /home/apache_kylin/kylin/tomcat/logs
+        - name: filebeat-config
+          mountPath: /usr/share/filebeat/config
+          readOnly: true
+        resources:
+          requests:
+            memory: 1Gi
+            cpu: 300m
+          limits:
+            memory: 1Gi
+            cpu: 300m
+      volumes:
+      - name: kylin-logs
+        emptyDir:
+          sizeLimit: 3Gi
+      - name: tomcat-logs
+        emptyDir:
+          sizeLimit: 2Gi
+      - configMap:

Review comment:
       The hadoop configmap is not used. Should we delete them or add the volume mount for
hdp configuration?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message