flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8981) Add end-to-end test for running on YARN with Kerberos
Date Fri, 20 Jul 2018 08:42:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550503#comment-16550503
] 

ASF GitHub Bot commented on FLINK-8981:
---------------------------------------

Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6377#discussion_r203973314
  
    --- Diff: flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/Dockerfile
---
    @@ -0,0 +1,159 @@
    +################################################################################
    +# Licensed to the Apache Software Foundation (ASF) under one
    +# or more contributor license agreements.  See the NOTICE file
    +# distributed with this work for additional information
    +# regarding copyright ownership.  The ASF licenses this file
    +# to you under the Apache License, Version 2.0 (the
    +# "License"); you may not use this file except in compliance
    +# with the License.  You may obtain a copy of the License at
    +#
    +#     http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +################################################################################
    +#
    +# This image is modified version of Knappek/docker-hadoop-secure
    +#   * Knappek/docker-hadoop-secure <https://github.com/Knappek/docker-hadoop-secure>
    +#
    +# With bits and pieces added from Lewuathe/docker-hadoop-cluster to extend it to start
a proper kerberized Hadoop cluster:
    +#   * Lewuathe/docker-hadoop-cluster <https://github.com/Lewuathe/docker-hadoop-cluster>
    +#
    +# Author: Aljoscha Krettek
    +# Date:   2018 May, 15
    +#
    +# Creates multi-node, kerberized Hadoop cluster on Docker
    +
    +FROM sequenceiq/pam:ubuntu-14.04
    +MAINTAINER aljoscha
    +
    +USER root
    +
    +RUN addgroup hadoop
    +RUN useradd -d /home/hdfs -ms /bin/bash -G hadoop -p hdfs hdfs
    +RUN useradd -d /home/yarn -ms /bin/bash -G hadoop -p yarn yarn
    +RUN useradd -d /home/mapred -ms /bin/bash -G hadoop -p mapred mapred
    +
    +RUN useradd -d /home/hadoop-user -ms /bin/bash -p hadoop-user hadoop-user
    +
    +# install dev tools
    +RUN apt-get update
    +RUN apt-get install -y curl tar sudo openssh-server openssh-client rsync unzip
    +
    +# Kerberos client
    +RUN apt-get install krb5-user -y
    +RUN mkdir -p /var/log/kerberos
    +RUN touch /var/log/kerberos/kadmind.log
    +
    +# passwordless ssh
    +RUN rm -f /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_rsa_key /root/.ssh/id_rsa
    +RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key
    +RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key
    +RUN ssh-keygen -q -N "" -t rsa -f /root/.ssh/id_rsa
    +RUN cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
    +
    +# java
    +RUN mkdir -p /usr/java/default && \
    +     curl -Ls 'http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz'
-H 'Cookie: oraclelicense=accept-securebackup-cookie' | \
    +     tar --strip-components=1 -xz -C /usr/java/default/
    +
    +ENV JAVA_HOME /usr/java/default
    +ENV PATH $PATH:$JAVA_HOME/bin
    +
    +RUN curl -LOH 'Cookie: oraclelicense=accept-securebackup-cookie' 'http://download.oracle.com/otn-pub/java/jce/8/jce_policy-8.zip'
    +RUN unzip jce_policy-8.zip
    +RUN cp /UnlimitedJCEPolicyJDK8/local_policy.jar /UnlimitedJCEPolicyJDK8/US_export_policy.jar
$JAVA_HOME/jre/lib/security
    +
    +ENV HADOOP_VERSION=2.8.4
    +
    +# ENV HADOOP_URL https://www.apache.org/dist/hadoop/common/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz
    --- End diff --
    
    remove


> Add end-to-end test for running on YARN with Kerberos
> -----------------------------------------------------
>
>                 Key: FLINK-8981
>                 URL: https://issues.apache.org/jira/browse/FLINK-8981
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Security, Tests
>    Affects Versions: 1.5.0
>            Reporter: Till Rohrmann
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.6.0
>
>
> We should add an end-to-end test which verifies Flink's integration with Kerberos security.
In order to do this, we should start a Kerberos secured Hadoop, ZooKeeper and Kafka cluster.
Then we should start a Flink cluster with HA enabled and run a job which reads from and writes
to Kafka. We could use a simple pipe job for that purpose which has some state for checkpointing
to HDFS.
> See [security docs| https://ci.apache.org/projects/flink/flink-docs-master/ops/security-kerberos.html]
for how more information about Flink's Kerberos integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message