flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8981) Add end-to-end test for running on YARN with Kerberos
Date Fri, 20 Jul 2018 11:57:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550699#comment-16550699
] 

ASF GitHub Bot commented on FLINK-8981:
---------------------------------------

Github user dawidwys commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6377#discussion_r204020749
  
    --- Diff: flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/README.md ---
    @@ -0,0 +1,118 @@
    +# Apache Hadoop Docker image with Kerberos enabled
    +
    +This image is modified version of Knappek/docker-hadoop-secure
    + * Knappek/docker-hadoop-secure <https://github.com/Knappek/docker-hadoop-secure>
    +
    +With bits and pieces added from Lewuathe/docker-hadoop-cluster to extend it to start
a proper kerberized Hadoop cluster:
    + * Lewuathe/docker-hadoop-cluster <https://github.com/Lewuathe/docker-hadoop-cluster>
    +
    +And a lot of added stuff for making this an actual, properly configured, kerberized cluster
with proper user/permissions structure.
    +
    +Versions
    +--------
    +
    +* JDK8
    +* Hadoop 2.8.3
    +
    +Default Environment Variables
    +-----------------------------
    +
    +| Name | Value | Description |
    +| ---- | ----  | ---- |
    +| `KRB_REALM` | `EXAMPLE.COM` | The Kerberos Realm, more information [here](https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html#)
|
    +| `DOMAIN_REALM` | `example.com` | The Kerberos Domain Realm, more information [here](https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html#)
|
    +| `KERBEROS_ADMIN` | `admin/admin` | The KDC admin user |
    +| `KERBEROS_ADMIN_PASSWORD` | `admin` | The KDC admin password |
    +
    +You can simply define these variables in the `docker-compose.yml`.
    +
    +Run image
    +---------
    +
    +Clone the [Github project](https://github.com/aljoscha/docker-hadoop-secure-cluster)
and run
    +
    +```
    +docker-compose up
    +```
    +
    +Usage
    +-----
    +
    +Get the container name with `docker ps` and login to the container with
    +
    +```
    +docker exec -it <container-name> /bin/bash
    +```
    +
    +
    +To obtain a Kerberos ticket, execute
    +
    +```
    +kinit -kt /home/hadoop-user/hadoop-user.keytab hadoop-user
    +```
    +
    +Afterwards you can use `hdfs` CLI like
    +
    +```
    +hdfs dfs -ls /
    +```
    +
    +
    +Known issues
    +------------
    +
    +### Unable to obtain Kerberos password
    +
    +#### Error
    +docker-compose up fails for the first time with the error
    +
    +```
    +Login failure for nn/hadoop.docker.com@EXAMPLE.COM from keytab /etc/security/keytabs/nn.service.keytab:
javax.security.auth.login.LoginException: Unable to obtain password from user
    +```
    +
    +#### Solution
    +
    +Stop the containers with `docker-compose down` and start again with `docker-compose up
-d`.
    +
    +
    +### JDK 8
    +
    +Make sure you use download a JDK version that is still available. Old versions can be
deprecated by Oracle and thus the download link won't be able anymore.
    +
    +Get the latest JDK8 Download URL with
    +
    +```
    +curl -s https://lv.binarybabel.org/catalog-api/java/jdk8.json
    +```
    +
    +### Java Keystore
    +
    +If the Keystroe has been expired, then create a new `keystore.jks`:
    --- End diff --
    
    Keystroe -> Keystore
    
    Won't it be a problem in tests? Will the test start failing one day because of the keystore
expired?


> Add end-to-end test for running on YARN with Kerberos
> -----------------------------------------------------
>
>                 Key: FLINK-8981
>                 URL: https://issues.apache.org/jira/browse/FLINK-8981
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Security, Tests
>    Affects Versions: 1.5.0
>            Reporter: Till Rohrmann
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.6.0
>
>
> We should add an end-to-end test which verifies Flink's integration with Kerberos security.
In order to do this, we should start a Kerberos secured Hadoop, ZooKeeper and Kafka cluster.
Then we should start a Flink cluster with HA enabled and run a job which reads from and writes
to Kafka. We could use a simple pipe job for that purpose which has some state for checkpointing
to HDFS.
> See [security docs| https://ci.apache.org/projects/flink/flink-docs-master/ops/security-kerberos.html]
for how more information about Flink's Kerberos integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message