flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Razvan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-9540) Apache Flink 1.4.2 S3 Hadoop library for Hadoop 2.7 is built for Hadoop 2.8 and fails
Date Fri, 08 Jun 2018 17:52:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506323#comment-16506323
] 

Razvan edited comment on FLINK-9540 at 6/8/18 5:51 PM:
-------------------------------------------------------

Hi [~aljoscha],

 

  Thanks for the hint. I managed to get 1.4.2 working with S3A by copying necessary libraries
to lib/ for Hadoop 2.8 only (but it's not straightforward and it's not correctly documented
in my opinion). My goal here is also to clean up the documentation a bit. So I want *to define
exactly what is needed for each version of Apache Flink (1.4.* - 1.6.* ) in order to get it
working with S3/A/N for each of Hadoop versions (2.4 - 2.8).* I believe this will increase
the number of users of the framework and contributors. So if you / anyone could help me define
the dependencies I can make PR for documentation. 


was (Author: razvan):
Hi [~aljoscha],

 

  Thanks for the hint. I managed to get 1.4.2 working with S3A by copying necessary libraries
to lib/ (but it's not straightforward and it's not correctly documented in my opinion). My
goal here is also to clean up the documentation a bit. So I want *to define exactly what is
needed for each version of Apache Flink (1.4.* - 1.6.* ) in order to get it working with S3/A/N
for each of Hadoop versions (2.4 - 2.8).* I believe this will increase the number of users
of the framework and contributors. So if you / anyone could help me define the dependencies
I can make PR for documentation. 

> Apache Flink 1.4.2 S3 Hadoop library for Hadoop 2.7 is built for Hadoop 2.8 and fails
> -------------------------------------------------------------------------------------
>
>                 Key: FLINK-9540
>                 URL: https://issues.apache.org/jira/browse/FLINK-9540
>             Project: Flink
>          Issue Type: Bug
>            Reporter: Razvan
>            Priority: Blocker
>
> Download Apache Flink 1.4.2 for Hadoop 2.7 or earlier. Go to opt/ inside flink-s3-fs-hadoop-1.4.2.jar,
open common version info .properties inside you get:
>  
> #
>  # Licensed to the Apache Software Foundation (ASF) under one
>  # or more contributor license agreements. See the NOTICE file
>  # distributed with this work for additional information
>  # regarding copyright ownership. The ASF licenses this file
>  # to you under the Apache License, Version 2.0 (the
>  # "License"); you may not use this file except in compliance
>  # with the License. You may obtain a copy of the License at
>  #
>  # [http://www.apache.org/licenses/LICENSE-2.0]
>  #
>  # Unless required by applicable law or agreed to in writing, software
>  # distributed under the License is distributed on an "AS IS" BASIS,
>  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>  # See the License for the specific language governing permissions and
>  # limitations under the License.
>  #
> *version=2.8.1*
>  revision=1e6296df38f9cd3d9581c8af58a2a03a6e4312be
>  *branch=branch-2.8.1-private*
>  user=vinodkv
>  date=2017-06-07T21:22Z
>  *url=[https://git-wip-us.apache.org/repos/asf/hadoop.git*]
>  srcChecksum=60125541c2b3e266cbf3becc5bda666
>  protocVersion=2.5.0
>  
> This will fail to work with dependencies described in ([https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html)]
>  
> "Depending on which file system you use, please add the following dependencies. You can
find these as part of the Hadoop binaries in {{hadoop-2.7/share/hadoop/tools/lib}}:
>  * {{S3AFileSystem}}:
>  ** {{hadoop-aws-2.7.3.jar}}
>  ** {{aws-java-sdk-s3-1.11.183.jar}} and its dependencies:
>  *** {{aws-java-sdk-core-1.11.183.jar}}
>  *** {{aws-java-sdk-kms-1.11.183.jar}}
>  *** {{jackson-annotations-2.6.7.jar}}
>  *** {{jackson-core-2.6.7.jar}}
>  *** {{jackson-databind-2.6.7.jar}}
>  *** {{joda-time-2.8.1.jar}}
>  *** {{httpcore-4.4.4.jar}}
>  *** {{httpclient-4.5.3.jar}}
>  * {{NativeS3FileSystem}}:
>  ** {{hadoop-aws-2.7.3.jar}}
>  ** {{guava-11.0.2.jar"}}{{}}
>  
> {{  I presume the build task is flawed, it should be built for Apache Hadoop 2.7 can
someone please check it?}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message