Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Thu, 13 Apr 2017 12:40:41 +0000 (UTC)
From: "liyuntian (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.12982767.1466803180000.275422.1492087241614@Atlassian.JIRA>
In-Reply-To: <JIRA.12982767.1466803180000@Atlassian.JIRA>
References: <JIRA.12982767.1466803180000@Atlassian.JIRA> <JIRA.12982767.1466803180678@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-375) HadoopIO and runners-spark conflict
 with hadoop.version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 13 Apr 2017 12:40:47 -0000


    [ https://issues.apache.org/jira/browse/BEAM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967528#comment-15967528 ] 

liyuntian commented on BEAM-375:
--------------------------------

Does HdfsIO must use hadoop2.7.0 or above? I use Hadoop2.6.0 in our system,but It is blocking ,when I execute "pipeline.run()" with sparkrunner . If I change Hadoop version to 2.7.0, it can run very well with sparkrunner.So,I think I must use Hadoop2.7.0,yes?

> HadoopIO and runners-spark conflict with hadoop.version
> -------------------------------------------------------
>
>                 Key: BEAM-375
>                 URL: https://issues.apache.org/jira/browse/BEAM-375
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: Pei He
>            Assignee: Pei He
>
> HadoopIO currently uses 2.7.0 and runners-spark uses 2.2.0 for hadoop-client, hadoop-common.
> From [~amitsela]
> "Spark can be built against different hadoop versions, but the release in maven central is a 2.2.0 build (latest). ''
> For HadoopIO, I don't know why 2.7.0 is picked at the beginning. I can check if it will work with 2.2.0.
> I am creating this issue, since I think it there is a general question.
> In principle, HadoopIO and other sdks Sources should work with any runners. But, when one set of runners require version A, but the other set of runners require version B, we will need a general solution for it.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)