beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyuntian (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-375) HadoopIO and runners-spark conflict with hadoop.version
Date Fri, 14 Apr 2017 01:15:41 GMT

    [ https://issues.apache.org/jira/browse/BEAM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968410#comment-15968410
] 

liyuntian edited comment on BEAM-375 at 4/14/17 1:15 AM:
---------------------------------------------------------

yes,I get state by "pipelineResult.getState().isTerminal()" ,But ,IT IS running all the time,and
the spark executors Disk Used is 0.0B, but when I change hadoop version to 2.7.0,I can get
the result correctly.so I want to know  I must use Hadoop2.7.0 in beam0.5?perhaps I lost some
jars?


was (Author: yuntian):
yes,I get state by "pipelineResult.getState().isTerminal()" ,But ,IT IS running all the time,and
the spark executors Disk Used is 0.0B, but when I change hadoop version to 2.7.0,I can get
the result correctly.so I want to know  I must use Hadoop2.7.0 in beam0.5?

> HadoopIO and runners-spark conflict with hadoop.version
> -------------------------------------------------------
>
>                 Key: BEAM-375
>                 URL: https://issues.apache.org/jira/browse/BEAM-375
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: Pei He
>            Assignee: Pei He
>
> HadoopIO currently uses 2.7.0 and runners-spark uses 2.2.0 for hadoop-client, hadoop-common.
> From [~amitsela]
> "Spark can be built against different hadoop versions, but the release in maven central
is a 2.2.0 build (latest). ''
> For HadoopIO, I don't know why 2.7.0 is picked at the beginning. I can check if it will
work with 2.2.0.
> I am creating this issue, since I think it there is a general question.
> In principle, HadoopIO and other sdks Sources should work with any runners. But, when
one set of runners require version A, but the other set of runners require version B, we will
need a general solution for it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message