beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Recommended Flink Version
Date Wed, 12 Jul 2017 08:01:51 GMT
So this is a general bug in how Flink constructs the HADOOP_CONF_DIR path?

Best,
Aljoscha

> On 11. Jul 2017, at 20:14, Will Walters <wwalters@yahoo-inc.com> wrote:
> 
> I've managed to solve the problem I was having, which was with Flink not properly finding
my hdfs registrar. It turns out that in the config.sh file, it runs this line on the global
variable $HADOOP_CONF_DIR:
> 
> if [ -d "$HADOOP_HOME/etc/hadoop" ]; then
>         # Its Hadoop 2.2+
>         HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
> 
> This means that HADOOP_CONF_DIR is set to two filepaths concatenated with a colon. In
the function which reads in the registrars, it passes this string into the File() constructor,
which fails because the string isn't a valid filepath. Commenting out the line above solved
the problem, allowing a successful submission.
> 
> Thanks for your help!
> Will
> 
> 
> On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb@nanthrax.net> wrote:
> 
> 
> Good point, fair enough.
> 
> Regards
> JB
> 
> On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> > I think it’s a bug because if you start a Flink cluster on bare-metal it works,
just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster
should not affect how you can submit jobs to it.
> > 
> > Best,
> > Aljoscha
> > 
> >> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb@nanthrax.net <mailto:jb@nanthrax.net>>
wrote:
> >>
> >> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint,
it's not a bug, it's a feature request.
> >>
> >> Regards
> >> JB
> >>
> >> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
> >>> I also responded to a separate mail by Will. The problem is that currently
we cannot submit a job using the remote client to a Flink cluster that was started on YARN.
(It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
> >>> The workaround for that is to use the bin/flink command to submit a Beam
fat-jar on a Flink YARN cluster.
> >>> Best,
> >>> Aljoscha
> >>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb@nanthrax.net
<mailto:jb@nanthrax.net>> wrote:
> >>>>
> >>>> Hi Will,
> >>>>
> >>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1
by default. So, I would recommend this version or 1.2.x.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 06/28/2017 10:39 PM, Will Walters wrote:
> >>>>> Hello,
> >>>>> I've been attempting to run Beam through Flink on a Yarn cluster
and have run into trouble with getting a job to submit, partly because of incompatibility
between versions. Does anyone know what versions of Beam and Flink I should be using to give
myself the best chance of finding compatibility?
> >>>>> Thank you,
> >>>>> Will.
> >>>>
> >>>> -- 
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org <mailto:jbonofre@apache.org>
> >>>> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> >>>> Talend - http://www.talend.com <http://www.talend.com/>
> 
> >>
> >> -- 
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org <mailto:jbonofre@apache.org>
> >> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> >> Talend - http://www.talend.com <http://www.talend.com/>
> > 
> 
> -- 
> Jean-Baptiste Onofré
> jbonofre@apache.org <mailto:jbonofre@apache.org>
> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> Talend - http://www.talend.com <http://www.talend.com/>
> 
> 


Mime
View raw message