flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Features only compatible with hadoop 2.x
Date Mon, 25 Jan 2016 14:39:31 GMT
At some point, the state backends should probably have their own parent-
and subprojects, similar to the batch-connectors and streaming-connectors.

Then you have the flexibility to include and exclude them as needed.

Stephan


On Sat, Jan 23, 2016 at 6:42 PM, Gyula Fóra <gyula.fora@gmail.com> wrote:

> Alright, thanks.
>
> My dilemma is that currently all the out-of-core state stuff I put under
> streaming-contrib. So I either make the whole contrib package hadoop 2 only
> which is probably kind of awkward, or I move the main interfaces from
> contrib to somewhere else then I can put the HDFS backend stuff into a
> module that is hadoop 2 only.
>
> Gyula
>
> Robert Metzger <rmetzger@apache.org> ezt írta (időpont: 2016. jan. 22., P,
> 22:31):
>
> > Hi Gyula,
> >
> > will the out of core backend be in a separate maven module? If so, you
> can
> > include the module only in the "hadoop-2" profile.
> >
> > As you can see in the main pom.xml,  "flink-yarn" and "flink-fs-tests"
> are
> > also "hadoop2" only modules:
> >
> > <profile>
> >    <id>hadoop-2</id>
> >    <activation>
> >       <property>
> >          <!-- Please do not remove the 'hadoop2' comment. See
> > ./tools/generate_specific_pom.sh -->
> >          <!--hadoop2--><name>!hadoop.profile</name>
> >       </property>
> >    </activation>
> >    <properties>
> >       <hadoop.version>${hadoop-two.version}</hadoop.version>
> >       <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name
> >
> >       <shading-artifact-module.name>flink-shaded-hadoop2</
> > shading-artifact-module.name>
> >    </properties>
> >    <modules>
> >       <module>flink-yarn</module>
> >       <!-- Include the flink-fs-tests project only for HD2.
> >          The HDFS minicluster interfaces changed between the two
> versions.
> >        -->
> >       <module>flink-fs-tests</module>
> >    </modules>
> > </profile>
> >
> >
> > If the backend is not in a separate maven module, you can use reflection.
> > Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush"
> > only if the method is available ;)
> >
> >
> >
> > On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <gyfora@apache.org> wrote:
> >
> > > Hi,
> > >
> > > While developing the out-of-core state backend that will store state
> > > directly to hdfs (either TFiles or BloomMapFiles), I realised that some
> > > file formats and features I use are hadoop 2.x only.
> > >
> > > What is the suggested way to handle features that use hadoop 2.x api?
> Can
> > > these be excluded from the travis build for the hadoop 1 profile
> somehow?
> > >
> > > Thanks,
> > > Gyula
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message