flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: Features only compatible with hadoop 2.x
Date Sat, 23 Jan 2016 17:42:43 GMT
Alright, thanks.

My dilemma is that currently all the out-of-core state stuff I put under
streaming-contrib. So I either make the whole contrib package hadoop 2 only
which is probably kind of awkward, or I move the main interfaces from
contrib to somewhere else then I can put the HDFS backend stuff into a
module that is hadoop 2 only.

Gyula

Robert Metzger <rmetzger@apache.org> ezt írta (időpont: 2016. jan. 22., P,
22:31):

> Hi Gyula,
>
> will the out of core backend be in a separate maven module? If so, you can
> include the module only in the "hadoop-2" profile.
>
> As you can see in the main pom.xml,  "flink-yarn" and "flink-fs-tests" are
> also "hadoop2" only modules:
>
> <profile>
>    <id>hadoop-2</id>
>    <activation>
>       <property>
>          <!-- Please do not remove the 'hadoop2' comment. See
> ./tools/generate_specific_pom.sh -->
>          <!--hadoop2--><name>!hadoop.profile</name>
>       </property>
>    </activation>
>    <properties>
>       <hadoop.version>${hadoop-two.version}</hadoop.version>
>       <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name>
>       <shading-artifact-module.name>flink-shaded-hadoop2</
> shading-artifact-module.name>
>    </properties>
>    <modules>
>       <module>flink-yarn</module>
>       <!-- Include the flink-fs-tests project only for HD2.
>          The HDFS minicluster interfaces changed between the two versions.
>        -->
>       <module>flink-fs-tests</module>
>    </modules>
> </profile>
>
>
> If the backend is not in a separate maven module, you can use reflection.
> Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush"
> only if the method is available ;)
>
>
>
> On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <gyfora@apache.org> wrote:
>
> > Hi,
> >
> > While developing the out-of-core state backend that will store state
> > directly to hdfs (either TFiles or BloomMapFiles), I realised that some
> > file formats and features I use are hadoop 2.x only.
> >
> > What is the suggested way to handle features that use hadoop 2.x api? Can
> > these be excluded from the travis build for the hadoop 1 profile somehow?
> >
> > Thanks,
> > Gyula
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message