flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: env.readFile with enumeratenestedFields
Date Wed, 20 Jul 2016 16:29:12 GMT
Hi,
the configuration has to be passed using
env.readFile(...).withParameters(ifConf). The InputFormat will then be
properly configured at runtime.

However, Kostas just enhanced the FileInputFormats to allow setting the
parameters directly on the input format. In 1.1-SNAPSHOT and the upcoming
1.1 you should be able to use inputFormat.setNestedFileEnumeration(true).

Cheers,
Aljoscha

On Wed, 20 Jul 2016 at 17:55 Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> Hi to all,
>
> in my job I'm doing the following to recursively read the files inside a
> dir:
>
>  TextInputFormat inputFormat = new TextInputFormat(new Path(inputDir));
>     org.apache.flink.configuration.Configuration ifConf =
>         new org.apache.flink.configuration.Configuration();
>     ifConf.setBoolean(FileInputFormat.ENUMERATE_NESTED_FILES_FLAG, true);
>     inputFormat.configure(ifConf);
>
>     ExecutionEnvironment env =
> ExecutionEnvironment.getExecutionEnvironment();
>      env.readFile(inputFormat, inputDir).print();
>
> While inputFormat.configure() sets correctly the enumeratenestedFields
> field within TextInputFormat, the execution of the job seems to forget this
> parameter and reset it to false.
>
> Am I doing something wrong or there's a bug here (I'm using Flink 1.0.2)?
>
> Best,
> Flavio
>
>
>

Mime
View raw message