flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Odd flink behaviour
Date Tue, 01 Aug 2017 00:09:33 GMT
I have a very simple program that just reads all the files in the path.
However, flink is not working as expected.

Everytime I execute this job I only see flink reading 2 files, even though
there are more in that directory. On closer look it appears that it might
be related to:

[flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.taskmanager.TaskManager - TaskManager has 2 task
slot(s).

My question is, isn't flink supposed to iterate over the directory after
those 2 slots become free again? I am assuming this problem is caused
because there are only 2 slots.


Code ---

  PDFFileInputFormat format = new PDFFileInputFormat();
  format.setFilePath(args[0]);
  format.setNestedFileEnumeration(true);
  logger.info("Number of splits " + format.getNumSplits());

  // logger.info(Paths.get(".").toAbsolutePath().normalize().toString());

  env.createInput(format, TypeInformation.of(StringValue.class)).print();

Mime
View raw message