flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <Ni...@basjes.nl>
Subject Checking for existance of output directory/files before running a batch job
Date Thu, 18 Aug 2016 12:02:34 GMT

I have a batch job that I run on yarn that creates files in HDFS.
I want to avoid running this job at all if the output already exists.

So in my code (before submitting the job into yarn-session) I do this:

    String directory = "foo";

    Path directory = new Path(directoryName);    FileSystem fs =
    if (!fs.exists(directory)) {

        // run the job


What I found is that this code apparently checks the 'wrong' file
system. (I always get 'false' even if it exists in hdfs)

I checked the API of the execution environment yet I was unable to get
the 'correct' filesystem from there.

What is the proper way to check this?

Best regards / Met vriendelijke groeten,

Niels Basjes

View raw message