hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andreina j <andrein...@huawei.com>
Subject RE: [HDFS] Programatic way to determine is a file is under construction?
Date Fri, 30 Oct 2015 03:32:45 GMT
Hi Aaron,

    DistributedFileSystem dfs = (DistributedFileSystem) FileSystem.get(conf);
    Path file=new Path("/foo/bar");
boolean fileClosed = dfs.isFileClosed(file);

will help you in finding whether the files is open/closed.

Andreina J.

From: Aaron.Dossett [mailto:Aaron.Dossett@target.com]
Sent: Friday, October 30, 2015 9:54 AM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: [HDFS] Programatic way to determine is a file is under construction?


I'm looking for a programatic way to determine if an HDFS file is open for writing (or under
construction).  `hadoop fsck` has options to show files that are open, but looking through
the HDFS source code none of the methods and classes used to determine if a file is open are
part of the public HDFS interface (for example FSDirectory has method to retrieve an INodeFile
for a given path, which can tell you if a file is under construction - but there is no way
to instantiate an FDDirectory object outside of the HDFS package).

In a perfect world, a public method like FileStatus.isUnderConstruction() would exist... are
there any alternatives?

Thanks in advance for any advice! -Aaron

View raw message