hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richards Peter <hbkricha...@gmail.com>
Subject Using apache commons-vfs to read files from hdfs
Date Thu, 31 Jul 2014 08:32:19 GMT
Hi,

I am having a usecase to read files from hdfs and local file system
depending on a configuration parameter. I found that apache commons-vfs
supports various file systems and the latest developer release has an
implementation for hdfs also (though only read support is provided
currently). I find commons-vfs really useful to access txt files from hdfs
and local file system.

However I am not able to access RC Files because commons-vfs exposes only
java.io.InputStream and java.io.OutputStream on FileContent interface to
create readers and writers:
http://commons.apache.org/proper/commons-vfs/apidocs/index.html?org/apache/commons/vfs2/FileContent.html

Since the constructors of RCFile.Reader and RCFile.Writer classes do not
accept any InputStreams or OutputStreams, I am not able to access such
files using commons-vfs.

Is it possible to have constructors accepting FSDataInputStream and
FSDataOutputStream as arguments in these classes?

Is there a better way to access files(of any format) from hdfs and local
file system using a common api?

Thanks,
Richards.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message