camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sorokin, Alexey" <a.soro...@cleverdata.ru>
Subject HDFS2 Component and NORMAL_FILE type
Date Mon, 23 Mar 2015 14:31:43 GMT
Hi, all!

I'm looking at ways to use hdfs2 component to read files stored in a Hadoop
directory. As a quite new Hadoop user I assume that simplest way is when
data is stored in normal file format.

I was looking at code in
'org.apache.camel.component.hdfs2.HdfsFileType#NORMAL_FILE' class that is
responsible for creating the input stream and noticed that it will copy the
whole file to the local file system (in temp file) before opening input
stream (the case when using 'hdfs://' URI).

I wonder what is the reason behind this? Isn't it possible that file can be
very large and then this operation will be quite costly? Maybe I missing
some basic restrictions on using normal files in Hadoop?

Thanks in advance
Alexey

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message