Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: domain of chen.song.82@gmail.com
 designates 209.85.223.172 as permitted sender)
MIME-Version: 1.0
Date: Tue, 30 Apr 2013 14:00:44 -0400
Message-ID: 
 <CAGF+3rZWJPokfKHtRYoK2Hq4CWKqFsmDSN0dZ3KRieTfA1OqaQ@mail.gmail.com>
Subject: Programmatically write files into HDFS with Flume
From: Chen Song <chen.song.82@gmail.com>
To: user@flume.apache.org
Content-Type: multipart/alternative; boundary=047d7b418e99c120da04db97c716

--047d7b418e99c120da04db97c716
Content-Type: text/plain; charset=ISO-8859-1

I am looking at options in Java programs that can write files into HDFS
with the following requirements.

1) Transaction Support: Each file, when being written, either fully written
successfully or failed totally without any partial file blocks written.

2) Compression Support/File Formats: Can specify compression type or file
format when writing contents.

I know how to write data into a file on HDFS by opening a FSDataOutputStream
 shown here<http://stackoverflow.com/questions/13457934/writing-to-a-file-in-hdfs-in-hadoop>.
Just wondering if there is some libraries of out of the box solutions that
provides the support I mentioned above.

I stumbled upon Flume, which provides HDFS sink that can support
transaction, compression, file rotation, etc. But it doesn't seem to
provide an API to be used as a library. The features Flume provides are
highly coupled with the Flume architectural components, like source,
channel, and sinks and doesn't seem to be usable independently. All I need
is merely on the HDFS loading part.

Does anyone have some good suggestions?

-- 
Chen Song

--047d7b418e99c120da04db97c716
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><p style=3D"clear:both;vertical-align:baseline;line-height=
:18px;font-size:14px;font-family:Arial,&#39;Liberation Sans&#39;,&#39;DejaV=
u Sans&#39;,sans-serif;margin:0px 0px 1em;word-wrap:break-word;border:0px;p=
adding:0px">
I am looking at options in Java programs that can write files into HDFS wit=
h the following requirements.</p><p style=3D"clear:both;vertical-align:base=
line;line-height:18px;font-size:14px;font-family:Arial,&#39;Liberation Sans=
&#39;,&#39;DejaVu Sans&#39;,sans-serif;margin:0px 0px 1em;word-wrap:break-w=
ord;border:0px;padding:0px">
1) Transaction Support: Each file, when being written, either fully written=
 successfully or failed totally without any partial file blocks written.</p=
><p style=3D"clear:both;vertical-align:baseline;line-height:18px;font-size:=
14px;font-family:Arial,&#39;Liberation Sans&#39;,&#39;DejaVu Sans&#39;,sans=
-serif;margin:0px 0px 1em;word-wrap:break-word;border:0px;padding:0px">
2) Compression Support/File Formats: Can specify compression type or file f=
ormat when writing contents.</p><p style=3D"clear:both;vertical-align:basel=
ine;line-height:18px;font-size:14px;font-family:Arial,&#39;Liberation Sans&=
#39;,&#39;DejaVu Sans&#39;,sans-serif;margin:0px 0px 1em;word-wrap:break-wo=
rd;border:0px;padding:0px">
I know how to write data into a file on HDFS by opening a=A0<code style=3D"=
margin:0px;padding:1px 5px;border:0px;font-size:14px;vertical-align:baselin=
e;background-color:rgb(238,238,238);font-family:Consolas,Menlo,Monaco,&#39;=
Lucida Console&#39;,&#39;Liberation Mono&#39;,&#39;DejaVu Sans Mono&#39;,&#=
39;Bitstream Vera Sans Mono&#39;,&#39;Courier New&#39;,monospace,serif">FSD=
ataOutputStream</code>=A0shown=A0<a href=3D"http://stackoverflow.com/questi=
ons/13457934/writing-to-a-file-in-hdfs-in-hadoop" target=3D"_blank" style=
=3D"color:rgb(74,107,130);margin:0px;padding:0px;border:0px;font-size:14px;=
vertical-align:baseline;background-color:transparent;text-decoration:none">=
here</a>. Just wondering if there is some libraries of out of the box solut=
ions that provides the support I mentioned above.</p>
<p style=3D"clear:both;vertical-align:baseline;line-height:18px;font-size:1=
4px;font-family:Arial,&#39;Liberation Sans&#39;,&#39;DejaVu Sans&#39;,sans-=
serif;margin:0px 0px 1em;word-wrap:break-word;border:0px;padding:0px">I stu=
mbled upon Flume, which provides HDFS sink that can support transaction, co=
mpression, file rotation, etc. But it doesn&#39;t seem to provide an API to=
 be used as a library. The features Flume provides are highly coupled with =
the Flume architectural components, like source, channel, and sinks and doe=
sn&#39;t seem to be usable independently. All I need is merely on the HDFS =
loading part.</p>
<p style=3D"clear:both;vertical-align:baseline;line-height:18px;font-size:1=
4px;font-family:Arial,&#39;Liberation Sans&#39;,&#39;DejaVu Sans&#39;,sans-=
serif;margin:0px 0px 1em;word-wrap:break-word;border:0px;padding:0px">Does =
anyone have some good suggestions?</p>
<div><br></div>-- <br>Chen Song<br><div><br></div>
</div>

--047d7b418e99c120da04db97c716--