Israel,=

<= /p>

They are text files and I= did exactly as you said, I unzipped the file use ‘GZIPInputStream= 217; and created an event for the full content of the file and compressed it as part of the sink. This work around is fine but there maybe cases tha= t you are forced to keep the original format and you just want to channel t= he data and not unzip it and avoid the extra headers.

<= /p>

Thanks<= /p>

Peyan

<= /p>

From: Israel E= kpo [mailto:israel@aicer.org]
Sent: Thursday, July 18, 2013 7:00 PM
To: user@flume.apache.org
Subject: Re: HDF Sink Additional Bytes added for File Events

Peyman,

I would like to understand the original types for th= e gzip files your custom source is consuming. Are these binary files or tex= t files before they are compressed?

Is the entire file a single event, or does it contai= n delimiters that mark where one event ends and another one starts?

You may be able to get around consuming the gzip fil= es by decompressing it first before reading it.

This way, the uncompressed bytes are not corrupted i= f additional data gets appended to the event body or headers.

Here are some tools that could help:

http://docs.oracle.com/javase/6/docs/ap= i/java/util/zip/GZIPInputStream.html

http://docs.oracle.com/javase/6/docs/api= /java/util/zip/ZipInputStream.html

http://commons.apache.org/proper/commons-compress/apidocs/org/apac= he/commons/compress/compressors/gzip/package-summary.html

What do you think about this direction?

Author and Instructor for the Upcoming Book and L= ecture Series

Massive Log Data Aggregation, Processing, Searchi= ng and Visualization with Open Source Software

http://massivelogdata.com

On 12 July 2013 19:36, Peyman Mohajerian <Peyman.Mohajerian@f= ox.com> wrote:

Hi Guys,<= /o:p>

I have a custom sourc= e and consuming whole ‘gz’ files as byte arrays and each file i= s a single event. I’d like to write the file to HDFS. During the writ= e some additional bytes are added and therefore file is corrupted, not able to unzip it any more. I know this is not a good use= case for Flume but I’d like to keep a consistent data collection desi= gn and was hoping I could pass full gz files to HDFS without the file being= corrupted. Either the ‘timestamp’ header is causing issue or the ‘text’ file format, but I’m not = sure. Any solution?

Thanks,

Peyman

XXX.sources =3D xx= x

XXX.channels =3D MemC= hannel

XXX.sinks =3D HDFS

XXX.sources.xxx.type = =3D com.xxx.xxx.xxx.Source

XXX.sources.xxx.chann= els =3D MemChannel

XXX.sinks.HDFS.channe= l =3D MemChannel

XXX.sinks.HDFS.type = =3D hdfs

XXX.sinks.HDFS.hdfs.p= ath =3D hdfs://xxxx/user/xxx/xxx/gzfiles/%Y/%m/%d/

XXX.sinks.HDFS.hdfs.f= ileType =3D DataStream

XXX.sinks.HDFS.hdfs.f= ilePrefix =3D xxxx

XXX.sinks.HDFS.hdfs.b= atchSize =3D 1

XXX.sinks.HDFS.hdfs.r= ollSize =3D 0

XXX.sinks.HDFS.hdfs.i= dleTimeout =3D 3

XXX.sinks.HDFS.hdfs.r= ollInterval =3D 0

XXX.sinks.HDFS.hdfs.r= ollCount =3D 1

XXX.channels.MemChann= el.type =3D memory

XXX.channels.MemChann= el.capacity =3D 1

XXX.channels.MemChann= el.transactionCapacity =3D 1

XXX.channels= .MemChannel.byteCapacityBufferPercentag =3D 100

InputStream in =3D To= olbox.inputStreamUrlConnection(url, account.getAuth1(), account.getAuth2()); = ;

outputStream =3D <= span style=3D"color:#7F0055">new ByteArrayOutputStream();=

byte= [] buf =3D new byte[1024]; // optimize the size of buffer to your need

int<= /span> num;

whil= e ((num = =3D in.read(buf)) !=3D -1) {

&nb= sp; outputStream.write(buf, 0, num);

}

headers.put("timestamp", String.valueOf(new Date().getTime()));

Event e =3D EventBuil= der.withBody(outputStream.toByteArray(), headers);=

getChannelPr= ocessor().processEvent(e);