flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cormier, Christopher" <christopher.corm...@teamaol.com>
Subject RE: Flume/HDFS Encoding
Date Fri, 14 Dec 2012 20:59:18 GMT
Thanks Brock,
When I run it as a DataStream I still get some weird characters between records.

[DATA_HERE]ÿÿÿÿ×ùÎ0ÆÜ9Ig::¬                  ;)
 [DATA_HERE]ÿÿÿÿ×ùÎ0ÆÜ9Ig::¬
                                         ;)
                                           Î[DATA_HERE]ÿÿÿÿ×ùÎ0ÆÜ9Ig::¬
                                                                                         
                                                                                         
                       ;0
                                                                                         
                                                                                         
                         ½[DATA_HERE]ÿÿÿÿ×ùÎ0ÆÜ9Ig::¬
;0
:[DATA_HERE]

I was hoping to avoid the ÿÿÿÿ and spaces (I'm assuming they're characters that are encoded
such that -cat won't show them).

Any thoughts?

Thanks again,

Chris

-----Original Message-----
From: Brock Noland [mailto:brock@cloudera.com] 
Sent: Friday, December 14, 2012 3:52 PM
To: user@flume.apache.org
Subject: Re: Flume/HDFS Encoding

Hi,

On Fri, Dec 14, 2012 at 2:48 PM, Cormier, Christopher <christopher.cormier@teamaol.com>
wrote:
> SEQ!org.apache.hadoop.io.LongWritableorg.apache.hadoop.io.TextY] 
> õpµ^R÷ﳬÕ
>

This is a SequenceFile.

>
> requestToHDFS.sinks.HDFS.hdfs.file.Type = DataStream
>
> # also tried...
>
> #requestToHDFS.sinks.HDFS.hdfs.file.Type = SequenceFile
>

The parameter is hdfs.fileType. See here:

http://flume.apache.org/FlumeUserGuide.html#hdfs-sink

It sounds like you want a text file so you should use DataStream.

Brock

Mime
View raw message