flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "guangtaozhuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-2968) After transfer the entire file it add a ‘\n’ to the file
Date Thu, 11 Aug 2016 02:35:20 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15416407#comment-15416407
] 

guangtaozhuang commented on FLUME-2968:
---------------------------------------

thanks, my config is following:



agent1.sources=source1
agent1.sinks=sink1
agent1.channels=channel1

agent1.sources.source1.channels =channel1
agent1.sources.source1.type = spooldir
agent1.sources.source1.fileSuffix = .COMPLETED
agent1.sources.source1.deletePolicy = never
agent1.sources.source1.spoolDir = /data/log/
agent1.sources.source1.ignorePattern = ^$
agent1.sources.source1.consumeOrder = oldest
agent1.sources.source1.deserializer = org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder
agent1.sources.source1.batchsize = 100
agent1.sources.source1.basenameHeader = true

agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.hdfs.path = hdfs://127.0.0.1:9000/test/
agent1.sinks.sink1.hdfs.fileType = DataStream
agent1.sinks.sink1.hdfs.rollInterval = 600
agent1.sinks.sink1.hdfs.filePrefix = %{basename}
agent1.sinks.sink1.hdfs.useLocalTimeStamp = true
agent1.sinks.sink1.hdfs.idleTimeout = 60000
agent1.sinks.sink1.channel = channel1

agent1.channels = channel1
agent1.channels.channel1.type = file
agent1.channels.channel1.checkpointDir = /data/flume/checkpoint
agent1.channels.channel1.dataDirs  = /data/flume/data
~                                                       

> After  transfer the entire file it add a  ‘\n’ to the file
> ----------------------------------------------------------
>
>                 Key: FLUME-2968
>                 URL: https://issues.apache.org/jira/browse/FLUME-2968
>             Project: Flume
>          Issue Type: Bug
>            Reporter: guangtaozhuang
>
> After the transfer the entire file , add a space
> I want to transfer the entire to hdfs and calculate the MD5. But I found that the source
file and the file in hdfs MD5 are different. I found that the flume add the '\n' to the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message