flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Surage <csur...@gmail.com>
Subject Re: Spooling Directory Source Stuck in Exception [Serializer has been closed]
Date Wed, 30 Oct 2013 13:47:33 GMT
I am having the same problem except that I am only copying files one at a
time and this occurs after I copy the first file. My configs are as follows.

1) First box

# Name the compnents of the agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


###############Describe/configure the source#################
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/desktop/flume_test/
a1.sources.r1.deletePolicy = immediate
a1.sources.r1.trackerDir = /home/desktop/.flume/
a1.sources.r1.deserializer.maxLineLength = 10000

# how many events to transfer to the channel at a time
a1.sources.r1.batchSize = 1000
a1.sources.r1.channels = c1

##############describe the sink#######################
# Avro sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = box2
a1.sinks.k1.port = 9313

# How many events to take from the channel at a time
a1.sinks.k1.batch-size = 1000

# Channel the sink connects to
a1.sinks.k1.channel = c1

################describe the channel##################
# use a channel which buffers events in memory
a1.channels.c1.type = memory

# How many events the channel holds
a1.channels.c1.capacity = 10000

# Max number of events to give to sink or take from source
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.byteCapacity = 0


2) Second Box

# Name the compnents of the agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


###############Describe/configure the source#################
a1.sources.r1.type = avro
a1.sources.r1.bind = box2
a1.sources.r1.port = 9313
a1.sources.r1.channels = c1


##############describe the sink#######################
# HDFS sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /user/hive
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.rollSize = 0
a1.sinks.k1.hdfs.rollCount = 5000000
a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.idleTimeout = 30
a1.sinks.k1.hdfs.batchSize = 1000

# Channel the sink connects to
a1.sinks.k1.channel = c1


################describe the channel##################
# use a channel which buffers events in memory
a1.channels.c1.type = memory

# How many events the channel can hold
a1.channels.c1.capacity = 10000

# Max number of events the channel can take from source
# or give to a sink
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.byteCapacity = 0


any help would be appreciated.

Chris

Mime
View raw message