flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikhil Gs <gsnikhil1432...@gmail.com>
Subject Urgent help: Flume Uncaught Exception
Date Wed, 22 Jul 2015 13:41:17 GMT
Hello Everyone,

Facing a problem with flume spool.
Below is my configuration,

# Please paste flume.conf here. Example:

# Sources, channels, and sinks are defined per
# agent name, in this case 'pnm'.
pnm.sources  = SPOOL
pnm.channels = MemChannel
pnm.sinks    = AVRO

# For each source, channel, and sink, set
# standard properties.
pnm.sources.SPOOL.type          = spooldir
pnm.sources.SPOOL.spoolDir      = /home/s_sdldalplhdxxxedh/pnm-poll-results
pnm.sources.SPOOL.channels      = MemChannel MemChannel2
pnm.sources.SPOOL.fileHeader    = true
pnm.sources.SPOOL.deletePolicy  = immediate
pnm.sources.SPOOL.consumeOrder  = oldest
pnm.sources.SPOOL.batchSize     = 1

pnm.sources.SPOOL.interceptors = time
pnm.sources.SPOOL.interceptors.time.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
pnm.sources.SPOOL.deserializer  =
com.suddenlink.flume.WholeFileDeserializer$Builder

pnm.sinks.AVRO.type         = avro
pnm.sinks.AVRO.channel      = MemChannel
pnm.sinks.AVRO.hostname = sdldalplhdw01.suddenlink.cequel3.com
pnm.sinks.AVRO.port     = 40001
pnm.sinks.AVRO.batchSize = 1
pnm.sinks.AVRO.connect-timeout = 40000


# pnm.sinks.HDFS.type         = hdfs
# pnm.sinks.HDFS.channel      = MemChannel2
# pnm.sinks.HDFS.hdfs.path = /user/flume/poll/%Y/%m/%d/%H/
# pnm.sinks.HDFS.hdfs.fileType = DataStream
# pnm.sinks.HDFS.hdfs.writeFormat = Text
# pnm.sinks.HDFS.hdfs.batchSize = 100
# pnm.sinks.HDFS.hdfs.rollSize = 0
# pnm.sinks.HDFS.hdfs.rollCount = 1000
# pnm.sinks.HDFS.hdfs.rollInterval = 600

# Other properties are specific to each type of
# source, channel, or sink. In this case, we
# specify the capacity of the memory channel.

pnm.channels.MemChannel.capacity = 1000000
pnm.channels.MemChannel.type   = memory

# pnm.channels.MemChannel2.capacity = 10000
# pnm.channels.MemChannel2.type   = memory


*Error Log file*

org.apache.flume.source.SpoolDirectorySource
FATAL: Spool Directory source SPOOL: { spoolDir:
/home/s_sdldalplhdxxxedh/pnm-poll-results }: Uncaught exception in
SpoolDirectorySource thread. Restart or reconfigure Flume to continue
processing.
java.lang.IllegalStateException: File should not roll when commit is
outstanding.
at
org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:235)
at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:228)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

I have tried to restart the agent, but the same error is appearing.

Regards,
Nikhil

Mime
View raw message