flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Constant Flume Error
Date Mon, 31 Aug 2015 03:02:51 GMT
The exception is coming from your de-serializer, from following line

if (i > 1) throw new RuntimeException("WholeFileDeserializer creates
one event for the whole file");

Best is to see what is going on inside in your code using some test cases.

On Sun, Aug 30, 2015 at 7:56 PM, Nikhil Gs <gsnikhil1432010@gmail.com> wrote:
> Hello Team,
>
> The current flume version used in my cluster is Flume 1.5.0-cdh5.4.1. Can
> this uncaught exception be resolved in 1.6 version?
>
> Thanks in advance.
>
> Regards,
> Nik.
>
> On Sun, Aug 30, 2015 at 1:29 PM, Nikhil Gs <gsnikhil1432010@gmail.com>
> wrote:
>>
>> Hello Team,
>>
>> Thank you in advance for your cooperation and time.
>>
>> Facing this issue several times. Tried to restart flume and build the jar.
>> But files are not deleting from the flume spool directory and below is my
>> error and flume config file.
>>
>> Any suggestions would be great helpful. It is urgent because of
>> production.
>>
>>
>> # Please paste flume.conf here. Example:
>>
>> # Sources, channels, and sinks are defined per
>> # agent name, in this case 'pnmtest2'.
>> pnmtest2.sources  = SPOOL
>> pnmtest2.channels = MemChanneltest2
>> pnmtest2.sinks    = AVRO
>>
>> # For each source, channel, and sink, set
>> # standard properties.
>> pnmtest2.sources.SPOOL.type          = spooldir
>> pnmtest2.sources.SPOOL.spoolDir      = /home/a_nikhil.gopishetti/pnm
>> #pnmtest2.sources.SPOOL.spoolDir      =
>> /home/s_sdldalplhdxxxedh/pnmtest2-poll-results
>> #pnmtest2.sources.SPOOL.spoolDir  =
>> /home/s_sdldalplhdxxxedh/pnm-poll-results-uat
>> pnmtest2.sources.SPOOL.ignorePattern = \.*tmp$
>> pnmtest2.sources.SPOOL.channels      = MemChanneltest2
>> pnmtest2.sources.SPOOL.fileHeader    = true
>> pnmtest2.sources.SPOOL.deletePolicy  = immediate
>> pnmtest2.sources.SPOOL.consumeOrder  = oldest
>> pnmtest2.sources.SPOOL.batchSize     = 100
>>
>> pnmtest2.sources.SPOOL.interceptors = time
>> pnmtest2.sources.SPOOL.interceptors.time.type =
>> org.apache.flume.interceptor.TimestampInterceptor$Builder
>> pnmtest2.sources.SPOOL.deserializer  =
>> com.suddenlink.flume.WholeFileDeserializer$Builder
>>
>> pnmtest2.sinks.AVRO.type         = avro
>> pnmtest2.sinks.AVRO.channel      = MemChanneltest2
>> pnmtest2.sinks.AVRO.hostname = sdldalplhdw02.suddenlink.cequel3.com
>> pnmtest2.sinks.AVRO.port     = 40002
>> pnmtest2.sinks.AVRO.batch-size = 100
>> pnmtest2.sinks.AVRO.connect-timeout = 40000
>>
>> #pnmtest2.channels.MemChanneltest1.capacity = 10000
>> #pnmtest2.channels.MemChanneltest1.type   = memory
>>
>> pnmtest2.channels.MemChanneltest2.capacity = 1000000
>> pnmtest2.channels.MemChanneltest2.type   = memory
>>
>>
>> Flume Log Error
>>
>>
>> 12:02:57.363 PMINFOorg.mortbay.log
>>
>> Started SelectChannelConnector@0.0.0.0:41414
>>
>> 12:02:57.621 PMINFOorg.apache.flume.sink.AbstractRpcSink
>>
>> Rpc sink AVRO started.
>>
>> 12:07:30.095 PMERRORorg.apache.flume.source.SpoolDirectorySource
>>
>> FATAL: Spool Directory source SPOOL: { spoolDir:
>> /home/a_nikhil.gopishetti/pnm/ }: Uncaught exception in SpoolDirectorySource
>> thread. Restart or reconfigure Flume to continue processing.
>> java.lang.RuntimeException: WholeFileDeserializer creates one event for
>> the whole file
>> 	at
>> com.suddenlink.flume.WholeFileDeserializer.readEvents(WholeFileDeserializer.java:59)
>> 	at
>> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:252)
>> 	at
>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:228)
>> 	at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> 	at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> 	at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> 	at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:745)
>>
>> 1:12:31.161 PMINFOorg.apache.flume.lifecycle.LifecycleSupervisor
>>
>> Stopping lifecycle supervisor 11
>>
>> 1:12:31.163 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Component type: CHANNEL, name: MemChanneltest2 stopped
>>
>> 1:12:31.163 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: CHANNEL, name: MemChanneltest2.
>> channel.start.time == 1440954177242
>>
>> 1:12:31.163 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: CHANNEL, name: MemChanneltest2.
>> channel.stop.time == 1440958351163
>>
>> 1:12:31.163 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: CHANNEL, name: MemChanneltest2. channel.capacity
>> == 1000000
>>
>> 1:12:31.163 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: CHANNEL, name: MemChanneltest2.
>> channel.current.size == 0
>>
>> 1:12:31.164 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: CHANNEL, name: MemChanneltest2.
>> channel.event.put.attempt == 0
>>
>> 1:12:31.164 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: CHANNEL, name: MemChanneltest2.
>> channel.event.put.success == 0
>>
>> 1:12:31.164 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: CHANNEL, name: MemChanneltest2.
>> channel.event.take.attempt == 523
>>
>> 1:12:31.164 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: CHANNEL, name: MemChanneltest2.
>> channel.event.take.success == 0
>>
>> 1:12:31.164
>> PMINFOorg.apache.flume.node.PollingPropertiesFileConfigurationProvider
>>
>> Configuration provider stopping
>>
>> 1:12:31.164 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Component type: SOURCE, name: SPOOL stopped
>>
>> 1:12:31.164 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. source.start.time ==
>> 1440954177279
>>
>> 1:12:31.164 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. source.stop.time ==
>> 1440958351164
>>
>> 1:12:31.164 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. src.append-batch.accepted
>> == 0
>>
>> 1:12:31.165 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. src.append-batch.received
>> == 0
>>
>> 1:12:31.165 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. src.append.accepted == 0
>>
>> 1:12:31.165 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. src.append.received == 0
>>
>> 1:12:31.165 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. src.events.accepted == 0
>>
>> 1:12:31.165 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. src.events.received == 0
>>
>> 1:12:31.165 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SOURCE, name: SPOOL. src.open-connection.count
>> == 0
>>
>> 1:12:31.165 PMINFOorg.apache.flume.source.SpoolDirectorySource
>>
>> SpoolDir source SPOOL stopped. Metrics:
>> SOURCE:SPOOL{src.events.accepted=0, src.events.received=0,
>> src.append.accepted=0, src.append-batch.accepted=0,
>> src.open-connection.count=0, src.append-batch.received=0,
>> src.append.received=0}
>>
>> 1:12:31.165 PMINFOorg.apache.flume.sink.AbstractRpcSink
>>
>> Rpc sink AVRO stopping...
>>
>> 1:12:31.170 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Component type: SINK, name: AVRO stopped
>>
>> 1:12:31.170 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.start.time ==
>> 1440954177243
>>
>> 1:12:31.170 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.stop.time ==
>> 1440958351170
>>
>> 1:12:31.170 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.batch.complete == 0
>>
>> 1:12:31.170 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.batch.empty == 523
>>
>> 1:12:31.171 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.batch.underflow == 0
>>
>> 1:12:31.171 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.connection.closed.count
>> == 1
>>
>> 1:12:31.171 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.connection.creation.count
>> == 1
>>
>> 1:12:31.171 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.connection.failed.count
>> == 0
>>
>> 1:12:31.171 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.event.drain.attempt == 0
>>
>> 1:12:31.171 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Shutdown Metric for type: SINK, name: AVRO. sink.event.drain.sucess == 0
>>
>> 1:12:31.171 PMINFOorg.apache.flume.sink.AbstractRpcSink
>>
>> Rpc sink AVRO stopped. Metrics: SINK:AVRO{sink.connection.closed.count=1,
>> sink.event.drain.attempt=0, sink.batch.underflow=0,
>> sink.connection.failed.count=0, sink.connection.creation.count=1,
>> sink.event.drain.sucess=0, sink.batch.empty=523, sink.batch.complete=0}
>>
>> 1:12:31.171 PMINFOorg.mortbay.log
>>
>> Stopped SelectChannelConnector@0.0.0.0:41414
>>
>> 1:12:46.490
>> PMINFOorg.apache.flume.node.PollingPropertiesFileConfigurationProvider
>>
>> Configuration provider starting
>>
>> 1:12:46.513
>> PMINFOorg.apache.flume.node.PollingPropertiesFileConfigurationProvider
>>
>> Reloading configuration
>> file:/var/run/cloudera-scm-agent/process/5683-flume-AGENT/flume.conf
>>
>> 1:12:46.520 PMINFOorg.apache.flume.conf.FlumeConfiguration
>>
>> Processing:AVRO
>>
>> 1:12:46.521 PMINFOorg.apache.flume.conf.FlumeConfiguration
>>
>> Processing:AVRO
>>
>> 1:12:46.521 PMINFOorg.apache.flume.conf.FlumeConfiguration
>>
>> Processing:AVRO
>>
>> 1:12:46.521 PMINFOorg.apache.flume.conf.FlumeConfiguration
>>
>> Processing:AVRO
>>
>> 1:12:46.522 PMINFOorg.apache.flume.conf.FlumeConfiguration
>>
>> Added sinks: AVRO Agent: pnmtest2
>>
>> 1:12:46.522 PMINFOorg.apache.flume.conf.FlumeConfiguration
>>
>> Processing:AVRO
>>
>> 1:12:46.522 PMINFOorg.apache.flume.conf.FlumeConfiguration
>>
>> Processing:AVRO
>>
>> 1:12:46.554 PMINFOorg.apache.flume.conf.FlumeConfiguration
>>
>> Post-validation flume configuration contains configuration for agents:
>> [pnmtest2]
>>
>> 1:12:46.554 PMINFOorg.apache.flume.node.AbstractConfigurationProvider
>>
>> Creating channels
>>
>> 1:12:46.564 PMINFOorg.apache.flume.channel.DefaultChannelFactory
>>
>> Creating instance of channel MemChanneltest2 type memory
>>
>> 1:12:46.569 PMINFOorg.apache.flume.node.AbstractConfigurationProvider
>>
>> Created channel MemChanneltest2
>>
>> 1:12:46.570 PMINFOorg.apache.flume.source.DefaultSourceFactory
>>
>> Creating instance of source SPOOL, type spooldir
>>
>> 1:12:46.597 PMINFOorg.apache.flume.sink.DefaultSinkFactory
>>
>> Creating instance of sink: AVRO, type: avro
>>
>> 1:12:46.604 PMINFOorg.apache.flume.sink.AbstractRpcSink
>>
>> Connection reset is set to 0. Will not reset connection to next hop
>>
>> 1:12:46.606 PMINFOorg.apache.flume.node.AbstractConfigurationProvider
>>
>> Channel MemChanneltest2 connected to [SPOOL, AVRO]
>>
>> 1:12:46.615 PMINFOorg.apache.flume.node.Application
>>
>> Starting new configuration:{ sourceRunners:{SPOOL=EventDrivenSourceRunner:
>> { source:Spool Directory source SPOOL: { spoolDir:
>> /home/a_nikhil.gopishetti/pnm } }} sinkRunners:{AVRO=SinkRunner: {
>> policy:org.apache.flume.sink.DefaultSinkProcessor@452d9a56 counterGroup:{
>> name:null counters:{} } }}
>> channels:{MemChanneltest2=org.apache.flume.channel.MemoryChannel{name:
>> MemChanneltest2}} }
>>
>> 1:12:46.615 PMINFOorg.apache.flume.node.Application
>>
>> Starting Channel MemChanneltest2
>>
>> 1:12:46.674 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Monitored counter group for type: CHANNEL, name: MemChanneltest2:
>> Successfully registered new MBean.
>>
>> 1:12:46.675 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Component type: CHANNEL, name: MemChanneltest2 started
>>
>> 1:12:46.675 PMINFOorg.apache.flume.node.Application
>>
>> Starting Sink AVRO
>>
>> 1:12:46.675 PMINFOorg.apache.flume.sink.AbstractRpcSink
>>
>> Starting RpcSink AVRO { host: sdldalplhdw02.suddenlink.cequel3.com, port:
>> 40002 }...
>>
>> 1:12:46.675 PMINFOorg.apache.flume.node.Application
>>
>> Starting Source SPOOL
>>
>> 1:12:46.676 PMINFOorg.apache.flume.source.SpoolDirectorySource
>>
>> SpoolDirectorySource source starting with directory:
>> /home/a_nikhil.gopishetti/pnm
>>
>> 1:12:46.676 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Monitored counter group for type: SINK, name: AVRO: Successfully
>> registered new MBean.
>>
>> 1:12:46.676 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Component type: SINK, name: AVRO started
>>
>> 1:12:46.676 PMINFOorg.apache.flume.sink.AbstractRpcSink
>>
>> Rpc sink AVRO: Building RpcClient with hostname:
>> sdldalplhdw02.suddenlink.cequel3.com, port: 40002
>>
>> 1:12:46.676 PMINFOorg.apache.flume.sink.AvroSink
>>
>> Attempting to create Avro Rpc client.
>>
>> 1:12:46.699 PMWARNorg.apache.flume.api.NettyAvroRpcClient
>>
>> Using default maxIOWorkers
>>
>> 1:12:46.706 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Monitored counter group for type: SOURCE, name: SPOOL: Successfully
>> registered new MBean.
>>
>> 1:12:46.706 PMINFOorg.apache.flume.instrumentation.MonitoredCounterGroup
>>
>> Component type: SOURCE, name: SPOOL started
>>
>> 1:12:46.718 PMINFOorg.mortbay.log
>>
>> Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>>
>> 1:12:46.783 PMINFOorg.mortbay.log
>>
>> jetty-6.1.26.cloudera.4
>>
>> 1:12:46.829 PMINFOorg.mortbay.log
>>
>> Started SelectChannelConnector@0.0.0.0:41414
>>
>> 1:12:47.068 PMINFOorg.apache.flume.sink.AbstractRpcSink
>>
>> Rpc sink AVRO started.
>>
>>
>>
>> Regards,
>> Nik.
>
>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message