flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johny Rufus <jru...@cloudera.com>
Subject Re: Urgent help: Flume Uncaught Exception
Date Wed, 22 Jul 2015 17:10:19 GMT
Sure, then we can confirm that FLUME-1934 is the issue, and upgrading to
flume-1.6 should take care of it.

Thanks,
Rufus

On Wed, Jul 22, 2015 at 10:02 AM, Nikhil Gs <gsnikhil1432010@gmail.com>
wrote:

> Yes Johny. I can see some of the files with 0 size.
>
> Thanks!
>
> Regards,
> Nikhil Gopishetti Sadanand.
> Illinois, USA.
> 857-253-9553.
>
> On Wed, Jul 22, 2015 at 11:55 AM, Johny Rufus <jrufus@cloudera.com> wrote:
>
>> Can you confirm one more thing, do you have some files in the spool
>> directory that are 0 bytes ?
>>
>> Thanks,
>> Rufus
>>
>> On Wed, Jul 22, 2015 at 9:48 AM, Nikhil Gs <gsnikhil1432010@gmail.com>
>> wrote:
>>
>>> We are deleting the files from the spool directory only after
>>> consumption of files. So now, if I upgrade my Flume to 1.6... Will it be
>>> solved?
>>>
>>> Regards,
>>> Nikhil Gopishetti Sadanand.
>>> Illinois, USA.
>>> 857-253-9553.
>>>
>>> On Wed, Jul 22, 2015 at 10:35 AM, Johny Rufus <jrufus@cloudera.com>
>>> wrote:
>>>
>>>>  [image: Boxbe] <https://www.boxbe.com/overview> This message is
>>>> eligible for Automatic Cleanup! (jrufus@cloudera.com) Add cleanup rule
>>>> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Ftoken%3DI1AGGNBeRixnJqSFO%252B2Azw4T5PQV5DpBzTm%252FcvBKi8os7xt4EueN0aTFQl4EnHRGPzdDVz88BqNDmQT1cyrwKbpunPRBSVULGIrYk%252FyYGv1CV8Y7f0WSZ8HWozaRjKymoKYMBGjt3%252FY%253D%26key%3DBdlCAxP8uw46NdLhIif5DMc4DAtRhSsTF2T7Hg85pEo%253D&tc_serial=22052737395&tc_rand=1998199434&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>>>> | More info
>>>> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=22052737395&tc_rand=1998199434&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>>>>
>>>> A couple of things.
>>>> 1) You should not delete the file while the file is still being read
>>>> from. This could lead to the IlegalStateException
>>>> 2) As you pointed out, you could end up with this issue when 0 byte
>>>> files are placed in the spool directory and if the fix for FLUME-1934 is
>>>> not there.
>>>>
>>>> Thanks,
>>>> Rufus
>>>>
>>>> On Wed, Jul 22, 2015 at 8:25 AM, Nikhil Gs <gsnikhil1432010@gmail.com>
>>>> wrote:
>>>>
>>>>> Deleting the file after placed in the spooling directory.
>>>>>
>>>>> Is this because of my flume version i.e. Flume 1.5.0-cdh5.4.1
>>>>> Because, I have noticed the same issue that has been resolved in Flume
>>>>> 1.6.
>>>>> https://issues.apache.org/jira/browse/FLUME-1934
>>>>>
>>>>> Thanks,
>>>>> Flume User.
>>>>>
>>>>>
>>>>> On Wed, Jul 22, 2015 at 10:16 AM, Johny Rufus <jrufus@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Are you renaming or deleting the file that has been placed in the
>>>>>> spooling directory ?
>>>>>>
>>>>>> Thanks,
>>>>>> Rufus
>>>>>>
>>>>>> On Wed, Jul 22, 2015 at 6:41 AM, Nikhil Gs <gsnikhil1432010@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hello Everyone,
>>>>>>>
>>>>>>> Facing a problem with flume spool.
>>>>>>> Below is my configuration,
>>>>>>>
>>>>>>> # Please paste flume.conf here. Example:
>>>>>>>
>>>>>>> # Sources, channels, and sinks are defined per
>>>>>>> # agent name, in this case 'pnm'.
>>>>>>> pnm.sources  = SPOOL
>>>>>>> pnm.channels = MemChannel
>>>>>>> pnm.sinks    = AVRO
>>>>>>>
>>>>>>> # For each source, channel, and sink, set
>>>>>>> # standard properties.
>>>>>>> pnm.sources.SPOOL.type          = spooldir
>>>>>>> pnm.sources.SPOOL.spoolDir      =
>>>>>>> /home/s_sdldalplhdxxxedh/pnm-poll-results
>>>>>>> pnm.sources.SPOOL.channels      = MemChannel MemChannel2
>>>>>>> pnm.sources.SPOOL.fileHeader    = true
>>>>>>> pnm.sources.SPOOL.deletePolicy  = immediate
>>>>>>> pnm.sources.SPOOL.consumeOrder  = oldest
>>>>>>> pnm.sources.SPOOL.batchSize     = 1
>>>>>>>
>>>>>>> pnm.sources.SPOOL.interceptors = time
>>>>>>> pnm.sources.SPOOL.interceptors.time.type =
>>>>>>> org.apache.flume.interceptor.TimestampInterceptor$Builder
>>>>>>> pnm.sources.SPOOL.deserializer  =
>>>>>>> com.suddenlink.flume.WholeFileDeserializer$Builder
>>>>>>>
>>>>>>> pnm.sinks.AVRO.type         = avro
>>>>>>> pnm.sinks.AVRO.channel      = MemChannel
>>>>>>> pnm.sinks.AVRO.hostname = sdldalplhdw01.suddenlink.cequel3.com
>>>>>>> pnm.sinks.AVRO.port     = 40001
>>>>>>> pnm.sinks.AVRO.batchSize = 1
>>>>>>> pnm.sinks.AVRO.connect-timeout = 40000
>>>>>>>
>>>>>>>
>>>>>>> # pnm.sinks.HDFS.type         = hdfs
>>>>>>> # pnm.sinks.HDFS.channel      = MemChannel2
>>>>>>> # pnm.sinks.HDFS.hdfs.path = /user/flume/poll/%Y/%m/%d/%H/
>>>>>>> # pnm.sinks.HDFS.hdfs.fileType = DataStream
>>>>>>> # pnm.sinks.HDFS.hdfs.writeFormat = Text
>>>>>>> # pnm.sinks.HDFS.hdfs.batchSize = 100
>>>>>>> # pnm.sinks.HDFS.hdfs.rollSize = 0
>>>>>>> # pnm.sinks.HDFS.hdfs.rollCount = 1000
>>>>>>> # pnm.sinks.HDFS.hdfs.rollInterval = 600
>>>>>>>
>>>>>>> # Other properties are specific to each type of
>>>>>>> # source, channel, or sink. In this case, we
>>>>>>> # specify the capacity of the memory channel.
>>>>>>>
>>>>>>> pnm.channels.MemChannel.capacity = 1000000
>>>>>>> pnm.channels.MemChannel.type   = memory
>>>>>>>
>>>>>>> # pnm.channels.MemChannel2.capacity = 10000
>>>>>>> # pnm.channels.MemChannel2.type   = memory
>>>>>>>
>>>>>>>
>>>>>>> *Error Log file*
>>>>>>>
>>>>>>> org.apache.flume.source.SpoolDirectorySource
>>>>>>> FATAL: Spool Directory source SPOOL: { spoolDir:
>>>>>>> /home/s_sdldalplhdxxxedh/pnm-poll-results }: Uncaught exception
in
>>>>>>> SpoolDirectorySource thread. Restart or reconfigure Flume to
continue
>>>>>>> processing.
>>>>>>> java.lang.IllegalStateException: File should not roll when commit
is
>>>>>>> outstanding.
>>>>>>> at
>>>>>>> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:235)
>>>>>>> at
>>>>>>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:228)
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> I have tried to restart the agent, but the same error is appearing.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Nikhil
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message