flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shady Xu <shad...@gmail.com>
Subject write beyond end of stream
Date Sun, 06 Sep 2015 09:27:08 GMT
Hi all,

Have anyone experienced the exception below? I am using LZO compression and
when local data that hasn't been uploaded to HDFS accumulate to size of Gs,
this error happens and Flume can not recover from it.

29 Aug 2015 18:50:37,031 WARN
 [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
(org.apache.flume.sink.hdfs.BucketWriter.append:555)  - Caught IOException
writing to HDFSWriter (write beyond end of stream). Closing file
(/user/log/data.1440845433925.lzo.tmp) and rethrowing exception.
29 Aug 2015 18:50:37,039 INFO
 [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
(org.apache.flume.sink.hdfs.BucketWriter.close:363)  - Closing
/user/log/data.1440845433925.lzo.tmp
29 Aug 2015 18:50:37,064 INFO  [hdfs-sink2-call-runner-3]
(org.apache.flume.sink.hdfs.BucketWriter$8.call:629)  - Renaming
/user/log/data.1440845433925.lzo.tmp to /user/log/data.1440845433925.lzo
29 Aug 2015 18:50:37,069 INFO
 [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
(org.apache.flume.sink.hdfs.HDFSEventSink$1.run:394)  - Writer callback
called.
29 Aug 2015 18:50:37,086 WARN
 [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
(org.apache.flume.sink.hdfs.HDFSEventSink.process:455)  - HDFS IO error
java.io.IOException: write beyond end of stream
        at
com.hadoop.compression.lzo.LzopOutputStream.write(LzopOutputStream.java:134)
        at java.io.OutputStream.write(OutputStream.java:75)
        at
org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
        at
org.apache.flume.sink.hdfs.HDFSCompressedDataStream.append(HDFSCompressedDataStream.java:126)
        at
org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550)
        at
org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547)
        at
org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at
org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at
org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

Mime
View raw message