incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eran Kutner <e...@gigya.com>
Subject Re: Collector going crazy...
Date Wed, 10 Aug 2011 14:34:29 GMT
Great!
By the way, I've opened FLUME-734 about this issue, it has some more log
data there.

Thanks.

-eran



On Wed, Aug 10, 2011 at 16:55, Jonathan Hsieh <jon@cloudera.com> wrote:

> Eran,
>
> I've dug into this and I think I've found the root cause of this bug.  I'll
> try to patch this today.
>
> Jon.
>
>
> On Mon, Aug 8, 2011 at 7:41 AM, Eran Kutner <eran@gigya.com> wrote:
>
>> Hi,
>> I have 3 collectors running, all with this configuration:
>> collectorSource(54001) | collector(600000) {
>> escapedFormatDfs("hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/",
>> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec")) }
>>
>> Right now tall three are going crazy, opening multiple HDFS files per
>> second. All failing with this error: "OutputFormat instance can only write
>> to the same OutputStream".
>> This configuration used to work fine, for a few days, but I've seen this
>> wild behavior happen a few times before. Usually restarting the service was
>> enough to stop it. Now that doesn't seem to help.
>>
>> The logs look like this:
>>
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Creating HDFS file:
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy.tmp
>> 11/08/08 10:33:12 INFO compress.CodecPool: Got brand-new compressor
>> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Opening
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Creating HDFS file:
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy.tmp
>> 11/08/08 10:33:12 INFO debug.StubbornAppendSink: append failed on event
>> 'WEB514 [INFO Sun Aug 07 09:51:43 EDT 2011] { City :  } { CookieID :  } {
>> Country : US } { Date : (long)-721216654471344504  (string) '??????F #??'
>> (double)-2.2850700827524182E260 } { ErrCode : 0 '' } { Event : 637534208 '&'
>> } { GigyaUID :  } { IP : 75.107.140.48 } { IPNum :
>> (long)5434591513138954240  (string) 'Kk???0' (double)2.110829276777375E55 }
>> { Info :  } { PartnerData :  } { PartnerData_hash : (long)0  (string) ''
>> (double)0.0 } { PartnerID : (long)8768254487304339456  (string) 'y??? '
>> (double)1.3781410651737394E278 } { ProviderUID :  } { Referrer :
>> http://www.heavy.com/action/girls/2011/08/the-20-hottest-photos-of-jessica-jane-clement/}
{ ReqID : d3b9c9f27a1848919003e25268b90c34 } { RetryCount : 0 '' } {
>> RowType : GSEvent } { Server : WEB514 } { SiteAccountID :  } {
>> SiteAccountSNID : 0 '' } { State :  } { Status : 50331648 ' ' } { TotalTime
>> : 0 '' } { Type : js } { UCID : tSUUKaA13PGc62n9R9xn4PbqBwHxfeeuV15smIXotZY=
>> } { UUID :  } { UserAge : 48 } { UserGender : 0 '' } { Val : 0 '' } { dbTime
>> : 0 '' } { log :  } { memcacheTime : 0 '' } { provider : 0 '' } { request :
>> } { siteID : (long)8919959604146733056  (string) '{??? '
>> (double)1.9842753626727537E288 } { source : 1315925605 'None' } ' with
>> error: OutputFormat instance can only write to the same OutputStream
>> 11/08/08 10:33:12 INFO rolling.RollSink: closing RollSink
>> 'escapedFormatDfs( "hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/",
>> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec") )'
>> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Closing
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Closing HDFS file:
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy.tmp
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: done writing raw file to hdfs
>> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Closing
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Closing HDFS file:
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312089-0400.5436285984294774.00000040-col1.snappy.tmp
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: done writing raw file to hdfs
>> 11/08/08 10:33:12 INFO rolling.RollSink: opening RollSink
>> 'escapedFormatDfs( "hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/",
>> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec") )'
>> 11/08/08 10:33:12 INFO debug.InsistentOpenDecorator: Opened MaskDecorator
>> on try 0
>> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Opening
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Creating HDFS file:
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-07/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy.tmp
>> 11/08/08 10:33:12 INFO compress.CodecPool: Got brand-new compressor
>> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Opening
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Creating HDFS file:
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy.tmp
>> 11/08/08 10:33:12 INFO debug.StubbornAppendSink: append failed on event
>> 'WEB204 [INFO Mon Aug 08 10:13:54 EDT 2011] { City :  } { CookieID :  } {
>> Country : DE } { Date : (long)6591305623858826888  (string) '[y??????#??'
>> (double)4.440113505901932E132 } { IP : 93.231.131.246 } { IPNum :
>> (long)6766522057732653056  (string) ']??????' (double)2.2940434972655672E144
>> } { Referrer :  } { RowType : GSLoad } { Server : WEB204 } { State :  } {
>> UCID : Jyg2gzYokIn3id2wDfv71Q== } { isHTTPS : 48 } { partnerID :
>> (long)-7147478690450898944  (string) '??????' (double)-6.42864763300399E-170
>> } { siteID : (long)5554354511524397056  (string) 'M'
>> (double)2.162938607572601E63 } ' with error: OutputFormat instance can only
>> write to the same OutputStream
>> 11/08/08 10:33:12 INFO rolling.RollSink: closing RollSink
>> 'escapedFormatDfs( "hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/",
>> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec") )'
>> 11/08/08 10:33:12 INFO hdfs.EscapedCustomDfsSink: Closing
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: Closing HDFS file:
>> hdfs://hadoop1-m1:8020/raw-events/2011-08-08/events-20110808-103312478-0400.5436286373283774.00000040-col1.snappy.tmp
>> 11/08/08 10:33:12 INFO hdfs.CustomDfsSink: done writing raw file to hdfs
>>
>>
>>
>>
>>
>> -eran
>>
>>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
>

Mime
View raw message