chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈镇海 <>
Subject Re: UDPAdaptor
Date Fri, 30 Dec 2011 06:38:36 GMT
Hi Eric,
When no collector is available,data is stored in memory queue.In this
case , if the amount of data is large and the memory size is limited.
Will it be "out of memory"  and whether the data will be lost?

2011/12/30 Eric Yang <>:
> Hi,
> Data is stored in Agent in memory queue.  Agent queues messages if no
> collector is available.  The reason that data is out of order in
> chukwa/repos because data does not contain a time stamp.  The demux
> parser does not know how to sort the given data, hence the data is
> stored in random order.  You might be able to improve the order of the
> data by modifying the demux parser to use SeqID for ordering to get
> original order.  Hope this helps.
> regards,
> Eric
> On Thu, Dec 29, 2011 at 6:22 PM, 陈镇海 <> wrote:
>> Hello,
>> I'm using chukwa-0.4.0. The agent and collector are in the same
>> machine. When I use UDPAdaptor, I found a problem.
>> The initial_adaptor is written "add UDPAdaptor Packets 1234 0". After
>> start agent,collector and start_data_processor, I use "nc" to send
>> some data to this udp port as followed:
>> echo "hello" | nc -u
>> echo "world" | nc -u
>> echo "this is a test" | nc -u
>> echo "good job" | nc -u
>> echo "OK" | nc -u
>> After it works for a while, I found something was written in HDFS. In
>> the directory "/chukwa/dataSinkArchives", I found the data was written
>> in correct order. But in the directory "/chukwa/repos", I found the
>> data was written in a wrong order as followed:
>> ............body this is  a test
>> ............body OK
>> ............body good job
>> ............body hello
>> ............body world
>> How it happened?
>> Another problem,when I keep the agent running and stop the collector,
>> I continue to send data to the udp port.After a while,when I start the
>> collector,I found the data was not lost.I want to know how and where
>> the data is stored.
>> Thanks a lot.

View raw message