incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <>
Subject Re: UDPAdaptor
Date Fri, 30 Dec 2011 05:04:49 GMT

Data is stored in Agent in memory queue.  Agent queues messages if no
collector is available.  The reason that data is out of order in
chukwa/repos because data does not contain a time stamp.  The demux
parser does not know how to sort the given data, hence the data is
stored in random order.  You might be able to improve the order of the
data by modifying the demux parser to use SeqID for ordering to get
original order.  Hope this helps.


On Thu, Dec 29, 2011 at 6:22 PM, 陈镇海 <> wrote:
> Hello,
> I'm using chukwa-0.4.0. The agent and collector are in the same
> machine. When I use UDPAdaptor, I found a problem.
> The initial_adaptor is written "add UDPAdaptor Packets 1234 0". After
> start agent,collector and start_data_processor, I use "nc" to send
> some data to this udp port as followed:
> echo "hello" | nc -u
> echo "world" | nc -u
> echo "this is a test" | nc -u
> echo "good job" | nc -u
> echo "OK" | nc -u
> After it works for a while, I found something was written in HDFS. In
> the directory "/chukwa/dataSinkArchives", I found the data was written
> in correct order. But in the directory "/chukwa/repos", I found the
> data was written in a wrong order as followed:
> ............body this is  a test
> ............body OK
> ............body good job
> ............body hello
> ............body world
> How it happened?
> Another problem,when I keep the agent running and stop the collector,
> I continue to send data to the udp port.After a while,when I start the
> collector,I found the data was not lost.I want to know how and where
> the data is stored.
> Thanks a lot.

View raw message