flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Percy <mpe...@apache.org>
Subject Re: Constant Traffic on port 35872
Date Thu, 17 Jan 2013 03:37:14 GMT
I doubt it's the Thrift RPC layer. It's most likely the app.


On Wed, Jan 16, 2013 at 5:53 PM, James Stewart <James.Stewart@optiver.com.au
> wrote:

> I thought it was only ‘heartbeats’ that were supposed to go via port
> 35872, so I reduced flume.config.heartbeat.period on the nodes to 60 sec.
> According to the master it’s only seeing heartbeats every 60 sec now, and
> yet I still get constantly spammed with data on port 35872 from every node.
> ****
>
> ** **
>
> It does look like metric collection or config reporting of some kind, like
> it’s reporting the configuration of the sources/sinks and even data about
> the JVM:****
>
> ** **
>
> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013...
> rt.vmname...!Java HotSpot(TM) 64-Bit Server
> VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun
> Microsystems Inc.****
>
> ..........max.....@
> ......mem.other.used................mem.other.committed.....s......mem.heap.max...........
> ****
>
> ** **
>
> But it’s just the same data over and over again every second. This traffic
> is travelling across a WAN and with a lot of nodes it’s a significant
> enough amount of data to be a problem.****
>
> ** **
>
> I don’t know much about Java, but could this be something to do with
> Thrift?****
>
> ** **
>
> ** **
>
> *From:* Mike Percy [mailto:mpercy@apache.org]
> *Sent:* Thursday, 17 January 2013 12:19 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Constant Traffic on port 35872****
>
> ** **
>
> I know next to nothing about Flume OG but if I had to guess I'd say it's
> either a heartbeat or metrics collection. Why do you want it to stop?****
>
> ** **
>
> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart <
> James.Stewart@optiver.com.au> wrote:****
>
> Hello all,****
>
>  ****
>
> I’m using flume 0.9.4 – before anybody mentions it, we aren’t in a
> position to upgrade at the moment due to custom decorators + sinks.****
>
>  ****
>
> I’m seeing constant traffic from my various flume nodes back to my master
> on port 35872. Even after increasing my timeout period to 60 sec and
> disabling all custom sources/sinks/decorators, I am still constantly
> receiving packets from all of my nodes back to my master. I have included a
> dump of the tcp packets below – I receive this same traffic from every node
> every 1-2 sec.****
>
>  ****
>
> Does anybody have any idea what is generating this traffic and how I can
> stop it?****
>
>  ****
>
> Many thanks,****
>
>  ****
>
> James****
>
>  ****
>
>  ****
>
> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu
> Jan 01 10:00:00 EST
> 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu
> Jan 17 11:45:44 EST 2013...****
>
> 11:58:22.630881 IP my_physical_node.my_domain.com.53367 > my_master.35872:
> . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp 1456047935
> 3787992397>****
>
> E.....@.<...****
>
> I..****
>
> B...w. ."..CC1L...........****
>
>
> ...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.physicalnode....
> my_physical_node.my_domain.com****
>
> .......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number
> of
> bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber
> of events............****
>
> NullSource....type...****
>
> ...lSource****
>
> ............name....NullSink....type....NullSinksr0202_apps.NullSink****
>
> ............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSource*
> ***
>
> NullSource....type...****
>
> ...lSource****
>
>
> ............name....NullSink....type....NullSinksr0202.aus.optiver.com.NullSink
> ****
>
> ...........(pn-my_logical_node.aus.optiumber of bytes........****
>
> 11:58:22.630891 IP my_physical_node.my_domain.com.53367 > my_master.35872:
> P 333225:333700(475) ack 2613 win 6 <nop,nop,timestamp 1456047935
> 3787992397>****
>
> E.....@.<..c****
>
> I..****
>
> ..w. .".:CC1L...........****
>
> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013...
> rt.vmname...!Java HotSpot(TM) 64-Bit Server
> VM....name...(pn-my_physical_node.my_domain.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun
> Microsystems Inc.****
>
>
> ..........max.....@......mem.other.used................mem.other.committed.....s......mem.heap.max
> ...........****
>
> ** **
> ------------------------------
>
> Information contained in this communication (including any attachments) is
> confidential and may be privileged or subject to copyright. If you have
> received this communication in error you are not authorised to use the
> information in any way and Optiver requests that you notify the sender by
> return email, destroy all copies and delete the information from your
> system. Optiver does not represent, warrant or guarantee that this
> communication is free from computer viruses or other defects or that the
> integrity of this communication has been maintained. Any views expressed in
> this communication are those of the individual sender. Optiver does not
> accept liability for any loss or damage caused directly or indirectly by
> this communication or its use.
>
> Please consider the environment before printing this email.****
>
> ** **
>

Mime
View raw message