hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: 3D Cluster Performance Visualization
Date Fri, 25 Sep 2009 15:09:08 GMT
Brian Bockelman wrote:
> ;) Unfortunately, I'm going to go out on a limb and guess that we don't 
> want to add OpenGL to the dependency list for the namenode...  The viz 
> application actually doesn't depend on the namenode, it uses the datanodes.
> Here's the source:
> svn://t2.unl.edu/brian/HadoopViz/trunk
> The server portion is a bit hardcoded to our site (simply a python 
> server); the client application is pretty cross-platform.  I actually 
> compile and display the application on my Mac.
> Here's how it works:
> 1) Client issues read() request
> 2) Datanode services it.  Logs it with log4j
> 3) One of the log4j appenders is syslog pointing at a separate server
> 4) Separate log server recieves UDP packets; one packet per read()
> 5) Log server parses packets and decides whether they are within the 
> cluster or going to the internet
>   - Currently a Pentium 4 throw-away machine; handles up to 4-5k packets 
> per second before it starts dropping
> 6) Each client opens a TCP stream to the server and receives the 
> transfer type, source, and dest, then renders appropriately
> It's pretty danged close to real-time; the time the client issues the 
> read() request to seeing something plotted is on the order of 1 second.
> I'd really like to see this on a big (Yahoo, Facebook, any takers?) 
> cluster.
> Brian

Ok, so this is really an example of a datacentre back-end for Log4J, 
pushing out UDP packets to something else in the datacentre.  A nice 
side-line to the classic hadoop management displays. Add something about 
jobs executing and you are laughing. Do it all in Java3D and you even 
have cross platformness

View raw message