hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: 3D Cluster Performance Visualization
Date Fri, 25 Sep 2009 15:09:08 GMT
Brian Bockelman wrote:
> ;) Unfortunately, I'm going to go out on a limb and guess that we don't 
> want to add OpenGL to the dependency list for the namenode...  The viz 
> application actually doesn't depend on the namenode, it uses the datanodes.
> 
> Here's the source:
> svn://t2.unl.edu/brian/HadoopViz/trunk
> 
> The server portion is a bit hardcoded to our site (simply a python 
> server); the client application is pretty cross-platform.  I actually 
> compile and display the application on my Mac.
> 
> Here's how it works:
> 
> 1) Client issues read() request
> 2) Datanode services it.  Logs it with log4j
> 3) One of the log4j appenders is syslog pointing at a separate server
> 4) Separate log server recieves UDP packets; one packet per read()
> 5) Log server parses packets and decides whether they are within the 
> cluster or going to the internet
>   - Currently a Pentium 4 throw-away machine; handles up to 4-5k packets 
> per second before it starts dropping
> 6) Each client opens a TCP stream to the server and receives the 
> transfer type, source, and dest, then renders appropriately
> 
> It's pretty danged close to real-time; the time the client issues the 
> read() request to seeing something plotted is on the order of 1 second.
> 
> I'd really like to see this on a big (Yahoo, Facebook, any takers?) 
> cluster.
> 
> Brian
> 


Ok, so this is really an example of a datacentre back-end for Log4J, 
pushing out UDP packets to something else in the datacentre.  A nice 
side-line to the classic hadoop management displays. Add something about 
jobs executing and you are laughing. Do it all in Java3D and you even 
have cross platformness

Mime
View raw message