chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: Visualization backend - still MySQL?
Date Tue, 15 Dec 2009 00:44:15 GMT
Hi Jiaqi,

The visualization code is moving from MySQL to REST API + media files
(images, video, etc).  The extra layer of abstraction is for improving data
source connectivities.  My goal is to be able to serve data from HDFS and
cached RAM buffers.  I am writing a SocketDataLoader for demo which enable
streaming of data from collector and direct render json data on the browser.
It will be my pet project for this Christmas.

It is definitely useful to generate static images for large data.  The
management of static images over extensive period of time should be
discussed.  I think it could use something similar to the current demux
output partitioned by:

year/month/day

And store different resolutions at each partition.  Those graphs could be
stored either on disk or HDFS.

Regards,
Eric

On 12/14/09 4:23 PM, "Jiaqi Tan" <tanjiaqi@gmail.com> wrote:

> Hi all,
> 
> Is the visualization code for HICC currently pulling data from MySQL?
> Is there any quick fix to swap that out for a more lightweight
> backend, and/or are there plans from anyone to change that (e.g. serve
> from HDFS) in the near future, and if so, any estimates on a schedule?
> 
> I'm also thinking of having a separate workflow/pipeline of sorts that
> generates static visualizations (e.g. JPGs or PDFs or PNGs) which
> would facilitate off-line post-mortem, would people find that useful?
> 
> Thanks,
> Jiaqi


Mime
View raw message