Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Date: Tue, 26 Feb 2013 19:30:41 +0100
From: Matthias Friedrich <matt@mafr.de>
To: dev@crunch.apache.org
Subject: Re: About status web page
Message-ID: <20130226183041.GA18293@mafr.de>
References: <BLU0-SMTP12513C2A3C2A7C5E15B6E89ADFC0@phx.gbl>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BLU0-SMTP12513C2A3C2A7C5E15B6E89ADFC0@phx.gbl>
User-Agent: Mutt/1.5.21 (2010-09-15)

Hi Chao,

sounds interesting - just a couple of things that come to mind:

I this intended as debugging aid or for operational monitoring?

A Crunch job is a temporary thing, to me this doesn't sound like a
good match for a web service because it disappears after a (possibly
short) time. Also, when multiple jobs are executed concurrently from
the same machine, you can't work with a well-known port, you'd have to
pick an unused port for each job.

It also looks to me like this has security implications? Right now,
Crunch is just a client library and we're part of Hadoop's security
framework. A web service we might have to secure in some way.

Regards,
  Matthias

On Tuesday, 2013-02-26, Chao Shi wrote:
> Hi Crunch Devs,
> 
> I'm interested in adding a web status page to crunch. I'm working on a
> prototype first, which simply runs a jetty server and renders the dot file
> produced by DotFileWriter at browser. The dot rendering work is done by
> viz.js <https://github.com/mdaines/viz.js>. It can successfully render the
> plan into SVG.
> 
> I think there are 2 issues I hit with viz.js:
> 
> 1. The license of viz.js is unclear. It is compiled from GraphViz source
> code with emscripten. GraphViz is Eclipse Public License 1.0.
> 
> 2. viz.js is big and slow. It is a 1.4MB compressed JS. It takes 1 or 2
> seconds on my laptop to render my pipeline (30+ MRs). I think it good to
> have the graph refresh frequently and show the running status of the
> pipeline (i.e. whether MRs are done or not). Thus the rendering time would
> be too slow.
> 
> Another approach is to call graphviz command at server side, if viz.js is
> not possible. I can't find any pure Java implementation of graphviz.
> 
> Looking forward to your advices.
> 
> Thanks,
> Chao