Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8;
 format=flowed
Content-Transfer-Encoding: 8bit
Date: Thu, 20 Aug 2015 12:34:16 +0200
From: =?UTF-8?Q?Cl=C3=A9ment_MATHIEU?= <clement@unportant.info>
To: user@crunch.apache.org
Subject: Crunch API to run code at JVM startup / shutdown
Message-ID: <5ad34c2c46c9f1c110a6c7520e0af502@webmail.webfaction.com>
User-Agent: Roundcube Webmail/1.1.2

Hi,

I am trying to setup something to automatically profile my Crunch jobs 
on an Hadoop cluster.

I have been a long time user of hprof & "mapred.task.profile" because it 
is so easy to use on Hadoop. However, I am now moving away from it:

  - will be removed from Java 9
  - suffers from safe point bias
  - does not allow to profile native code
  - gathering other metrics than stack trace samples can be useful

I had like to replace hprof by Flight Recorder and/or perf. Unlike 
hprof, both need to be started and stopped programmatically since there 
is not glue for them in Hadoop. I can see three options:


1. Hack the app

It can be done using DoFn.initialize/cleanup. Or all DoFns invoke the 
same idempotent code, or dedicated DoFns are inserted at specific 
points. Both seems horrific and disgusting :)


2. Java agent

Profiling is not tied to Crunch and any tool can be profiled. Main 
drawbacks are that the agent must be deployed on all the nodes and that 
it does not have easy access to metadata like user, job name, stage etc.

A good example of such agent is statsd-jvm-profiler, see 
https://github.com/etsy/statsd-jvm-profiler. They even have a small 
bridge to push Cascading metadata to the agent, see 
https://github.com/etsy/statsd-jvm-profiler/blob/master/example/StatsDProfilerFlowListener.scala.


3. Dedicated Crunch API

Some code needs to be executed on JVM startup / shutdown. AFAIK it is 
not currently possible but could be added (however I am not sure how to 
implement it on Spark). Unlike a javaagent, it does not require to 
deploy something on the nodes, metadata can be pushed to the services 
(ie. ctx) and it is more flexible.


I believe that allowing users to easily run code at JVM startup / 
shutdown would be an useful improvement. Any opinion ?

Clément MATHIEU