crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nithin Asokan <anithi...@gmail.com>
Subject Question about Spark Job/Stage names
Date Mon, 28 Sep 2015 21:46:33 GMT
I'm fairly new to Spark, and would like to understand about stage/job names
when using Crunch on Spark. When I submit my Spark application, I see a set
of stage names like *mapToPair at PGroupedTableImpl.java:108. *I would like
to understand if it possible by user code to update these stage names
dynamically? Perhaps, is it possible to have DoFn names as Stage names?

I did a little bit of digging and the closest thing I can find to modify
stage name is using

sparkContext.setCallSite(String)

However, this updates all stage and job names to same text. I tried looking
at MRPipeline's implementation to understand how JobNames are built, and I
believe for SparkPipeline crunch does not create DAG and we don't create a
job name.

But does anyone with Spark expertise know if it's possible in Crunch to
create job/stage names based on DoFn names?

Thank you!
Nithin

Mime
View raw message