spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kurian vs <>
Subject [spark-sql] Custom Query Execution listener via conf properties
Date Fri, 16 Feb 2018 08:43:53 GMT

I was trying to create a custom Query execution listener by extending
the  org.apache.spark.sql.util.QueryExecutionListener class. My custom
listener just contains some logging statements. But i do not see those
logging statements when i run a spark job.

Here are the steps that i did:

   1. Create a custom listener by extending the QueryExecutionLIstener class
   2. Created a jar file for the above project
   3. Edited spark-defaults.conf to add the following properties:
   com.customListener.spark.customSparkListener spark.driver.extraClassPath
   4. Restarted everything using SPARK-HOME/sbin/
   5. Ran a sample job using spark-submit

I do not see any of the logging statements from the custom listener being
printed[i don't see them in the console]

   - Is there anything else that i need to do to make it work other than
   the above steps?
   - I'm adding this in the config properties because i need some info from
   all the spark jobs being executed on that cluster. My assumption is that
   this will prevent the need to do it from the code by adding an extra
   .register(customListener)  line. Is this assumption correct?
   - From which version of spark is this supported? (i'm using spark V

Can someone point me in the right direction?

View raw message