spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From brightsparc <>
Subject Handling expirying state in UDF
Date Mon, 12 Oct 2015 00:14:27 GMT

I have created a python UDF to make an API which requires an expirying OAuth
token which requires refreshing every 600 seconds which is longer than any
given stage.

Due to the nature of threads and local state, if I use a global variable,
the variable goes out of scope regularly. 

I look into using a broadcast variable, but this doesn't support the ability
to expire/refresh the variable.  So I looked into using setLocalProperty and
getLocalProperty on the spark context, but this can't be accessed within a

Is there a recommended way to handle this scenario in PySpark?


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message