spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: SPIP: Executor Plugin (SPARK-24918)
Date Fri, 31 Aug 2018 06:11:34 GMT
I actually had a similar use case a while ago, but not entirely the same.
In my use case, Spark is already up, but I want to make sure all existing
(and new) executors run some specific code. Can we update the API to
support that? I think that's doable if we split the design into two: one is
the ability to do what I just mentioned, and second is the ability to
register via config class when Spark starts to run the code.


On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung <felixcheung_m@hotmail.com>
wrote:

> +1
> ------------------------------
> *From:* Mridul Muralidharan <mridul@gmail.com>
> *Sent:* Wednesday, August 29, 2018 1:27:27 PM
> *To:* dev@spark.apache.org
> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918)
>
> +1
> I left a couple of comments in NiharS's PR, but this is very useful to
> have in spark !
>
> Regards,
> Mridul
> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid
> <irashid@cloudera.com.invalid> wrote:
> >
> > I'd like to propose adding a plugin api for Executors, primarily for
> instrumentation and debugging (
> https://issues.apache.org/jira/browse/SPARK-24918).  The changes are
> small, but as its adding a new api, it might be spip-worthy.  I mentioned
> it as well in a recent email I sent about memory monitoring
> >
> > The spip proposal is here (and attached to the jira as well):
> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing
> >
> > There are already some comments on the jira and pr, and I hope to get
> more thoughts and opinions on it.
> >
> > thanks,
> > Imran
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message