spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stavros Kontopoulos (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
Date Fri, 31 Aug 2018 20:33:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599258#comment-16599258
] 

Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:32 PM:
----------------------------------------------------------------------

[~eje] Personally I will just stick to the facts (the ones I am aware of):

1) Several weeks ago I asked if this feature (in one of the  meetings) should go in 2.4 and
you responded that this cannot be the case, as it needs testing etc. I had no objection.
It seems now that Palantir is pushing this for their own reasons and will be marked as experimental.
The PR created is not that big we had time back then even with the old dates of the 2.4 cut.

2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit
comment in this Jira above:

Stavros [3:27 PM]
 @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks,
work more on that after.

eje [5:04 PM]
 @Stavros thanks!

liyinan926 [7:29 PM]
 @Stavros Thanks for working on that!

As you see you agreed that im working on it no?

In any other Jira I have seen before, people just need to state that they are working on something.
They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code
to declare assignment ). 

3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit]
 * Custom YAML
 * Stavros on vacation
 * Complete it without him?
 * Palantir will look at completing

4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK.
Fine.

5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of
things, also in the minutes doc I dont see any decision about who is going to do the PR.

I think the reasonable thing to do is ask what I have done, so people dont do double effort. 

If the whole things looks ok, then fine.


was (Author: skonto):
[~eje] Personally I will just stick to the facts:

1) Several weeks ago I asked if this feature (in one of the  meetings) should go in 2.4 and
you responded that this cannot be the case, as it needs testing etc. I had no objection.
It seems now that Palantir is pushing this for their own reasons and will be marked as experimental.
The PR created is not that big we had time back then even with the old dates of the 2.4 cut.

2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit
comment in this Jira above:

Stavros [3:27 PM]
@liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks,
work more on that after.

eje [5:04 PM]
@Stavros thanks!

liyinan926 [7:29 PM]
@Stavros Thanks for working on that!

As you see you agreed that im working on it no?

In any other Jira I have seen before, people just need to state that they are working on something.
They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code
to declare assignment ). 

3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit]
 * 


 * Custom YAML
 * Stavros on vacation
 * Complete it without him?
 * Palantir will look at completing

4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK.
Fine.

5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of
things, also in the minutes doc I dont see any decision about who is going to do the PR.

I think the reasonable thing to do is ask what I have done, so people dont do double effort. 

If the whole things looks ok, then fine.

> Support user-specified driver and executor pod templates
> --------------------------------------------------------
>
>                 Key: SPARK-24434
>                 URL: https://issues.apache.org/jira/browse/SPARK-24434
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes
>    Affects Versions: 2.4.0
>            Reporter: Yinan Li
>            Priority: Major
>
> With more requests for customizing the driver and executor pods coming, the current approach
of adding new Spark configuration options has some serious drawbacks: 1) it means more Kubernetes
specific configuration options to maintain, and 2) it widens the gap between the declarative
model used by Kubernetes and the configuration model used by Spark. We should start designing
a solution that allows users to specify pod templates as central places for all customization
needs for the driver and executor pods. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message