spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szilard Nemeth (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs
Date Tue, 13 Feb 2018 09:09:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360904#comment-16360904
] 

Szilard Nemeth edited comment on SPARK-20327 at 2/13/18 9:08 AM:
-----------------------------------------------------------------

Hey [~vanzin]!

I see what you said about compatibility.

By reflection, you meant a similar solution like this? [https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L268]

By the way, isn't it a good way to go forward  if users specify any of these new resource
configs and the Hadoop version Spark currently depends on is 2.x then we handle this case
as an error explicitly and inform the user about this?

In general, is there a conventional way to detect Hadoop version in Spark code?

Thanks!

 


was (Author: snemeth):
Hey [~vanzin]!

I see what you said about compatibility.

By reflection, you meant a similar solution like this? [https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L268]

By the way, isn't it a good way to go forward  if users specify any of these new resource
configs and the Hadoop version Spark currently depends on is 2.x then we handle this case
as an error explicitly and inform the user about this?

In general, is that a conventional way to detect Hadoop version in Spark code?

Thanks!

 

> Add CLI support for YARN custom resources, like GPUs
> ----------------------------------------------------
>
>                 Key: SPARK-20327
>                 URL: https://issues.apache.org/jira/browse/SPARK-20327
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Shell, Spark Submit
>    Affects Versions: 2.1.0
>            Reporter: Daniel Templeton
>            Priority: Major
>              Labels: newbie
>
> YARN-3926 adds the ability for administrators to configure custom resources, like GPUs.
 This JIRA is to add support to Spark for requesting resources other than CPU virtual cores
and memory.  See YARN-3926.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message