spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server
Date Fri, 31 Oct 2014 21:16:33 GMT


Marcelo Vanzin commented on SPARK-1537:

bq. This is proposed to improve the Java libs by adding GET APIs. They are used to query data,
NOT to put data.

Spark needs both to put and read data, otherwise the ATS is useless for Spark. The current
goal of Spark is to use the ATS as a store for its history data, since the data itself is
not considered public and stable itself.

So there is no point in integration if you can only write data. (I know you can read data
through other means, but I don't want to write a custom REST client just to get ATS support

bq.  It is reported for 2.5, and is probably no longer valid after we fixed a bunch of security
issues for 2.6.

I'm not sure why you say it's security-related since there nothing security-related in the
example code I posted. And if something doesn't work in 2.5 but works in 2.6, it means we
(and by that I mean Spark) have to restrict our support to the versions where things work
- even if the underlying API is exactly the same.

> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>                 Key: SPARK-1537
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: YARN
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
> It would be nice to have Spark integrate with Yarn's Application Timeline Server (see
YARN-321, YARN-1530). This would allow users running Spark on Yarn to have a single place
to go for all their history needs, and avoid having to manage a separate service (Spark's
built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch, although
there is still some ongoing work. But the basics are there, and I wouldn't expect them to
change (much) at this point.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message