spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <>
Subject [jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server
Date Fri, 31 Oct 2014 21:10:33 GMT


Zhijie Shen commented on SPARK-1537:

bq. BTW, if you want a list of things I think are important for Spark, here are some quick

Thanks for sharing the details, which are more helpful to clean up the puzzles than some big
but vague statement. Let me go through the aforementioned Jiras:

* YARN-2521: I'd like to keep it open for some further client improvement, such as local timeline
data caching, while YARN-2673 already made the client retry when the server temporally doesn't
respond. Please note that "I think it's pretty critical when you can't upload your data because
the server is down" is *no longer true* after YARN-2673. On the other side, At the point of
view of the API, it should keep stable.

* YARN-2423: This is proposed to improve the Java libs by adding GET APIs. They are used to
query data, NOT to put data. We do this to help the use case that the developers write Java
code to implement the UI to analyze the timeline data. Framework integration mainly deals
with PUT APIs, and the Java client libs are already there. Take one step back, apart from
the client libs, the RESTful APIs are always there, which is programming language neutral,
and useful to non-Java developers.

* YARN-2444: It's may be a bug or an improper use case. According to the exception, the user
doesn't pass the authorization for some reason. It is reported for 2.5, and is probably no
longer valid after we fixed a bunch of security issues for 2.6. We need to do more validation
for this issue before a conclusion. Anyway, it's obviously an internal issue happening in
secure mode only, which should not the API CHANGES.

bq. I understand it doesn't affect the client API and we can still have the code in,

It seems that we have the agreement that the current timeline service offering is not blocking
the Spark integration work.

> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>                 Key: SPARK-1537
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: YARN
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
> It would be nice to have Spark integrate with Yarn's Application Timeline Server (see
YARN-321, YARN-1530). This would allow users running Spark on Yarn to have a single place
to go for all their history needs, and avoid having to manage a separate service (Spark's
built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch, although
there is still some ongoing work. But the basics are there, and I wouldn't expect them to
change (much) at this point.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message