spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pillis Work <pillis.w...@gmail.com>
Subject Re: About Spark job web ui persist(JIRA-969)
Date Mon, 13 Jan 2014 22:43:52 GMT
My primary concern was having too many watches for history.
But I do not have an argument against stand-alone mode.

Also questions about automatic lifecycle management of history server -
when to start / stop etc. in a cluster environment would be difficult when
one is not the cluster manager.

I agree that JSON files are not easily readable.
Maybe a stand-alone history server which makes it implicit that it is only
for that host would be fine.





On Mon, Jan 13, 2014 at 1:32 PM, Patrick Wendell <pwendell@gmail.com> wrote:

> Pillis - I agree we need to decouple the representation from a
> particular history server. But why not provide as simple history
> server people can (optionally) run if they aren't using Yarn or Mesos?
> For people running the standalone cluster scheduler this seems
> important. Giving them only a JSON dump isn't super consumable for
> most users.
>
> - Patrick
>
> On Mon, Jan 13, 2014 at 10:43 AM, Pillis Work <pillis.work@gmail.com>
> wrote:
> > The listeners in SparkUI which update the counters can trigger saves
> along
> > the way.
> > The save can be on a 500ms delay after the last update, to batch changes.
> > This solution would not require save on stop().
> >
> >
> >
> > On Mon, Jan 13, 2014 at 6:15 AM, Tom Graves <tgraves_cs@yahoo.com>
> wrote:
> >
> >> So the downside to just saving stuff at the end is that if the app
> crashes
> >> or exits badly you don't have anything.   Hadoop has taken the approach
> of
> >> saving events along the way.  But Hadoop also uses that history file to
> >> start where it left off at if something bad happens and it gets
> restarted.
> >>  I don't think the latter really applies to spark though.
> >>
> >> Does mesos have a history server?
> >>
> >> Tom
> >>
> >>
> >>
> >> On Sunday, January 12, 2014 9:22 PM, Pillis Work <pillis.work@gmail.com
> >
> >> wrote:
> >>
> >> IMHO from a pure Spark standpoint, I don't know if having a dedicated
> >> history service makes sense as of now - considering that cluster
> managers
> >> have their own history servers. Just showing UI of history runs might be
> >> too thin a requirement for a full service. Spark should store history
> >> information that can later be exposed in required ways.
> >>
> >> Since each SparkContext is the logical entry and exit point for doing
> >> something useful in Spark, during its stop(), it should serialize that
> >> run's statistics into a JSON file - like
> "sc_run_[name]_[start-time].json".
> >> When SparkUI.stop() is called, it in turn asks its UI objects (which
> should
> >> implement a trait) to provide either a flat or hierarchical Map of
> String
> >> key/value pairs. This map (flat, hierarchical) is then serialized to a
> >> configured path (default being "var/history").
> >>
> >> With regards to Mesos or YARN, their applications during shutdown can
> have
> >> API to import this Spark history into their history servers - by making
> API
> >> calls etc.
> >>
> >> This way Spark's history information is persisted independent of cluster
> >> framework, and cluster frameworks can import the history when/as needed.
> >> Hope this helps.
> >> Regards,
> >> pillis
> >>
> >>
> >>
> >> On Thu, Jan 9, 2014 at 6:13 AM, Tom Graves <tgraves_cs@yahoo.com>
> wrote:
> >>
> >> > Note that it looks like we are planning on adding support for
> application
> >> > specific frameworks to YARN sooner rather then later. There is an
> initial
> >> > design up here: https://issues.apache.org/jira/browse/YARN-1530. Note
> >> > this has not been reviewed yet so changes are likely but gives an
> idea of
> >> > the general direction.  If anyone has comments on how that might work
> >> with
> >> > SPARK I encourage you to post to the jira.
> >> >
> >> > As Sandy mentioned it would be very nice if the solution could be
> >> > compatible with that.
> >> >
> >> > Tom
> >> >
> >> >
> >> >
> >> > On Wednesday, January 8, 2014 12:44 AM, Sandy Ryza <
> >> > sandy.ryza@cloudera.com> wrote:
> >> >
> >> > Hey,
> >> >
> >> > YARN-321 is targeted for the Hadoop 2.4.  The minimum feature set
> doesn't
> >> > include application-specific data, so that probably won't be part of
> 2.4
> >> > unless other things delay the release for a while.  There are no APIs
> for
> >> > it yet and pluggable UIs have been discussed but not agreed upon.  I
> >> think
> >> > requirements from Spark could be useful in helping shape what gets
> done
> >> > there.
> >> >
> >> > -Sandy
> >> >
> >> >
> >> >
> >> > On Tue, Jan 7, 2014 at 4:13 PM, Patrick Wendell <pwendell@gmail.com>
> >> > wrote:
> >> >
> >> > > Hey Sandy,
> >> > >
> >> > > Do you know what the status is for YARN-321 and what version of YARN
> >> > > it's targeted for? Also, is there any kind of documentation or API
> for
> >> > > this? Does it control the presentation of the data itself (e.g. it
> >> > > actually has its own UI)?
> >> > >
> >> > > @Tom - having an optional history server sounds like a good idea.
> >> > >
> >> > > One question is what format to use for storing the data and how the
> >> > > persisted format relates to XML/HTML generation in the live UI. One
> >> > > idea would be to add JSON as an intermediate format inside of the
> >> > > current WebUI, and then any JSON page could be persisted and
> rendered
> >> > > by the history server using the same code. Once a SparkContext exits
> >> > > it could dump a series of named paths each with a JSON file. Then
> the
> >> > > history server could load those paths and pass them through the
> second
> >> > > rendering stage (JSON => XML) to create each page.
> >> > >
> >> > > It would be good if SPARK-969 had a good design doc before anyone
> >> > > starts working on it.
> >> > >
> >> > > - Patrick
> >> > >
> >> > > On Tue, Jan 7, 2014 at 3:18 PM, Sandy Ryza <sandy.ryza@cloudera.com
> >
> >> > > wrote:
> >> > > > As a sidenote, it would be nice to make sure that whatever done
> here
> >> > will
> >> > > > work with the YARN Application History Server (YARN-321), a
> generic
> >> > > history
> >> > > > server that functions similarly to MapReduce's JobHistoryServer.
>  It
> >> > will
> >> > > > eventually have the ability to store application-specific data.
> >> > > >
> >> > > > -Sandy
> >> > > >
> >> > > >
> >> > > > On Tue, Jan 7, 2014 at 2:51 PM, Tom Graves <tgraves_cs@yahoo.com>
> >> > wrote:
> >> > > >
> >> > > >> I don't think you want to save the html/xml files. I would
rather
> >> see
> >> > > the
> >> > > >> info saved into a history file in like a json format that
could
> then
> >> > be
> >> > > >> re-read and the web ui display the info, hopefully without
much
> >> change
> >> > > to
> >> > > >> the UI parts.  For instance perhaps the history server could
read
> >> the
> >> > > file
> >> > > >> and populate the appropriate Spark data structures that the
web
> ui
> >> > > already
> >> > > >> uses.
> >> > > >>
> >> > > >> I would suggest making it so the history server is an optional
> >> server
> >> > > and
> >> > > >> could be run on any node. That way if the load on a particular
> node
> >> > > becomes
> >> > > >> to much it could be moved, but you also could run it on the
same
> >> node
> >> > as
> >> > > >> the Master.  All it really needs to know is where to get
the
> history
> >> > > files
> >> > > >> from and have access to that location.
> >> > > >>
> >> > > >> Hadoop actually has a history server for MapReduce which
works
> very
> >> > > >> similar to what I mention above.   One thing to keep in minds
> here
> >> is
> >> > > >> security.  You want to make sure that the history files can
only
> be
> >> > > read by
> >> > > >> users who have the appropriate permissions.  The history
server
> >> itself
> >> > > >> could run as  a superuser who has permission to server up
the
> files
> >> > > based
> >> > > >> on the acls.
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> On Tuesday, January 7, 2014 8:06 AM, "Xia, Junluan" <
> >> > > junluan.xia@intel.com>
> >> > > >> wrote:
> >> > > >>
> >> > > >> Hi all
> >> > > >>          Spark job web ui will not be available when job
is over,
> >> but
> >> > it
> >> > > >> is convenient for developer to debug with persisting job
web ui.
> I
> >> > just
> >> > > >> come up with draft for this issue.
> >> > > >>
> >> > > >> 1.       We could simply save the web page with html/xml
> >> > > >> format(stages/executors/storages/environment) to certain
location
> >> when
> >> > > job
> >> > > >> finished
> >> > > >>
> >> > > >> 2.       But it is not easy for user to review the job info
with
> #1,
> >> > we
> >> > > >> could build extra job history service for developers
> >> > > >>
> >> > > >> 3.       But where will we build this history service? In
Driver
> >> node
> >> > or
> >> > > >> Master node?
> >> > > >>
> >> > > >> Any suggestions about this improvement?
> >> > > >>
> >> > > >> regards,
> >> > > >> Andrew
> >> > > >>
> >> > >
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message