spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewor14 <...@git.apache.org>
Subject [GitHub] spark pull request: [WIP] SPARK-2450 Adds exeuctor log links to We...
Date Tue, 09 Dec 2014 21:46:46 GMT
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/3486#issuecomment-66363953
  
    Hey @ksakellis I think we should explore a different approach that doesn't involve adding
new listener events. Have you considered something like the following:
    
    The AM is the one that launches these executors, and it has global knowledge of which
executor maps to which container IDs and thus log URLs. The next step then is to pass this
knowledge to the driver, which contains the UI.
    
    In cluster mode, this is easy because the AM already has a reference to `sc`, which is
launched within the same JVM. In client mode, there is an existing communication path through
the `AMActor` in `ApplicationMaster`. Instead of passing these URLs through env vars and listeners
we can just have the `AMActor` send a message to the driver every time a new executor is launched.
This bypasses passing the log URLs to the executors themselves, which don't consume them except
for passing them along to the driver.
    
    What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message