spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pgandhi999 <...@git.apache.org>
Subject [GitHub] spark pull request #18741: [Spark-21541][YARN]: Spark Logs show incorrect jo...
Date Wed, 26 Jul 2017 22:54:23 GMT
GitHub user pgandhi999 opened a pull request:

    https://github.com/apache/spark/pull/18741

    [Spark-21541][YARN]: Spark Logs show incorrect job status for a job that does not create
SparkContext

    If you run a spark job without creating the SparkSession or SparkContext, the spark job
logs says it succeeded but yarn says it fails and retries 3 times. Also, since, Application
Master unregisters with Resource Manager and exits successfully, it deletes the spark staging
directory, so when yarn makes subsequent retries, it fails to find the staging directory and
thus, the retries fail.
    
    Added a flag to check whether user has initialized SparkContext. If it is true, we let
Application Master unregister with Resource Manager else, we do not let AM unregister with
RM.
    
    ## How was this patch tested?
    Manually tested the fix.
    Before:
    <img width="1253" alt="screen shot-before" src="https://user-images.githubusercontent.com/22228190/28647214-69bf81e2-722b-11e7-9ed0-d416d2bf23be.png">
    
    After:
    <img width="1319" alt="screen shot-after" src="https://user-images.githubusercontent.com/22228190/28647220-70f9eea2-722b-11e7-85c6-e56276b15614.png">
    
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/pgandhi999/spark SPARK-21541

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18741.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18741
    
----
commit 172fc20898896058b7288360eb5292ed9df9d79c
Author: pgandhi <pgandhi@yahoo-inc.com>
Date:   2017-07-21T21:00:22Z

    [SPARK-21503]: Fixed the issue
    
    Added the case ExecutorLostFailure which was previously not there, thus, the default case
would be executed in which case, task would be marked as completed.

commit 81422e0f634c0f06eb2ea29fba4281176a1ab528
Author: pgandhi <pgandhi@yahoo-inc.com>
Date:   2017-07-25T14:54:41Z

    [SPARK-21503][UI]: Adding changes as per comments

commit 55c6c37d09b41ae6914edb5d067e7f2c252ac92a
Author: pgandhi999 <parthkgandhi9@gmail.com>
Date:   2017-07-26T21:26:27Z

    Merge pull request #1 from apache/master
    
    Apache Spark Pull Request - July 26, 2017

commit f454c8933e07967548095e068063bd313ae4845c
Author: pgandhi <pgandhi@yahoo-inc.com>
Date:   2017-07-26T21:41:16Z

    [SPARK-21541]: Spark Logs show incorrect job status for a job that does not create SparkContext
    
    Added a flag to check whether user has initialized Spark Context. If it is true, then
we let Application Master unregister with Resource Manager else we do not.

commit 6b7d5c6e2565c7c4dd97f31fe404c59e73c7474c
Author: pgandhi <pgandhi@yahoo-inc.com>
Date:   2017-07-26T21:58:27Z

    Revert "[SPARK-21541]: Spark Logs show incorrect job status for a job that does not create
SparkContext"
    
    This reverts commit f454c8933e07967548095e068063bd313ae4845c.
    
    "Merged another issue to this one by mistake"

commit 5c52c72949efdf14a40ff60e8edbe98e011bc084
Author: pgandhi <pgandhi@yahoo-inc.com>
Date:   2017-07-26T22:23:30Z

    [SPARK-21541]: Making Changes
    
    Added a flag to check whether user has initialized SparkContext. If it is true, we let
Application Master unregister with Resource Manager else, we do not let AM unregister with
RM.

commit cfd9f71497c5411f697555c9d604923d998cf406
Author: pgandhi <pgandhi@yahoo-inc.com>
Date:   2017-07-26T22:40:34Z

    Reverting Commits from Master Branch

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message