spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From thomastechs <>
Subject [GitHub] spark pull request #16743: [SPARK-19379][CORE] SparkAppHandle.getState not r...
Date Mon, 30 Jan 2017 17:50:18 GMT
GitHub user thomastechs opened a pull request:

    [SPARK-19379][CORE] SparkAppHandle.getState not registering FAILED state upon Spark app
failure in Local mode

    ## What changes were proposed in this pull request?
    As per the bug, the state corresponding to app failures are not getting registered with
call back to the launcher.So, the already available isFailed() check of TaskState.scala, is
utilized to send the explicit status of SparkAppHandle.State.FAILED back to the launcherBackend,
which under the hood updates the status for (LOST == state) || (FAILED == state).
    The already available TaskState.isFinished() is used to set the value SparkAppHandle.State.FINISHED,
as under the hood, it considers FINISHED, FAILED, KILLED, LOST as part of FINISHED state.
    Since the isFailed() is checked first, the isFinished is checked in the else if condition
to get the mutually exclusive scenarios.
    ## How was this patch tested?
    This patch was tested manually. The tests included, by simply submitting local jobs, the
states are set correctly. Also, the tests included with some fail prone codes executed and
tested the status set as FAILED. It looks difficult to write testcase to create an error prone
scnario code and check the status.

You can merge this pull request into a Git repository by running:

    $ git pull thomastechs-19379

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #16743
commit 52ac0083f14a2454b1109b5f535c05ac7fd5a2b4
Author: thomastechs <>
Date:   2016-02-22T17:16:15Z

    Adding new test suite for [SPARK-12941][SQL]
    Added this new integration suite for creating the docker based test suite for the bug
fix in SPARK-12941

commit b711688be233ab78fd92f42b22bcfb852a0bb1b9
Author: thomastechs <>
Date:   2016-02-22T17:20:19Z

    Update pom.xml for [SPARK-12941] docker test suite addition
    Updated this pom.xml with the ojdbc jar related dependency to test the OracleIntegrationSuite
related to SPARK-12941 bug fix

commit c2c6d14353cab4e72b66f9125f964b215a977446
Author: thomastechs <>
Date:   2016-02-22T17:46:51Z

    Update DockerJDBCIntegrationSuite.scala for [SPARK-12941] docker test suite addition
    Updated DockerJDBCIntegrationSuite.scala with the time out interval, as it is required
to be atleast 200 sec for Oracle instance to be up

commit 54b2d4e466e309c8c12e58cefcdf31ca690183f1
Author: thomastechs <>
Date:   2016-02-22T17:51:21Z

    Update OracleDialect.scala for [SPARK-12941] docker test suite addition
    Updated OracleDialect.scala for [SPARK-12941]  for the fix of adding a data type mapping
to Oracle

commit bde06fa2189dfe9f9484d86a942eb6cfa50ffad8
Author: thomastechs <>
Date:   2016-02-22T17:57:59Z

    Update OracleIntegrationSuite.scala for [SPARK-12941] docker test suite addition
    Updated  OracleIntegrationSuite.scala for [SPARK-12941]  with Oracle Docker test cases

commit 0861d72c3b3ad377807edb96f4016231291e792a
Author: thomastechs <>
Date:   2016-02-23T02:57:32Z

    Update OracleIntegrationSuite.scala for [SPARK-12941] Style check fixes and ignore test
    Updated the file with style check fixes and ignore test cases, as the ojdbc jar is not
present in the maven.

commit ca7ef4e3de03f70c7f37ede0075e31daf17ff4cc
Author: thomastechs <>
Date:   2016-02-23T02:59:23Z

    Update OracleIntegrationSuite.scala for [SPARK-12941] Style check fixes
    Style check fixes

commit 28a9e9fc7793f0ae7dfc0fc0fae17d6ef8acd94e
Author: thomastechs <>
Date:   2016-02-23T03:14:09Z

    Update OracleIntegrationSuite.scala for [SPARK-12941] Style check fixes
    Updated the order of imports

commit 87e3d7f4de872852a3f16bf6a31761a8cfd1d46b
Author: thomastechs <>
Date:   2016-02-23T06:35:28Z

    Update pom.xml
    Updating this pom.xml to comment the changes related oracle ojdbc jar dependency as the
maven repository does not contain this.

commit 7b12418ca7f75bcc17941bb713286573b7209225
Author: thomastechs <>
Date:   2016-02-23T09:43:37Z

    Update OracleIntegrationSuite.scala
    Updated the OracleIntegrationSuite.scala with the doc and override method

commit 250591dfb7c63f446cfb09702587642da2ec026d
Author: thomastechs <>
Date:   2016-02-23T09:48:02Z

    Update OracleIntegrationSuite.scala
    updated doc

commit d77d5852b06c4003661757207868c693723210bd
Author: thomastechs <>
Date:   2016-02-23T11:13:24Z

    Update OracleIntegrationSuite.scala for [SPARK-12941] Style check fixes
    Updated style check fixes

commit 7fcb16cac7e0f32c74db1c5d9e3d4968c9da183a
Author: thomastechs <>
Date:   2016-02-24T02:38:26Z

    Update OracleIntegrationSuite.scala for [SPARK-12941] Updated Comments
    Updated OracleIntegrationSuite.scala for [SPARK-12941] for the edits in the comments section

commit b3de7f493cc14e258cd699f395bd1241912beb4b
Author: thomastechs <>
Date:   2016-02-24T02:38:36Z

    Update pom.xml for [SPARK-12941] added comments
    Added comments on why the oracle dependency was commented out

commit 345555f777ab09cd6426685938e5b5c935749472
Author: thomastechs <>
Date:   2016-02-24T06:11:17Z

    Update DockerJDBCIntegrationSuite.scala
    Updated DockerJDBCIntegrationSuite.scala- Reverted the timeout and interval parameter
to 60 and 1. This would be required to be increased for Oracle based testing.

commit 3f0f3b8105b358f580b6225f9cf1592b44e17450
Author: thomastechs <>
Date:   2016-02-24T06:11:33Z

    Updated OracleIntegrationSuite.scala
    Updated the comments about the time interval change required in DockerJDBCIntegrationSuite.scala
when testing with Oracle tests

commit 5683dcabfd283f6008f4c1294ff41a07e3f40b8f
Date:   2016-03-03T03:55:31Z

    Merge remote-tracking branch 'upstream/master'

commit 154e10444b4796fd52ff3e0ddf7b6b7a86cc4a87
Date:   2016-11-24T18:52:43Z

    Merge remote-tracking branch 'upstream/master'

commit e6d431789b7f390612b82ba8305439de3f047de1
Author: thomastechs <>
Date:   2017-01-30T17:30:25Z

    Updated LocalSchedulerBackend.scala for fix of JIRA:19379
    The setState methods of the launcher backend is being invoked for the status updates related
to Failed and Finished. The isFailed Test explicity captures the failure scenario and updates
the status. and n else if is given to maintain the as is scenario of choosing isFinished conditions


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message