spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From logc <...@git.apache.org>
Subject [GitHub] spark pull request: Add unit test to spark_ec2 script
Date Thu, 13 Mar 2014 09:51:21 GMT
GitHub user logc opened a pull request:

    https://github.com/apache/spark/pull/134

    Add unit test to spark_ec2 script

    Add a unittest for a refactored function to destroy cluster.  Relies on mock and moto
dependencies to avoid sending out EC2 requests.
    
    We noticed at work that the supplied script does not work always when destroying clusters,
specially for regions outside "us-east-1".  This pull request adds a test on the command.
If this is accepted, we can continue debugging why exactly are such clusters not terminated
correctly.
    
    Run the test with:
    
    ```console
    $ python ec2/tests.py
    Searching for existing cluster cluster_name...
    Terminating master...
    Terminating slaves...
    Deleting security groups (this will take some time)...
    Attempt 1
    .
    ----------------------------------------------------------------------
    Ran 1 test in 1.040s
    ```
    
    Please notice that the test relies on [moto](https://github.com/spulec/moto) and [mock](https://pypi.python.org/pypi/mock)
to run. Since I did not know how you handle such dependencies, I have not added a `requirements.txt`
or a `setup.py` script, but can update the pull request in that sense.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/logc/spark tlpmaster

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/134.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #134
    
----
commit 2e7ca995dce6a8822fed4389be4d69223bd671d2
Author: Luis Osa <logc@tid.es>
Date:   2014-03-12T15:32:30Z

    Add unittest for destroy cluster command
    
    Add a unittest for refactored function to destroy cluster.  Relies on
    mock and moto dependencies to avoid sending out EC2 requests.

commit 534edcc96e33889275ad278dc3cfaf24faf44604
Author: Luis Osa <logc@tid.es>
Date:   2014-03-12T15:34:13Z

    Add pyc files to gitignore

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message