whirr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (WHIRR-698) Implement the installation and configuration of Spark and its dependencies as a Whirr service
Date Thu, 07 Feb 2013 00:38:12 GMT

     [ https://issues.apache.org/jira/browse/WHIRR-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eli Reisman updated WHIRR-698:
------------------------------

    Attachment: WHIRR-698-2.patch

This is a much cleaned-up and functional version of this service. One problem is that it sometimes
seems to timeout during an otherwise successful Spark install when Spark is using SBT to download
a bunch of Spark/Scala dependencies. The error shows up in whirr.log as this typical except:

{code}
+ sbt/sbt assembly
+ sbt/sbt package
+ cp conf/spark-env.sh.template conf/spark-env.sh
+ echo 'export SCALA_HOME=/usr/local/scala'
+ echo 'export SPARK_JAR=/home/users/ereisman/spark/core/target/spark-core-assembly-0.6.0.jar'
+ exit 0
+ echo 0
, exitStatus=0}
2013-02-06 16:10:45,287 ERROR [net.schmizz.sshj.transport.TransportImpl] (reader) Dying because
- java.net.SocketTimeoutException: Read timed out
{code}

Otherwise it works great.
                
> Implement the installation and configuration of Spark and its dependencies as a Whirr
service
> ---------------------------------------------------------------------------------------------
>
>                 Key: WHIRR-698
>                 URL: https://issues.apache.org/jira/browse/WHIRR-698
>             Project: Whirr
>          Issue Type: New Feature
>    Affects Versions: 0.8.1
>            Reporter: Eli Reisman
>            Priority: Minor
>         Attachments: WHIRR-698-1.patch, WHIRR-698-2.patch
>
>
> I would like to Launch the Spark Project on a Hadoop YARN cluster on EC2 (or similar
cloud service.) and this service + recipe file make that happen. This is my first service,
and Spark is still learning to play nice with YARN and Whirr, so suggestions are welcome.
> Thanks!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message