spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Costantini <silvio.costant...@granatads.com>
Subject Re: AWS Spark-ec2 script with different user
Date Wed, 09 Apr 2014 16:38:57 GMT
Perfect. Now I know what to do. Thanks to your help!

Many thanks,
Marco.


On Wed, Apr 9, 2014 at 12:27 PM, Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:

> The AMI should automatically switch between PVM and HVM based on the
> instance type you specify on the command line. For reference (note you
> don't need to specify this on the command line), the PVM ami id
> is ami-5bb18832 in us-east-1.
>
> FWIW we maintain the list of AMI Ids (across regions and pvm, hvm) at
> https://github.com/mesos/spark-ec2/tree/v2/ami-list
>
> Thanks
> Shivaram
>
>
> On Wed, Apr 9, 2014 at 9:12 AM, Marco Costantini <
> silvio.costantini@granatads.com> wrote:
>
>> Ah, tried that. I believe this is an HVM AMI? We are exploring
>> paravirtual AMIs.
>>
>>
>> On Wed, Apr 9, 2014 at 11:17 AM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> And for the record, that AMI is ami-35b1885c. Again, you don't need to
>>> specify it explicitly; spark-ec2 will default to it.
>>>
>>>
>>> On Wed, Apr 9, 2014 at 11:08 AM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> Marco,
>>>>
>>>> If you call spark-ec2 launch without specifying an AMI, it will default
>>>> to the Spark-provided AMI.
>>>>
>>>> Nick
>>>>
>>>>
>>>> On Wed, Apr 9, 2014 at 9:43 AM, Marco Costantini <
>>>> silvio.costantini@granatads.com> wrote:
>>>>
>>>>> Hi there,
>>>>> To answer your question; no there is no reason NOT to use an AMI that
>>>>> Spark has prepared. The reason we haven't is that we were not aware such
>>>>> AMIs existed. Would you kindly point us to the documentation where we
can
>>>>> read about this further?
>>>>>
>>>>> Many many thanks, Shivaram.
>>>>> Marco.
>>>>>
>>>>>
>>>>> On Tue, Apr 8, 2014 at 4:42 PM, Shivaram Venkataraman <
>>>>> shivaram@eecs.berkeley.edu> wrote:
>>>>>
>>>>>> Is there any reason why you want to start with a vanilla amazon AMI
>>>>>> rather than the ones we build and provide as a part of Spark EC2
scripts ?
>>>>>> The AMIs we provide are close to the vanilla AMI but have the root
account
>>>>>> setup properly and install packages like java that are used by Spark.
>>>>>>
>>>>>> If you wish to customize the AMI, you could always start with our
AMI
>>>>>> and add more packages you like -- I have definitely done this recently
and
>>>>>> it works with HVM and PVM as far as I can tell.
>>>>>>
>>>>>> Shivaram
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 8, 2014 at 8:50 AM, Marco Costantini <
>>>>>> silvio.costantini@granatads.com> wrote:
>>>>>>
>>>>>>> I was able to keep the "workaround" ...around... by overwriting
the
>>>>>>> generated '/root/.ssh/authorized_keys' file with a known good
one, in the
>>>>>>> '/etc/rc.local' file
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 8, 2014 at 10:12 AM, Marco Costantini <
>>>>>>> silvio.costantini@granatads.com> wrote:
>>>>>>>
>>>>>>>> Another thing I didn't mention. The AMI and user used: naturally
>>>>>>>> I've created several of my own AMIs with the following characteristics.
>>>>>>>> None of which worked.
>>>>>>>>
>>>>>>>> 1) Enabling ssh as root as per this guide (
>>>>>>>> http://blog.tiger-workshop.com/enable-root-access-on-amazon-ec2-instance/).
>>>>>>>> When doing this, I do not specify a user for the spark-ec2
script. What
>>>>>>>> happens is that, it works! But only while it's alive. If
I stop the
>>>>>>>> instance, create an AMI, and launch a new instance based
from the new AMI,
>>>>>>>> the change I made in the '/root/.ssh/authorized_keys' file
is overwritten
>>>>>>>>
>>>>>>>> 2) adding the 'ec2-user' to the 'root' group. This means
that the
>>>>>>>> ec2-user does not have to use sudo to perform any operations
needing root
>>>>>>>> privilidges. When doing this, I specify the user 'ec2-user'
for the
>>>>>>>> spark-ec2 script. An error occurs: rsync fails with exit
code 23.
>>>>>>>>
>>>>>>>> I believe HVMs still work. But it would be valuable to the
>>>>>>>> community to know that the root user work-around does/doesn't
work any more
>>>>>>>> for paravirtual instances.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Marco.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 8, 2014 at 9:51 AM, Marco Costantini <
>>>>>>>> silvio.costantini@granatads.com> wrote:
>>>>>>>>
>>>>>>>>> As requested, here is the script I am running. It is
a simple
>>>>>>>>> shell script which calls spark-ec2 wrapper script. I
execute it from the
>>>>>>>>> 'ec2' directory of spark, as usual. The AMI used is the
raw one from the
>>>>>>>>> AWS Quick Start section. It is the first option (an Amazon
Linux
>>>>>>>>> paravirtual image). Any ideas or confirmation would be
GREATLY appreciated.
>>>>>>>>> Please and thank you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #!/bin/sh
>>>>>>>>>
>>>>>>>>> export AWS_ACCESS_KEY_ID=MyCensoredKey
>>>>>>>>> export AWS_SECRET_ACCESS_KEY=MyCensoredKey
>>>>>>>>>
>>>>>>>>> AMI_ID=ami-2f726546
>>>>>>>>>
>>>>>>>>> ./spark-ec2 -k gds-generic -i ~/.ssh/gds-generic.pem
-u ec2-user
>>>>>>>>> -s 10 -v 0.9.0 -w 300 --no-ganglia -a ${AMI_ID} -m m3.2xlarge
-t m3.2xlarge
>>>>>>>>> launch marcotest
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Apr 7, 2014 at 6:21 PM, Shivaram Venkataraman
<
>>>>>>>>> shivaram.venkataraman@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hmm -- That is strange. Can you paste the command
you are using
>>>>>>>>>> to launch the instances ? The typical workflow is
to use the spark-ec2
>>>>>>>>>> wrapper script using the guidelines at
>>>>>>>>>> http://spark.apache.org/docs/latest/ec2-scripts.html
>>>>>>>>>>
>>>>>>>>>> Shivaram
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 7, 2014 at 1:53 PM, Marco Costantini
<
>>>>>>>>>> silvio.costantini@granatads.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Shivaram,
>>>>>>>>>>>
>>>>>>>>>>> OK so let's assume the script CANNOT take a different
user and
>>>>>>>>>>> that it must be 'root'. The typical workaround
is as you said, allow the
>>>>>>>>>>> ssh with the root user. Now, don't laugh, but,
this worked last Friday, but
>>>>>>>>>>> today (Monday) it no longer works. :D Why? ...
>>>>>>>>>>>
>>>>>>>>>>> ...It seems that NOW, when you launch a 'paravirtual'
ami, the
>>>>>>>>>>> root user's 'authorized_keys' file is always
overwritten. This means the
>>>>>>>>>>> workaround doesn't work anymore! I would LOVE
for someone to verify this.
>>>>>>>>>>>
>>>>>>>>>>> Just to point out, I am trying to make this work
with a
>>>>>>>>>>> paravirtual instance and not an HVM instance.
>>>>>>>>>>>
>>>>>>>>>>> Please and thanks,
>>>>>>>>>>> Marco.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 7, 2014 at 4:40 PM, Shivaram Venkataraman
<
>>>>>>>>>>> shivaram.venkataraman@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Right now the spark-ec2 scripts assume that
you have root
>>>>>>>>>>>> access and a lot of internal scripts assume
have the user's home directory
>>>>>>>>>>>> hard coded as /root.   However all the Spark
AMIs we build should have root
>>>>>>>>>>>> ssh access -- Do you find this not to be
the case ?
>>>>>>>>>>>>
>>>>>>>>>>>> You can also enable root ssh access in a
vanilla AMI by editing
>>>>>>>>>>>> /etc/ssh/sshd_config and setting "PermitRootLogin"
to yes
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Shivaram
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 7, 2014 at 11:14 AM, Marco Costantini
<
>>>>>>>>>>>> silvio.costantini@granatads.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>> On the old Amazon Linux EC2 images, the
user 'root' was
>>>>>>>>>>>>> enabled for ssh. Also, it is the default
user for the Spark-EC2 script.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Currently, the Amazon Linux images have
an 'ec2-user' set up
>>>>>>>>>>>>> for ssh instead of 'root'.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can see that the Spark-EC2 script allows
you to specify
>>>>>>>>>>>>> which user to log in with, but even when
I change this, the script fails
>>>>>>>>>>>>> for various reasons. And the output SEEMS
that the script is still based on
>>>>>>>>>>>>> the specified user's home directory being
'/root'.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am I using this script wrong?
>>>>>>>>>>>>> Has anyone had success with this 'ec2-user'
user?
>>>>>>>>>>>>> Any ideas?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please and thank you,
>>>>>>>>>>>>> Marco.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message