hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yusaku Sako <yus...@hortonworks.com>
Subject Re: Need help get the hadoop cluster started in EC2
Date Sat, 29 Mar 2014 04:56:12 GMT
Hi Max,

Not sure if you have already, but you might also want to look into
Apache Ambari [1] for provisioning, managing, and monitoring Hadoop
clusters.
Many have successfully deployed Hadoop clusters on EC2 using Ambari.

[1] http://ambari.apache.org/

Yusaku

On Fri, Mar 28, 2014 at 7:07 PM, Max Zhao <gz123forhadoop@gmail.com> wrote:
> Hi Everybody,
>
> I am trying to get my first hadoop cluster started using the Amazon EC2. I
> tried quite a few times and searched the web for the solutions, yet I still
> cannot get it up. I hope somebody can help out here.
>
> Here is what I did based on the Apache Whirr Quick Guide
> (http://whirr.apache.org/docs/0.8.1/quick-start-guide.html):
>
> 1) I downloaded a Whirr tar ball and installed it.
>                     bin/whirr version shows the following:  Apache Whirr
> 0.8.2  &  jclouds 1.5.8
> 2) I created the ./whirr directory and edit the credential file with my
> Amazon "PROVIDER", "IDENTITY" and "CREDENTIAL"
>                "IDENTITY=AAAAAASSSSSSSSS", with no extra quotes or curly
> quotes around the actual key_id
> 3) I used the following command to creat the key pair for whirr and stored
> it at the folder .ssh
>                ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr
> 4) I think I am ready to use one of the properties file provided with whirr
> in the recipes folder. Here is the command I ran:
>                 bin/whirr launch-cluster --config
> recipes/hadoop-yarn-ec2.properties --private-key-file ~/.ssh/id_rsa_whi
> The command ran into the error and did not bring up the hadoop.  My questin
> is: Do we need to change anything the default properties provided in the
> recipes folder in the "whirr-0.8.2" folder, such as the
> "hadoop-yarn-ec2.properties " I used?
>
> Here are the error messages:
>
> ---------------------------------------------------
> [ec2-user@ip-172-31-20-120 whirr-0.8.2]$ bin/whirr launch-cluster --config
> recipes/hadoop-yarn-ec2.properties  --private-key-file ~/.ssh/id_rsa_whirr
> Running on provider aws-ec2 using identity AKIAJLFVRARQ3IZE3KGF
> Unable to start the cluster. Terminating all nodes.
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.inject.CreationException: Guice creation errors:
> 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot
> be used as a key; It is not fully specified.
> 1 error
>         at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
>         at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
>         at
> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
>         at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
>         at
> com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884)
>         at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88)
>         at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80)
>         at
> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110)
>         at
> org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137)
>         at
> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
>         at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
>         at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
>         at org.apache.whirr.cli.Main.run(Main.java:69)
>         at org.apache.whirr.cli.Main.main(Main.java:102)
> Caused by: com.google.inject.CreationException: Guice creation errors:
> 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot
> be used as a key; It is not fully specified.
> 1 error
>         at
> com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
>         at
> com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
>         at
> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
>         at com.google.inject.Guice.createInjector(Guice.java:95)
>         at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401)
>         at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325)
>         at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600)
>         at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580)
>         at
> org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119)
>         at
> org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98)
>         at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
>         at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
>         at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
>         at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
>         ... 13 more
> Unable to load cluster state, assuming it has no running nodes.
> java.io.FileNotFoundException: /home/ec2-user/.whirr/hadoop-yarn/instances
> (No such file or directory)
>         at java.io.FileInputStream.open(Native Method)
>         at java.io.FileInputStream.<init>(FileInputStream.java:140)
>         at com.google.common.io.Files$1.getInput(Files.java:109)
>         at com.google.common.io.Files$1.getInput(Files.java:106)
>         at com.google.common.io.CharStreams$2.getInput(CharStreams.java:93)
>         at com.google.common.io.CharStreams$2.getInput(CharStreams.java:90)
>         at com.google.common.io.CharStreams.readLines(CharStreams.java:307)
>         at com.google.common.io.Files.readLines(Files.java:543)
>         at
> org.apache.whirr.state.FileClusterStateStore.load(FileClusterStateStore.java:55)
>         at
> org.apache.whirr.state.ClusterStateStore.tryLoadOrEmpty(ClusterStateStore.java:59)
>         at
> org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226)
>         at
> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121)
>         at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
>         at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
>         at org.apache.whirr.cli.Main.run(Main.java:69)
>         at org.apache.whirr.cli.Main.main(Main.java:102)
> Exception in thread "main"
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.inject.CreationException: Guice creation errors:
> 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot
> be used as a key; It is not fully specified.
> 1 error
>         at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
>         at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
>         at
> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
>         at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
>         at
> com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884)
>         at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88)
>         at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80)
>         at
> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110)
>         at
> org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:233)
>         at
> org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226)
>         at
> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121)
>         at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
>         at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
>         at org.apache.whirr.cli.Main.run(Main.java:69)
>         at org.apache.whirr.cli.Main.main(Main.java:102)
> Caused by: com.google.inject.CreationException: Guice creation errors:
> 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot
> be used as a key; It is not fully specified.
> 1 error
>         at
> com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
>         at
> com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
>         at
> com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
>         at com.google.inject.Guice.createInjector(Guice.java:95)
>         at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401)
>         at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325)
>         at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600)
>         at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580)
>         at
> org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119)
>         at
> org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98)
>         at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
>         at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
>         at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
>         at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
> I would really appreciate if somebody can help pinpoint my problem here.
>
> Thanks
> Max

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message