hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Max Zhao <gz123forhad...@gmail.com>
Subject Need help get the hadoop cluster started in EC2
Date Sat, 29 Mar 2014 02:07:48 GMT
Hi Everybody,

I am trying to get my first hadoop cluster started using the Amazon EC2. I
tried quite a few times and searched the web for the solutions, yet I still
cannot get it up. I hope somebody can help out here.

Here is what I did based on the Apache Whirr Quick Guide (
http://whirr.apache.org/docs/0.8.1/quick-start-guide.html):

1) I downloaded a Whirr tar ball and installed it.
                    bin/whirr version shows the following:  Apache Whirr
0.8.2  &  jclouds 1.5.8
2) I created the ./whirr directory and edit the credential file with my
Amazon "PROVIDER", "IDENTITY" and "CREDENTIAL"
               "IDENTITY=AAAAAASSSSSSSSS", with no extra quotes or curly
quotes around the actual key_id
3) I used the following command to creat the key pair for whirr and stored
it at the folder .ssh
               ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr
4) I think I am ready to use one of the properties file provided with whirr
in the recipes folder. Here is the command I ran:
                bin/whirr launch-cluster --config
recipes/hadoop-yarn-ec2.properties --private-key-file ~/.ssh/id_rsa_whi
The command ran into the error and did not bring up the hadoop.  My questin
is: Do we need to change anything the default properties provided in the
recipes folder in the "whirr-0.8.2" folder, such as the
"hadoop-yarn-ec2.properties
" I used?

Here are the error messages:

---------------------------------------------------
[ec2-user@ip-172-31-20-120 whirr-0.8.2]$ bin/whirr launch-cluster --config
recipes/hadoop-yarn-ec2.properties  --private-key-file ~/.ssh/id_rsa_whirr
Running on provider aws-ec2 using identity AKIAJLFVRARQ3IZE3KGF
Unable to start the cluster. Terminating all nodes.
com.google.common.util.concurrent.UncheckedExecutionException:
com.google.inject.CreationException: Guice creation errors:
1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot
be used as a key; It is not fully specified.
1 error
        at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
        at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
        at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
        at
com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884)
        at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88)
        at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80)
        at
org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110)
        at
org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137)
        at
org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
        at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
        at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
        at org.apache.whirr.cli.Main.run(Main.java:69)
        at org.apache.whirr.cli.Main.main(Main.java:102)
Caused by: com.google.inject.CreationException: Guice creation errors:
1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot
be used as a key; It is not fully specified.
1 error
        at
com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
        at
com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
        at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
        at com.google.inject.Guice.createInjector(Guice.java:95)
        at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401)
        at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325)
        at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600)
        at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580)
        at
org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119)
        at
org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98)
        at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
        at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
        at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
        at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
        ... 13 more
Unable to load cluster state, assuming it has no running nodes.
java.io.FileNotFoundException: /home/ec2-user/.whirr/hadoop-yarn/instances
(No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:140)
        at com.google.common.io.Files$1.getInput(Files.java:109)
        at com.google.common.io.Files$1.getInput(Files.java:106)
        at com.google.common.io.CharStreams$2.getInput(CharStreams.java:93)
        at com.google.common.io.CharStreams$2.getInput(CharStreams.java:90)
        at com.google.common.io.CharStreams.readLines(CharStreams.java:307)
        at com.google.common.io.Files.readLines(Files.java:543)
        at
org.apache.whirr.state.FileClusterStateStore.load(FileClusterStateStore.java:55)
        at
org.apache.whirr.state.ClusterStateStore.tryLoadOrEmpty(ClusterStateStore.java:59)
        at
org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226)
        at
org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121)
        at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
        at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
        at org.apache.whirr.cli.Main.run(Main.java:69)
        at org.apache.whirr.cli.Main.main(Main.java:102)
Exception in thread "main"
com.google.common.util.concurrent.UncheckedExecutionException:
com.google.inject.CreationException: Guice creation errors:
1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot
be used as a key; It is not fully specified.
1 error
        at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
        at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
        at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
        at
com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884)
        at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88)
        at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80)
        at
org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110)
        at
org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:233)
        at
org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226)
        at
org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121)
        at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
        at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
        at org.apache.whirr.cli.Main.run(Main.java:69)
        at org.apache.whirr.cli.Main.main(Main.java:102)
Caused by: com.google.inject.CreationException: Guice creation errors:
1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot
be used as a key; It is not fully specified.
1 error
        at
com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
        at
com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
        at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
        at com.google.inject.Guice.createInjector(Guice.java:95)
        at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401)
        at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325)
        at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600)
        at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580)
        at
org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119)
        at
org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98)
        at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
        at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
        at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
        at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
I would really appreciate if somebody can help pinpoint my problem here.

Thanks
Max

Mime
View raw message