hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lincoln Ritter" <linc...@lincolnritter.com>
Subject Namenode Exceptions with S3
Date Tue, 01 Jul 2008 23:34:47 GMT
Hello,

I am trying to use S3 with Hadoop 0.17.0 on EC2.  Using this style of
configuration:

<property>
  <name>fs.default.name</name>
  <value>s3://$HDFS_BUCKET</value>
</property>

<property>
  <name>fs.s3.awsAccessKeyId</name>
  <value>$AWS_ACCESS_KEY_ID</value>
</property>

<property>
  <name>fs.s3.awsSecretAccessKey</name>
  <value>$AWS_SECRET_ACCESS_KEY</value>
</property>

on startup of the cluster with the bucket having no non-alphabetic
characters, I get:

2008-07-01 16:10:49,171 ERROR org.apache.hadoop.dfs.NameNode:
java.lang.RuntimeException: Not a host:port pair: XXXXX
	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:121)
	at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:121)
	at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:178)
	at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:164)
	at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:848)
	at org.apache.hadoop.dfs.NameNode.main(NameNode.java:857)

If I use this style of configuration:

<property>
  <name>fs.default.name</name>
  <value>s3://$AWS_ACCESS_KEY:$AWS_SECRET_ACCESS_KEY@$HDFS_BUCKET</value>
</property>

I get (where the all-caps portions are the actual values...):

2008-07-01 19:05:17,540 ERROR org.apache.hadoop.dfs.NameNode:
java.lang.NumberFormatException: For input string:
"AWS_SECRET_ACCESS_KEY@HDFS_BUCKET"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
	at java.lang.Integer.parseInt(Integer.java:447)
	at java.lang.Integer.parseInt(Integer.java:497)
	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
	at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:121)
	at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:178)
	at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:164)
	at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:848)
	at org.apache.hadoop.dfs.NameNode.main(NameNode.java:857)

These exceptions are taken from the namenode log.  The datanode logs
show the same exceptions.

Other than the above configuration changes, the configuration is
identical to that generate by the hadoop image creation script found
in the 0.17.0 distribution.

Can anybody point me in the right direction here?

-lincoln

--
lincolnritter.com

Mime
View raw message