cheers,

Dhiraj

On Wed, Sep 10, 2014 at 3:13 PM, Harsh J <harsh@= cloudera.com> wrote:

>= ; Incorrect configuration: namenode address dfs.namenode.servicerpc-address= or dfs.namenode.rpc-address is not configured.
> Starting namenodes on []

NameNode/DataNode are part of a HDFS service. It makes no sense to t= ry
and run them over an S3 URL default, which is a distributed filesystem
in itself. The services need fs.defaultFS to be set to a HDFS URI to
be able to start up.

> but unable to get an s3 config started via hadoop

You can run jobs over S3 input and output data by running a regular = MR
cluster on HDFS - just pass the right URI as input and output
parameters of the job. Set your S3 properties in core-site.xml but let
the fs.defaultFS be of HDFS type, to do this.

> There is an s3.impl until 1.2.1 release. So does the 2.5.0 release sup= port s3 or do i need to do anything else.

In Apache Hadoop 2 we dynamically load the FS classes, so we do not<= br> need the fs.NAME.impl configs anymore as we did in Apache Hadoop 1.

On Wed, Sep 10, 2014 at 1:15 PM, Dhiraj <jarihd@gmail.com> wrote:
> Hi,
>
> I have downloaded hadoop-2.5.0 and am trying to get it working for s3<= br> > backend (single-node in a pseudo-distributed mode).
> I have made changes to the core-site.xml according to
> = https://wiki.apache.org/hadoop/AmazonS3
>
> I have an backend object store running on my machine that supports S3.=
>
> I get the following message when i try to start the daemons
> Incorrect configuration: namenode address dfs.namenode.servicerpc-addr= ess or
> dfs.namenode.rpc-address is not configured.
>
>
> root@ubuntu:/build/hadoop/hadoop-2.5.0# ./sbin/start-dfs.sh
> Incorrect configuration: namenode address dfs.namenode.servicerpc-addr= ess or
> dfs.namenode.rpc-address is not configured.
> Starting namenodes on []
> localhost: starting namenode, logging to
> /build/hadoop/hadoop-2.5.0/logs/hadoop-root-namenode-ubuntu.out
> localhost: starting datanode, logging to
> /build/hadoop/hadoop-2.5.0/logs/hadoop-root-datanode-ubuntu.out
> Starting secondary namenodes [0.0.0.0]
> 0.0.0.0: starting sec= ondarynamenode, logging to
> /build/hadoop/hadoop-2.5.0/logs/hadoop-root-secondarynamenode-ubuntu.o= ut
> root@ubuntu:/build/hadoop/hadoop-2.5.0#
>
> The deamons dont start after the above.
> i get the same error if i add the property "fs.defaultFS" an= d set its value
> to the s3 bucket but if i change the defaultFS to hdfs:// it works fin= e - am
> able to launch the daemons.
>
> my core-site.xml:
> <configuration>
>=C2=A0 =C2=A0 =C2=A0<property>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<name>fs.defaultFS</name>=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<value>s3://bucket1</value&g= t;
>=C2=A0 =C2=A0 =C2=A0</property>
>=C2=A0 =C2=A0 =C2=A0<property>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<name>fs.s3.awsAccessKeyId</= name>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<value>abcd</value>
>=C2=A0 =C2=A0 =C2=A0</property>
>=C2=A0 =C2=A0 =C2=A0<property>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<name>fs.s3.awsSecretAccessKey&= lt;/name>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<value>1234</value>
>=C2=A0 =C2=A0 =C2=A0</property>
> </configuration>
>
>
> I am able to list the buckets and its contents via s3cmd and boto; but=
> unable to get an s3 config started via hadoop
>
> Also from the following core-file.xml listed on the website; i dont se= e an
> implementation for s3
> http://hadoop.apache.org/d= ocs/current/hadoop-project-dist/hadoop-common/core-default.xml
>
> There is an s3.impl until 1.2.1 release. So does the 2.5.0 release sup= port
> s3 or do i need to do anything else.
>
> cheers,
> Dhiraj
>
>
>

--
Harsh J