Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of fozziethebeat@gmail.com
 designates 74.125.82.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADjdZ8ZgQYq7vhLv7rkmJvW5FUuiZ5mUDo6p2kJN6WAFcCE1rg@mail.gmail.com>
References: 
 <CADjdZ8YURB-aa2H5T+ByWHxx9-ScHWYwdAzXJ2cy3Hg4kL5qmg@mail.gmail.com>
	<CAOcnVr0Zmu8Q=XPzOZ52Ex_4vPM=caRLb++H1-K6pRZja0+fzw@mail.gmail.com>
	<CADjdZ8ZgQYq7vhLv7rkmJvW5FUuiZ5mUDo6p2kJN6WAFcCE1rg@mail.gmail.com>
Date: Sun, 11 Mar 2012 21:34:53 -0700
Message-ID: 
 <CADjdZ8aRakzp4T8cdOP1SZX7G+xAufP_PBDD2oKewmjXn_5vBA@mail.gmail.com>
Subject: Re: Setting up MapReduce 2 on a test cluster
From: Keith Stevens <fozziethebeat@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d04426f147ca68004bb04437b

--f46d04426f147ca68004bb04437b
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Just to double check that I'm checking the logs correctly,

I need the hadoop-hdfs-0.23.0-cdh4b1.jar specifically, yes?  The logs for
my local nodes are reporting that this is included in the class path for
the resource master and node masters.  Is there some other task that might
be missing the jar?

On Sun, Mar 11, 2012 at 9:26 PM, Keith Stevens <fozziethebeat@gmail.com>wro=
te:

> Hi Harsh,
>
> Thanks for getting back to me on this on a sunday.
>
> Your guess is the same as mine, but i'm not sure where this is happening
> or how.
>
> I installed this manually using the tarballs because the cluster i'm
> working on is mostly cut off from the internet.  I also can't seem to
> install createrepo to create a local yum repository.
>
> Is there a way to install the cdh4 packages using just yum install if I
> downloaded them all?  I tried to do this but yum says there the yarn rpm
> depends on the hadoop rpm and visa versa.
>
> Thanks,
> --Keith
>
>
> On Sun, Mar 11, 2012 at 8:53 PM, Harsh J <harsh@cloudera.com> wrote:
>
>> Hey Keith,
>>
>> You're most likely missing the HDFS jar somehow. I use the package
>> installation and am able to run the following successfully:
>>
>> hadoop jar /usr/lib/hadoop/hadoop-mapreduce-examples.jar randomwriter
>> -Dmapreduce.job.user.name=3D$USER
>> -Dmapreduce.clientfactory.class.name
>> =3Dorg.apache.hadoop.mapred.YarnClientFactory
>> -Dmapreduce.randomwriter.bytespermap=3D10000 -Ddfs.blocksize=3D536870912
>> -Ddfs.block.size=3D536870912 -libjars
>> /usr/lib/hadoop/hadoop-mapreduce-client-jobclient.jar output
>>
>> My `hadoop classpath` looks like:
>>
>> /etc/hadoop/conf:/usr/lib/hadoop:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/*=
:/usr/lib/hadoop/share/hadoop/common/*:/usr/lib/hadoop/share/hadoop/hdfs/*:=
/usr/lib/hadoop/share/hadoop/mapreduce/*
>>
>> How have you installed this?
>>
>> On Mon, Mar 12, 2012 at 6:48 AM, Keith Stevens <fozziethebeat@gmail.com>
>> wrote:
>> > Hi All,
>> >
>> > I've been trying to setup Cloudera's Ch4 Beta 1 release of MapReduce
>> 2.0 on
>> > a small cluster for testing but i'm not having much luck getting thing=
s
>> > running.  I've been following the guides on
>> >
>> http://hadoop.apache.org/common/docs/r0.23.1/hadoop-yarn/hadoop-yarn-sit=
e/ClusterSetup.htmlto
>> > configure everything.  hdfs seems to be working properly in that I can
>> > access the file system, load files, and read them.
>> >
>> > However, running jobs doesn't seem to work correctly.  I'm trying to r=
un
>> > just a sample job with
>> >
>> > hadoop jar
>> >
>> /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.=
0-cdh4b1.jar
>> > randomwriter -Dmapreduce.job.user.name=3D$USER -
>> > Dmapreduce.clientfactory.class.name
>> =3Dorg.apache.hadoop.mapred.YarnClientFactory
>> > -Dmapreduce.randomwriter.bytespermap=3D10000 -Ddfs.blocksize=3D5368709=
12
>> > -Ddfs.block.size=3D536870912 -libjars
>> >
>> /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclie=
nt-0.23.0-cdh4b1.jar
>> > output
>> >
>> > When running I get a ClassNotFoundException:
>> > org.apache.hadoop.hdfs.DistributedFileSystem exception on the local no=
de
>> > running the task.  I have fs.hdfs.impl set to be
>> > org.apache.hadoop.hdfs.DistributedFileSystem which i believe is to be
>> > correct.  But i'm not sure why the node isn't finding the class.
>> >
>> > In my setup, everything is located under /usr/local/hadoop on all the
>> nodes
>> > and all the relevant environment variables point to that directly.  So
>> when
>> > the local nodes start up they include this:
>> >
>> >    -classpath
>> >
>> /usr/local/hadoop/conf:/usr/local/hadoop/conf:/usr/local/hadoop/conf:/us=
r/local/hadoop-0.23.0-cdh4b1/sbin/..:/usr/local/hadoop-0.23.0-cdh4b1/sbin/.=
./lib/*:/usr/local/hadoop-0.23.0-cdh4b1/sbin/../*:/usr/local/hadoop-0.23.0-=
cdh4b1/sbin/../share/hadoop/common/lib/*:/usr/local/hadoop-0.23.0-cdh4b1/sb=
in/../share/hadoop/common/*:/usr/local/hadoop-0.23.0-cdh4b1/sbin/../share/h=
adoop/hdfs:/usr/local/hadoop-0.23.0-cdh4b1/sbin/../share/hadoop/hdfs/lib/*:=
/usr/local/hadoop-0.23.0-cdh4b1/sbin/../share/hadoop/hdfs/*:/usr/local/hado=
op/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:=
/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/m=
apreduce/lib/*:/usr/local/hadoop/conf/nm-config/log4j.properties
>> >
>> > which looks to be correct.  So I'm not exactly sure where the problem =
is
>> > coming from.
>> >
>> > Any suggestions on what might be wrong or how to further diagnose the
>> > problem would be greatly appreciated.
>> >
>> > Thanks!
>> > --Keith
>>
>>
>>
>> --
>> Harsh J
>>
>
>

--f46d04426f147ca68004bb04437b--