Return-Path: 
 <common-user-return-30075-apmail-hadoop-common-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-user-archive@www.apache.org
Delivered-To: apmail-hadoop-common-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDC4A78FC
	for <apmail-hadoop-common-user-archive@www.apache.org>;
 Fri, 16 Sep 2011 04:48:35 +0000 (UTC)
Received: (qmail 43761 invoked by uid 500); 16 Sep 2011 04:48:32 -0000
Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org
Received: (qmail 43694 invoked by uid 500); 16 Sep 2011 04:48:32 -0000
Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-user@hadoop.apache.org>
List-Id: <common-user.hadoop.apache.org>
Reply-To: common-user@hadoop.apache.org
Delivered-To: mailing list common-user@hadoop.apache.org
Received: (qmail 43670 invoked by uid 99); 16 Sep 2011 04:48:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Sep 2011 04:48:32 +0000
X-ASF-Spam-Status: No, hits=1.6 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of arunk786@gmail.com designates
 209.85.161.172 as permitted sender)
Received: from [209.85.161.172] (HELO mail-gx0-f172.google.com)
 (209.85.161.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Sep 2011 04:48:26 +0000
Received: by gxk19 with SMTP id 19so3925095gxk.3
        for <multiple recipients>; Thu, 15 Sep 2011 21:48:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        bh=OE51aEHwd67rAEOrEVCFWrE5Cdpgp8NQ7p0SOUJRjf8=;
        b=P2Q7QKcfIJx0Yyp0WgCHuNKXG8/b0jy9pTTzXLvoY8HVVPqwJ5Sk+6z8jSlQH3ac7e
         +tIyE6nlRm17lj8d/1V5YPybf61V3c+Qex7I+9W4zXJ+bAbWNKhJbV3Vn7UMei1M92T0
         fWz5Y+4kRpjFtKA0kUbosw+ZxY0K8U1cbobe8=
Received: by 10.236.75.165 with SMTP id z25mr11671266yhd.68.1316148484414;
 Thu, 15 Sep 2011 21:48:04 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.236.203.198 with HTTP; Thu, 15 Sep 2011 21:47:44 -0700 (PDT)
In-Reply-To: 
 <CAOcnVr1dM5R_C67SJZDae0ShB_+LOcgNvxUPj+ja2g5VYx6mtA@mail.gmail.com>
References: 
 <CAO7gJehqjtsiHqSZ7SSPDmGYd8UVihWwTifonkvVoMAC6wyL2Q@mail.gmail.com>
 <CA962183.1139C%tgraves@yahoo-inc.com>
 <CAO7gJeiQHpskp1vM-Euwnpj0U5obwarOWF=ZoDfCWw=+wVKOtA@mail.gmail.com>
 <CAOcnVr1dM5R_C67SJZDae0ShB_+LOcgNvxUPj+ja2g5VYx6mtA@mail.gmail.com>
From: arun k <arunk786@gmail.com>
Date: Fri, 16 Sep 2011 10:17:44 +0530
Message-ID: 
 <CAO7gJehkw9W5zQjHT+47+RoRGbpVqv5VLcQ7qKumw2jPTeY1JQ@mail.gmail.com>
Subject: Re: Running example application with capacity scheduler ?
To: mapreduce-user@hadoop.apache.org,
	common-user <common-user@hadoop.apache.org>, harsh@cloudera.com
Content-Type: multipart/alternative; boundary=20cf300fab35e29ea904ad07b28c
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf300fab35e29ea904ad07b28c
Content-Type: text/plain; charset=ISO-8859-1

Hi all !

Harsh ! Namenode appears to be out of safe mode :
In http://nn-host:50070 i see in time

T1>Safe mode is ON. The ratio of reported blocks 0.0000 has not reached the
threshold 0.9990. Safe mode will be turned off automatically.
7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / 966.69
MB (1%)

T2>Safe mode is ON. The ratio of reported blocks 1.0000 has reached the
threshold 0.9990. Safe mode will be turned off automatically in 17 seconds.
7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / 966.69
MB (1%)

T3>9 files and directories, 3 blocks = 12 total. Heap Size is 15.06 MB /
966.69 MB (1%)

Added properties :

 mapred.jobtracker.taskScheduler org.apache.hadoopertiep.mapred.CTS

 mapred.queue.names                          myqueue1,myqueue2
 mapred.capacity-scheduler.queue.myqueue1.capacity               25
 mapred.capacity-scheduler.queue.myqueue1.capacity               75

${HADOOP_HOME}$ bin/hadoop jar hadoop*examples*.jar wordcount -
Dmapred.job.queue.name <http://dmapred.job.queue.name/>=
myqueue1 /user/hduser/wcinput /user/hduser/wcoutput

I get the error:
java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local
exception: java.io.IOException: Connection reset by peer
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
    at org.apache.hadoop.ipc.Client.call(Client.java:1033)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
        ...................

When i give
$jps
32463 NameNode
32763 SecondaryNameNode
32611 DataNode
931 Jps


The jobracker log gives info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2011-09-16 00:21:42,012 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-16 00:21:42,014 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory:
hdfs://localhost:54310/app203/hadoop203/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app203/hadoop203/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
mode will be turned off automatically in 6 seconds.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1851)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1831)
2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker:
java.io.IOException: Queue 'myqueue1' doesn't have configured capacity!
    at
org.apache.hadoop.mapred.CapacityTaskScheduler.parseQueues(CapacityTaskScheduler.java:905)
    at
org.apache.hadoop.mapred.CapacityTaskScheduler.start(CapacityTaskScheduler.java:822)
    at
org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2563)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4957)

2011-09-16 00:21:52,322 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Even if i submit the job to "myqueue2" i see the same error of "myqueue1"
2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker:
java.io.IOException: Queue 'myqueue1' doesn't have configured capacity!

Thanks,
Arun



On Thu, Sep 15, 2011 at 5:23 PM, Harsh J <harsh@cloudera.com> wrote:

> Hello Arun,
>
> To me it looks like your HDFS isn't setup properly, in this case. Can
> you ensure all DNs are properly up? Your NN appears to have gotten
> stuck somehow into a safemode. Check out your http://nn-host:50070
> page for more details on why.
>
> Your JT won't come up until the NN is properly up and out of safemode
> (for which it needs the DNs). And once it comes up, I think you should
> be good to go, keeping in mind the changes Thomas mentioned earlier.
>
> On Thu, Sep 15, 2011 at 3:58 PM, arun k <arunk786@gmail.com> wrote:
> > Hi all !
> >
> > Thanks Thomas ! it's working in terminal.
> > I saw the queues in web UI of JT.
> > when i try to run normally again (default) i get this error :
> > i tried formatting namenode and making safemode off and restart but
> didn't
> > work.
> >
> > hduser@arun-Presario-C500-RU914PA-ACJ:/usr/local/hadoop$ bin/hadoop jar
> > hadoop*examples*.jar wordcount  /user/hduser/wcinput
> /user/hduser/wcoutput6
> > java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local
> > exception: java.io.IOException: Connection reset by peer
> >
> > The log of JobTracker shows :
> > 2011-09-15 12:46:13,346 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker
> > up at: 54311
> > 2011-09-15 12:46:13,347 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker
> > webserver: 50030
> > 2011-09-15 12:46:13,634 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-15 12:46:13,646 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> > hdfs://localhost:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> >
> > Thanks,
> > Arun
> >
> >
> >
> >
> >
> >
> > On Wed, Sep 14, 2011 at 7:46 PM, Thomas Graves <tgraves@yahoo-inc.com>
> > wrote:
> >>
> >> I believe it defaults to submit a job to the default queue if you don't
> >> specify it.  You don't have the default queue defined in your list of
> >> mapred.queue.names.  So add -Dmapred.job.queue.name=myqueue1 (or
> another
> >> queue you have defined) to the wordcount command like:
> >>
> >> bin/hadoop jar
> >> > hadoop*examples*.jar wordcount -Dmapred.job.queue.name=myqueue1
> >> /user/hduser/wcinput /user/hduser/wcoutput5
> >>
> >> Tom
> >>
> >>
> >> On 9/14/11 5:57 AM, "arun k" <arunk786@gmail.com> wrote:
> >>
> >> > Hi !
> >> >
> >> > I have set up single-node cluster using
> >> >
> >> >
> http://www.google.co.in/url?sa=t&source=web&cd=1&ved=0CB0QFjAA&url=http%3A%2F%
> >> >
> >> > 2Fwww.michael-noll.com
> %2Ftutorials%2Frunning-hadoop-on-ubuntu-linux-single-nod
> >> >
> >> >
> e-cluster%2F&rct=j&q=michael%20noll%20single%20node&ei=b4ZwTvrCLsOrrAei-N32Bg&
> >> >
> >> >
> usg=AFQjCNGhuvv0tNdvPj4u23bbj-qXJDlixg&sig2=7ij8Dy7aQZUkBwhTnS1rLw&cad=rja
> >> > and could run wordcount example application.
> >> > I was trying to run this application using capacity scheduler.
> >> > As per
> >> > http://hadoop.apache.org/common/docs/current/capacity_scheduler.htmli
> >> > have done :
> >> > 1.Copied the hadoop-capacity-scheduler-*.jar from *
> >> > contrib/capacity-scheduler* directory to HADOOP_HOME/lib
> >> > 2.Set mapred.jobtracker.taskScheduler
> >> > 3.Set *mapred.queue.names to myqueue1,myqueue2.
> >> > 4.Set *mapred.capacity-scheduler.queue.<queue-name>.capacity to 30 and
> >> > 70
> >> > for two queues.
> >> >
> >> > When i run i get the error :
> >> > hduser@arun-Presario-C500-RU914PA-ACJ:/usr/local/hadoop$ bin/hadoop
> jar
> >> > hadoop*examples*.jar wordcount /user/hduser/wcinput
> >> > /user/hduser/wcoutput5
> >> > 11/09/14 16:00:56 INFO input.FileInputFormat: Total input paths to
> >> > process :
> >> > 4
> >> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: Queue
> >> > "default"
> >> > does not exist
> >> >     at
> >> > org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2998)
> >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> >     at
> >> >
> >> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >     at
> >> >
> >> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> >> > ava:25)
> >> >     at java.lang.reflect.Method.invoke(Method.java:597)
> >> >     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> >> >     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> >> >     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> >> >     at java.security.AccessController.doPrivileged(Native Method)
> >> >     at javax.security.auth.Subject.doAs(Subject.java:396)
> >> >     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >> >
> >> >     at org.apache.hadoop.ipc.Client.call(Client.java:740)
> >> >     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> >> >     at org.apache.hadoop.mapred.$Proxy0.submitJob(Unknown Source)
> >> >     at
> >> >
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
> >> >     at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
> >> >     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
> >> >     at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> >     at
> >> >
> >> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >     at
> >> >
> >> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> >> > ava:25)
> >> >     at java.lang.reflect.Method.invoke(Method.java:597)
> >> >     at
> >> >
> >> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.j
> >> > ava:68)
> >> >     at
> >> > org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> >> >     at
> >> > org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> >     at
> >> >
> >> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >     at
> >> >
> >> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> >> > ava:25)
> >> >     at java.lang.reflect.Method.invoke(Method.java:597)
> >> >     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >> >
> >> > I didn't submit jobs to a particular queue as such. Do i need to do it
> ?
> >> > How
> >> > can i do it ?
> >> > Any help ?
> >> >
> >> > Thanks,
> >> > Arun
> >> >
> >> >
> >> > *
> >> > *
> >>
> >
> >
>
>
>
> --
> Harsh J
>

--20cf300fab35e29ea904ad07b28c--