Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D334EF076 for ; Sat, 23 Mar 2013 11:10:48 +0000 (UTC) Received: (qmail 45142 invoked by uid 500); 23 Mar 2013 11:10:43 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 44969 invoked by uid 500); 23 Mar 2013 11:10:40 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 44935 invoked by uid 99); 23 Mar 2013 11:10:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Mar 2013 11:10:39 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of reduno1985@googlemail.com designates 209.85.212.171 as permitted sender) Received: from [209.85.212.171] (HELO mail-wi0-f171.google.com) (209.85.212.171) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Mar 2013 11:10:33 +0000 Received: by mail-wi0-f171.google.com with SMTP id hn17so8676997wib.16 for ; Sat, 23 Mar 2013 04:10:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=7XanCgwjjWXwoxZVhD8Jhg0SS4dD5hYrfIkUKRvCi+Y=; b=SH3DIxrQh2DBfMf5T9HqxM4K0/K+X50lt+28WH1kUFBMcUbInPIfIRl5U+Ckh1hnDX u+ZKvy6db8YSJrXGYcorg9/J/DoFs2nw1iubCNX+C9hm0GaLMalhfMfMimpzonOc1FGT VogLRdpaiwL8CFqCVo4QqlhYEwM+Q09U7KwLLyvAf4AQvB5ilJJB8pU5WRDbAKxqStDr rAk+XdFGJCVursV1DzmA6MnIZvnpvSkCoHM8Cz7vjdOStQpWH8AXpWBiw93hpC/ck1p3 SyA/RUBgbb1AnoulqL/EamSuGb1R7bkgoMBp4uYRRMtyRDZ2H8sqDX+MpFpYBRynzc4v UgVw== MIME-Version: 1.0 X-Received: by 10.180.97.166 with SMTP id eb6mr16086101wib.20.1364037013363; Sat, 23 Mar 2013 04:10:13 -0700 (PDT) Received: by 10.194.64.130 with HTTP; Sat, 23 Mar 2013 04:10:13 -0700 (PDT) In-Reply-To: References: Date: Sat, 23 Mar 2013 12:10:13 +0100 Message-ID: Subject: Re: About running a simple wordcount mapreduce From: Redwane belmaati cherkaoui To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d04430664a462e804d8959d55 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04430664a462e804d8959d55 Content-Type: text/plain; charset=ISO-8859-1 This the content of the jobtracker log file : 2013-03-23 12:06:48,912 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201303231139_0001 = 6950001. Number of splits = 7 2013-03-23 12:06:48,925 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201303231139_0001_m_000000 has split on node:/default-rack/hadoop0.novalocal 2013-03-23 12:06:48,927 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201303231139_0001_m_000001 has split on node:/default-rack/hadoop0.novalocal 2013-03-23 12:06:48,930 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201303231139_0001_m_000002 has split on node:/default-rack/hadoop0.novalocal 2013-03-23 12:06:48,931 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201303231139_0001_m_000003 has split on node:/default-rack/hadoop0.novalocal 2013-03-23 12:06:48,933 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201303231139_0001_m_000004 has split on node:/default-rack/hadoop0.novalocal 2013-03-23 12:06:48,934 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201303231139_0001_m_000005 has split on node:/default-rack/hadoop0.novalocal 2013-03-23 12:06:48,939 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201303231139_0001_m_000006 has split on node:/default-rack/hadoop0.novalocal 2013-03-23 12:06:48,950 INFO org.apache.hadoop.mapred.JobInProgress: job_201303231139_0001 LOCALITY_WAIT_FACTOR=0.5 2013-03-23 12:06:48,978 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201303231139_0001 initialized successfully with 7 map tasks and 1 reduce tasks. 2013-03-23 12:06:50,855 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip task_201303231139_0001_m_000008, for tracker 'tracker_hadoop0.novalocal:hadoop0.novalocal/127.0.0.1:44879' 2013-03-23 12:08:00,340 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201303231139_0001_m_000008_0' has completed task_201303231139_0001_m_000008 successfully. 2013-03-23 12:08:00,538 WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we expect map to take 1317624576693539401 2013-03-23 12:08:00,543 WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we expect map to take 1317624576693539401 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we expect map to take 1317624576693539401 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we expect map to take 1317624576693539401 2013-03-23 12:08:01,264 WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node hadoop1.novalocal has 8807518208 bytes free; but we expect map to take 1317624576693539401 The value in we excpect map to take is too huge 1317624576693539401 bytes !!!!!!! On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui < reduno1985@googlemail.com> wrote: > The estimated value that the hadoop compute is too huge for the simple > example that i am running . > > ---------- Forwarded message ---------- > From: Redwane belmaati cherkaoui > Date: Sat, Mar 23, 2013 at 11:32 AM > Subject: Re: About running a simple wordcount mapreduce > To: Abdelrahman Shettia > Cc: user@hadoop.apache.org, reduno1985 > > > This the output that I get I am running two machines as you can see do u > see anything suspicious ? > Configured Capacity: 21145698304 (19.69 GB) > Present Capacity: 17615499264 (16.41 GB) > DFS Remaining: 17615441920 (16.41 GB) > DFS Used: 57344 (56 KB) > DFS Used%: 0% > Under replicated blocks: 0 > Blocks with corrupt replicas: 0 > Missing blocks: 0 > > ------------------------------------------------- > Datanodes available: 2 (2 total, 0 dead) > > Name: 11.1.0.6:50010 > Decommission Status : Normal > Configured Capacity: 10572849152 (9.85 GB) > DFS Used: 28672 (28 KB) > Non DFS Used: 1765019648 (1.64 GB) > DFS Remaining: 8807800832(8.2 GB) > DFS Used%: 0% > DFS Remaining%: 83.31% > Last contact: Sat Mar 23 11:30:10 CET 2013 > > > Name: 11.1.0.3:50010 > Decommission Status : Normal > Configured Capacity: 10572849152 (9.85 GB) > DFS Used: 28672 (28 KB) > Non DFS Used: 1765179392 (1.64 GB) > DFS Remaining: 8807641088(8.2 GB) > DFS Used%: 0% > DFS Remaining%: 83.3% > Last contact: Sat Mar 23 11:30:08 CET 2013 > > > On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Shettia < > ashettia@hortonworks.com> wrote: > >> Hi Redwane, >> >> Please run the following command as hdfs user on any datanode. The output >> will be something like this. Hope this helps >> >> hadoop dfsadmin -report >> Configured Capacity: 81075068925 (75.51 GB) >> Present Capacity: 70375292928 (65.54 GB) >> DFS Remaining: 69895163904 (65.09 GB) >> DFS Used: 480129024 (457.89 MB) >> DFS Used%: 0.68% >> Under replicated blocks: 0 >> Blocks with corrupt replicas: 0 >> Missing blocks: 0 >> >> Thanks >> -Abdelrahman >> >> >> On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 wrote: >> >>> >>> I have my hosts running on openstack virtual machine instances each >>> instance has 10gb hard disc . Is there a way too see how much space is in >>> the hdfs without web ui . >>> >>> >>> Sent from Samsung Mobile >>> >>> Serge Blazhievsky wrote: >>> Check web ui how much space you have on hdfs??? >>> >>> Sent from my iPhone >>> >>> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia < >>> ashettia@hortonworks.com> wrote: >>> >>> Hi Redwane , >>> >>> It is possible that the hosts which are running tasks are do not have >>> enough space. Those dirs are confiugred in mapred-site.xml >>> >>> >>> >>> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui < >>> reduno1985@googlemail.com> wrote: >>> >>>> >>>> >>>> ---------- Forwarded message ---------- >>>> From: Redwane belmaati cherkaoui >>>> Date: Fri, Mar 22, 2013 at 4:39 PM >>>> Subject: About running a simple wordcount mapreduce >>>> To: mapreduce-issues@hadoop.apache.org >>>> >>>> >>>> Hi >>>> I am trying to run a wordcount mapreduce job on several files (<20 mb) >>>> using two machines . I get stuck on 0% map 0% reduce. >>>> The jobtracker log file shows the following warning: >>>> WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. >>>> Node hadoop0.novalocal has 8791384064 bytes free; but we expect map to take >>>> 1317624576693539401 >>>> >>>> Please help me , >>>> Best Regards, >>>> >>>> >>> >> > > --f46d04430664a462e804d8959d55 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable This the content of the jobtracker log file :
2013-03-23 12:06:48,= 912 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201= 303231139_0001 =3D 6950001. Number of splits =3D 7
2013-03-23 12:= 06:48,925 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_20130323113= 9_0001_m_000000 has split on node:/default-rack/hadoop0.novalocal
2013-03-23 12:06:48,927 INFO org.apache.hadoop.mapred.JobInProgress: t= ip:task_201303231139_0001_m_000001 has split on node:/default-rack/hadoop0.= novalocal
2013-03-23 12:06:48,930 INFO org.apache.hadoop.mapred.J= obInProgress: tip:task_201303231139_0001_m_000002 has split on node:/defaul= t-rack/hadoop0.novalocal
2013-03-23 12:06:48,931 INFO org.apache.hadoop.mapred.JobInProgress: t= ip:task_201303231139_0001_m_000003 has split on node:/default-rack/hadoop0.= novalocal
2013-03-23 12:06:48,933 INFO org.apache.hadoop.mapred.J= obInProgress: tip:task_201303231139_0001_m_000004 has split on node:/defaul= t-rack/hadoop0.novalocal
2013-03-23 12:06:48,934 INFO org.apache.hadoop.mapred.JobInProgress: t= ip:task_201303231139_0001_m_000005 has split on node:/default-rack/hadoop0.= novalocal
2013-03-23 12:06:48,939 INFO org.apache.hadoop.mapred.J= obInProgress: tip:task_201303231139_0001_m_000006 has split on node:/defaul= t-rack/hadoop0.novalocal
2013-03-23 12:06:48,950 INFO org.apache.hadoop.mapred.JobInProgress: j= ob_201303231139_0001 LOCALITY_WAIT_FACTOR=3D0.5
2013-03-23 12:06:= 48,978 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201303231139_00= 01 initialized successfully with 7 map tasks and 1 reduce tasks.
2013-03-23 12:06:50,855 INFO org.apache.hadoop.mapred.JobTracker: Addi= ng task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip t= ask_201303231139_0001_m_000008, for tracker 'tracker_hadoop0.novalocal:= hadoop0.novalocal/127.0.0.1:44879= 9;
2013-03-23 12:08:00,340 INFO org.apache.hadoop.mapred.JobInProgress: T= ask 'attempt_201303231139_0001_m_000008_0' has completed task_20130= 3231139_0001_m_000008 successfully.
2013-03-23 12:08:00,538 WARN = org.apache.hadoop.mapred.JobInProgress: No room for map task. Node hadoop0.= novalocal has 8791543808 bytes free; but we expect map to take 131762457669= 3539401
2013-03-23 12:08:00,543 WARN org.apache.hadoop.mapred.JobInProgress: N= o room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but = we expect map to take 1317624576693539401
2013-03-23 12:08:00,544= WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node ha= doop0.novalocal has 8791543808 bytes free; but we expect map to take 131762= 4576693539401
2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: N= o room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but = we expect map to take 1317624576693539401
2013-03-23 12:08:01,264= WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. Node ha= doop1.novalocal has 8807518208 bytes free; but we expect map to take 131762= 4576693539401


The value in we excpect map to take is t= oo huge =A0=A01317624576693539401 bytes =A0!!!!!!!

On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui <reduno1985@googlemail.com> wrote:
The estimated value that the hadoop compute = is too huge for the simple example that i am running .

---------- Forwarded message ----------
From: Redwane belmaati cherkaoui <= reduno1985@g= ooglemail.com>
Date: Sat, Mar 23, 2013 at 11:32 AM
Subject: Re: About running a simple = wordcount mapreduce
To: Abdelrahman Shettia <ashettia@hortonworks.com>
Cc: user@hadoop= .apache.org, reduno1985 <reduno1985@gmail.com>


This the output that I get I am running two machines =A0as you can = see =A0do u see anything suspicious ?
Configured Capacity: 2114569= 8304 (19.69 GB)
Present Capacity: 17615499264 (16.41 GB)
DFS Remaining: 17615441920 (16.41 GB)
DFS Used: 57344 (56 KB)
DFS Used%: 0%
Under r= eplicated blocks: 0
Blocks with corrupt replicas: 0
Mis= sing blocks: 0

-----------------------------= --------------------
Datanodes available: 2 (2 total, 0 dead)

Name= : 11.1.0.6:50010
Decommission Status : Normal
Configured Capacity: 1057284= 9152 (9.85 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 1765019648 (1.64 GB)
DFS Remaining: 8807800832(8.2 GB)
DFS Used%: 0%
DFS Remaining%: 83.31%
Last contact: Sat Mar 23 11:30:10 CET 20= 13


Decommission Status : Normal=
Configured Capacity: 10572849152 (9.85 GB)
DFS Used: 2= 8672 (28 KB)
Non DFS Used: 1765179392 (1.64 GB)
DFS Remaining: 8807641088= (8.2 GB)
DFS Used%: 0%
DFS Remaining%: 83.3%
= Last contact: Sat Mar 23 11:30:08 CET 2013


On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Sh= ettia <ashettia@hortonworks.com> wrote:
Hi Redwane,=A0

Please run the following= command as hdfs user on any datanode. The output will be something like th= is. Hope this helps=A0

hadoop dfsadmin -repor= t
Configured Capacity: 81075068925 (75.51 GB)
Present Capacity= : 70375292928 (65.54 GB)
DFS Remaining: 69895163904 (65.09 GB)
DFS Used: 480129024 (457.89 MB)
DFS Used%: 0.68%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

Thanks
-Abdelrahman=A0


On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 <reduno1985@googlema= il.com> wrote:

I have my hosts running on openstack virtual machi= ne instances each instance has 10gb hard disc . Is there a way too see how = much space is in the hdfs without web ui .


Sent from Samsung Mob= ile

Serge Blazhievsky <hadoop.ca@gmail.com> wrote:
Check web ui how much space you have on hdfs???

Sent from my iP= hone

On Mar 22, 2013, at 11:41 AM, Abdelrahman She= ttia <ashe= ttia@hortonworks.com> wrote:

Hi Redwane ,=A0
It is possible that the hosts which are running tasks are= do not have enough space. Those dirs are confiugred in mapred-site.xml



On Fri, Mar 22, 2013 at 8:42 AM, Redwane= belmaati cherkaoui <reduno1985@googlemail.com> wrot= e:


---------= - Forwarded message ----------
From: Redwa= ne belmaati cherkaoui <reduno1985@googlemail.com>
Date: Fri, Mar 22, 2013 at 4:39 PM
Subject: About running a simple wordc= ount mapreduce
To: mapreduce-issues@hadoop.apache.org


Hi I am trying to run=A0 a wordcount mapreduce job on several files (<20 mb= ) using two machines . I get stuck on 0% map 0% reduce.
The jobtracker log file shows the following warning:
=A0WARN org.apache.= hadoop.mapred.JobInProgress: No room for map task. Node hadoop0.novalocal h= as 8791384064 bytes free; but we expect map to take 1317624576693539401

Please help me ,
Best Regards,






--f46d04430664a462e804d8959d55--