hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Yamijala <yhema...@thoughtworks.com>
Subject Re: About running a simple wordcount mapreduce
Date Mon, 25 Mar 2013 06:11:30 GMT
Which version of Hadoop are you using. A quick search shows me a bug
https://issues.apache.org/jira/browse/HADOOP-5241 that seems to show
similar symptoms. However, that was fixed a long while ago.


On Sat, Mar 23, 2013 at 4:40 PM, Redwane belmaati cherkaoui <
reduno1985@googlemail.com> wrote:

> This the content of the jobtracker log file :
> 2013-03-23 12:06:48,912 INFO org.apache.hadoop.mapred.JobInProgress: Input
> size for job job_201303231139_0001 = 6950001. Number of splits = 7
> 2013-03-23 12:06:48,925 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201303231139_0001_m_000000 has split on
> node:/default-rack/hadoop0.novalocal
> 2013-03-23 12:06:48,927 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201303231139_0001_m_000001 has split on
> node:/default-rack/hadoop0.novalocal
> 2013-03-23 12:06:48,930 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201303231139_0001_m_000002 has split on
> node:/default-rack/hadoop0.novalocal
> 2013-03-23 12:06:48,931 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201303231139_0001_m_000003 has split on
> node:/default-rack/hadoop0.novalocal
> 2013-03-23 12:06:48,933 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201303231139_0001_m_000004 has split on
> node:/default-rack/hadoop0.novalocal
> 2013-03-23 12:06:48,934 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201303231139_0001_m_000005 has split on
> node:/default-rack/hadoop0.novalocal
> 2013-03-23 12:06:48,939 INFO org.apache.hadoop.mapred.JobInProgress:
> tip:task_201303231139_0001_m_000006 has split on
> node:/default-rack/hadoop0.novalocal
> 2013-03-23 12:06:48,950 INFO org.apache.hadoop.mapred.JobInProgress:
> job_201303231139_0001 LOCALITY_WAIT_FACTOR=0.5
> 2013-03-23 12:06:48,978 INFO org.apache.hadoop.mapred.JobInProgress: Job
> job_201303231139_0001 initialized successfully with 7 map tasks and 1
> reduce tasks.
> 2013-03-23 12:06:50,855 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip
> task_201303231139_0001_m_000008, for tracker
> 'tracker_hadoop0.novalocal:hadoop0.novalocal/127.0.0.1:44879'
> 2013-03-23 12:08:00,340 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201303231139_0001_m_000008_0' has completed
> task_201303231139_0001_m_000008 successfully.
> 2013-03-23 12:08:00,538 WARN org.apache.hadoop.mapred.JobInProgress: No
> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we
> expect map to take 1317624576693539401
> 2013-03-23 12:08:00,543 WARN org.apache.hadoop.mapred.JobInProgress: No
> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we
> expect map to take 1317624576693539401
> 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: No
> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we
> expect map to take 1317624576693539401
> 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: No
> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we
> expect map to take 1317624576693539401
> 2013-03-23 12:08:01,264 WARN org.apache.hadoop.mapred.JobInProgress: No
> room for map task. Node hadoop1.novalocal has 8807518208 bytes free; but we
> expect map to take 1317624576693539401
>
>
> The value in we excpect map to take is too huge   1317624576693539401
> bytes  !!!!!!!
>
> On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui <
> reduno1985@googlemail.com> wrote:
>
>> The estimated value that the hadoop compute is too huge for the simple
>> example that i am running .
>>
>> ---------- Forwarded message ----------
>> From: Redwane belmaati cherkaoui <reduno1985@googlemail.com>
>>  Date: Sat, Mar 23, 2013 at 11:32 AM
>> Subject: Re: About running a simple wordcount mapreduce
>> To: Abdelrahman Shettia <ashettia@hortonworks.com>
>> Cc: user@hadoop.apache.org, reduno1985 <reduno1985@gmail.com>
>>
>>
>> This the output that I get I am running two machines  as you can see  do
>> u see anything suspicious ?
>> Configured Capacity: 21145698304 (19.69 GB)
>> Present Capacity: 17615499264 (16.41 GB)
>> DFS Remaining: 17615441920 (16.41 GB)
>> DFS Used: 57344 (56 KB)
>> DFS Used%: 0%
>> Under replicated blocks: 0
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>>
>> -------------------------------------------------
>> Datanodes available: 2 (2 total, 0 dead)
>>
>> Name: 11.1.0.6:50010
>> Decommission Status : Normal
>> Configured Capacity: 10572849152 (9.85 GB)
>> DFS Used: 28672 (28 KB)
>> Non DFS Used: 1765019648 (1.64 GB)
>> DFS Remaining: 8807800832(8.2 GB)
>> DFS Used%: 0%
>> DFS Remaining%: 83.31%
>> Last contact: Sat Mar 23 11:30:10 CET 2013
>>
>>
>> Name: 11.1.0.3:50010
>> Decommission Status : Normal
>> Configured Capacity: 10572849152 (9.85 GB)
>> DFS Used: 28672 (28 KB)
>> Non DFS Used: 1765179392 (1.64 GB)
>> DFS Remaining: 8807641088(8.2 GB)
>> DFS Used%: 0%
>> DFS Remaining%: 83.3%
>> Last contact: Sat Mar 23 11:30:08 CET 2013
>>
>>
>> On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Shettia <
>> ashettia@hortonworks.com> wrote:
>>
>>> Hi Redwane,
>>>
>>> Please run the following command as hdfs user on any datanode. The
>>> output will be something like this. Hope this helps
>>>
>>> hadoop dfsadmin -report
>>> Configured Capacity: 81075068925 (75.51 GB)
>>> Present Capacity: 70375292928 (65.54 GB)
>>> DFS Remaining: 69895163904 (65.09 GB)
>>> DFS Used: 480129024 (457.89 MB)
>>> DFS Used%: 0.68%
>>> Under replicated blocks: 0
>>> Blocks with corrupt replicas: 0
>>> Missing blocks: 0
>>>
>>> Thanks
>>> -Abdelrahman
>>>
>>>
>>> On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 <reduno1985@googlemail.com>wrote:
>>>
>>>>
>>>> I have my hosts running on openstack virtual machine instances each
>>>> instance has 10gb hard disc . Is there a way too see how much space is in
>>>> the hdfs without web ui .
>>>>
>>>>
>>>> Sent from Samsung Mobile
>>>>
>>>> Serge Blazhievsky <hadoop.ca@gmail.com> wrote:
>>>> Check web ui how much space you have on hdfs???
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia <
>>>> ashettia@hortonworks.com> wrote:
>>>>
>>>> Hi Redwane ,
>>>>
>>>> It is possible that the hosts which are running tasks are do not have
>>>> enough space. Those dirs are confiugred in mapred-site.xml
>>>>
>>>>
>>>>
>>>> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui <
>>>> reduno1985@googlemail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Redwane belmaati cherkaoui <reduno1985@googlemail.com>
>>>>> Date: Fri, Mar 22, 2013 at 4:39 PM
>>>>> Subject: About running a simple wordcount mapreduce
>>>>> To: mapreduce-issues@hadoop.apache.org
>>>>>
>>>>>
>>>>> Hi
>>>>> I am trying to run  a wordcount mapreduce job on several files (<20
>>>>> mb) using two machines . I get stuck on 0% map 0% reduce.
>>>>> The jobtracker log file shows the following warning:
>>>>>  WARN org.apache.hadoop.mapred.JobInProgress: No room for map task.
>>>>> Node hadoop0.novalocal has 8791384064 bytes free; but we expect map to
take
>>>>> 1317624576693539401
>>>>>
>>>>> Please help me ,
>>>>> Best Regards,
>>>>>
>>>>>
>>>>
>>>
>>
>>
>

Mime
View raw message