Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Lanati, Matteo" <Matteo.Lanati@lrz.de>
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Subject: Re: 
Thread-Index: AQHOYKpdZMmnsWlgmkWoL84agNkpOw==
Date: Mon, 3 Jun 2013 22:33:26 +0000
Message-ID: <11F6F9CB-D131-4355-B78C-C423822B2B6B@lrz.de>
References: 
 <31270DDF18CC924782A594673481B9154BD3897B@BADWLRZ-SWMBX11.ads.mwn.de>
 <CAOcnVr3Us3wrcagJR9rivpd3tL-Y5sHsStEj3nmOcCnXYcU8Fw@mail.gmail.com>
 <CAEo-6+R=d9ZwyH4rBuN5zADB23LCYFWLs9eibGiwQ5azX_OCFQ@mail.gmail.com>
 <CAOcnVr29toU83oNW9VpB3RdO9VH+tvWjBnuiLDdSn=usOMgH3A@mail.gmail.com>
 <CAEo-6+SS-5T6bAS0xbrtvAtN0+UCFm5rKCDAhbGXk3yZjq6tbg@mail.gmail.com>
 <CALr1C9qBOSxEG88SMgZso7UqZCeck6Yv=-2UAYE0jhPiTiMOnw@mail.gmail.com>
In-Reply-To: 
 <CALr1C9qBOSxEG88SMgZso7UqZCeck6Yv=-2UAYE0jhPiTiMOnw@mail.gmail.com>
Accept-Language: en-GB, it-IT, en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <889F0C33865BD343B7F8433872E52DCB@ads.mwn.de>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Azuryy,

thanks for the update. Sorry for the silly question, but where can I downlo=
ad the patched version?
If I look into the closest mirror (i.e. http://mirror.netcologne.de/apache.=
org/hadoop/common/), I can see that the Hadoop 1.1.2 version was last updat=
ed on Jan. 31st.
Thanks in advance,

Matteo

PS: just to confirm that I tried a minimal Hadoop 1.2.0 setup, so without a=
ny security, and the problem is there.

On Jun 3, 2013, at 3:02 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:

> can you upgrade to 1.1.2, which is also a stable release, and fixed the b=
ug you facing now.
>=20
> --Send from my Sony mobile.
>=20
> On Jun 2, 2013 3:23 AM, "Shahab Yunus" <shahab.yunus@gmail.com> wrote:
> Thanks Harsh for the reply. I was confused too that why security is causi=
ng this.
>=20
> Regards,
> Shahab
>=20
>=20
> On Sat, Jun 1, 2013 at 12:43 PM, Harsh J <harsh@cloudera.com> wrote:
> Shahab - I see he has mentioned generally that security is enabled
> (but not that it happens iff security is enabled), and the issue here
> doesn't have anything to do with security really.
>=20
> Azurry - Lets discuss the code issues on the JIRA (instead of here) or
> on the mapreduce-dev lists.
>=20
> On Sat, Jun 1, 2013 at 10:05 PM, Shahab Yunus <shahab.yunus@gmail.com> wr=
ote:
> > HI Harsh,
> >
> > Quick question though: why do you think it only happens if the OP 'uses
> > security' as he mentioned?
> >
> > Regards,
> > Shahab
> >
> >
> > On Sat, Jun 1, 2013 at 11:49 AM, Harsh J <harsh@cloudera.com> wrote:
> >>
> >> Does smell like a bug as that number you get is simply Long.MAX_VALUE,
> >> or 8 exbibytes.
> >>
> >> Looking at the sources, this turns out to be a rather funny Java issue
> >> (there's a divide by zero happening and [1] suggests Long.MAX_VALUE
> >> return in such a case). I've logged a bug report for this at
> >> https://issues.apache.org/jira/browse/MAPREDUCE-5288 with a
> >> reproducible case.
> >>
> >> Does this happen consistently for you?
> >>
> >> [1]
> >> http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(dou=
ble)
> >>
> >> On Sat, Jun 1, 2013 at 7:27 PM, Lanati, Matteo <Matteo.Lanati@lrz.de>
> >> wrote:
> >> > Hi all,
> >> >
> >> > I stumbled upon this problem as well while trying to run the default
> >> > wordcount shipped with Hadoop 1.2.0. My testbed is made up of 2 virt=
ual
> >> > machines: Debian 7, Oracle Java 7, 2 GB RAM, 25 GB hard disk. One no=
de is
> >> > used as JT+NN, the other as TT+DN. Security is enabled. The input fi=
le is
> >> > about 600 kB and the error is
> >> >
> >> > 2013-06-01 12:22:51,999 WARN org.apache.hadoop.mapred.JobInProgress:=
 No
> >> > room for map task. Node 10.156.120.49 has 22854692864 bytes free; bu=
t we
> >> > expect map to take 9223372036854775807
> >> >
> >> > The logfile is attached, together with the configuration files. The
> >> > version I'm using is
> >> >
> >> > Hadoop 1.2.0
> >> > Subversion
> >> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -=
r
> >> > 1479473
> >> > Compiled by hortonfo on Mon May  6 06:59:37 UTC 2013
> >> > From source with checksum 2e0dac51ede113c1f2ca8e7d82fb3405
> >> > This command was run using
> >> > /home/lu95jib/hadoop-exmpl/hadoop-1.2.0/hadoop-core-1.2.0.jar
> >> >
> >> > If I run the default configuration (i.e. no securty), then the job
> >> > succeeds.
> >> >
> >> > Is there something missing in how I set up my nodes? How is it possi=
ble
> >> > that the envisaged value for the needed space is so big?
> >> >
> >> > Thanks in advance.
> >> >
> >> > Matteo
> >> >
> >> >
> >> >
> >> >>Which version of Hadoop are you using. A quick search shows me a bug
> >> >>https://issues.apache.org/jira/browse/HADOOP-5241 that seems to show
> >> >>similar symptoms. However, that was fixed a long while ago.
> >> >>
> >> >>
> >> >>On Sat, Mar 23, 2013 at 4:40 PM, Redwane belmaati cherkaoui <
> >> >>reduno1985@googlemail.com> wrote:
> >> >>
> >> >>> This the content of the jobtracker log file :
> >> >>> 2013-03-23 12:06:48,912 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> Input
> >> >>> size for job job_201303231139_0001 =3D 6950001. Number of splits =
=3D 7
> >> >>> 2013-03-23 12:06:48,925 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> tip:task_201303231139_0001_m_000000 has split on
> >> >>> node:/default-rack/hadoop0.novalocal
> >> >>> 2013-03-23 12:06:48,927 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> tip:task_201303231139_0001_m_000001 has split on
> >> >>> node:/default-rack/hadoop0.novalocal
> >> >>> 2013-03-23 12:06:48,930 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> tip:task_201303231139_0001_m_000002 has split on
> >> >>> node:/default-rack/hadoop0.novalocal
> >> >>> 2013-03-23 12:06:48,931 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> tip:task_201303231139_0001_m_000003 has split on
> >> >>> node:/default-rack/hadoop0.novalocal
> >> >>> 2013-03-23 12:06:48,933 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> tip:task_201303231139_0001_m_000004 has split on
> >> >>> node:/default-rack/hadoop0.novalocal
> >> >>> 2013-03-23 12:06:48,934 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> tip:task_201303231139_0001_m_000005 has split on
> >> >>> node:/default-rack/hadoop0.novalocal
> >> >>> 2013-03-23 12:06:48,939 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> tip:task_201303231139_0001_m_000006 has split on
> >> >>> node:/default-rack/hadoop0.novalocal
> >> >>> 2013-03-23 12:06:48,950 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> job_201303231139_0001 LOCALITY_WAIT_FACTOR=3D0.5
> >> >>> 2013-03-23 12:06:48,978 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> Job
> >> >>> job_201303231139_0001 initialized successfully with 7 map tasks an=
d 1
> >> >>> reduce tasks.
> >> >>> 2013-03-23 12:06:50,855 INFO org.apache.hadoop.mapred.JobTracker:
> >> >>> Adding
> >> >>> task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip
> >> >>> task_201303231139_0001_m_000008, for tracker
> >> >>> 'tracker_hadoop0.novalocal:hadoop0.novalocal/127.0.0.1:44879'
> >> >>> 2013-03-23 12:08:00,340 INFO org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> Task
> >> >>> 'attempt_201303231139_0001_m_000008_0' has completed
> >> >>> task_201303231139_0001_m_000008 successfully.
> >> >>> 2013-03-23 12:08:00,538 WARN org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> No
> >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes fre=
e;
> >> >>> but we
> >> >>> expect map to take 1317624576693539401
> >> >>> 2013-03-23 12:08:00,543 WARN org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> No
> >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes fre=
e;
> >> >>> but we
> >> >>> expect map to take 1317624576693539401
> >> >>> 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> No
> >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes fre=
e;
> >> >>> but we
> >> >>> expect map to take 1317624576693539401
> >> >>> 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> No
> >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes fre=
e;
> >> >>> but we
> >> >>> expect map to take 1317624576693539401
> >> >>> 2013-03-23 12:08:01,264 WARN org.apache.hadoop.mapred.JobInProgres=
s:
> >> >>> No
> >> >>> room for map task. Node hadoop1.novalocal has 8807518208 bytes fre=
e;
> >> >>> but we
> >> >>> expect map to take 1317624576693539401
> >> >>>
> >> >>>
> >> >>> The value in we excpect map to take is too huge   1317624576693539=
401
> >> >>> bytes  !!!!!!!
> >> >>>
> >> >>> On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui <
> >> >>> reduno1985@googlemail.com> wrote:
> >> >>>
> >> >>>> The estimated value that the hadoop compute is too huge for the
> >> >>>> simple
> >> >>>> example that i am running .
> >> >>>>
> >> >>>> ---------- Forwarded message ----------
> >> >>>> From: Redwane belmaati cherkaoui <reduno1985@googlemail.com>
> >> >>>>  Date: Sat, Mar 23, 2013 at 11:32 AM
> >> >>>> Subject: Re: About running a simple wordcount mapreduce
> >> >>>> To: Abdelrahman Shettia <ashettia@hortonworks.com>
> >> >>>> Cc: user@hadoop.apache.org, reduno1985 <reduno1985@gmail.com>
> >> >>>>
> >> >>>>
> >> >>>> This the output that I get I am running two machines  as you can =
see
> >> >>>> do
> >> >>>> u see anything suspicious ?
> >> >>>> Configured Capacity: 21145698304 (19.69 GB)
> >> >>>> Present Capacity: 17615499264 (16.41 GB)
> >> >>>> DFS Remaining: 17615441920 (16.41 GB)
> >> >>>> DFS Used: 57344 (56 KB)
> >> >>>> DFS Used%: 0%
> >> >>>> Under replicated blocks: 0
> >> >>>> Blocks with corrupt replicas: 0
> >> >>>> Missing blocks: 0
> >> >>>>
> >> >>>> -------------------------------------------------
> >> >>>> Datanodes available: 2 (2 total, 0 dead)
> >> >>>>
> >> >>>> Name: 11.1.0.6:50010
> >> >>>> Decommission Status : Normal
> >> >>>> Configured Capacity: 10572849152 (9.85 GB)
> >> >>>> DFS Used: 28672 (28 KB)
> >> >>>> Non DFS Used: 1765019648 (1.64 GB)
> >> >>>> DFS Remaining: 8807800832(8.2 GB)
> >> >>>> DFS Used%: 0%
> >> >>>> DFS Remaining%: 83.31%
> >> >>>> Last contact: Sat Mar 23 11:30:10 CET 2013
> >> >>>>
> >> >>>>
> >> >>>> Name: 11.1.0.3:50010
> >> >>>> Decommission Status : Normal
> >> >>>> Configured Capacity: 10572849152 (9.85 GB)
> >> >>>> DFS Used: 28672 (28 KB)
> >> >>>> Non DFS Used: 1765179392 (1.64 GB)
> >> >>>> DFS Remaining: 8807641088(8.2 GB)
> >> >>>> DFS Used%: 0%
> >> >>>> DFS Remaining%: 83.3%
> >> >>>> Last contact: Sat Mar 23 11:30:08 CET 2013
> >> >>>>
> >> >>>>
> >> >>>> On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Shettia <
> >> >>>> ashettia@hortonworks.com> wrote:
> >> >>>>
> >> >>>>> Hi Redwane,
> >> >>>>>
> >> >>>>> Please run the following command as hdfs user on any datanode. T=
he
> >> >>>>> output will be something like this. Hope this helps
> >> >>>>>
> >> >>>>> hadoop dfsadmin -report
> >> >>>>> Configured Capacity: 81075068925 (75.51 GB)
> >> >>>>> Present Capacity: 70375292928 (65.54 GB)
> >> >>>>> DFS Remaining: 69895163904 (65.09 GB)
> >> >>>>> DFS Used: 480129024 (457.89 MB)
> >> >>>>> DFS Used%: 0.68%
> >> >>>>> Under replicated blocks: 0
> >> >>>>> Blocks with corrupt replicas: 0
> >> >>>>> Missing blocks: 0
> >> >>>>>
> >> >>>>> Thanks
> >> >>>>> -Abdelrahman
> >> >>>>>
> >> >>>>>
> >> >>>>> On Fri, Mar 22, 2013 at 12:35 PM, reduno1985
> >> >>>>> <reduno1985@googlemail.com>wrote:
> >> >>>>>
> >> >>>>>>
> >> >>>>>> I have my hosts running on openstack virtual machine instances =
each
> >> >>>>>> instance has 10gb hard disc . Is there a way too see how much s=
pace
> >> >>>>>> is in
> >> >>>>>> the hdfs without web ui .
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> Sent from Samsung Mobile
> >> >>>>>>
> >> >>>>>> Serge Blazhievsky <hadoop.ca@gmail.com> wrote:
> >> >>>>>> Check web ui how much space you have on hdfs???
> >> >>>>>>
> >> >>>>>> Sent from my iPhone
> >> >>>>>>
> >> >>>>>> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia <
> >> >>>>>> ashettia@hortonworks.com> wrote:
> >> >>>>>>
> >> >>>>>> Hi Redwane ,
> >> >>>>>>
> >> >>>>>> It is possible that the hosts which are running tasks are do no=
t
> >> >>>>>> have
> >> >>>>>> enough space. Those dirs are confiugred in mapred-site.xml
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui <
> >> >>>>>> reduno1985@googlemail.com> wrote:
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> ---------- Forwarded message ----------
> >> >>>>>>> From: Redwane belmaati cherkaoui <reduno1985@googlemail.com>
> >> >>>>>>> Date: Fri, Mar 22, 2013 at 4:39 PM
> >> >>>>>>> Subject: About running a simple wordcount mapreduce
> >> >>>>>>> To: mapreduce-issues@hadoop.apache.org
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> Hi
> >> >>>>>>> I am trying to run  a wordcount mapreduce job on several files
> >> >>>>>>> (<20
> >> >>>>>>> mb) using two machines . I get stuck on 0% map 0% reduce.
> >> >>>>>>> The jobtracker log file shows the following warning:
> >> >>>>>>>  WARN org.apache.hadoop.mapred.JobInProgress: No room for map
> >> >>>>>>> task.
> >> >>>>>>> Node hadoop0.novalocal has 8791384064 bytes free; but we expec=
t
> >> >>>>>>> map to
> >> >>take
> >> >>>>>>> 1317624576693539401
> >> >>>>>>>
> >> >>>>>>> Please help me ,
> >> >>>>>>> Best Regards,
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>
> >> >
> >> >
> >> > Matteo Lanati
> >> > Distributed Resources Group
> >> > Leibniz-Rechenzentrum (LRZ)
> >> > Boltzmannstrasse 1
> >> > 85748 Garching b. M=FCnchen (Germany)
> >> > Phone: +49 89 35831 8724
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>=20
>=20
>=20
> --
> Harsh J
>=20

Matteo Lanati
Distributed Resources Group
Leibniz-Rechenzentrum (LRZ)
Boltzmannstrasse 1
85748	Garching b. M=FCnchen	(Germany)
Phone: +49 89 35831 8724