Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of
 jrottinghuis@ebay.com)
DomainKey-Signature: s=corp; d=ebay.com; c=nofws; q=dns;
  h=X-EBay-Corp:X-IronPort-AV:Received:Received:From:To:CC:
   Date:Subject:Thread-Topic:Thread-Index:Message-ID:
   References:In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:acceptlanguage:
   x-ems-proccessed:x-ems-stamp:Content-Type:
   Content-Transfer-Encoding:MIME-Version:X-CFilter;
  b=pSgxhKuWmBvzHbIJm50Sbnqu5ynwg/mIMWjhDKpA7Ruul28n1pET0WXz
   t3NDnORd1eefpSreAzXc+ZXjy3S/pD9PHW95JeRQ/zU9o7ohChQyAAW+8
   aSsigtNPACRyBw+;
From: "Rottinghuis, Joep" <jrottinghuis@ebay.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>, CDH Users
	<cdh-user@cloudera.org>
CC: "wlangiewicz@gmail.com" <wlangiewicz@gmail.com>
Date: Thu, 17 Mar 2011 12:01:34 -0700
Subject: RE: java.io.IOException: Split metadata size exceeded 10000000
Thread-Topic: java.io.IOException: Split metadata size exceeded 10000000
Thread-Index: Acvi/HgegM3CCCS3TXyDOASoYHwwSwBzYtww
Message-ID: 
 <F4FCE59AE46B2243922F224426BC869B01BD745BBA@RHV-MEXMS-002.corp.ebay.com>
References: <4D7F378E.30402@gmail.com>
 <AANLkTikHAG-0YYsHNpKTT_dMjQQ2Y07oXngYqZNHfZZY@mail.gmail.com>
In-Reply-To: <AANLkTikHAG-0YYsHNpKTT_dMjQQ2Y07oXngYqZNHfZZY@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Doubt this is a CDH3 issue.
We saw the same with a large job using the 0.20-security branch.

There is a property (mapreduce.jobtracker.split.metainfo.maxsize) that can =
be used to override the default of 10^6.
We found that passing this along with the job has no effect, this worked on=
ly when setting this property on the jobtracker node. Not sure if this is a=
 feature or a bug.

Cheers,

Joep

-----Original Message-----
From: Harsh J [mailto:qwertymaniac@gmail.com]=20
Sent: Tuesday, March 15, 2011 3:33 AM
To: CDH Users
Cc: wlangiewicz@gmail.com
Subject: Re: java.io.IOException: Split metadata size exceeded 10000000

Moving this discussion to the CDH users list at cdh-user [at]
cloudera.org since it could be a CDH specific issue.

[Bcc: general]

On Tue, Mar 15, 2011 at 3:25 PM, Wojciech Langiewicz
<wlangiewicz@gmail.com> wrote:
> Hello,
> I'm having this problem running mapreduce jobs over about 10TB of data
> (smaller jobs are ok):
> 2011-03-15 07:48:22,031 ERROR org.apache.hadoop.mapred.JobTracker: Job
> initialization failed:
> java.io.IOException: Split metadata size exceeded 10000000. Aborting job
> job_201103141436_0058
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(S=
plitMetaInfoReader.java:48)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.JobInProgress.createSplits(JobInProgress.java:73=
2)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:633)
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.=
java:3965)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(Eage=
rTaskInitializationListener.java:79)
> =A0 =A0 =A0 =A0at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor=
.java:886)
> =A0 =A0 =A0 =A0at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav=
a:908)
> =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619)
>
> 2011-03-15 07:48:22,031 INFO org.apache.hadoop.mapred.JobTracker: Failing
> job job_201103141436_0058
>
> What settings should I change to run this job?
> I'm using CDH3b3.
> Thanks for all answers.
>
> --
> Wojciech Langiewcz
>


--=20
Harsh J
http://harshj.com