Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (athena.apache.org: domain of static.void.dev@gmail.com
 designates 209.85.220.170 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=q++C/hEiX9sexDjCa0cw+RimNOKm5Whra2hUeFcLAkjEfPWzRsgsI9ivk+NOByFrDI
         eKC9vcgQDOnDFbVWIzYwkqEMw5hX5HldV7WrJizwHykvt3wZ/5hxg9qeoelvByySsB7I
         REpNQdeHL4QglWHLkEha0B/7JkUIsHngGrmJc=
Message-ID: <4D75557A.7020705@gmail.com>
Date: Mon, 07 Mar 2011 14:00:26 -0800
From: Mark <static.void.dev@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
 rv:1.9.2.15) Gecko/20110303 Lightning/1.0b2 Thunderbird/3.1.9
MIME-Version: 1.0
To: user@mahout.apache.org
Subject: Re: Max JobConf exceeded
References: <4D74FC30.3060003@gmail.com>
 <AANLkTinPDdnozA29P3j0_8zxc5NEwF1H-DioDBsWuirM@mail.gmail.com>
In-Reply-To: <AANLkTinPDdnozA29P3j0_8zxc5NEwF1H-DioDBsWuirM@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Its when the 2nd job starts...

11/03/07 13:56:17 INFO mapred.JobClient: Job complete: job_201103011155_0070
11/03/07 13:56:17 INFO mapred.JobClient: Counters: 23
11/03/07 13:56:17 INFO mapred.JobClient:   Job Counters
11/03/07 13:56:17 INFO mapred.JobClient:     Launched reduce tasks=1
11/03/07 13:56:17 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=1439077
11/03/07 13:56:17 INFO mapred.JobClient:     Total time spent by all 
reduces waiting after reserving slots (ms)=0
11/03/07 13:56:17 INFO mapred.JobClient:     Total time spent by all 
maps waiting after reserving slots (ms)=0
11/03/07 13:56:17 INFO mapred.JobClient:     Rack-local map tasks=2
11/03/07 13:56:17 INFO mapred.JobClient:     Launched map tasks=17
11/03/07 13:56:17 INFO mapred.JobClient:     Data-local map tasks=15
11/03/07 13:56:17 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=272929
11/03/07 13:56:17 INFO mapred.JobClient:   FileSystemCounters
11/03/07 13:56:17 INFO mapred.JobClient:     FILE_BYTES_READ=386802233
11/03/07 13:56:17 INFO mapred.JobClient:     HDFS_BYTES_READ=1028395435
11/03/07 13:56:17 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=479789858
11/03/07 13:56:17 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=166731091
11/03/07 13:56:17 INFO mapred.JobClient:   Map-Reduce Framework
11/03/07 13:56:17 INFO mapred.JobClient:     Reduce input groups=6364696
11/03/07 13:56:17 INFO mapred.JobClient:     Combine output records=83890945
11/03/07 13:56:17 INFO mapred.JobClient:     Map input records=15448494
11/03/07 13:56:17 INFO mapred.JobClient:     Reduce shuffle bytes=86260981
11/03/07 13:56:17 INFO mapred.JobClient:     Reduce output records=6364696
11/03/07 13:56:17 INFO mapred.JobClient:     Spilled Records=148689438
11/03/07 13:56:17 INFO mapred.JobClient:     Map output bytes=1856646788
11/03/07 13:56:17 INFO mapred.JobClient:     Combine input records=162172630
11/03/07 13:56:17 INFO mapred.JobClient:     Map output records=103539365
11/03/07 13:56:17 INFO mapred.JobClient:     SPLIT_RAW_BYTES=2112
11/03/07 13:56:17 INFO mapred.JobClient:     Reduce input records=25257680
11/03/07 13:56:41 INFO pfpgrowth.PFPGrowth: No of Features: 2266800
11/03/07 13:57:14 WARN mapred.JobClient: Use GenericOptionsParser for 
parsing the arguments. Applications should implement Tool for the same.
11/03/07 13:57:15 INFO input.FileInputFormat: Total input paths to 
process : 1
11/03/07 13:57:17 INFO mapred.JobClient: Cleaning up the staging area 
hdfs://hadoop1.testing.com:9000/var/hadoop/tmp/mapred/staging/root/.staging/job_201103011155_0071
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: 
java.io.IOException: java.io.IOException: Exceeded max jobconf size: 
94340533 limit: 5242880
     at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3759)
     at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
     at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1416)
     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1412)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
Caused by: java.io.IOException: Exceeded max jobconf size: 94340533 
limit: 5242880
     at 
org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:405)
     at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3757)
     ... 10 more

     at org.apache.hadoop.ipc.Client.call(Client.java:1104)
     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
     at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:904)
     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
     at 
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
     at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
     at 
org.apache.mahout.fpm.pfpgrowth.PFPGrowth.startTransactionSorting(PFPGrowth.java:345)
     at 
org.apache.mahout.fpm.pfpgrowth.PFPGrowth.runPFPGrowth(PFPGrowth.java:198)
     at 
org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.main(FPGrowthDriver.java:166)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.hadoop.util.RunJar.main(RunJar.java:186)


On 3/7/11 9:13 AM, Sean Owen wrote:
> Can you give more of the stack trace?
> Something is putting a huge amount of data in the
> Configuration/JobConf object but I don't know the code well enough to
> say what that may be.
>
> On Mon, Mar 7, 2011 at 3:39 PM, Mark<static.void.dev@gmail.com>  wrote:
>> I'm running the Mahout Frequent Pattern Mining Job
>> (org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver) and I keep receiving the
>> following:
>>
>> Caused by: java.io.IOException: Exceeded max jobconf size: 94278797 limit:
>> 524288
>>
>> Can someone explain the cause of this and more importantly the resolution?
>>
>> Thanks
>>