mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark <static.void....@gmail.com>
Subject Re: Max JobConf exceeded
Date Mon, 07 Mar 2011 22:00:26 GMT
Its when the 2nd job starts...

11/03/07 13:56:17 INFO mapred.JobClient: Job complete: job_201103011155_0070
11/03/07 13:56:17 INFO mapred.JobClient: Counters: 23
11/03/07 13:56:17 INFO mapred.JobClient:   Job Counters
11/03/07 13:56:17 INFO mapred.JobClient:     Launched reduce tasks=1
11/03/07 13:56:17 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=1439077
11/03/07 13:56:17 INFO mapred.JobClient:     Total time spent by all 
reduces waiting after reserving slots (ms)=0
11/03/07 13:56:17 INFO mapred.JobClient:     Total time spent by all 
maps waiting after reserving slots (ms)=0
11/03/07 13:56:17 INFO mapred.JobClient:     Rack-local map tasks=2
11/03/07 13:56:17 INFO mapred.JobClient:     Launched map tasks=17
11/03/07 13:56:17 INFO mapred.JobClient:     Data-local map tasks=15
11/03/07 13:56:17 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=272929
11/03/07 13:56:17 INFO mapred.JobClient:   FileSystemCounters
11/03/07 13:56:17 INFO mapred.JobClient:     FILE_BYTES_READ=386802233
11/03/07 13:56:17 INFO mapred.JobClient:     HDFS_BYTES_READ=1028395435
11/03/07 13:56:17 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=479789858
11/03/07 13:56:17 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=166731091
11/03/07 13:56:17 INFO mapred.JobClient:   Map-Reduce Framework
11/03/07 13:56:17 INFO mapred.JobClient:     Reduce input groups=6364696
11/03/07 13:56:17 INFO mapred.JobClient:     Combine output records=83890945
11/03/07 13:56:17 INFO mapred.JobClient:     Map input records=15448494
11/03/07 13:56:17 INFO mapred.JobClient:     Reduce shuffle bytes=86260981
11/03/07 13:56:17 INFO mapred.JobClient:     Reduce output records=6364696
11/03/07 13:56:17 INFO mapred.JobClient:     Spilled Records=148689438
11/03/07 13:56:17 INFO mapred.JobClient:     Map output bytes=1856646788
11/03/07 13:56:17 INFO mapred.JobClient:     Combine input records=162172630
11/03/07 13:56:17 INFO mapred.JobClient:     Map output records=103539365
11/03/07 13:56:17 INFO mapred.JobClient:     SPLIT_RAW_BYTES=2112
11/03/07 13:56:17 INFO mapred.JobClient:     Reduce input records=25257680
11/03/07 13:56:41 INFO pfpgrowth.PFPGrowth: No of Features: 2266800
11/03/07 13:57:14 WARN mapred.JobClient: Use GenericOptionsParser for 
parsing the arguments. Applications should implement Tool for the same.
11/03/07 13:57:15 INFO input.FileInputFormat: Total input paths to 
process : 1
11/03/07 13:57:17 INFO mapred.JobClient: Cleaning up the staging area 
hdfs://hadoop1.testing.com:9000/var/hadoop/tmp/mapred/staging/root/.staging/job_201103011155_0071
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: 
java.io.IOException: java.io.IOException: Exceeded max jobconf size: 
94340533 limit: 5242880
     at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3759)
     at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
     at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1416)
     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1412)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
Caused by: java.io.IOException: Exceeded max jobconf size: 94340533 
limit: 5242880
     at 
org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:405)
     at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3757)
     ... 10 more

     at org.apache.hadoop.ipc.Client.call(Client.java:1104)
     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
     at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:904)
     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
     at 
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
     at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
     at 
org.apache.mahout.fpm.pfpgrowth.PFPGrowth.startTransactionSorting(PFPGrowth.java:345)
     at 
org.apache.mahout.fpm.pfpgrowth.PFPGrowth.runPFPGrowth(PFPGrowth.java:198)
     at 
org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.main(FPGrowthDriver.java:166)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.hadoop.util.RunJar.main(RunJar.java:186)


On 3/7/11 9:13 AM, Sean Owen wrote:
> Can you give more of the stack trace?
> Something is putting a huge amount of data in the
> Configuration/JobConf object but I don't know the code well enough to
> say what that may be.
>
> On Mon, Mar 7, 2011 at 3:39 PM, Mark<static.void.dev@gmail.com>  wrote:
>> I'm running the Mahout Frequent Pattern Mining Job
>> (org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver) and I keep receiving the
>> following:
>>
>> Caused by: java.io.IOException: Exceeded max jobconf size: 94278797 limit:
>> 524288
>>
>> Can someone explain the cause of this and more importantly the resolution?
>>
>> Thanks
>>

Mime
View raw message