mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tom pierce <...@apache.org>
Subject Re: Build failed in Jenkins: Mahout-Examples-Cluster-Reuters #58
Date Thu, 01 Mar 2012 23:01:37 GMT
What's the best way to even begin tracking this down?  To my eye, this 
looks like something has gone wrong on the server (full disk, maybe?).

-tom

On 03/01/2012 03:06 PM, Apache Jenkins Server wrote:
> See<https://builds.apache.org/job/Mahout-Examples-Cluster-Reuters/58/changes>
>
> Changes:
>
> [tcp] cleanups to make Jenkins a little happier
>
> [tcp] MAHOUT-980: Fix DistributedCache usage to allow EMR deployment
>
> ------------------------------------------
> [...truncated 8490 lines...]
> 12/03/01 20:00:13 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:16 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:16 INFO mapred.JobClient:  map 39% reduce 0%
> 12/03/01 20:00:19 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:22 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:22 INFO mapred.JobClient:  map 40% reduce 0%
> 12/03/01 20:00:25 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:28 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:28 INFO mapred.JobClient:  map 41% reduce 0%
> 12/03/01 20:00:31 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:34 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:34 INFO mapred.JobClient:  map 42% reduce 0%
> 12/03/01 20:00:37 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:40 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:40 INFO mapred.JobClient:  map 43% reduce 0%
> 12/03/01 20:00:43 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:46 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:46 INFO mapred.JobClient:  map 44% reduce 0%
> 12/03/01 20:00:49 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:52 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:55 INFO mapred.LocalJobRunner:
> 12/03/01 20:00:55 INFO mapred.JobClient:  map 45% reduce 0%
> 12/03/01 20:00:58 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:01 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:01 INFO mapred.JobClient:  map 46% reduce 0%
> 12/03/01 20:01:04 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:07 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:07 INFO mapred.JobClient:  map 47% reduce 0%
> 12/03/01 20:01:10 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:12 INFO mapred.MapTask: Spilling map output: buffer full= true
> 12/03/01 20:01:12 INFO mapred.MapTask: bufstart = 0; bufend = 79691690; bufvoid = 99614720
> 12/03/01 20:01:12 INFO mapred.MapTask: kvstart = 0; kvend = 203102; length = 327680
> 12/03/01 20:01:13 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:13 INFO mapred.JobClient:  map 48% reduce 0%
> 12/03/01 20:01:16 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:18 INFO mapred.MapTask: Finished spill 0
> 12/03/01 20:01:19 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:19 INFO mapred.JobClient:  map 49% reduce 0%
> 12/03/01 20:01:22 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:25 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:28 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:28 INFO mapred.JobClient:  map 50% reduce 0%
> 12/03/01 20:01:31 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:31 INFO mapred.JobClient:  map 51% reduce 0%
> 12/03/01 20:01:34 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:37 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:40 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:40 INFO mapred.JobClient:  map 52% reduce 0%
> 12/03/01 20:01:43 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:46 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:46 INFO mapred.JobClient:  map 53% reduce 0%
> 12/03/01 20:01:49 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:52 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:52 INFO mapred.JobClient:  map 54% reduce 0%
> 12/03/01 20:01:55 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:58 INFO mapred.LocalJobRunner:
> 12/03/01 20:01:58 INFO mapred.JobClient:  map 55% reduce 0%
> 12/03/01 20:02:01 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:04 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:07 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:07 INFO mapred.JobClient:  map 56% reduce 0%
> 12/03/01 20:02:10 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:13 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:13 INFO mapred.JobClient:  map 57% reduce 0%
> 12/03/01 20:02:16 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:19 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:19 INFO mapred.JobClient:  map 58% reduce 0%
> 12/03/01 20:02:22 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:25 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:25 INFO mapred.JobClient:  map 59% reduce 0%
> 12/03/01 20:02:28 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:31 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:34 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:34 INFO mapred.JobClient:  map 60% reduce 0%
> 12/03/01 20:02:37 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:40 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:40 INFO mapred.JobClient:  map 61% reduce 0%
> 12/03/01 20:02:43 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:46 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:46 INFO mapred.JobClient:  map 62% reduce 0%
> 12/03/01 20:02:49 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:52 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:52 INFO mapred.JobClient:  map 63% reduce 0%
> 12/03/01 20:02:55 INFO mapred.LocalJobRunner:
> 12/03/01 20:02:58 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:01 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:01 INFO mapred.JobClient:  map 64% reduce 0%
> 12/03/01 20:03:04 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:07 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:10 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:10 INFO mapred.JobClient:  map 65% reduce 0%
> 12/03/01 20:03:13 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:16 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:19 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:19 INFO mapred.JobClient:  map 66% reduce 0%
> 12/03/01 20:03:22 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:25 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:28 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:28 INFO mapred.JobClient:  map 67% reduce 0%
> 12/03/01 20:03:31 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:34 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:37 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:37 INFO mapred.JobClient:  map 68% reduce 0%
> 12/03/01 20:03:40 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:43 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:43 INFO mapred.JobClient:  map 69% reduce 0%
> 12/03/01 20:03:46 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:49 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:49 INFO mapred.JobClient:  map 70% reduce 0%
> 12/03/01 20:03:52 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:55 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:58 INFO mapred.LocalJobRunner:
> 12/03/01 20:03:58 INFO mapred.JobClient:  map 71% reduce 0%
> 12/03/01 20:04:01 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:04 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:04 INFO mapred.JobClient:  map 72% reduce 0%
> 12/03/01 20:04:07 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:10 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:10 INFO mapred.JobClient:  map 73% reduce 0%
> 12/03/01 20:04:13 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:16 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:16 INFO mapred.JobClient:  map 74% reduce 0%
> 12/03/01 20:04:19 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:22 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:22 INFO mapred.JobClient:  map 75% reduce 0%
> 12/03/01 20:04:25 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:28 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:31 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:31 INFO mapred.JobClient:  map 76% reduce 0%
> 12/03/01 20:04:34 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:37 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:37 INFO mapred.JobClient:  map 77% reduce 0%
> 12/03/01 20:04:40 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:43 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:43 INFO mapred.JobClient:  map 78% reduce 0%
> 12/03/01 20:04:46 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:49 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:49 INFO mapred.JobClient:  map 79% reduce 0%
> 12/03/01 20:04:52 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:55 INFO mapred.LocalJobRunner:
> 12/03/01 20:04:55 INFO mapred.JobClient:  map 80% reduce 0%
> 12/03/01 20:04:58 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:01 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:01 INFO mapred.JobClient:  map 81% reduce 0%
> 12/03/01 20:05:04 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:07 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:07 INFO mapred.JobClient:  map 82% reduce 0%
> 12/03/01 20:05:10 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:13 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:13 INFO mapred.JobClient:  map 83% reduce 0%
> 12/03/01 20:05:16 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:19 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:19 INFO mapred.JobClient:  map 84% reduce 0%
> 12/03/01 20:05:22 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:25 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:25 INFO mapred.JobClient:  map 85% reduce 0%
> 12/03/01 20:05:28 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:31 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:31 INFO mapred.JobClient:  map 86% reduce 0%
> 12/03/01 20:05:34 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:37 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:37 INFO mapred.JobClient:  map 87% reduce 0%
> 12/03/01 20:05:40 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:43 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:46 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:46 INFO mapred.JobClient:  map 88% reduce 0%
> 12/03/01 20:05:49 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:52 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:52 INFO mapred.JobClient:  map 89% reduce 0%
> 12/03/01 20:05:55 INFO mapred.LocalJobRunner:
> 12/03/01 20:05:58 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:01 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:01 INFO mapred.JobClient:  map 90% reduce 0%
> 12/03/01 20:06:04 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:07 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:07 INFO mapred.JobClient:  map 91% reduce 0%
> 12/03/01 20:06:10 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:13 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:13 INFO mapred.JobClient:  map 92% reduce 0%
> 12/03/01 20:06:16 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:19 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:19 INFO mapred.JobClient:  map 93% reduce 0%
> 12/03/01 20:06:22 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:25 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:28 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:28 INFO mapred.JobClient:  map 94% reduce 0%
> 12/03/01 20:06:31 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:34 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:37 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:37 INFO mapred.JobClient:  map 95% reduce 0%
> 12/03/01 20:06:40 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:43 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:46 INFO mapred.MapTask: Spilling map output: buffer full= true
> 12/03/01 20:06:46 INFO mapred.MapTask: bufstart = 79691690; bufend = 59768438; bufvoid
= 99614720
> 12/03/01 20:06:46 INFO mapred.MapTask: kvstart = 203102; kvend = 85998; length = 327680
> 12/03/01 20:06:46 WARN mapred.LocalJobRunner: job_local_0005
> java.io.IOException: Spill failed
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
> 	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansClusterer.emitPointProbToCluster(FuzzyKMeansClusterer.java:165)
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansMapper.map(FuzzyKMeansMapper.java:45)
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansMapper.map(FuzzyKMeansMapper.java:35)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/spill1.out
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127)
> 	at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:121)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1392)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
> 12/03/01 20:06:46 INFO mapred.LocalJobRunner:
> 12/03/01 20:06:46 INFO mapred.JobClient:  map 96% reduce 0%
> 12/03/01 20:06:46 INFO mapred.JobClient: Job complete: job_local_0005
> 12/03/01 20:06:46 INFO mapred.JobClient: Counters: 11
> 12/03/01 20:06:46 INFO mapred.JobClient:   File Input Format Counters
> 12/03/01 20:06:46 INFO mapred.JobClient:     Bytes Read=4734976
> 12/03/01 20:06:46 INFO mapred.JobClient:   FileSystemCounters
> 12/03/01 20:06:46 INFO mapred.JobClient:     FILE_BYTES_READ=398738970
> 12/03/01 20:06:46 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=357205174
> 12/03/01 20:06:46 INFO mapred.JobClient:   Map-Reduce Framework
> 12/03/01 20:06:46 INFO mapred.JobClient:     Map output materialized bytes=0
> 12/03/01 20:06:46 INFO mapred.JobClient:     Combine output records=20
> 12/03/01 20:06:46 INFO mapred.JobClient:     Map input records=20685
> 12/03/01 20:06:46 INFO mapred.JobClient:     Spilled Records=20
> 12/03/01 20:06:46 INFO mapred.JobClient:     Map output bytes=159384560
> 12/03/01 20:06:46 INFO mapred.JobClient:     SPLIT_RAW_BYTES=155
> 12/03/01 20:06:46 INFO mapred.JobClient:     Map output records=413680
> 12/03/01 20:06:46 INFO mapred.JobClient:     Combine input records=203102
> Exception in thread "main" java.lang.InterruptedException: Fuzzy K-Means Iteration failed
processing /tmp/mahout-work-jenkins/reuters-fkmeans/clusters-4
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.runIteration(FuzzyKMeansDriver.java:252)
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.buildClustersMR(FuzzyKMeansDriver.java:426)
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.buildClusters(FuzzyKMeansDriver.java:345)
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.run(FuzzyKMeansDriver.java:295)
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.run(FuzzyKMeansDriver.java:125)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> 	at org.apache.mahout.clustering.fuzzykmeans.FuzzyKMeansDriver.main(FuzzyKMeansDriver.java:67)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> 	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> 	at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188)
> Build step 'Execute shell' marked build as failure
>


Mime
View raw message