incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillermo Pérez <bi...@tuenti.com>
Subject Launching different record & reducers from mapper
Date Mon, 08 Mar 2010 15:05:49 GMT
I'm launching several chukwa records with different keys and reducers,
so I can generate some aggregated records data directly while loading
data.

The map / redux works well, but the data that passes to the aggregator
reducer is not stored to HDFS. Anybody knows why?

2010-03-08 15:28:28,784 INFO main JobClient - Job complete:
job_201002191418_1463
2010-03-08 15:28:28,800 INFO main JobClient - Counters: 29
2010-03-08 15:28:28,800 INFO main JobClient -   DemuxReduceOutput
2010-03-08 15:28:28,800 INFO main JobClient -     total records=1416018
2010-03-08 15:28:28,800 INFO main JobClient -     ActionLog records=1416018
2010-03-08 15:28:28,800 INFO main JobClient -   DemuxMapOutput
2010-03-08 15:28:28,800 INFO main JobClient -
ActionLogAggregateWeights records=1416018
2010-03-08 15:28:28,800 INFO main JobClient -     total records=2832036
2010-03-08 15:28:28,800 INFO main JobClient -     ActionLog records=1416018
2010-03-08 15:28:28,801 INFO main JobClient -   Job Counters
2010-03-08 15:28:28,801 INFO main JobClient -     Launched reduce tasks=9
2010-03-08 15:28:28,801 INFO main JobClient -     Rack-local map tasks=1
2010-03-08 15:28:28,801 INFO main JobClient -     Launched map tasks=2
2010-03-08 15:28:28,801 INFO main JobClient -     Data-local map tasks=1
2010-03-08 15:28:28,801 INFO main JobClient -   DemuxMapInput
2010-03-08 15:28:28,801 INFO main JobClient -     ActionLog chunks=610
2010-03-08 15:28:28,801 INFO main JobClient -     total chunks=610
2010-03-08 15:28:28,801 INFO main JobClient -   DemuxReduceInput
2010-03-08 15:28:28,801 INFO main JobClient -     total distinct keys=58400
2010-03-08 15:28:28,801 INFO main JobClient -     ActionLog total
distinct keys=57600
2010-03-08 15:28:28,801 INFO main JobClient -
ActionLogAggregateWeights total distinct keys=800
2010-03-08 15:28:28,801 INFO main JobClient -   FileSystemCounters
2010-03-08 15:28:28,801 INFO main JobClient -     FILE_BYTES_READ=1001600403
2010-03-08 15:28:28,802 INFO main JobClient -     HDFS_BYTES_READ=85558794
2010-03-08 15:28:28,802 INFO main JobClient -     FILE_BYTES_WRITTEN=1501914817
2010-03-08 15:28:28,802 INFO main JobClient -     HDFS_BYTES_WRITTEN=325688807
2010-03-08 15:28:28,802 INFO main JobClient -   Map-Reduce Framework
2010-03-08 15:28:28,802 INFO main JobClient -     Reduce input groups=58400
2010-03-08 15:28:28,802 INFO main JobClient -     Combine output records=0
2010-03-08 15:28:28,802 INFO main JobClient -     Map input records=610
2010-03-08 15:28:28,802 INFO main JobClient -     Reduce shuffle bytes=342907105
2010-03-08 15:28:28,802 INFO main JobClient -     Reduce output records=1416018
2010-03-08 15:28:28,802 INFO main JobClient -     Spilled Records=8496108
2010-03-08 15:28:28,802 INFO main JobClient -     Map output bytes=493557805
2010-03-08 15:28:28,802 INFO main JobClient -     Map input bytes=85558588
2010-03-08 15:28:28,802 INFO main JobClient -     Combine input records=0
2010-03-08 15:28:28,802 INFO main JobClient -     Map output records=2832036
2010-03-08 15:28:28,802 INFO main JobClient -     Reduce input records=2832036

ActionLog is stored in the repository dir, but I can't find anything
about ActionLogAggregateWeights...


-- 
Guille -ℬḭṩḩø- <bisho@tuenti.com>
:wq

Mime
View raw message