hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgene...@yahoo.com>
Subject Re: [VOTE] Release candidate 0.20.203.0-rc1
Date Fri, 06 May 2011 00:42:14 GMT
+1 on 0.20.203.0-rc1.

I downloaded the release, verified signature, verified some message digests, 
started a one-node cluster and tested it with my MapReduce Math Library.  In 
particular, I have run

*DistMpMultTest*
It runs 5 jobs:
- 2 forward FFT jobs for calculating two 4086-dimensional DFTs.
- 1 backward FFT job for calculating componentwise multiplication using a c 
program with GMP lib, and a 4086-dimensional inverse DFTs.
- 1 componentwise summation job.
- 1 job for carrying.

*DistBbp*
It runs 14 jobs for computing the 1024 bits of pi right after the billionth bit 
positions.

Please see the details below.

Regards,
Nicholas


****** DistMpMultTest ******************************************************

distfft.VERSION = 20110124
distmpmult.VERSION = 20110307
distmpmult.LOCAL_THRESHOLD = 2^16 (=65536)

args = [#=2
  0: 18
  1: gen
]
dir              = DistMpMultTest-20110505-222621889
NEW: Zahlen(bitsPerDigit=32, digitsPerArray=4096, numArrays=128)

SchonhageStrassen.Factory.valueOf(numDigits=262144): 
  digitsPerOperand = 2^18 (=262144) (highest=262144)
  bitsPerElement   = 2^12 (=4096)
  D                = 2^12 (=4096)
  modulusExponent  = 10240 (ss_e=8224)
  efficiency       = 0.801171875 (N=2^24 (=16777216), n=10240)
NEW: SchonhageStrassen[modulas=2^10240 + 1, D=2^12 (=4096), bitsPerElement=2^12 
(=4096), Z=Zahlen(bitsPerDigit=2^5, digitLimit=2^32, digitsPerArray=2^12, 
numArrays=128)]
NEW: Zahlen(bitsPerDigit=32, digitsPerArray=4096, numArrays=1)
NEW: SchonhageStrassen[modulas=2^10240 + 1, D=2^12 (=4096), bitsPerElement=2^12 
(=4096), Z=Zahlen(bitsPerDigit=2^5, digitLimit=2^32, digitsPerArray=2^12, 
numArrays=1)]

numDigits        = 2^18 (=262144) (e=18)
digitsPerOperand = 262144
J                = 2^6 (=64)
K                = 2^6 (=64)

Verifier: STARTED
WorkGroup: Created DistMpMultTest with 2 threads.

multiply_mapreduce: c; x = [a, b]
  distfft: forward
    inverse           = false
    distmpbase: 
      Z                 = Zahlen
        bitsPerDigit    = 2^5 (=32)
        digitLimit      = 2^32 (=4294967296)
        digitsPerArray  = 2^12 (=4096)
        numArrays       = 1
        digitsSupported = 4096
      schonhagestrassen = SchonhageStrassen
        D              = 2^12 (=4096)
        D^(-1)         = -2^10228
        modulus        = 2^10240 + 1
        bitsPerElement = 2^12 (=4096)
        zeta           = [2^0, 2^5, 2^10, 2^15, 2^20, 2^25, 2^30, 2^35, 2^40, 
2^45, ...]
      J                 = 2^6 (=64)
      K                 = 2^6 (=64)
      dir               = DistMpMultTest-20110505-222621889
  digitsPerOperand = 262144
  descriptor = FunctionDescriptor(input=a, output=a')
  descriptor = FunctionDescriptor(input=b, output=b')
  forward a: DistFft(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889, inverse=false)
  GmpMultiplier(Thread-260, count=0): cmd=[./gmp_mult, 16]
  GmpMultiplier(Thread-260, count=1): messager started.
  GmpMultiplier(Thread-260, count=1): 
  GmpMultiplier(Thread-260, count=1): START: Thu May  5 22:26:28 2011
  GmpMultiplier(Thread-260, count=1): VERSION=20110122b, GMP4.1.4
  GmpMultiplier(Thread-260, count=1): 
  GmpMultiplier(Thread-260, count=1): argc=2, argv=[./gmp_mult, 16]
  SUBMIT JOB: DistMpMultTest-20110505-222621889: a' = dft(a)
  forward b: DistFft(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889, inverse=false)
  SUBMIT JOB: DistMpMultTest-20110505-222621889: b' = dft(b)
  Verifier: expected = + 0129ED2D B8CF7E1D A9BF33D8 9A294511 8508413C ...  
4EA85F30 EF79A552 1C080D0C 82F6882C 8CE9F355 (524,288 digits, 16,777,209 bits)
  descriptor = FunctionDescriptor(input=a' ** b', output=c')
  backward: DistFft(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889, inverse=true)
  SUBMIT JOB: DistMpMultTest-20110505-222621889: c' = dft^-1(a' ** b')
          2272875ms (=37:52.875)     : backward, 
DistMpMultTest-20110505-222621889: c' = dft^-1(a' ** b')
  -----------------------------------------------------
  descriptor = FunctionDescriptor(input=c'_0 + c'_1 + c'_2, output=c'')
  summation: DistMpSum(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889/c')
  SUBMIT JOB: DistMpMultTest-20110505-222621889/c': c'' = c'_0 + c'_1 + c'_2
          2805177ms (=46:45.177)     : summation, 
DistMpMultTest-20110505-222621889/c': c'' = c'_0 + c'_1 + c'_2
  -----------------------------------------------------
  descriptor = FunctionDescriptor(input=c'', output=c)
  carrying: DistCarrying(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889/c')
  SUBMIT JOB: DistMpMultTest-20110505-222621889/c': c = carry(c'')
          3456765ms (=57:36.765)     : carrying, 
DistMpMultTest-20110505-222621889/c': c = carry(c'')
  -----------------------------------------------------
multiply_mapreduce: returns ZahlenDescriptor(numParts=2^6 (=64), 
elementsPerPart=64)
c = ZahlenDescriptor(numParts=2^6 (=64), elementsPerPart=64)
        3456769ms (=57:36.769)     : DistMpMult

           maxMemory = 963.00 MB
         totalMemory = 64.88 MB
          freeMemory = 26.32 MB

computed = + 0129ED2D B8CF7E1D A9BF33D8 9A294511 8508413C ...  4EA85F30 EF79A552 
1C080D0C 82F6882C 8CE9F355 (524,288 digits, 16,777,209 bits)
        3457374ms (=57:37.374)     : DONE (dir = 
DistMpMultTest-20110505-222621889)




****** DistBbp *************************************************************
Create file b1000000000-p1024-20110506-000707662.log

STARTUP Fri May 06 00:07:07 UTC 2011
  Started at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
  Printed at org.apache.hadoop.mp.pi.DistSum.<init>(DistSum.java:428)
WorkGroup: Created DistSum.OutputProcessor with 1 threads.
DistBbp.VERSION = 20100731

b = 1,000,000,000 (bits skipped)

precision = 1,024
nWorkers  = 14
nJobs     = 1
machine   = MapSide(p1,t1), m1t1
remoteDir = 100b-1024-test
tmpDir    = ../100b-1024-test


DistBbp.processExistingJobOutputs = true

Check existing job outputs from hdfs://NAMENODE/user/100b-1024-test ...

Read existing results from hdfs://NAMENODE/user/tsz/100b-1024-test ...
DistBbp.bellard = null
ADD : P8_1: -[n:value=1,delta=8,limit=400000393; 
e:value=999999999,delta=-20,limit=-996], parts.length=1
ADD : P8_3: -[n:value=3,delta=8,limit=400000395; 
e:value=999999994,delta=-20,limit=-996], parts.length=1
ADD : P8_5: +[n:value=5,delta=8,limit=400000397; 
e:value=999999989,delta=-20,limit=-996], parts.length=1
ADD : P8_7: +[n:value=7,delta=8,limit=400000399; 
e:value=999999984,delta=-20,limit=-996], parts.length=1
ADD : P20_21: +[n:value=21,delta=20,limit=1000000981; 
e:value=999999982,delta=-20,limit=-995], parts.length=1
ADD : P20_3: -[n:value=3,delta=20,limit=1000000983; 
e:value=1000000000,delta=-20,limit=-995], parts.length=1
ADD : P20_5: -[n:value=5,delta=20,limit=1000000985; 
e:value=999999996,delta=-20,limit=-995], parts.length=1
ADD : P20_7: -[n:value=7,delta=20,limit=1000000987; 
e:value=999999996,delta=-20,limit=-995], parts.length=1
ADD : P20_9: +[n:value=9,delta=20,limit=1000000989; 
e:value=999999994,delta=-20,limit=-995], parts.length=1
ADD : P20_11: -[n:value=11,delta=20,limit=1000000991; 
e:value=999999992,delta=-20,limit=-995], parts.length=1
ADD : P20_13: +[n:value=13,delta=20,limit=1000000993; 
e:value=999999990,delta=-20,limit=-995], parts.length=1
ADD : P20_15: +[n:value=15,delta=20,limit=1000000995; 
e:value=999999986,delta=-20,limit=-995], parts.length=1
ADD : P20_17: +[n:value=17,delta=20,limit=1000000997; 
e:value=999999986,delta=-20,limit=-995], parts.length=1
ADD : P20_19: -[n:value=19,delta=20,limit=1000000979; 
e:value=999999984,delta=-20,limit=-995], parts.length=1
           1102ms (=1.102s)        : EXECUTOR: 14 computation(s)
           1298ms (=1.298s)        : P20_5.job0006-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=1
P20_5.job0006-20110506-000707683> sleep(5s)
           7667ms (=7.667s)        : P20_15.job0011-20110506-000707683> 
starting, steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=2
P20_15.job0011-20110506-000707683> sleep(5s)
          13327ms (=13.327s)       : P8_7.job0003-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=3
P8_7.job0003-20110506-000707683> sleep(5s)
          18670ms (=18.670s)       : P20_3.job0005-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=4
P20_3.job0005-20110506-000707683> sleep(5s)
          24281ms (=24.281s)       : P8_5.job0002-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=5
P8_5.job0002-20110506-000707683> sleep(5s)
          29693ms (=29.693s)       : P8_1.job0000-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=6
P8_1.job0000-20110506-000707683> sleep(5s)
          34917ms (=34.917s)       : P20_19.job0013-20110506-000707683> 
starting, steps/cores = 50000048/1 = 50,000,048, ++nSubmittedJobs=7
P20_19.job0013-20110506-000707683> sleep(5s)
          40135ms (=40.135s)       : P20_21.job0004-20110506-000707683> 
starting, steps/cores = 50000048/1 = 50,000,048, ++nSubmittedJobs=8
P20_21.job0004-20110506-000707683> sleep(5s)
          45458ms (=45.458s)       : P20_13.job0010-20110506-000707683> 
starting, steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=9
P20_13.job0010-20110506-000707683> sleep(5s)
          50666ms (=50.666s)       : P20_11.job0009-20110506-000707683> 
starting, steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=10
P20_11.job0009-20110506-000707683> sleep(5s)
          55942ms (=55.942s)       : P20_7.job0007-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=11
P20_7.job0007-20110506-000707683> sleep(5s)
          61261ms (=1:01.261)      : P20_9.job0008-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=12
P20_9.job0008-20110506-000707683> sleep(5s)
          66577ms (=1:06.577)      : P20_17.job0012-20110506-000707683> 
starting, steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=13
P20_17.job0012-20110506-000707683> sleep(5s)
          71752ms (=1:11.752)      : P8_3.job0001-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=14
P8_3.job0001-20110506-000707683> sleep(5s)
         187749ms (=3:07.749)      : P20_5.job0006-20110506-000707683> 
timetaken=3:06.451, --nSubmittedJobs=13
         187753ms (=3:07.753)      : ++DistSum.OutputProcessor.count=1 for 
P20_5.job0006-20110506-000707683
> found 3 items in 
>hdfs://NAMENODE/user/100b-1024-test/P20_5.job0006-20110506-000707683/out
> write the result to 100b-1024-test/P20_5.job0006-20110506-000707683.writable
DistSum P20_5.job0006-20110506-000707683> duration=142210(2:22.210), 
sigma=-[n:value=5,delta=20,limit=1000000985; 
e:value=999999996,delta=-20,limit=-995], value=[32:7AB75B81 4B81A170 48F78314 
C1B677EF 6D8AA841 77D75680 AE63063C 378B6899 DBF6BB41 113B3C9E F6ADC215 2AA38330 
14796B10 16862533 C17A6A17 4893ED2C 23570119 471AADFC 493994FA CBA6DBF4 C60A3E46 
48E27ADC 6B2A442E 797C4011 40FCA6AF FE97AFAF 3FF6CB94 532F818F E2EA5E7F 7BC2FADF 
83B2F466 07F9A5C7 ]
         187804ms (=3:07.804)      : --DistSum.OutputProcessor.count=0 for 
P20_5.job0006-20110506-000707683
EXECUTOR: 1/14 done, estimated time: remaining=40.94 minutes, total=44.09 
minutes, sleep=2.300s
         193403ms (=3:13.403)      : P20_15.job0011-20110506-000707683> 
timetaken=3:05.735, --nSubmittedJobs=12
         193403ms (=3:13.403)      : ++DistSum.OutputProcessor.count=1 for 
P20_15.job0011-20110506-000707683
> found 3 items in 
>hdfs://NAMENODE/user/100b-1024-test/P20_15.job0011-20110506-000707683/out
> write the result to 100b-1024-test/P20_15.job0011-20110506-000707683.writable
DistSum P20_15.job0011-20110506-000707683> duration=141830(2:21.830), 
sigma=+[n:value=15,delta=20,limit=1000000995; 
e:value=999999986,delta=-20,limit=-995], value=[32:FDC65838 CB87FD4C 0C6883F1 
F6E7A8F4 903A3888 6C377451 138823A3 9B8AFB5D F1F60326 F5300E57 DFDDF086 83309616 
8FBB61F1 C3D2CC7A 5414D92E 3A27B56C 305F6A64 BD1BE14D CDDE3433 AD57BF9C C3CCCF98 
BB7AF093 A42E5EB8 7182B1A0 09058BF0 5C2EF773 A8B4D621 ADD9C960 32272AAD 0B060FFA 
D0932CC6 ACAF0A05 ]
         193419ms (=3:13.419)      : --DistSum.OutputProcessor.count=0 for 
P20_15.job0011-20110506-000707683
EXECUTOR: 2/14 done, estimated time: remaining=19.36 minutes, total=22.58 
minutes, sleep=2.200s
         338790ms (=5:38.790)      : P8_7.job0003-20110506-000707683> 
timetaken=5:25.463, --nSubmittedJobs=11


...


        1087176ms (=18:07.176)     : --DistSum.OutputProcessor.count=0 for 
P20_17.job0012-20110506-000707683
EXECUTOR: 13/14 done, estimated time: remaining=1.39 minutes, total=19.52 
minutes, sleep=1.100s
        1097443ms (=18:17.443)     : P8_3.job0001-20110506-000707683> 
timetaken=17:05.691, --nSubmittedJobs=0
        1097443ms (=18:17.443)     : ++DistSum.OutputProcessor.count=1 for 
P8_3.job0001-20110506-000707683
> found 3 items in 
>hdfs://NAMENODE/user/100b-1024-test/P8_3.job0001-20110506-000707683/out
> write the result to 100b-1024-test/P8_3.job0001-20110506-000707683.writable
DistSum P8_3.job0001-20110506-000707683> duration=142280(2:22.280), 
sigma=-[n:value=3,delta=8,limit=400000395; 
e:value=999999994,delta=-20,limit=-996], value=[32:9850C816 183C9765 11830320 
A12AB99B 81D93E98 061769A9 0FBBCAF7 3AEE5079 5673D8FA 0A352C71 C025AC12 5D482AC1 
276E834F C862BF85 D9B79DE5 1264B724 A5C5FD90 2677498B DDF05B2B D298E033 9AB3DC52 
2A3DB5AA 2835506D DF795BCB 743CC0FE 27AA4F1F 1F711BAB 52E4F4E6 5410CE20 CD27ACB6 
6C7F6810 95F97608 ]
        1097461ms (=18:17.461)     : --DistSum.OutputProcessor.count=0 for 
P8_3.job0001-20110506-000707683
EXECUTOR: 14/14 done, estimated time: remaining=0.00 ms, total=18.31 minutes, 
sleep=1s
Write to hdfs://NAMENODE/user/tsz/100b-1024-test/b1000000000-p1024 ...

b = 1,000,000,000 (bits skipped)

CPU time = 1991670ms = 33:11.670
END Fri May 06 00:25:26 UTC 2011
[32:
  3E08FF2B 03F1829E 05C038A3 2884A9C4 E7DEF417 875B1C22 D2DDDA11 D99573D8 
43F80107 AB3A56CC 

  C975C849 2DC5AD43 1D191704 9064DE41 67DE5C6E 0F264D33 D903CE49 F2324781 
D9D7ED45 FAA9272D 

  CAA42278 E8EF2FFF 450E2183 25FA9AB3 459D01AF AEF8AF79 DD6CC766 12152C31 
2F31ABCD DCF01DEC 

  67332105 643A438D 
]
****** END ******************************************************

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message