hama-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hama Wiki] Update of "SpMV" by Mikalai Parafeniuk
Date Sat, 18 Aug 2012 15:52:02 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hama Wiki" for change notification.

The "SpMV" page has been changed by Mikalai Parafeniuk:
http://wiki.apache.org/hama/SpMV?action=diff&rev1=11&rev2=12

  /user/hduser/spmv/vector-seq
  /user/hduser/spmv/vector-txt
  }}}
- Suffix `seq` denotes that directory contains sequence files. Suffix `txt` denotes that directory
contains human-readable text files. 
+ Suffix `seq` denotes that directory contains sequence files. Suffix `txt` denotes that directory
contains human-readable text files. <<BR>>
+ Also I defined some shell variables in .bashrc file of hadoop user to simplify following
scripts.
+ {{{
+ export HAMA_EXAMPLES=$HAMA_HOME/hama-examples*.jar
+ export SPMV=/user/hduser/spmv
+ }}}
+ First variable allows fast access to jar with hama examples, which plased in hama home directory,
second variable is prefix in HDFS for tests in this tutorial. If you not defined this variables
just substitute appropriate values into following scripts.
  ==== Usage with RandomMatrixGenerator ====
  `RandomMatrixGenerator` as a `SpMV` works with sequence file format. So, to multiply random
matrix with random vector we will do the following: generate matrix and vector; convert matrix,
vector and result to text file; view matrix, vector and result. This sequence is described
by the following code snippet:
  {{{
- 1: bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar rmgenerator /user/hduser/spmv/matrix-seq
4 4 0.3 4
- 2: bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar rmgenerator /user/hduser/spmv/vector-seq
1 4 0.8 1
- 3: bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar spmv /user/hduser/spmv/matrix-seq /user/hduser/spmv/vector-seq
/user/hduser/spmv/result-seq 4
- 4: ../hadoop/bin/hadoop dfs -rmr /user/hduser/spmv/result-seq/part
- 5: bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar matrixtotext /user/hduser/spmv/matrix-seq
/user/hduser/spmv/matrix-txt
- 6: bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar matrixtotext /user/hduser/spmv/vector-seq
/user/hduser/spmv/vector-txt
- 7: bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar matrixtotext /user/hduser/spmv/result-seq
/user/hduser/spmv/result-txt
+ 0:    hadoop dfs -rmr $SPMV/*/*
+ 1:    hama jar $HAMA_EXAMPLES rmgenerator $SPMV/matrix-seq 6 6 0.4 4
+ 2:    hama jar $HAMA_EXAMPLES rmgenerator $SPMV/vector-seq 1 6 0.9 4
+ 3:    hama jar $HAMA_EXAMPLES spmv $SPMV/matrix-seq $SPMV/vector-seq $SPMV/result-seq 4
+ 4:    hadoop dfs -rmr $SPMV/result-seq/part
+ 5:    hama jar $HAMA_EXAMPLES matrixtotext $SPMV/matrix-seq $SPMV/matrix-txt
+ 6:    hama jar $HAMA_EXAMPLES matrixtotext $SPMV/vector-seq $SPMV/vector-txt
+ 7:    hama jar $HAMA_EXAMPLES matrixtotext $SPMV/result-seq $SPMV/result-txt
- 8: ../hadoop/bin/hadoop dfs -cat /user/hduser/spmv/matrix-txt/*
+ 8:    hadoop dfs -cat /user/hduser/spmv/matrix-txt/*
-     0	 4 1 2 0.09775514911904559
-     3	 4 1 1 0.22718006778335464
-     1	 4 1 1 0.796916052801057
-     3	 4 1 2 0.9680719390036476
-     2	 4 1 0 0.44269679525022
+    0	 6 3 5 0.24316243288531214 2 0.638622414091597 3 0.5480468710898891
+    3	 6 2 5 0.5054043538570098 2 0.03911646523753309
+    1	 6 3 4 0.5077528966368161 5 0.5780340816354201 3 0.4626752204959449
+    4	 6 2 1 0.6512355661856207 4 0.08804976645891671
+    2	 6 2 4 0.7200271909735554 1 0.3510851368183805
+    5	 6 2 2 0.5848717104309032 3 0.0889791409798859
+ 
- 9: ../hadoop/bin/hadoop dfs -cat /user/hduser/spmv/vector-txt/*
+ 9:    hadoop dfs -cat /user/hduser/spmv/vector-txt/*
-     0	 4 4 0 0.3111172930766064 1 0.27242674637140374 2 0.11394746764097863 3 0.0
+    0	 6 6 0 0.3365077672167889 1 0.17498609722570935 2 0.32806410950648845 3 0.6016567879100464
4 0.786158850847722 5 0.6856872945972037
- 10: ../hadoop/bin/hadoop dfs -cat /user/hduser/spmv/result-txt/*
+ 10:   hadoop dfs -cat /user/hduser/spmv/result-txt/*
-     0	 5 5 0 0.011138951690981488 1 0.21710124739573375 2 0.1377306285919371 3 0.11030934594375758
4 0.0
+    0	 6 6 0 0.7059786044267415 1 1.0738967463653346 2 0.6274907669206862 3 0.35938205240905363
4 0.18317827331814918 5 0.24541032101100438
  }}}
- Line 1-2: Generation of input matrix and vector.<<BR>>
+ Line 0: Clean up of directories related to SpMV tests.<<BR>>
+ Line 1-2: Generation of input matrix and vector. In this example we test 6x6 matrix and
1x6 vector multiplication<<BR>>
  Line 3: SpMV algorithm.<<BR>>
  Line 4: Deletion of part files from output directory at line 4. NOTE: `matrixtotext` will
fail if this step will not be performed, because `result-seq` will containg part folder and
`matrixtotext` don't know how to deal with it yet.<<BR>>
  Line 5-7: Convertion of input matrix, input vector and result to text format.<<BR>>
- Line 8-10: Showing the result. NOTE: currently SpMV is buggy - you can see, that output
vector has length of 5, and the last cell of vector has incorrect value. This will be fixed
soon.
+ Line 8-10: Showing the result. 
  
  ==== Usage with arbitrary text files ====
  SpMV works with `SequenceFile`, so we need to provide tools to convert input and output
of SpMV between sequence file format and text format. These tools are `matrixtoseq` and `matrixtotext`.
This programs are included in example driver, so they can be launched like any other example.
`matrixtoseq` converts matrix, represented in text file to sequence file format. Also this
program gives choice to choose target writable: `DenseVectorWritable` and `SparseVectorWritable`.

Mime
View raw message