hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-815) Hama Pipes uses C++ templates
Date Sat, 23 Nov 2013 23:47:35 GMT

    [ https://issues.apache.org/jira/browse/HAMA-815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13830804#comment-13830804
] 

Hudson commented on HAMA-815:
-----------------------------

SUCCESS: Integrated in Hama-Nightly-for-Hadoop-2.x #101 (See [https://builds.apache.org/job/Hama-Nightly-for-Hadoop-2.x/101/])
HAMA-815: Hama Pipes uses C++ templates (fixed glibc detection) (millecker: rev 1544764)
* /hama/trunk/c++/src/main/native/pipes/impl/Pipes.cc
HAMA-815: Hama Pipes uses C++ templates (millecker: rev 1544761)
* /hama/trunk/CHANGES.txt
* /hama/trunk/c++/README.txt
* /hama/trunk/c++/src/CMakeLists.txt
* /hama/trunk/c++/src/main/native/examples/README.txt
* /hama/trunk/c++/src/main/native/examples/conf/matrixmultiplication.xml
* /hama/trunk/c++/src/main/native/examples/conf/piestimator.xml
* /hama/trunk/c++/src/main/native/examples/conf/summation.xml
* /hama/trunk/c++/src/main/native/examples/impl/DenseDoubleVector.cc
* /hama/trunk/c++/src/main/native/examples/impl/DenseDoubleVector.hh
* /hama/trunk/c++/src/main/native/examples/impl/matrixmultiplication.cc
* /hama/trunk/c++/src/main/native/examples/impl/piestimator.cc
* /hama/trunk/c++/src/main/native/examples/impl/summation.cc
* /hama/trunk/c++/src/main/native/examples/input
* /hama/trunk/c++/src/main/native/pipes/api/hama/Pipes.hh
* /hama/trunk/c++/src/main/native/pipes/api/hama/TemplateFactory.hh
* /hama/trunk/c++/src/main/native/pipes/impl/HamaPipes.cc
* /hama/trunk/c++/src/main/native/pipes/impl/Pipes.cc
* /hama/trunk/c++/src/main/native/utils/api/hadoop/SerialUtils.hh
* /hama/trunk/c++/src/main/native/utils/impl/SerialUtils.cc
* /hama/trunk/c++/src/main/native/utils/impl/StringUtils.cc
* /hama/trunk/commons/src/main/java/org/apache/hama/commons/io/KeyValueWritable.java
* /hama/trunk/commons/src/main/java/org/apache/hama/commons/io/PipesKeyValueWritable.java
* /hama/trunk/commons/src/main/java/org/apache/hama/commons/io/PipesVectorWritable.java
* /hama/trunk/commons/src/main/java/org/apache/hama/commons/io/VectorWritable.java
* /hama/trunk/core/pom.xml
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJob.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJobClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPTask.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/PartitioningRunner.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/PipesApplicable.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/PipesApplication.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/PipesBSP.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/PipesPartitioner.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/Submitter.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/protocol/BinaryProtocol.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/protocol/DownwardProtocol.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/protocol/MessageType.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/protocol/StreamingProtocol.java
* /hama/trunk/core/src/main/java/org/apache/hama/pipes/protocol/UplinkReader.java
* /hama/trunk/examples/src/main/java/org/apache/hama/examples/util/Generator.java
* /hama/trunk/examples/src/main/java/org/apache/hama/examples/util/VectorWritableMatrixGen.java
* /hama/trunk/src/assemble/bin.xml


> Hama Pipes uses C++ templates
> -----------------------------
>
>                 Key: HAMA-815
>                 URL: https://issues.apache.org/jira/browse/HAMA-815
>             Project: Hama
>          Issue Type: New Feature
>          Components: bsp core, pipes
>    Affects Versions: 0.6.3
>            Reporter: Martin Illecker
>            Assignee: Martin Illecker
>             Fix For: 0.7.0
>
>         Attachments: HAMA-815.patch
>
>
> *Extending Hama Pipes to use C++ templates*
> Currently all messages are converted to *strings* before they are transferred over a
socket communication between C++ and Java and vice versa.
> To take advantage of the binary socket communication we will serialize and deserialize
basic types like *int, long, float, double* directly without converting to strings. This will
minimize the risk of type conversation errors. Other types (except these basic types) are
transferred as strings. 
> It's also possible to create custom *Writables* and serialize and deserialize the object
to string by overriding the following methods. (e.g., *PipesVectorWritable* and *PipesKeyValueWritable*)
> {code}
> @Override public void readFields(DataInput in) throws IOException
> @Override public void write(DataOutput out) throws IOException
> {code}
> Hama Streaming, which depends on Hama Pipes, is still using strings.
> The following methods change
> {{virtual void sendMessage(const string& peerName, const string& msg)}}
> {{virtual const string& getCurrentMessage()}}
> {{virtual void write(const string& key, const string& value)}}
> {{virtual bool readNext(string& key, string& value)}}
> to support C++ templates:
> {{virtual void sendMessage(const string& peer_name, *const M& msg*)}}
> {{virtual *M* getCurrentMessage()}}
> {{virtual void write(*const K2& key, const V2& value*)}}
> {{virtual bool readNext(*K1& key, V1& value*)}}
> Also *SequenceFile* functions uses templates:
> {{bool sequenceFileReadNext(int32_t file_id, *K& key, V& value*)}}
> {{bool sequenceFileAppend(int32_t file_id, *const K& key, const V& value*)}}
> And the native *Partitioner* supports it:
> {code}
>   template<class K1, class V1, class K2, class V2, class M>
>   class Partitioner {
>   public:
>     virtual int partition(const K1& key, const V1& value, int32_t num_tasks)
= 0;
>     virtual ~Partitioner() {}
>   };
> {code}
> This will minimize type conversation errors and change the compilation procedure. Because
of the nature of C++ templates, static libraries are not possible anymore. The compiler will
substitute all templates at compile time.
> The compile command will look like:
> {code}
> g++ -m64 -Ic++/src/main/native/utils/api \
>       -Ic++/src/main/native/pipes/api \
>       -Lc++/target/native \
>       -lhadooputils -lpthread \
>       PROGRAM.cc \
>       -o PROGRAM \
>       -g -Wall -O2
> {code}
> Finally the job configuration supports the following properties:
> {code}
>   <property>
>     <name>bsp.input.format.class</name>
>     <value>org.apache.hama.bsp.KeyValueTextInputFormat</value>
>   </property>
>   <property>
>     <name>bsp.input.key.class</name>
>     <value>org.apache.hadoop.io.Text</value>
>   </property>
>   <property>
>     <name>bsp.input.value.class</name>
>     <value>org.apache.hadoop.io.Text</value>
>   </property>
>   <property>
>     <name>bsp.output.format.class</name>
>     <value>org.apache.hama.bsp.SequenceFileOutputFormat</value>
>   </property>
>   <property>
>     <name>bsp.output.key.class</name>
>     <value>org.apache.hadoop.io.Text</value>
>   </property>
>   <property>
>     <name>bsp.output.value.class</name>
>     <value>org.apache.hadoop.io.DoubleWritable</value>
>   </property>
>   <property>
>     <name>bsp.message.class</name>
>     <value>org.apache.hadoop.io.DoubleWritable</value>
>   </property>
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message