hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Jungblut <thomas.jungb...@gmail.com>
Subject Re: [jira] [Updated] (HAMA-619) Hama Pipes
Date Tue, 11 Sep 2012 17:35:03 GMT
Ah I remember, it was because of the center writer in my kmeans example!
Sorry. However I don't think writing sequence file via a binary protocol is
smart.
Am 11.09.2012 18:35 schrieb "Thomas Jungblut (JIRA)" <jira@apache.org>:

>
>      [
> https://issues.apache.org/jira/browse/HAMA-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Thomas Jungblut updated HAMA-619:
> ---------------------------------
>
>     Attachment: HAMA-619.patch
>
> Hey Martin, I just generated a patch for the java part. Would like to
> start with streaming soon, so I need to finalize the protocol.
>
> I have a question about the implementation of the read and write from and
> to sequence files.
>
> Why aren't you using the plain "WRITE_KEYVALUE" method?
>
> You can configure the job to output sequence files, there is no need to
> implement the stuff on your own.
> Personally I would delete this, but I don't know what parts of your
> implementations are grounding on it.
>
> > Hama Pipes
> > ----------
> >
> >                 Key: HAMA-619
> >                 URL: https://issues.apache.org/jira/browse/HAMA-619
> >             Project: Hama
> >          Issue Type: New Feature
> >          Components: bsp core, messaging
> >    Affects Versions: 0.6.0
> >            Reporter: Martin Illecker
> >            Assignee: Martin Illecker
> >         Attachments: HAMA-619.patch
> >
> >
> > *Adapting Hadoop Pipes to Hama for C++ integration.*
> > The package *org.apache.hama.pipes* contains all pipes specific code.
> The C++ implementation is located in the *c++* folder under *pipes* and
> *utils*. There are also some minor modifications in other packages but you
> can checkout the full Hama source here: [
> https://svn.illecker.at/hadoop/hama-0.5.0-gpu/]
> > The changelog can be found here: [here|
> http://hadoop.illecker.at/?page_id=6]
> > Hama Pipes: [
> https://svn.illecker.at/hadoop/hama-0.5.0-gpu/core/src/main/java/org/apache/hama/pipes/
> ]
> > Hama Pipes C++: [
> https://svn.illecker.at/hadoop/hama-0.5.0-gpu/c++/pipes/]
> > Hama Pipes provides the following methods for C++ integration: (similar
> to Hama BSP)
> > *{{virtual void sendMessage(const string& peerName, const string& msg) =
> 0;}}*
> > bq. Send a data with a tag to another BSPSlave corresponding to
> hostname. Messages sent by this method are not guaranteed to be received in
> a sent order.
> > *{{virtual const string& getCurrentMessage() = 0;}}*
> > bq. Returns a message from the peer's received messages queue (a FIFO).
> > *{{virtual int getNumCurrentMessages() = 0;}}*
> > bq. Returns the number of messages in the peer's received messages queue.
> > *{{virtual void sync() = 0;}}*
> > {quote}
> > Barrier Synchronization.
> > Sends all the messages in the outgoing message queues to the
> corresponding remote peers.
> > {quote}
> > *{{virtual long getSuperstepCount() = 0;}}*
> > bq. Returns the count of current super-step
> >
> > *{{virtual const string& getPeerName() = 0;}}*
> > bq. Returns the name of this peer in the format "hostname:port".
> >
> > *{{virtual const string& getPeerName(int index) = 0;}}*
> > bq. Returns the name of n-th peer from sorted array by name.
> >
> > *{{virtual int getPeerIndex() = 0;}}*
> > bq. Returns the index of this peer from sorted array by name.
> > *{{virtual vector<string> getAllPeerNames() = 0;}}*
> > bq. Returns the names of all the peers executing tasks from the same job
> (including this peer).
> >
> > *{{virtual int getNumPeers() = 0;}}*
> > bq. Returns the number of peers
> >
> > *{{virtual void clear() = 0;}}*
> > bq. Clears all queues entries.
> >
> > *{{virtual void write(const string& key, const string& value) = 0;}}*
> > bq. Writes a key/value pair to the output collector
> >
> > *{{virtual bool readNext(string& key, string& value) = 0;}}*
> > bq. Deserializes the next input key value into the given objects;
> > *{{virtual void reopenInput() = 0;}}*
> > bq. Closes the input and opens it right away, so that the file pointer
> is at the beginning again.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message