flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachin Goel <sachingoel0...@gmail.com>
Subject Re: Communicating between nodes at runtime
Date Tue, 21 Jul 2015 13:15:58 GMT
@Stephan, why would there be a problem with the reliability of updates?

Also, what would be the best way to achieve this functionality? Preferably
using the current API.

Cheers!
Sachin

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685

On Tue, Jul 21, 2015 at 4:17 PM, Stephan Ewen <sewen@apache.org> wrote:

> I see the use case for that, but I don't think that this should be realized
> through Flink's network stack. The network stack is designed for continuous
> streams, backpressure, and configurable throughput/latency. I think it is
> not the best solution for asynchronous messaging.
>
> Also, for the ML field, you will want to be able to send "fire and forget"
> messages, as exact and reliable updates are usually necessary for
> stochastic algorithms.
>
> To me, this calls for a different communication fabric.
>
>
> On Mon, Jul 20, 2015 at 5:22 PM, Sachin Goel <sachingoel0101@gmail.com>
> wrote:
>
> > @Max, broadcast variables have to be declared before the program is
> > executed. I want to be able to do something whereby I can send data
> inside
> > a map operation to all nodes. This would perhaps have an effect similar
> to
> > the recent discussion on making Accumulators available before Job
> > completion. [I don't know if it has been merged yet.]
> >
> > @Stephan, I was thinking more in the direction of adding a task at every
> > worker which works as an Event handler to receive data from other nodes.
> I
> > got the idea after going through the iterative runtime module which has a
> > separate task for handling synchronization. I followed it up with going
> > through the optimizer routine wherein it is added to some *auxiliary
> > vertices *list. Can you elaborate on that?
> >
> > Further, is it possible to add an event handler to the runtime context
> > itself? I wrote the boilerplate stuff for this but I haven't defined the
> > channels where the event would be written to the network so other workers
> > can also listen to it.  [
> > https://github.com/sachingoel0101/flink/tree/async_iter]. I want to be
> > able
> > to call broadcast and receive as getRuntimeContext.broadcast and
> > getRuntimeContext.receive, which ensures complete data transmission
> across
> > nodes, even at runtime.
> > I'm not sure how to add a separate task which handles event over the
> > network like the IterationSynchronization task [which is the best use I
> > could find of using event handlers].
> >
> > Let me know your thoughts. Or if you can think of a better way to achieve
> > sharing of data which is *only *evaluated at runtime.
> >
> > Cheers!
> > Sachin
> >
> > -- Sachin Goel
> > Computer Science, IIT Delhi
> > m. +91-9871457685
> >
> > On Mon, Jul 20, 2015 at 3:34 PM, Stephan Ewen <sewen@apache.org> wrote:
> >
> > > You are probably looking for a parameter server tool.
> > >
> > > How about setting up one of these memory grids to use that? Apache
> > Ignite,
> > > or Apache Geode, or one of those.
> > >
> > > On Fri, Jul 17, 2015 at 6:19 PM, Sachin Goel <sachingoel0101@gmail.com
> >
> > > wrote:
> > >
> > > > Hi all
> > > > Is it possible to send and receive key,value pairs at runtime? I
> would
> > > like
> > > > to broadcast values at runtime so they are available on every node.
> > This
> > > > somehow seems essential for an implementation of Asynchronous batch
> > > > iterations.
> > > >
> > > > What I would like is to have two functions in the RuntimeContext,
> > > > broadcast(key,value) and receive(key).
> > > > If such a thing doesn't exist now, how should I approach implementing
> > it?
> > > >
> > > > Cheers!
> > > > Sachin
> > > > -- Sachin Goel
> > > > Computer Science, IIT Delhi
> > > > m. +91-9871457685
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message