apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Operator Node Affinity
Date Thu, 02 Mar 2017 16:34:26 GMT
As Thomas says, attaching a JVM to an operator is do-able, but is against
the norm in a distributed cluster. A distributed OS cannot guarantee a
node. It could be down or not have resources, .... ZK way or any other way
to discover post deployment is the way to go. I think a webservice call
through Stram to get the specifics will work too.


E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*

www.datatorrent.com  |  apex.apache.org

*Join us at Apex Big Data World-San Jose
<http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
[image: http://www.apexbigdata.com/san-jose-register.html]

On Wed, Mar 1, 2017 at 8:16 PM, Thomas Weise <thw@apache.org> wrote:

> If I understand it correctly you want to run a server in an operator,
> discover its endpoint and push data to it? The preferred way of doing that
> would be to announce the endpoint through a discovery mechanism (such as
> ZooKeeper or a shared file) that the upstream entity can use to find the
> endpoint.
> If you are looking for a way to force deploy on a specific node, then have
> a look at the OperatorContext.LOCALITY_HOST attribute (and also
> AffinityRulesTest). AFAIK you can use a specific host name and the
> scheduler will make best effort to get a container on that host, but there
> isn't a guarantee. Generally, services running on the cluster shouldn't
> make assumptions about hosts and ports and use discovery instead.
> HTH,
> Thomas
> On Wed, Mar 1, 2017 at 7:53 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com
> >
> wrote:
> > Hello, all – is there any way to deploy a given operator to a specific
> > Node? E.g. if I’m trying to create a listener for a TCP socket that can
> > then pipe data to a DAG, is there any way for the location of that
> listener
> > to be deterministic so an upstream entity knows what to connect to?
> >
> >
> >
> > - Ilya Ganelin
> >
> > [image: id:image001.png@01D1F7A4.F3D42980]
> >
> > ------------------------------
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message