apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siyuan Hua <siy...@datatorrent.com>
Subject Re: Proposal: Variation for the Apex Api
Date Tue, 12 Apr 2016 17:08:39 GMT
Hey Sandesh,

I like your idea, but here is what I think.
 API Stability is VERY important!  The API we have to build a dag has been
used for a long time. If the new change doesn't help a lot, I won't
recommend to change the API

so +0 :)

On Mon, Apr 11, 2016 at 12:02 PM, Sandesh Hegde <sandesh@datatorrent.com>
wrote:

> Hello Chinmay,
>
> Idea is to follow natural order. Suppose you want to set attribute on a
> port, you will have to write something like below
>
> dag.setInputPortAttribute(deserializeJSON.input,
> Context.PortContext.PARTITION_PARALLEL, true);
>
> How about setting it directly on the ports?
> deserializeJSON.input.setAttribute(Context.PortContext.PARTITION_PARALLEL,
> true));
>
> Same thing goes for attributes of the Operators.
>
> These new set of APIs will support all the features of our
> existing API, but in a more natural way.
> It is not similar to High-Level API, it is just moonlighting idea about our
> existing API.
>
> Thanks
>
>
>
>
>
>
>
> On Sun, Apr 10, 2016 at 11:20 PM Chinmay Kolhatkar <chinmay@apache.org>
> wrote:
>
> > I feel that adding all operators to dag makes it explicit for a developer
> > who is looking at the code for the first time.
> >
> > Besides, one might want to do dag.getMeta on operators and ports to set
> > some attributes.
> > I'm not sure how much explicit will that be if we bypass adding operators
> > to the DAG.
> >
> > Though, I totally agree with the intention of simplifying the APIs for
> > which the proposal from Siyuan in another mail is a nice first step.
> >
> > Regards,
> > Chinmay.
> >
> >
> >
> > On Mon, Apr 11, 2016 at 10:48 AM, Sandesh Hegde <sandesh@datatorrent.com
> >
> > wrote:
> >
> > > Yes, that is the idea.
> > >
> > > On Sun, Apr 10, 2016, 9:28 PM Sandeep Deshmukh <
> sandeep@datatorrent.com>
> > > wrote:
> > >
> > > > Are you suggesting we add only the input adapters to the dag and then
> > all
> > > > the connected ones are added by simply traversing the DAG?
> > > >
> > > > Regards,
> > > > Sandeep
> > > >
> > > > On Fri, Apr 8, 2016 at 11:06 PM, Sandesh Hegde <
> > sandesh@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Hello Team,
> > > > >
> > > > > As we were talking about High Level api in another thread, here is
> a
> > > > small
> > > > > variation proposal for our existing apis,
> > > > >
> > > > > Take look at the current code and the proposed code
> > > > >
> > > > > Current Style
> > > > > ---------------------------
> > > > >    KafkaSinglePortStringInputOperator kafkaInput =
> > > > > dag.addOperator("kafkaInput", new
> > > KafkaSinglePortStringInputOperator());
> > > > >     DeserializeJSON deserializeJSON =
> dag.addOperator("deserialize",
> > > new
> > > > > DeserializeJSON());
> > > > >     RedisJoin redisJoin = dag.addOperator("redisJoin", new
> > > RedisJoin());
> > > > >     CampaignProcessorWithApexWindow campaignProcessor =
> > > > > dag.addOperator("campaignProcessor", new
> > > > > CampaignProcessorWithApexWindow());
> > > > >
> > > > >     // Connect the Ports in the Operators
> > > > >     dag.addStream("deserialize", kafkaInput.outputPort,
> > > > > deserializeJSON.input) ;
> > > > >     dag.addStream("redisJoin", filterFields.output,
> redisJoin.input)
> > ;
> > > > >     dag.addStream("output", redisJoin.output,
> > campaignProcessor.input);
> > > > > -------------------------------------
> > > > >
> > > > >
> > > > > Proposed Change ( Just create Operators and connect the output port
> > to
> > > > > input ports and then add the input operators to DAG )
> > > > > -------------------------------------------
> > > > >     KafkaSinglePortStringInputOperator
> > > > kafkaSinglePortStringInputOperator =
> > > > > new KafkaSinglePortStringInputOperator() ;
> > > > >     DeserializeJSON deserializeJSON = new DeserializeJSON();
> > > > >     RedisJoin redisJoin = new RedisJoin() ;
> > > > >     CampaignProcessor campaignProcessor =  new CampaignProcessor()
> ;
> > > > >
> > > > >     kafkaInput.outputPort.connect(deserializeJSON.input);
> > > > >     filterFields.output.connect(redisJoin.input) ;
> > > > >     redisJoin.output.connect(campaignProcessor.input) ;
> > > > >
> > > > >     dag.add(kafkaInput);
> > > > > -----------------------------------------
> > > > >
> > > > > Things to note : Name can be generated internally or user can
> > specify a
> > > > > name property of the operator ( like Dag.AddOperator("name", ...)
)
> > > > >
> > > > > Let me know your thoughts.
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message