apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: What-if analysis with apex
Date Wed, 27 Jan 2016 15:42:39 GMT
Amit,
Here are some answers
- Logic that you want to run can be coded as an utility, that is then
invoked by any other operator
- PopulateDAG() is today part of roll out of the app, i.e it is similar to
"compileTime" and not "runTime". You could do runTime, but then you will
need to go through dtcli. Today runTime changes via dtcli will need a lot
more coding. A very early version of runTime changes (based on system
metrics) exist, but the ask is for changes based on application data. That
ask is in the roadmap of module rollout (phase II?) and others can comment
on the roadmap for runtTime populateDAG.
- Outputs of many operators can be streamed as input to one operator in
following ways
   - Each output having different schema will mean different input ports on
that operator as port schema is fixed. This is fine, but will clutter the
DAG
   - If the schema of these output ports is same, there is a merge operator
that does that (
https://github.com/apache/incubator-apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/stream/StreamMerger.java).
You can write one for Nx1 merge by extending the above class.

Thks,
Amol


On Wed, Jan 27, 2016 at 6:03 AM, Amit Shah <amits.84@gmail.com> wrote:

> Thanks Ashwin for the follow up.
> I am not sure if I completely follow the query -> store -> query pattern.
> What does query mean here? Why would we need a in-memory store?
> Trying to list down the flow I came up with below points
>
>    1. We need to build a DAG after we get to know the cell (table, row
>    and column index) that is modified by the user.
>    2. Once we receive user input (i.e. once the user modifies a value in
>    a table) the populateDAG() method should be called.
>    3. The populateDAG() implementation would
>    1. Determine what cells should be updated across all tables
>       2. Create an Operator per cell that is affected by the change. From
>       the demo code I see dag.addOperator method instantiating an
>       operator. Since the logic to update an cell would be the same
>       across tables how do we create new operators per cell to have a graph that
>       looks what Bhupesh envisioned in his last email reply? In my view
>       the graph would like
>
>                     O1 (for user modified cell) -> O2 (table X, row Y,
> column index 2) -> O5 (table E, row F, column index 10000)
>                                                                O3 (table
> M, row N, column index 3)
>               ->  O6 (update UI)
>                                                                O4 (table
> P, row Q, column index 1)
>
>               3. We want the DAG to be evaluated instantly once the
> populateDAG() method finishes. How do we do it?
>               4. Can outputs from many operators be streamed as an input
> to one operator? From the above example outputs from O3, O4 and O5 need
> to go to O6.
>
> I appreciate your inputs on this.
>
> Thanks,
> Amit.
>
>
> On Wed, Jan 27, 2016 at 1:49 PM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>
>> Amit,
>>
>> Thanks for the response. You can use the query --> store --> query result
>> pattern to do the real time updates and lookups for what-if analysis.
>>
>> And you can also ingest your real time input data to the store operator.
>> input --> store.
>>
>> That way, you can keep ingesting your data into the store operator where
>> you will keep your OLAP dimensions and measures.
>>
>> For the query/query result pattern example, see this demo:
>>
>>
>> https://github.com/apache/incubator-apex-malhar/blob/master/demos/mobile/src/main/java/com/datatorrent/demos/mobile/Application.java
>>
>> Regards,
>> Ashwin.
>>
>> On Tue, Jan 26, 2016 at 9:52 PM, Amit Shah <amits.84@gmail.com> wrote:
>>
>>> Appreciate the discussion we are having on this topic.
>>>
>>> Bhupesh, If I understand the flow correctly, we would have to define one
>>> DAG per cell in the table that could be modified by the user. Given this,
>>> it would be right to define the DAG only when the table is presented to the
>>> user on the UI (not at definition time since there would be many tables).
>>> Would it be possible to define DAG at runtime i.e. defining & wiring the
>>> operators at runtime?
>>>
>>>
>>> Ashwin, I am glad to answer these questions
>>>
>>> 1. We are extending our OLTP based application by introducing analytical
>>> features that includes what-if kind of analysis. Other features do include
>>> performing OLAP kind of operations like aggregation, slice & dice, drill
>>> down/up, pivoting. Our first milestone is to target what-if kind of
>>> analysis. We don't have any implementation so far. We are exploring out
>>> solutions to these requirements
>>> 2. The technical challenges we have include having an in-memory
>>> calculation engine system that supports parallel writes and provides real
>>> time or near real time response time.
>>>
>>> Hope that answers your queries.
>>>
>>> Thanks,
>>> Amit.
>>>
>>>
>>> On Mon, Jan 25, 2016 at 10:26 PM, Ashwin Chandra Putta <
>>> ashwinchandrap@gmail.com> wrote:
>>>
>>>> Amit,
>>>>
>>>> I have a couple of questions if its not much.
>>>>
>>>> 1. What is the current implementation?
>>>> 2. What are the challenges you are facing?
>>>>
>>>> Regards,
>>>> Ashwin.
>>>> Hello,
>>>>
>>>> I am trying to evaluate apache apex for building an application that
>>>> supports what-if analysis support to users. This co-relates closed with
>>>> excel kind of functionality where changing a value in one cell triggers
>>>> changes in other cell values. In our case we would have multiple rows in
>>>> various tables getting updated when the user changes a row value. The
>>>> response needs to be in real-time or near real-time.
>>>>
>>>> Does Apex fit such an use-case? If so, what would be some of initial
>>>> steps to evaluate it for this use case?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> Ashwin.
>>
>
>

Mime
View raw message