apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: Visualize custom metrics in the dashboard
Date Thu, 03 Sep 2015 20:26:57 GMT
Hey Ilya,

We can discuss the simple AppData operators present in Malhar in this
thread. Operators which can perform aggregations over time buckets, store
results to hdfs using hdht, and make the aggregations available for
visualization are not open source and will have to be discussed offline if
you want to use them.

There are three components in enabling an application for visualization:

- A Query operator which receives queries from the UI
- A Store operator which receives queries from the Query operator and has
the data required to respond to the query.
- A Result operator which receives the data to reply to a query from the
Store operator and send the result to the ui

The sub-DAG of an application enabled application looks like this

(Query Operator) -> (Store Operator) -> (Result Operator)

- The Query operator to use is PubSubWebSocketAppDataQuery, The topic and
uri need to be set on this like in your code example.
- The Store operator to use is AppDataSnapshotServerMap. This operator
accepts a list of Map<String,Object> objects on its input port. The latest
list of maps is what is served in response to a query. Each Map in the list
represents a row in a table. The key of a map corresponds to a column name,
and the value corresponds to the value of that column for that row. The
Store requires a schema to be set on it. The schema defines the columns of
the table that the operator holds, and the types of the columns. An example
of a schema is the following:

{
  "values": [{"name": "hashtag", "type": "string"},
             {"name": "count", "type": "integer"}]
}

In this example the store has two columns hashtag and count of types string
and integer respectively. When the List of Maps is sent to the store, the
store will be expecting each map to have a "hashtag" key with a string
value and a "name" key with an integer value.

- The Result operator to use is PubSubWebSocketAppDataResult. The topic and
uri need to be set on this like in your code example.

The twitter demo in 3.0 is doing this:
https://github.com/apache/incubator-apex-malhar/blob/master/demos/twitter/src/main/java/com/datatorrent/demos/twitter/TwitterTopCounterApplication.java


      URI uri = URI.create("ws://" + gatewayAddress + "/pubsub");

      AppDataSnapshotServerMap snapshotServer = dag.addOperator("Snapshot
Server", new AppDataSnapshotServerMap());

      //Dont Worry about this conversion Map for your use case
      Map<String, String> conversionMap = Maps.newHashMap();
      conversionMap.put(alias, WindowedTopCounter.FIELD_TYPE);
      //This gets the json which is a schema that defines the types of the
data that the operator expects to receive
      String snapshotServerJSON =
SchemaUtils.jarResourceFileToString(schemaFile);

      //Set the schema on the store so that it knows the type of data to
expect
      snapshotServer.setSnapshotSchemaJSON(snapshotServerJSON);
      //Don't set the conversion map for your use case
      snapshotServer.setTableFieldToMapField(conversionMap);

      PubSubWebSocketAppDataQuery wsQuery = dag.addOperator("Query", new
PubSubWebSocketAppDataQuery());
      wsQuery.setUri(uri);
      Operator.OutputPort<String> queryPort = wsQuery.outputPort;
      PubSubWebSocketAppDataResult wsResult =
dag.addOperator("QueryResult", new PubSubWebSocketAppDataResult());
      wsResult.setUri(uri);
      Operator.InputPort<String> queryResultPort = wsResult.input;

      dag.addStream("MapProvider", topCount, snapshotServer.input);
      dag.addStream("Query", queryPort, snapshotServer.query);
      dag.addStream("Result", snapshotServer.queryResult, queryResultPort);


Once you have this portion of the Dag built you just need to have an
upstream operator periodically send the AppDataSnapshotServer (on its input
port)  a List of Maps containing the data that you want made available to
query. In the case of twitter, the Top 10 hashtags, urls, or words are sent
to the snapshot server at the end of each application window. The operator
which does this is this:

https://github.com/apache/incubator-apex-malhar/blob/master/demos/twitter/src/main/java/com/datatorrent/demos/twitter/WindowedTopCounter.java

On line 145 you'll be able to see the List of Maps being built and sent out
to the Snapshot Server.

Once you have your application built, you can launch it. After launching
you can click on the Application's name, near the top of the screen there
will be a Visualize button, which you can click. If you click generate
dashboard, a page should pop up and start displaying charts automatically.

Please let us know how it goes,
Tim


On Thu, Sep 3, 2015 at 7:07 AM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com>
wrote:

> Correct
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<mailto:tim@datatorrent.com>]
> Sent: Thursday, September 03, 2015 03:41 AM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Re: Visualize custom metrics in the dashboard
>
>
> Cool just to check though do you have the enterprise license for 3.0? 2.0
> does not include the visualization tools.
>
> Thanks,
> Tim
>
> On Wed, Sep 2, 2015 at 11:57 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com>
> wrote:
>
> > Hi Timothy - we do actually have an enterprise license and would
> > appreciate more details on how to use those operators. Thanks!
> >
> >
> >
> > Thank you,
> > Ilya Ganelin
> >
> >
> >
> > -----Original Message-----
> > From: Timothy Farkas [tim@datatorrent.com<mailto:tim@datatorrent.com>]
> > Sent: Thursday, September 03, 2015 12:15 AM Eastern Standard Time
> > To: dev@apex.incubator.apache.org
> > Subject: Re: Visualize custom metrics in the dashboard
> >
> >
> > Hi Ilya,
> >
> > I believe the overall approach is correct. Is {"type":"data",
> > "topic":"c1.citadel.latencies", "data":[{"Elapsed":"3987"}, {"Max
> > (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"}, {"Msgs/sec":"48"}]} the JSON
> > that you are seeing output by the PubSubWebsocketOutputOperator? Also is
> > the receiver of the message sending a {"type":"subscribe",
> > "topic":"c1.citadel.latencies"} message?
> >
> > A quick way to debug whether your messages can be sent or received
> properly
> > is to use Simple Websocket
> >
> >
> https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en
> >
> > -You can use it to connect to ws://your.gateway/pubsub
> > -Then enter {"type":"subscribe", "topic":"c1.citadel.latencies"}
> > -Then enter  {"type":"data", "topic":"c1.citadel.latencies",
> > "data":[{"Elapsed":"3987"},
> > {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will not echo the
> > result back to you
> > - If you enter  {"type":"publish", "topic":"c1.citadel.latencies",
> > "data":[{"Elapsed":"3987"},
> > {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will echo the
> result
> > back to you
> >
> > Note that if you have a fresh connection and you don't send the subscribe
> > message, you will not get the result echoed back to you.
> >
> > There is an easier way. Datatorrent already has operators which serve
> data
> > to the UI and Widgets which display the data in the UI, but that is in
> the
> > Enterprise edition.
> >
> > Hope this helps, please keep us posted
> > Tim
> >
> > On Wed, Sep 2, 2015 at 8:17 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com
> > >
> > wrote:
> >
> > > Hi Timothy - the idea was to surface our own metrics to the DataTorrent
> > > UI. We have been rolling with this approach since it was recommended to
> > us
> > > by DT folks. Is there a better way to visualize metrics in the UI?
> > >
> > >
> > >
> > > Thank you,
> > > Ilya Ganelin
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Timothy Farkas [tim@datatorrent.com<mailto:tim@datatorrent.com>]
> > > Sent: Wednesday, September 02, 2015 10:24 PM Eastern Standard Time
> > > To: dev@apex.incubator.apache.org
> > > Subject: Re: Visualize custom metrics in the dashboard
> > >
> > >
> > > Hi Ilya,
> > >
> > > It looks like you are using Gateway as your websocket server. The
> > protocol
> > > to send messages to gateway is to use "publish" as you "type" not
> "data".
> > > Also whoever is receiving data from the gateway needs to send a
> subscribe
> > > message like this to gateway first
> > {"type":"subscribe","topic":"myTopic"},
> > > before they will receive any data from a topic.
> > >
> > > Tim
> > >
> > > On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <
> > Ilya.Ganelin@capitalone.com
> > > >
> > > wrote:
> > >
> > > > Chetan, thanks! I fixed that error, and validated the JSON shown
> below,
> > > > any further ideas?
> > > >
> > > > {"type":"data", "topic":"c1.citadel.latencies",
> > > > "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
> > > > (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
> > > >
> > > > On 9/2/15, 12:28 PM, "Chetan Narsude" <chetan@datatorrent.com>
> wrote:
> > > >
> > > > >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
> > > > ><Ilya.Ganelin@capitalone.com>
> > > > >wrote:
> > > > >
> > > > >> Hi all ­ I¹m attempting to surface some output directly to
the
> > > existing
> > > > >> Apex dashboard. Based on the Twitter example here:
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
> > > >
> > >
> >
> >>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
> > > > >>ion.java
> > > > >> [snip]
> > > > >>
> > > > >>
> > > > >> An example of the JSON object I¹m outputting is below:
> > > > >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> > > > >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
> > > > >>
> > > > >>
> > > > >The highlighted portion (*"2000.00"ms*) above looks pretty wrong to
> me
> > > :-)
> > > > >
> > > > >--
> > > > >Chetan
> > > > >
> > > > >
> > > > >
> > > > >>
> > > > >> Any thoughts on what¹s going on here or how I can debug this?
The
> > job
> > > > >>does
> > > > >> not ever appear to even launch. Thanks!
> > > > >> ________________________________________________________
> > > > >>
> > > > >> The information contained in this e-mail is confidential and/or
> > > > >> proprietary to Capital One and/or its affiliates and may only
be
> > used
> > > > >> solely in performance of work or services for Capital One. The
> > > > >>information
> > > > >> transmitted herewith is intended only for use by the individual
or
> > > > >>entity
> > > > >> to which it is addressed. If the reader of this message is not
the
> > > > >>intended
> > > > >> recipient, you are hereby notified that any review,
> retransmission,
> > > > >> dissemination, distribution, copying or other use of, or taking
of
> > any
> > > > >> action in reliance upon this information is strictly prohibited.
> If
> > > you
> > > > >> have received this communication in error, please contact the
> sender
> > > and
> > > > >> delete the material from your computer.
> > > > >>
> > > >
> > > > ________________________________________________________
> > > >
> > > > The information contained in this e-mail is confidential and/or
> > > > proprietary to Capital One and/or its affiliates and may only be used
> > > > solely in performance of work or services for Capital One. The
> > > information
> > > > transmitted herewith is intended only for use by the individual or
> > entity
> > > > to which it is addressed. If the reader of this message is not the
> > > intended
> > > > recipient, you are hereby notified that any review, retransmission,
> > > > dissemination, distribution, copying or other use of, or taking of
> any
> > > > action in reliance upon this information is strictly prohibited. If
> you
> > > > have received this communication in error, please contact the sender
> > and
> > > > delete the material from your computer.
> > > >
> > > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message