apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhupesh Chawda <bhup...@datatorrent.com>
Subject [DISCUSS] Proposal for adapting Malhar operators for batch use cases
Date Mon, 16 Jan 2017 06:32:56 GMT
Hi All,

While design / implementation for custom control tuples is ongoing, I
thought it would be a good idea to consider its usefulness in one of the
use cases -  batch applications.

This is a proposal to adapt / extend existing operators in the Apache Apex
Malhar library so that it is easy to use them in batch use cases.
Naturally, this would be applicable for only a subset of operators like
File, JDBC and NoSQL databases.
For example, for a file based store, (say HDFS store), we could have
FileBatchInput and FileBatchOutput operators which allow easy integration
into a batch application. These operators would be extended from their
existing implementations and would be "Batch Aware", in that they may
understand the meaning of some specific control tuples that flow through
the DAG. Start batch and end batch seem to be the obvious candidates that
come to mind. On receipt of such control tuples, they may try to modify the
behavior of the operator - to reinitialize some metrics or finalize an
output file for example.

We can discuss the potential control tuples and actions in detail, but
first I would like to understand the views of the community for this

~ Bhupesh

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message