apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: how to connect to database having the DB properties file outside the resources folder or applcation
Date Thu, 17 Nov 2016 14:55:28 GMT
For Req1, a better approach might be to have a single "error reporting"
operator that connects to the DB;
all the other operators can send custom error tuples to it with appropriate
details of the type of error.
That way, if there are issues that are DB-related, you have 1 place to
debug them instead of 20.

With that approach, your DB operator will need 20 input ports and a new one
added each time you add
an additional operator to the DAG; an alternative is to use the
OperatorRequest mechanism to
communicate with the DB operator instead of the normal ports-and-tuples
method. This is illustrated
in the example at
where it is used to modulate the output data rate of the input operator
when the downstream operators
are unable to keep up.

For Req2, you can put the DB properties in an HDFS location and read it in
the setup() callback of the
operator. The path to that location can be in your regular properties file
within the app package.
A similar, though more elaborate method is to run a service on a known port
which can be queried
for those properties; the operators will then connect to that service in
setup() to retrieve the info.

For Req3, cluster-level issues are generally not made available to
user-level callbacks; you'll need
an external process(es) to monitor log files, invoke REST APIs etc.


On Thu, Nov 17, 2016 at 5:50 AM, Venky <venkatesh.keetha@gmail.com> wrote:

> Hi,
> Req1: I have around 20 operators. In every operator I want to connect to
> database to mark my processing as failed when there is an Io Exception or
> any cluster level exception. I wanted to use plain java class here, please
> let me know how we can implement it. If not suggest the better approach
> Req2: I don't want the DB properties file within the application package
> and wanted to maintain the DB properties outside in a properties file and
> able to use them while connecting to database to fulfill my Req1. Please
> let me know how we can maintain the DB properties file outside.
> Req3: How can I capture cluster level exceptions like RPC Failure and etc?
> Regards,
> Venkat.

View raw message