apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniruddha Thombare <anirud...@datatorrent.com>
Subject Re: how to connect to database having the DB properties file outside the resources folder or applcation
Date Thu, 17 Nov 2016 14:58:12 GMT
About req 3:
AFAIK CDH does provide REST access to cluster health parameters among many
other things.
Not sure about ambari though



Sent with difficulty, I mean handheld ;)

On 17 Nov 2016 8:25 pm, "Munagala Ramanath" <ram@datatorrent.com> wrote:

> For Req1, a better approach might be to have a single "error reporting"
> operator that connects to the DB;
> all the other operators can send custom error tuples to it with
> appropriate details of the type of error.
> That way, if there are issues that are DB-related, you have 1 place to
> debug them instead of 20.
> With that approach, your DB operator will need 20 input ports and a new
> one added each time you add
> an additional operator to the DAG; an alternative is to use the
> OperatorRequest mechanism to
> communicate with the DB operator instead of the normal ports-and-tuples
> method. This is illustrated
> in the example at https://github.com/DataTorrent/examples/tree/
> master/tutorials/throttle
> where it is used to modulate the output data rate of the input operator
> when the downstream operators
> are unable to keep up.
> For Req2, you can put the DB properties in an HDFS location and read it in
> the setup() callback of the
> operator. The path to that location can be in your regular properties file
> within the app package.
> A similar, though more elaborate method is to run a service on a known
> port which can be queried
> for those properties; the operators will then connect to that service in
> setup() to retrieve the info.
> For Req3, cluster-level issues are generally not made available to
> user-level callbacks; you'll need
> an external process(es) to monitor log files, invoke REST APIs etc.
> Ram
> On Thu, Nov 17, 2016 at 5:50 AM, Venky <venkatesh.keetha@gmail.com> wrote:
>> Hi,
>> Req1: I have around 20 operators. In every operator I want to connect to
>> database to mark my processing as failed when there is an Io Exception or
>> any cluster level exception. I wanted to use plain java class here, please
>> let me know how we can implement it. If not suggest the better approach
>> Req2: I don't want the DB properties file within the application package
>> and wanted to maintain the DB properties outside in a properties file and
>> able to use them while connecting to database to fulfill my Req1. Please
>> let me know how we can maintain the DB properties file outside.
>> Req3: How can I capture cluster level exceptions like RPC Failure and etc?
>> Regards,
>> Venkat.

View raw message