Mailing-List: contact users-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@apex.apache.org
MIME-Version: 1.0
In-Reply-To: <CAJ484BgHQMEHus14PDpSUoXY4UNoHrBrKEmqvLAgOXyff1mzQw@mail.gmail.com>
References: <CAG1PUuaPv1FcO4JpPZzwK4atCcZ6SfMf0g_kOdQP9opD+QFvcA@mail.gmail.com>
 <CAJ484BgHQMEHus14PDpSUoXY4UNoHrBrKEmqvLAgOXyff1mzQw@mail.gmail.com>
From: Aniruddha Thombare <aniruddha@datatorrent.com>
Date: Thu, 17 Nov 2016 20:28:12 +0530
Message-ID: <CAL8spAKmXQOez6eAQ10i9gKMSpXP-5p3RHurojiGrAheDOVT3w@mail.gmail.com>
Subject: Re: how to connect to database having the DB properties file outside
 the resources folder or applcation
To: users@apex.apache.org
Content-Type: multipart/alternative; boundary=001a114db5b8334a6e0541806b1f
archived-at: Thu, 17 Nov 2016 14:58:22 -0000

--001a114db5b8334a6e0541806b1f
Content-Type: text/plain; charset=UTF-8

About req 3:
AFAIK CDH does provide REST access to cluster health parameters among many
other things.
Not sure about ambari though

Thanks,

A

_____________________________________
Sent with difficulty, I mean handheld ;)

On 17 Nov 2016 8:25 pm, "Munagala Ramanath" <ram@datatorrent.com> wrote:

> For Req1, a better approach might be to have a single "error reporting"
> operator that connects to the DB;
> all the other operators can send custom error tuples to it with
> appropriate details of the type of error.
> That way, if there are issues that are DB-related, you have 1 place to
> debug them instead of 20.
>
> With that approach, your DB operator will need 20 input ports and a new
> one added each time you add
> an additional operator to the DAG; an alternative is to use the
> OperatorRequest mechanism to
> communicate with the DB operator instead of the normal ports-and-tuples
> method. This is illustrated
> in the example at https://github.com/DataTorrent/examples/tree/
> master/tutorials/throttle
> where it is used to modulate the output data rate of the input operator
> when the downstream operators
> are unable to keep up.
>
> For Req2, you can put the DB properties in an HDFS location and read it in
> the setup() callback of the
> operator. The path to that location can be in your regular properties file
> within the app package.
> A similar, though more elaborate method is to run a service on a known
> port which can be queried
> for those properties; the operators will then connect to that service in
> setup() to retrieve the info.
>
> For Req3, cluster-level issues are generally not made available to
> user-level callbacks; you'll need
> an external process(es) to monitor log files, invoke REST APIs etc.
>
> Ram
>
>
>
> On Thu, Nov 17, 2016 at 5:50 AM, Venky <venkatesh.keetha@gmail.com> wrote:
>
>> Hi,
>>
>> Req1: I have around 20 operators. In every operator I want to connect to
>> database to mark my processing as failed when there is an Io Exception or
>> any cluster level exception. I wanted to use plain java class here, please
>> let me know how we can implement it. If not suggest the better approach
>>
>> Req2: I don't want the DB properties file within the application package
>> and wanted to maintain the DB properties outside in a properties file and
>> able to use them while connecting to database to fulfill my Req1. Please
>> let me know how we can maintain the DB properties file outside.
>>
>> Req3: How can I capture cluster level exceptions like RPC Failure and etc?
>>
>> Regards,
>> Venkat.
>>
>
>

--001a114db5b8334a6e0541806b1f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">About req 3:<br>
AFAIK CDH does provide REST access to cluster health parameters among many =
other things.<br>
Not sure about ambari though<br></p>
<p dir=3D"ltr">Thanks,</p>
<p dir=3D"ltr">A<br></p>
<p dir=3D"ltr">_____________________________________<br>
Sent with difficulty, I mean handheld ;)</p>
<div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 17 Nov 2016 8:=
25 pm, &quot;Munagala Ramanath&quot; &lt;<a href=3D"mailto:ram@datatorrent.=
com">ram@datatorrent.com</a>&gt; wrote:<br type=3D"attribution"><blockquote=
 class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli=
d;padding-left:1ex"><div dir=3D"ltr">For Req1, a better approach might be t=
o have a single &quot;error reporting&quot; operator that connects to the D=
B;<div>all the other operators can send custom error tuples to it with appr=
opriate details of the type of error.</div><div>That way, if there are issu=
es that are DB-related, you have 1 place to debug them instead of 20.</div>=
<div><br></div><div>With that approach, your DB operator will need 20 input=
 ports and a new one added each time you add</div><div>an additional operat=
or to the DAG; an alternative is to use the OperatorRequest mechanism to</d=
iv><div>communicate with the DB operator instead of the normal ports-and-tu=
ples method. This is illustrated</div><div>in the example at=C2=A0<a href=
=3D"https://github.com/DataTorrent/examples/tree/master/tutorials/throttle"=
 target=3D"_blank">https://github.com/<wbr>DataTorrent/examples/tree/<wbr>m=
aster/tutorials/throttle</a></div><div>where it is used to modulate the out=
put data rate of the input operator when the downstream operators</div><div=
>are unable to keep up.</div><div><br></div><div>For Req2, you can put the =
DB properties in an HDFS location and read it in the setup() callback of th=
e</div><div>operator. The path to that location can be in your regular prop=
erties file within the app package.</div><div>A similar, though more elabor=
ate method is to run a service on a known port which can be queried</div><d=
iv>for those properties; the operators will then connect to that service in=
 setup() to retrieve the info.</div><div><br></div><div>For Req3, cluster-l=
evel issues are generally not made available to user-level callbacks; you&#=
39;ll need</div><div>an external process(es) to monitor log files, invoke R=
EST APIs etc.</div><div><br></div><div>Ram</div><div><br></div><div><br></d=
iv></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, =
Nov 17, 2016 at 5:50 AM, Venky <span dir=3D"ltr">&lt;<a href=3D"mailto:venk=
atesh.keetha@gmail.com" target=3D"_blank">venkatesh.keetha@gmail.com</a>&gt=
;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div cla=
ss=3D"m_228360632430395450m_8431300982685307911gmail_signature"><div dir=3D=
"ltr"><span style=3D"font-size:12.8px">Hi,</span><div style=3D"font-size:12=
.8px"><br></div><div style=3D"font-size:12.8px">Req1: I have around 20 oper=
ators. In every operator I want to connect to database to mark my processin=
g as failed when there is an Io Exception or any cluster level exception. I=
 wanted to use plain java class here, please let me know how we can impleme=
nt it. If not suggest the better approach</div><div style=3D"font-size:12.8=
px"><br></div><div style=3D"font-size:12.8px">Req2: I don&#39;t want the DB=
 properties file within the application package and wanted to maintain the =
DB properties outside in a properties file and able to use them while conne=
cting to database to fulfill my Req1. Please let me know how we can maintai=
n the DB properties file outside.</div><div style=3D"font-size:12.8px"><br>=
</div><div style=3D"font-size:12.8px">Req3: How can I capture cluster level=
 exceptions like RPC Failure and etc?</div><div style=3D"font-size:12.8px">=
<br></div><div style=3D"font-size:12.8px">Regards,</div><div style=3D"font-=
size:12.8px">Venkat.</div></div></div>
</div>
</blockquote></div><br></div>
</blockquote></div></div>

--001a114db5b8334a6e0541806b1f--