apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukkamula, Suryavamshivardhan (CWM-NR)" <suryavamshivardhan.mukkam...@rbc.com>
Subject RE: Information Needed
Date Tue, 17 May 2016 22:24:22 GMT
Hello Aswin/Ram,

I am trying to use the JdbcPOJOInputOperator operators from Malhar JDBC, when I launch the
application I am running into Null pointer exception. I am passing the DB connection details
from properties XML.

May I know where to look for the launch logs , I am running the DT application on my sandbox.


Regards,
Surya Vamshi

From: Mukkamula, Suryavamshivardhan (CWM-NR) [mailto:suryavamshivardhan.mukkamula@rbc.com]
Sent: 2016, May, 17 10:59 AM
To: users@apex.incubator.apache.org
Subject: RE: Information Needed

Hi,

Thank you for the inputs Ram. Understood how the properties are read.

Regards,
Surya Vamshi

From: Munagala Ramanath [mailto:ram@datatorrent.com]
Sent: 2016, May, 17 10:53 AM
To: users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>
Subject: Re: Information Needed

Declare fields in your operators with appropriate setters and getters, for example:

private String foo;
public void setFoo(String v) {foo = v;}
public String getFoo(){ return foo;}

Then in the properties file, include a stanza like this:
  <property>
    <name>dt.application.{appName}.operator.{opName}.prop.foo</name>
    <value>Hello World</value>
  </property>

where {appName} is the name of your application provided in the parameter of
@ApplicationAnnotation, {opName} is the operator name given in the addOperator() call.

The platform will automatically inject this value into your operator so that it is ready to
use in
all of the operator callbacks like setup(), beginWindow(), endWindow(), emitTuples(), etc.

You can also use "*" wildcards for application and operator names. In such cases all operators
that have the named field will have the value injected.

In git@github.com:datatorrent/examples<mailto:git@github.com:datatorrent/examples> there
are several sample applications. For example,
the fileIO application has src/main/resources/META-INF/properties.xml which defines several
properties for operators.

You can also use annotations like @NotNull on your fields to ensure that values have been
injected; if for some reason the value was not set (e.g. typo in the property name in the
properties file), the application will fail validation and will not even launch.

Ram

On Tue, May 17, 2016 at 7:24 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <suryavamshivardhan.mukkamula@rbc.com<mailto:suryavamshivardhan.mukkamula@rbc.com>>
wrote:
Hi Ram,

How would I read the custom properties in my operators? , are these properties available to
all the operators which run on different nodes of the cluster?

Regards,
Surya Vamshi

From: Munagala Ramanath [mailto:ram@datatorrent.com<mailto:ram@datatorrent.com>]
Sent: 2016, May, 17 9:58 AM
To: users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>
Subject: Re: Information Needed

You can add multiple XML configuration files at src/site/conf in your project and select one
of them at launch time.
This is described briefly at http://docs.datatorrent.com/application_packages/ under the section
entitled
Adding pre-set configurations

Ram

On Tue, May 17, 2016 at 6:43 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <suryavamshivardhan.mukkamula@rbc.com<mailto:suryavamshivardhan.mukkamula@rbc.com>>
wrote:
Hi Ashwin,

Thank you.

I see that result set is read once and tuples are emitted tuple by tuple. I will explore these
JDBC operators.

One more requirement, I would like to read custom properties-JDBC and set my dbconnection
and query in the properties itself, do you have sample code to read custom properties from
the class path.

Regards,
Surya Vamshi

From: Ashwin Chandra Putta [mailto:ashwinchandrap@gmail.com<mailto:ashwinchandrap@gmail.com>]
Sent: 2016, May, 16 5:21 PM
To: users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>
Subject: Re: Information Needed

Surya,

You can use the jdbc input and output operators to fetch from origin database and destination
database.

Check https://github.com/apache/incubator-apex-malhar/tree/master/library/src/main/java/com/datatorrent/lib/db/jdbc

They should have configurable batch sizes. In general batching is faster than tuple by tuple.

Regards,
Ashwin.

On Mon, May 16, 2016 at 1:20 PM, Mukkamula, Suryavamshivardhan (CWM-NR) <suryavamshivardhan.mukkamula@rbc.com<mailto:suryavamshivardhan.mukkamula@rbc.com>>
wrote:
Hi Ashwin,

It is to get all the 10k rows as a single batch and process them and insert into destination
database.

Which one is suggested ,  to get single row from database or batch of 10k rows and process
and then insert into destination database?

Regards,
Surya Vamshi

From: Ashwin Chandra Putta [mailto:ashwinchandrap@gmail.com<mailto:ashwinchandrap@gmail.com>]
Sent: 2016, May, 16 1:50 PM
To: users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>
Subject: Re: Information Needed

Surya,

By single row tuple, are you looking for a way to enrich/join your tuples with a single tuple
from another table? Or simply get all rows from the origin database table as a single batch
of 10000 tuples?

Regards,
Ashwin.

On Mon, May 16, 2016 at 8:54 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <suryavamshivardhan.mukkamula@rbc.com<mailto:suryavamshivardhan.mukkamula@rbc.com>>
wrote:
Hello Team,

Use Case : Batch ingestion of transaction data (rows = 10000) from DB2 to Vertica database.

Question : How Can I make DB2 Connection in such a way that I get a single row tuple only
? Do you have any sample code which reads from one database and writes into another database
?

Regards,
Surya Vamshi


_______________________________________________________________________

This [email] may be privileged and/or confidential, and the sender does not waive any related
rights and obligations. Any distribution, use or copying of this [email] or the information
it contains by other than an intended recipient is unauthorized. If you received this [email]
in error, please advise the sender (by return [email] or otherwise) immediately. You have
consented to receive the attached electronically at the above-noted address; please retain
a copy of this confirmation for future reference.



--

Regards,
Ashwin.

_______________________________________________________________________

This [email] may be privileged and/or confidential, and the sender does not waive any related
rights and obligations. Any distribution, use or copying of this [email] or the information
it contains by other than an intended recipient is unauthorized. If you received this [email]
in error, please advise the sender (by return [email] or otherwise) immediately. You have
consented to receive the attached electronically at the above-noted address; please retain
a copy of this confirmation for future reference.



--

Regards,
Ashwin.

_______________________________________________________________________

If you received this email in error, please advise the sender (by return email or otherwise)
immediately. You have consented to receive the attached electronically at the above-noted
email address; please retain a copy of this confirmation for future reference.

Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par
retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s)
ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver
une copie de cette confirmation pour les fins de reference future.


_______________________________________________________________________

If you received this email in error, please advise the sender (by return email or otherwise)
immediately. You have consented to receive the attached electronically at the above-noted
email address; please retain a copy of this confirmation for future reference.

Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par
retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s)
ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver
une copie de cette confirmation pour les fins de reference future.


_______________________________________________________________________

If you received this email in error, please advise the sender (by return email or otherwise)
immediately. You have consented to receive the attached electronically at the above-noted
email address; please retain a copy of this confirmation for future reference.

Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par
retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s)
ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver
une copie de cette confirmation pour les fins de reference future.
_______________________________________________________________________

This [email] may be privileged and/or confidential, and the sender does not waive any related
rights and obligations. Any distribution, use or copying of this [email] or the information
it contains by other than an intended recipient is unauthorized. If you received this [email]
in error, please advise the sender (by return [email] or otherwise) immediately. You have
consented to receive the attached electronically at the above-noted address; please retain
a copy of this confirmation for future reference.
Mime
View raw message