apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raja.Aravapalli <Raja.Aravapa...@target.com>
Subject Re: [EXTERNAL] Re: How to Change memory setting for a unifier?
Date Tue, 20 Jun 2017 21:13:36 GMT
Thanks a lot for the response.

Do we any web link that gives details about all the possibile configurations and properties
?



Thanks
Raja.


From: Pramod Immaneni <pramod@datatorrent.com>
Reply-To: "users@apex.apache.org" <users@apex.apache.org>
Date: Tuesday, June 20, 2017 at 3:22 PM
To: "users@apex.apache.org" <users@apex.apache.org>
Subject: [EXTERNAL] Re: How to Change memory setting for a unifier?

Hi Raja,

To set the memory for the unifier you would use the port to identify the unifier, so something
like

dt.operator.oper1.port.port1.unifier.MEMORY_MB

or use the setUnifierAttribute in DAG if setting it from java.

Thanks

On Tue, Jun 20, 2017 at 1:13 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Hi,

Can someone please help me with configuration property on how to increase memory setting for
Unifier?

I am using the below way, to increase the memory for Application Master and opertors of a
DAG:

<property>
    <name>dt.attr.MASTER_MEMORY_MB</name>
    <value>10240</value>
</property>

<property>
    <name>dt.operator.*.attr.MEMORY_MB</name>
    <value>10240</value>
</property>


UseCase/Scenario:

I am having an apex application that reads from Kafka in 3 partitions with the setting “ONE_TO_MANY”,
and sends the messages to following operators. But, the problem is, even though the messages
are being read thru multiple parititons, but I see a unifier operation happening after the
messages read in multiple kafka partitions. Which is becoming a bottle nect with the number
of increase in the number of messages being floated.

I have configured the memory all the operators to 10GB, with above given setting, but, the
unifier which has been cropped up, was taking only 2GB by default, and failing with heap space
problems.

Hence I wish to increase the memory for the unifier.

Thanks a lot in advance.


Regards,
Raja.




Mime
View raw message