storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shaik Asifullah <s.asifull...@gmail.com>
Subject Re: Tuning Apache Storm Trident topology
Date Thu, 27 Dec 2018 01:39:09 GMT
Hi Stig,

I hope you are doing good. My topology has been working great until I
restarted my Storm cluster. There were a few nimbus issues as it was
refusing to restart. Currently I am getting Java Heap OutOfMemory issues
with my Trident Topology. Can you please help me in increasing "Assigned
Memory" to my Trident Topology ? I have posted my question here very long
back
https://stackoverflow.com/questions/50443737/increasing-assigned-memory-for-a-topology-in-storm
. Please help me in resolving this issue.

Thank you,
Shaik Asifullah

On Tue, Jul 31, 2018 at 1:57 AM Stig Rohde Døssing <srdo@apache.org> wrote:

> Also forgot to mention, in Trident the offsetCommitPeriodMs and
> maxUncommittedOffsets don't do anything. Trident spouts "commit" into
> Storm's Zookeeper rather than Kafka, so the mentioned settings don't do
> anything.
>
> 2018-07-30 19:58 GMT+02:00 Shaik Asifullah <s.asifullah7@gmail.com>:
>
>> Hi,
>> I am using Apache Storm Trident (1.2.1) for micro-batching. The main idea
>> of my topology is to modify messages from Kafka a little and push it to
>> Database. Each event of mine in Kafka is around 3 KB which is to be
>> inserted in one of the tables in DB. I want 10K inserts per second in the
>> database. Currently I am able to send around 1K messages per second with
>> the current configuration.
>>
>> .setOffsetCommitPeriodMs(1_000)
>>
>> .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST)
>>
>> .setMaxUncommittedOffsets(1000)
>>
>> .setPollTimeoutMs(2000)
>>
>> .setEmitNullTuples(false)
>>
>> .setProp(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
>>
>> .setProcessingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE)
>>
>> .setProp("fetch.min.bytes",10*1024*1024)
>>
>> .setProp("max.partition.fetch.bytes",20*1024*1024)
>>
>> .setProp("fetch.max.wait.ms",5* 100)
>>
>>
>> I am using 8 worker nodes spread over 2 machines of 4 cores each and I
>> am using tumbling window of batch size 10K. I am using batch since writes
>> to DB takes around 1-2 seconds. Can you please help me in increasing it
>> performance ?
>>
>>
>> Thanks,
>>
>> Shaik Asifullah
>>
>>
>

Mime
View raw message