Mailing-List: contact dev-help@samza.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of
 prvs=3727635f4=criccomini@linkedin.com designates 69.28.149.81 as permitted
 sender)
From: Chris Riccomini <criccomini@linkedin.com.INVALID>
To: "dev@samza.incubator.apache.org" <dev@samza.incubator.apache.org>
Subject: Re: Samza threads issues
Thread-Topic: Samza threads issues
Thread-Index: AQHP8sS1WTnAHBBYNUSenMSiDer/SJxFqPEAgAFyp4CAADX+AA==
Date: Wed, 29 Oct 2014 17:12:55 +0000
Message-ID: <D0766D8B.35AF7%criccomini@linkedin.com>
References: 
 <CA+g=GHa9Rq5AUazmw-OuqVCGsN4arJTBhhDTr18ANzZMXJfWtw@mail.gmail.com>
 <D0750852.3595D%criccomini@linkedin.com>
 <CA+g=GHbqB4rnRUoGr8ZRKPCU5n_Vy+Tv_0BbfrH9Ye-=+7orzQ@mail.gmail.com>
In-Reply-To: 
 <CA+g=GHbqB4rnRUoGr8ZRKPCU5n_Vy+Tv_0BbfrH9Ye-=+7orzQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.4.5.141003
Content-Type: text/plain; charset="us-ascii"
Content-ID: <6351095B7F1FD64F8D451B03F9295D9D@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hey Dotan,

> should we increase the kafka topic sizes to accommodate incoming data
>during these time gaps as opposed to the parallel GC?

You'll have to experiment and see. I doubt it, though.

> Or on a broader aspect - What are the best practices to measure and set
>the right size for the kafka topics? Can anyone share his experience on
>that?

There's a lot that goes into this. Some to consider:

1. Peak bytes/sec throughput.
2. Retention policy for the topic.
3. Parallelism requirements for consumers.

At LinkedIn, we start with a default of 8, and size up as needed. The need
could be that partitions are running too hot (either on reads or writes),
that the partitions are too large on disk (retention policy), or that the
downstream consumers can't keep up because their processing is slower than
the messages/sec on the partition.

Cheers,
Chris

On 10/28/14 11:59 PM, "Dotan Patrich" <dotanp@fortscale.com> wrote:

>Thanks Chris,
>We will test our product using SerialGC to see how it behave.
>
>One concern that I have is regarding the kafka topic sizes - Assuming
>"stop-the-world" GC stops will more noticable using SerialGC should we
>increase the kafka topic sizes to accommodate incoming data during these
>time gaps as opposed to the parallel GC?
>Or on a broader aspect - What are the best practices to measure and set
>the
>right size for the kafka topics? Can anyone share his experience on that?
>
>Thanks,
>Dotan
>
>On Tue, Oct 28, 2014 at 5:53 PM, Chris Riccomini <
>criccomini@linkedin.com.invalid> wrote:
>
>> Hey Dotan,
>>
>> We run all of our jobs using SerialGC by default. For a few of our
>> higher-throughput jobs, we've had better luck with parallel GC or G1,
>>but
>> in general, serial works fine.
>>
>> Cheers,
>> Chris
>>
>> On 10/28/14 8:34 AM, "Dotan Patrich" <dotanp@fortscale.com> wrote:
>>
>> >Hi All,
>> >
>> >I encountered some issues caused by having too many threads for a user
>>on
>> >linux CentOS. Investigating this deeper, it turned out that the JVM
>>spawn
>> >over 31 threads per process for GC. Having about 18 Samza processes
>> >running
>> >on the machine we soon got near to the 1000 threads limit per user.
>> >I was thinking of running the Samza JVM with SerialGC instead of
>>parallel
>> >GC to avoid having so many threads in the environment. In addition,
>> >theoretically this seems to be better fitted for situations where we
>> >prefer
>> >throughput over latency in a single-core environments (this is roughly
>> >what
>> >we Samza tasks is assigned with).
>> >
>> >Before doing so, I would really appreciate you insights - did anyone
>> >encountered this issue before? Does changing the GC to be serial is a
>>good
>> >solution?
>> >
>> >Thanks,
>> >Dotan
>>
>>