Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 270A618A6B for ; Thu, 28 May 2015 07:07:58 +0000 (UTC) Received: (qmail 9940 invoked by uid 500); 28 May 2015 07:07:55 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 9842 invoked by uid 500); 28 May 2015 07:07:55 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 9832 invoked by uid 99); 28 May 2015 07:07:55 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 May 2015 07:07:55 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C8AD7C9548 for ; Thu, 28 May 2015 07:07:54 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.001 X-Spam-Level: **** X-Spam-Status: No, score=4.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id TKCyr3q-S5F8 for ; Thu, 28 May 2015 07:07:46 +0000 (UTC) Received: from mail-ie0-f181.google.com (mail-ie0-f181.google.com [209.85.223.181]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 8B48143E3E for ; Thu, 28 May 2015 07:07:46 +0000 (UTC) Received: by iepj10 with SMTP id j10so31520773iep.3 for ; Thu, 28 May 2015 00:07:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=2rfNgmpzGC33ihFUqSgaUkvUjZI78ZSDN2r+pMqMw4w=; b=UZhZRbu/KF6FInYruKavUynkDDs5dTnJ0empD2QSzTTBT11ljUQRYSXCxWnPIJbE03 jDuJkCHhTc9x1MYk7AfDgVyPTxThKjeqxPF1BHJUyuq64mMdKBjnRxZ8qE/wCz3zOaP3 epXCE68JWYou/Pye8kD76Ql6BM4qtZ+pFkzHl8s9YAPFOaWrIqdKLHTPkHJCuMUornDm CLtue76SWbGREQloKVdaMTBJJgwZav8OqKILtrke/PRUbH5qxkOIqxY7xMi993WOMPiK Q8awvZoZxViU4xZ4O9K4xSwEqRY2+2bRSosKo5uo1HxiTHMqJKQoVY1r+s8TwGtUlgrM AcPA== X-Gm-Message-State: ALoCoQmMo+4xv3I1bp9qGYVy5f2au9VRb1osTkXHRmhZxQK39q3zPC47fghYnRFz/lzr43rXNU0v MIME-Version: 1.0 X-Received: by 10.43.133.137 with SMTP id hy9mr7699857icc.74.1432796866051; Thu, 28 May 2015 00:07:46 -0700 (PDT) Received: by 10.107.5.76 with HTTP; Thu, 28 May 2015 00:07:45 -0700 (PDT) In-Reply-To: References: Date: Thu, 28 May 2015 12:37:45 +0530 Message-ID: Subject: Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing From: Akhil Das To: Ji ZHANG Cc: User Content-Type: multipart/alternative; boundary=20cf307f33163cbcd305171f04e8 --20cf307f33163cbcd305171f04e8 Content-Type: text/plain; charset=UTF-8 Hi Zhang, Could you paste your code in a gist? Not sure what you are doing inside the code to fill up memory. Thanks Best Regards On Thu, May 28, 2015 at 10:08 AM, Ji ZHANG wrote: > Hi, > > Yes, I'm using createStream, but the storageLevel param is by default > MEMORY_AND_DISK_SER_2. Besides, the driver's memory is also growing. I > don't think Kafka messages will be cached in driver. > > > On Thu, May 28, 2015 at 12:24 AM, Akhil Das > wrote: > >> Are you using the createStream or createDirectStream api? If its the >> former, you can try setting the StorageLevel to MEMORY_AND_DISK (it might >> slow things down though). Another way would be to try the later one. >> >> Thanks >> Best Regards >> >> On Wed, May 27, 2015 at 1:00 PM, Ji ZHANG wrote: >> >>> Hi Akhil, >>> >>> Thanks for your reply. Accoding to the Streaming tab of Web UI, the >>> Processing Time is around 400ms, and there's no Scheduling Delay, so I >>> suppose it's not the Kafka messages that eat up the off-heap memory. Or >>> maybe it is, but how to tell? >>> >>> I googled about how to check the off-heap memory usage, there's a tool >>> called pmap, but I don't know how to interprete the results. >>> >>> On Wed, May 27, 2015 at 3:08 PM, Akhil Das >>> wrote: >>> >>>> After submitting the job, if you do a ps aux | grep spark-submit then >>>> you can see all JVM params. Are you using the highlevel consumer (receiver >>>> based) for receiving data from Kafka? In that case if your throughput is >>>> high and the processing delay exceeds batch interval then you will hit this >>>> memory issues as the data will keep on receiving and is dumped to memory. >>>> You can set StorageLevel to MEMORY_AND_DISK (but it slows things down). >>>> Another alternate will be to use the lowlevel kafka consumer >>>> or to use the >>>> non-receiver based directStream >>>> >>>> that comes up with spark. >>>> >>>> Thanks >>>> Best Regards >>>> >>>> On Wed, May 27, 2015 at 11:51 AM, Ji ZHANG wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm using Spark Streaming 1.3 on CDH5.1 with yarn-cluster mode. I find >>>>> out that YARN is killing the driver and executor process because of >>>>> excessive use of memory. Here's something I tried: >>>>> >>>>> 1. Xmx is set to 512M and the GC looks fine (one ygc per 10s), so the >>>>> extra memory is not used by heap. >>>>> 2. I set the two memoryOverhead params to 1024 (default is 384), but >>>>> the memory just keeps growing and then hits the limit. >>>>> 3. This problem is not shown in low-throughput jobs, neither in >>>>> standalone mode. >>>>> 4. The test job just receives messages from Kafka, with batch interval >>>>> of 1, do some filtering and aggregation, and then print to executor logs. >>>>> So it's not some 3rd party library that causes the 'leak'. >>>>> >>>>> Spark 1.3 is built by myself, with correct hadoop versions. >>>>> >>>>> Any ideas will be appreciated. >>>>> >>>>> Thanks. >>>>> >>>>> -- >>>>> Jerry >>>>> >>>> >>>> >>> >>> >>> -- >>> Jerry >>> >> >> > > > -- > Jerry > --20cf307f33163cbcd305171f04e8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Zhang,

Could you paste your code in a gist? Not sure what you are doing inside = the code to fill up memory.

Thanks
Be= st Regards

On Thu, May 28, 2015 at 10:08 AM, Ji ZHANG <= span dir=3D"ltr"><zhangji87@gmail.com> wrote:
Hi,

Yes, I'm using createStream= , but the storageLevel param is by default MEMORY_AND_DISK_SER_2. Besides, = the driver's memory is also growing. I don't think Kafka messages w= ill be cached in driver.


On Thu, May 28, 20= 15 at 12:24 AM, Akhil Das <akhil@sigmoidanalytics.com> wrote:
A= re you using the createStream or createDirectStream api? If its the former,= you can try setting the StorageLevel to MEMORY_AND_DISK (it might slow thi= ngs down though). Another way would be to try the later one.=C2=A0

= Thanks
Best Regards

On Wed, May 27, 2015 at 1:00 PM, Ji ZHANG <zhangji87@gmail.com> wrote:
Hi Akhil,

Thanks for your reply. Acc= oding to the Streaming tab of Web UI, the Processing Time is around 400ms, = and there's no Scheduling Delay, so I suppose it's not the Kafka me= ssages that eat up the off-heap memory. Or maybe it is, but how to tell?

I googled about how to check the off-heap memory usa= ge, there's a tool called pmap, but I don't know how to interprete = the results.

On Wed, May 27, 2015 at 3:08 PM, Akhil Das <akhi= l@sigmoidanalytics.com> wrote:
After submitting the job, if you do a ps a= ux | grep spark-submit then you can see all JVM params. Are you using the h= ighlevel consumer (receiver based) for receiving data from Kafka? In that c= ase if your throughput is high and the processing delay exceeds batch inter= val then you will hit this memory issues as the data will keep on receiving= and is dumped to memory. You can set StorageLevel to MEMORY_AND_DISK (but = it slows things down). Another alternate will be to use the lowlevel ka= fka consumer or to use the non-receiver based directStream that comes up with = spark.

Thanks
Best Regards

On Wed, May 27, 2015 at 11:51 AM, Ji ZHANG <= span dir=3D"ltr"><zhangji87@gmail.com> wrote:
Hi,

I'm using Spark Streaming 1= .3 on CDH5.1 with yarn-cluster mode. I find out that YARN is killing the dr= iver and executor process because of excessive use of memory. Here's so= mething I tried:

1. Xmx is set to 512M and the GC = looks fine (one ygc per 10s), so the extra memory is not used by heap.
2. I set the two memoryOverhead params to 1024 (default is 384), but = the memory just keeps growing and then hits the limit.
3. This pr= oblem is not shown in low-throughput jobs, neither in standalone mode.
4. The test job just receives messages from Kafka, with batch interva= l of 1, do some filtering and aggregation, and then print to executor logs.= So it's not some 3rd party library that causes the 'leak'.
=

Spark 1.3 is built by myself, with correct hadoop= versions.

Any ideas will be appreciated.

Thanks.

<= /div>--
Jerry




<= font color=3D"#888888">--
Jerry




--
Jerry
--20cf307f33163cbcd305171f04e8--