From dev-return-24002-archive-asf-public=cust-asf.ponee.io@spark.apache.org Fri Feb 9 08:57:08 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 4C718180654 for ; Fri, 9 Feb 2018 08:57:08 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 3C4FB160C4C; Fri, 9 Feb 2018 07:57:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7F932160C2E for ; Fri, 9 Feb 2018 08:57:07 +0100 (CET) Received: (qmail 53404 invoked by uid 500); 9 Feb 2018 07:57:05 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 53391 invoked by uid 99); 9 Feb 2018 07:57:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Feb 2018 07:57:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 95A81180164 for ; Fri, 9 Feb 2018 07:57:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.898 X-Spam-Level: * X-Spam-Status: No, score=1.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id GvqXPEy1eihg for ; Fri, 9 Feb 2018 07:57:03 +0000 (UTC) Received: from mail-io0-f176.google.com (mail-io0-f176.google.com [209.85.223.176]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 653395F5B3 for ; Fri, 9 Feb 2018 07:57:02 +0000 (UTC) Received: by mail-io0-f176.google.com with SMTP id n7so8792441iob.0 for ; Thu, 08 Feb 2018 23:57:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=EFTw3CdYVRPQ6USwRDyWIW+cEbb0OdYJzXbYzyXRjJ8=; b=EXuWz0yiRZqbyp8CnQc1furmNXn2SKZ9ripoRKMp4RVzRB3muKJ3oM+XtvGsFN/HP7 CcG9KbAwAuJzuR25B2eumISNiZGiICWYtW2944lkYDQ4dE834LpYee+FuBVij+ldXFmp Gc45mBW+7DZO9Jd5V0fM+zkatYuK/0s1RVw9skdoDEO5fbTVr69CGXhpMU7RkxVgLcbb +d/tvJ5TqbPtWLs5wwNPurzXaG0Q5rWhskwaZ7vYF69QHhRvar6rccH6l602rgdMOjvL nMTfNpD/06GoD0kbJh9j2vFMSCBgi2YBeZ7asm9D1CsK9GNOhzcc1Y2qhCkCsY5XIpdZ 3GVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=EFTw3CdYVRPQ6USwRDyWIW+cEbb0OdYJzXbYzyXRjJ8=; b=YDnaInps72hhfjRJhyAD+PkJBPzQTqNCUZTciaNAnlwn2wCUdWYOnvhbnrhX6YwmDx ieW8c1ePI7S6BO5CmJQXY7tLqv1gXS1v4lOFIYW9JrFWXws9fvCA1sIasUhYsK+SWFVF HixCGJLIh4Wke+NMROkFpBUbAWD98GyqyQObmfuQSsrzQ9KTZifPe1gV80lTPYVisS+4 71SxNewD+Pjx8ogOAMlTbEC0crJM+CSBSejJqy1aArj97aOE4roo8XPme9+nhNa/3q9c yqFMNQIZ0BEhu6aXUPgLwTzzJtL8ao7uabnVIAzJ95+Eo88OX4brBn4jSoC7YP9HN20a Jp+Q== X-Gm-Message-State: APf1xPCgG3d0XmMVGMYbwjbZXR8uIqbH5Q4Qqlm9zIG2NmhhwYdtQNmZ UXVaYY4bLcG0alU+R1IFQ3i7mhRHq3HvrRIHQOawKw== X-Google-Smtp-Source: AH8x226Gu72fpJ0/KnS4f/aZ4MFRZxWcmfJ92PwnG7WIRvDuKOo1eYh/NtT7iYJwHmpVDW9gLFS/pD6hmk3mt38V6Cg= X-Received: by 10.107.52.16 with SMTP id b16mr2187332ioa.103.1518163014523; Thu, 08 Feb 2018 23:56:54 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.206.13 with HTTP; Thu, 8 Feb 2018 23:56:54 -0800 (PST) From: SNEHASISH DUTTA Date: Fri, 9 Feb 2018 13:26:54 +0530 Message-ID: Subject: There is no space for new record To: dev@spark.apache.org Content-Type: multipart/alternative; boundary="001a1144180e311f900564c2dff5" --001a1144180e311f900564c2dff5 Content-Type: text/plain; charset="UTF-8" Hi , I am facing the following when running on EMR Caused by: java.lang.IllegalStateException: There is no space for new record at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.java:226) at org.apache.spark.sql.execution.UnsafeKVExternalSorter.(UnsafeKVExternalSorter.java:132) at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:250) I am using spark 2.2 , what spark configuration should be changed/modified to get this resolved Regards, Snehasish --001a1144180e311f900564c2dff5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi ,

I am facing the followi= ng when running on EMR

Caused by: java.lang.I= llegalStateException: There is no space for new record
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 at org.apache.spark.util.collection.unsafe.sort.UnsafeInM= emorySorter.insertRecord(UnsafeInMemorySorter.java:226)
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 at org.apache.spark.sql.execution.UnsafeKVExternalSort= er.<init>(UnsafeKVExternalSorter.java:132)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 at org.apache.spark.sql.execution.UnsafeFixedWidthAggregation= Map.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:250= )

I am using spark 2.2 , what spark configur= ation should be changed/modified to get this resolved=C2=A0

<= /div>
Regards,
Snehasish
--001a1144180e311f900564c2dff5--