spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SNEHASISH DUTTA <info.snehas...@gmail.com>
Subject Re: There is no space for new record
Date Tue, 13 Feb 2018 12:00:54 GMT
Hi,

Will it be possible to overcome this with some spark configuration tweak ,
since EMR has spark version available only till 2.2.1

Regards,
Snehasish

On Tue, Feb 13, 2018 at 2:00 PM, Marco Gaido <marcogaido91@gmail.com> wrote:

> You can check all the versions where the fix is available on the
> JIRA SPARK-23376. Anyway it will be available in the upcoming 2.3.0 release.
>
> Thanks.
>
> On 13 Feb 2018 9:09 a.m., "SNEHASISH DUTTA" <info.snehasish@gmail.com>
> wrote:
>
>> Hi,
>>
>> In which version of Spark will this fix  be available ?
>> The deployment is on EMR
>>
>> Regards,
>> Snehasish
>>
>> On Fri, Feb 9, 2018 at 8:51 PM, Wenchen Fan <cloud0fan@gmail.com> wrote:
>>
>>> It should be fixed by https://github.com/apache/spark/pull/20561 soon.
>>>
>>> On Fri, Feb 9, 2018 at 6:16 PM, Wenchen Fan <cloud0fan@gmail.com> wrote:
>>>
>>>> This has been reported before: http://apache-spark-de
>>>> velopers-list.1001551.n3.nabble.com/java-lang-IllegalStateEx
>>>> ception-There-is-no-space-for-new-record-tc20108.html
>>>>
>>>> I think we may have a real bug here, but we need a reproduce. Can you
>>>> provide one? thanks!
>>>>
>>>> On Fri, Feb 9, 2018 at 5:59 PM, SNEHASISH DUTTA <
>>>> info.snehasish@gmail.com> wrote:
>>>>
>>>>> Hi ,
>>>>>
>>>>> I am facing the following when running on EMR
>>>>>
>>>>> Caused by: java.lang.IllegalStateException: There is no space for new
>>>>> record
>>>>>         at org.apache.spark.util.collecti
>>>>> on.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMem
>>>>> orySorter.java:226)
>>>>>         at org.apache.spark.sql.execution
>>>>> .UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:132)
>>>>>         at org.apache.spark.sql.execution
>>>>> .UnsafeFixedWidthAggregationMap.destructAndCreateExternalSor
>>>>> ter(UnsafeFixedWidthAggregationMap.java:250)
>>>>>
>>>>> I am using pyspark 2.2 , what spark configuration should be
>>>>> changed/modified to get this resolved
>>>>>
>>>>>
>>>>> Regards,
>>>>> Snehasish
>>>>>
>>>>>
>>>>> Regards,
>>>>> Snehasish
>>>>>
>>>>> On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA <
>>>>> info.snehasish@gmail.com> wrote:
>>>>>
>>>>>> Hi ,
>>>>>>
>>>>>> I am facing the following when running on EMR
>>>>>>
>>>>>> Caused by: java.lang.IllegalStateException: There is no space for
>>>>>> new record
>>>>>>         at org.apache.spark.util.collecti
>>>>>> on.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMem
>>>>>> orySorter.java:226)
>>>>>>         at org.apache.spark.sql.execution
>>>>>> .UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:132)
>>>>>>         at org.apache.spark.sql.execution
>>>>>> .UnsafeFixedWidthAggregationMap.destructAndCreateExternalSor
>>>>>> ter(UnsafeFixedWidthAggregationMap.java:250)
>>>>>>
>>>>>> I am using spark 2.2 , what spark configuration should be
>>>>>> changed/modified to get this resolved
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Snehasish
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>

Mime
View raw message