spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Burak Yavuz (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-24525) Provide an option to limit MemorySink memory usage
Date Fri, 15 Jun 2018 20:59:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Burak Yavuz reassigned SPARK-24525:
-----------------------------------

    Assignee: Mukul Murthy

> Provide an option to limit MemorySink memory usage
> --------------------------------------------------
>
>                 Key: SPARK-24525
>                 URL: https://issues.apache.org/jira/browse/SPARK-24525
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.3.1
>            Reporter: Mukul Murthy
>            Assignee: Mukul Murthy
>            Priority: Major
>
> MemorySink stores stream results in memory and is mostly used for testing and displaying
streams, but for large streams, this can OOM the driver. We should add an option to limit
the number of rows and the total size of a memory sink and not add any new data once either
limit is hit. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message