geode-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GEODE-120) RDD.saveToGemfire() can not handle big dataset (1M entries per partition)
Date Mon, 27 Jul 2015 21:43:06 GMT

    [ https://issues.apache.org/jira/browse/GEODE-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14643459#comment-14643459
] 

ASF subversion and git services commented on GEODE-120:
-------------------------------------------------------

Commit 70448c5dffeb29ae285720b904cfc04f2ef377ec in incubator-geode's branch refs/heads/develop
from [~qihong]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=70448c5 ]

GEODE-120 Add batch size to RDD.saveToGemfire()


> RDD.saveToGemfire() can not handle big dataset (1M entries per partition)
> -------------------------------------------------------------------------
>
>                 Key: GEODE-120
>                 URL: https://issues.apache.org/jira/browse/GEODE-120
>             Project: Geode
>          Issue Type: Sub-task
>          Components: core, extensions
>    Affects Versions: 1.0.0-incubating
>            Reporter: Qihong Chen
>            Assignee: Qihong Chen
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> the connector use single region.putAll() call to save each RDD partition. But putAll()
doesn't  handle big dataset well (such as 1M record). Need to split the dataset into smaller
chunks, and invoke putAll() for each chunk. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message