geode-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Qihong Chen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (GEODE-120) RDD.saveToGemfire() can not handle big dataset (1M record per partition)
Date Wed, 15 Jul 2015 17:14:04 GMT
Qihong Chen created GEODE-120:
---------------------------------

             Summary: RDD.saveToGemfire() can not handle big dataset (1M record per partition)
                 Key: GEODE-120
                 URL: https://issues.apache.org/jira/browse/GEODE-120
             Project: Geode
          Issue Type: Sub-task
    Affects Versions: 1.0.0-incubating
            Reporter: Qihong Chen
            Assignee: Qihong Chen


the connector use single region.putAll() call to save each RDD partition. But putAll() doesn't
 handle big dataset well (such as 1M record). Need to split the dataset into smaller chunks,
and invoke putAll() for each chunk. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message