geode-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (GEODE-10) HDFS Integration
Date Wed, 27 Apr 2016 20:50:14 GMT


ASF subversion and git services commented on GEODE-10:

Commit 9f3f10fd2c2fc3d80d67f23e126480e853055b8b in incubator-geode's branch refs/heads/feature/GEODE-10
from [~upthewaterspout]
[;h=9f3f10f ]

GEODE-10: Reinstating HDFS persistence code

The HDFS persistence related code was removed from develop in several
pieces, first removing the API and then the underling internals.

This change reverts those commits and adds back all of the HDFS code on
this branch.

Revert "GEODE-1072: Removing HDFS related code"
This reverts commit 46535f28e4740ed9b6da87bbb27c39d0c13b3da4.
Revert "GEODE-429: Remove api for setting HdfsStore in Attributes"
This reverts commit 07d55bda1c1c9d641ca16b3b6804994ecb53bf9d.
Revert "GEODE-429: Remove HDFS persistence DataPolicy"
This reverts commit 1b4fd2fe872af1520027b8e0a84ffe84b9613f27.
Revert "GEODE-429: Remove HdfsStore parser in cache xml"
This reverts commit 12318e9cf862795e46540fdf72836fd8cbba262d.
Revert "GEODE-429: Remove hdfsStore gfsh commands"
This reverts commit 7f251978c9730c403534a62fb385e922eecc8e5b.
Revert "GEODE-429: Remove test category HoplogTests"
This reverts commit 8fb5edd349ac388fec2d5f665119f26244343703.
Revert "GEODE-429: Remove Cache.createHdfsStoreFactory method"
This reverts commit f2390a1ada2acbcabac28dd4226a67f7baf924ae.
Revert "GEODE-429: Remove HdfsStore Junit and Dunits"
This reverts commit 74c3156aaa0d29ccc4ec0b4c9a53659d2c9eb003.
Revert "GEODE-429: Remove RegionFactory.setHdfsStore"
This reverts commit 7bcc1e44cb7f0f69381c06d583b058926ca85331.
Revert "GEODE-429: Remove HDFS RegionShortcuts"
This reverts commit b3f838ea6a0b0eb150dcb92b7f6e46e5ee9db1e4.

> HDFS Integration
> ----------------
>                 Key: GEODE-10
>                 URL:
>             Project: Geode
>          Issue Type: New Feature
>          Components: hdfs
>            Reporter: Dan Smith
>            Assignee: Ashvin
>         Attachments: GEODE-HDFSPersistence-Draft-060715-2109-21516.pdf
> Ability to persist data on HDFS had been under development for GemFire. It was part of
the latest code drop, GEODE-8. As part of this feature we are proposing some changes to the
HdfsStore management API (see attached doc for details). 
> # The current API has nested configuration for compaction and async queue. This nested
structure forces user to execute multiple steps to manage a store. It also does not seem to
be consistent with other management APIs
> # Some member names in current API are confusing
> HDFS Integration: Geode as a transactional layer that microbatches data out to Hadoop.
This capability makes Geode a NoSQL store that can sit on top of Hadoop and parallelize the
process of moving data from the in memory tier into Hadoop, making it very useful for capturing
and processing fast data while making it available for Hadoop jobs relatively quickly. The
key requirements being met here are
> # Ingest data into HDFS parallely
> # Cache bloom filters and allow fast lookups of individual elements
> # Have programmable policies for deciding what stays in memory
> # Roll files in HDFS
> # Index data that is in memory
> # Have expiration policies that allows the transactional set to decay out older data
> # Solution needs to support replicated and partitioned regions

This message was sent by Atlassian JIRA

View raw message