Mailing-List: contact issues-help@geode.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@geode.incubator.apache.org
Date: Tue, 19 Jan 2016 19:20:40 +0000 (UTC)
From: "Dan Smith (JIRA)" <jira@apache.org>
To: issues@geode.incubator.apache.org
Message-ID: <JIRA.12828102.1431028579000.150534.1453231240008@Atlassian.JIRA>
In-Reply-To: <JIRA.12828102.1431028579000@Atlassian.JIRA>
References: <JIRA.12828102.1431028579000@Atlassian.JIRA>
 <JIRA.12828102.1431028579146@arcas>
Subject: [jira] [Commented] (GEODE-10) HDFS Integration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/GEODE-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107240#comment-15107240 ] 

Dan Smith commented on GEODE-10:
--------------------------------

The HDFS feature was disabled and removed from the public API with GEODE-429, because it was still unstable. 

It doesn't look to me like a feature branch was ever created to restore the public API for HDFS and fix the stability issues. I think the next step on this issue is probably to create that branch, revert the commits listed on GEODE-409 and fix the stability issues.

Otherwise, we should rip out the HDFS code if we are not planning on getting this working with the current design.

> HDFS Integration
> ----------------
>
>                 Key: GEODE-10
>                 URL: https://issues.apache.org/jira/browse/GEODE-10
>             Project: Geode
>          Issue Type: New Feature
>          Components: hdfs
>            Reporter: Dan Smith
>            Assignee: Ashvin
>             Fix For: 1.0.0-incubating.M1
>
>         Attachments: GEODE-HDFSPersistence-Draft-060715-2109-21516.pdf
>
>
> Ability to persist data on HDFS had been under development for GemFire. It was part of the latest code drop, GEODE-8. As part of this feature we are proposing some changes to the HdfsStore management API (see attached doc for details). 
> # The current API has nested configuration for compaction and async queue. This nested structure forces user to execute multiple steps to manage a store. It also does not seem to be consistent with other management APIs
> # Some member names in current API are confusing
> HDFS Integration: Geode as a transactional layer that microbatches data out to Hadoop. This capability makes Geode a NoSQL store that can sit on top of Hadoop and parallelize the process of moving data from the in memory tier into Hadoop, making it very useful for capturing and processing fast data while making it available for Hadoop jobs relatively quickly. The key requirements being met here are
> # Ingest data into HDFS parallely
> # Cache bloom filters and allow fast lookups of individual elements
> # Have programmable policies for deciding what stays in memory
> # Roll files in HDFS
> # Index data that is in memory
> # Have expiration policies that allows the transactional set to decay out older data
> # Solution needs to support replicated and partitioned regions


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)