apex-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sas...@apache.org
Subject incubator-apex-site git commit: Updating Apache Apex Roadmap
Date Mon, 11 Apr 2016 21:48:52 GMT
Repository: incubator-apex-site
Updated Branches:
  refs/heads/master 0e5c290ac -> fcc9d246c


Updating Apache Apex Roadmap


Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/commit/fcc9d246
Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/tree/fcc9d246
Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-site/diff/fcc9d246

Branch: refs/heads/master
Commit: fcc9d246cd31628cecac6c8f25055549f647e0f9
Parents: 0e5c290
Author: sashadt <sasha@datatorrent.com>
Authored: Mon Apr 11 14:48:51 2016 -0700
Committer: sashadt <sasha@datatorrent.com>
Committed: Mon Apr 11 14:48:51 2016 -0700

----------------------------------------------------------------------
 roadmap.json | 376 +++++++++++++++++++++++-------------------------------
 1 file changed, 161 insertions(+), 215 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/fcc9d246/roadmap.json
----------------------------------------------------------------------
diff --git a/roadmap.json b/roadmap.json
index 237caab..b4fa33e 100644
--- a/roadmap.json
+++ b/roadmap.json
@@ -5,87 +5,17 @@
     "jiras": [
       {
         "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12919181",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12919181",
-        "key": "APEXCORE-3",
-        "fields": {
-          "summary": "Ability for an operator to populate DAG at launch time",
-          "description": "Apex should have an operator API that lets the operator generate
DAG during launch time. This will mean the following\r\n\r\n- Logical DAG will have one operator.
This is the operator that will generate a DAG underneath\r\n- Physical plan will have the
DAG generated by the operator\r\n- Execution plan will mimic physical plan + container location
etc.\r\n\r\nFor example lets say we have three operators in a DAG (app) A->B->C\r\n\r\nB
during launch time generates a DAG B1->B2->B3, then the physical plan will be\r\n\r\nA->B1->B2->B3->C\r\n\r\nThis
should work irrespective of number of ports, etc. A typical flattening. The operators inside
of B (B1, B2, B3) should have properties and attributes just as any. Users should be able
to access these at run time and compile time. B itself should support properties and attributes
that B1, B2, B3 can inherit from.\r\n\r\nThis is a very critical feature as it will open up
users to plug-in their own engines and still tak
 e up complete operability support from Apex engine.",
-          "fixVersions": [
-            {
-              "self": "https://issues.apache.org/jira/rest/api/2/version/12333950",
-              "id": "12333950",
-              "name": "3.3.0",
-              "archived": false,
-              "released": false
-            }
-          ],
-          "priority": {
-            "self": "https://issues.apache.org/jira/rest/api/2/priority/2",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/critical.png",
-            "name": "Critical",
-            "id": "2"
-          },
-          "status": {
-            "self": "https://issues.apache.org/jira/rest/api/2/status/3",
-            "description": "This issue is being actively worked on at the moment by the assignee.",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/inprogress.png",
-            "name": "In Progress",
-            "id": "3",
-            "statusCategory": {
-              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/4",
-              "id": 4,
-              "key": "indeterminate",
-              "colorName": "yellow",
-              "name": "In Progress"
-            }
-          }
-        }
-      },
-      {
-        "expand": "operations,editmeta,changelog,transitions,renderedFields",
         "id": "12919188",
         "self": "https://issues.apache.org/jira/rest/api/2/issue/12919188",
         "key": "APEXCORE-10",
         "fields": {
           "summary": "Enable non-affinity of operators per node (not containers)",
           "description": "The issue happens on cloud which provides virtual cores with software
like Xen underneath. In effect if CPU intensive operators land up on same node we have a resource
bottleneck,\r\n\r\nNeed to create an attribute that does the following\r\n- Operators A &
B should not be on same node\r\n- Stram should use this attribute to try to get containers
on different node\r\n\r\nIt is understood that the user is making an explicit choice to use
NIC instead of stream local optimization",
-          "fixVersions": [],
-          "priority": {
-            "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
-            "name": "Major",
-            "id": "3"
-          },
-          "status": {
-            "self": "https://issues.apache.org/jira/rest/api/2/status/1",
-            "description": "The issue is open and ready for the assignee to start work on
it.",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/open.png",
-            "name": "Open",
-            "id": "1",
-            "statusCategory": {
-              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/2",
-              "id": 2,
-              "key": "new",
-              "colorName": "blue-gray",
-              "name": "New"
-            }
-          }
-        }
-      },
-      {
-        "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12919232",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12919232",
-        "key": "APEXCORE-60",
-        "fields": {
-          "summary": "Iterative processing support",
-          "description": "We would like to support iterative processing by introducing cycles
in the graph (known as DAG now, but no longer if we support iterative processing).\r\n\r\nInitial
idea is as follow:\r\n{noformat}\r\n     |----|\r\n     v    |\r\nA -> B -> C ->
D\r\n^         |\r\n|---------|\r\n{noformat} \r\n\r\nC has two separate backward streams
to A and B.  The input ports of A and B that C connects to will have a special attribute on
how many window IDs ahead the incoming windows should be treated as, and A and B will be responsible
for the initial data for such input ports.\r\n\r\nAnother idea is to have C advance the window
ID on its output ports and have C generate the initial data on its output ports to A and B.",
           "fixVersions": [
             {
-              "self": "https://issues.apache.org/jira/rest/api/2/version/12333950",
-              "id": "12333950",
-              "name": "3.3.0",
+              "self": "https://issues.apache.org/jira/rest/api/2/version/12334813",
+              "id": "12334813",
+              "name": "3.4.0",
               "archived": false,
               "released": false
             }
@@ -97,17 +27,17 @@
             "id": "3"
           },
           "status": {
-            "self": "https://issues.apache.org/jira/rest/api/2/status/1",
-            "description": "The issue is open and ready for the assignee to start work on
it.",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/open.png",
-            "name": "Open",
-            "id": "1",
+            "self": "https://issues.apache.org/jira/rest/api/2/status/5",
+            "description": "A resolution has been taken, and it is awaiting verification
by reporter. From here issues are either reopened, or are closed.",
+            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/resolved.png",
+            "name": "Resolved",
+            "id": "5",
             "statusCategory": {
-              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/2",
-              "id": 2,
-              "key": "new",
-              "colorName": "blue-gray",
-              "name": "New"
+              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/3",
+              "id": 3,
+              "key": "done",
+              "colorName": "green",
+              "name": "Complete"
             }
           }
         }
@@ -398,7 +328,46 @@
         "key": "APEXCORE-293",
         "fields": {
           "summary": "Add core and malhar documentation to project web site",
-          "description": null,
+          "description": "Migrate the subset of documentation that applies to Apex from http://docs.datatorrent.com/
",
+          "fixVersions": [
+            {
+              "self": "https://issues.apache.org/jira/rest/api/2/version/12334813",
+              "id": "12334813",
+              "name": "3.4.0",
+              "archived": false,
+              "released": false
+            }
+          ],
+          "priority": {
+            "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
+            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
+            "name": "Major",
+            "id": "3"
+          },
+          "status": {
+            "self": "https://issues.apache.org/jira/rest/api/2/status/6",
+            "description": "The issue is considered finished, the resolution is correct.
Issues which are closed can be reopened.",
+            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/closed.png",
+            "name": "Closed",
+            "id": "6",
+            "statusCategory": {
+              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/3",
+              "id": 3,
+              "key": "done",
+              "colorName": "green",
+              "name": "Complete"
+            }
+          }
+        }
+      },
+      {
+        "expand": "operations,editmeta,changelog,transitions,renderedFields",
+        "id": "12955374",
+        "self": "https://issues.apache.org/jira/rest/api/2/issue/12955374",
+        "key": "APEXCORE-414",
+        "fields": {
+          "summary": "Native support for event-time windowing",
+          "description": "Apex core has streaming windows that establish a boundary based
on arrival time of events. Many applications require boundaries based on the time of events,
which could be a field in the tuple. Some of the operators support this today (time bucketing),
but it would be good to provide more generic support for this in the engine itself. ",
           "fixVersions": [],
           "priority": {
             "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
@@ -424,12 +393,12 @@
       },
       {
         "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12924154",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12924154",
-        "key": "APEXCORE-295",
+        "id": "12955475",
+        "self": "https://issues.apache.org/jira/rest/api/2/issue/12955475",
+        "key": "APEXCORE-418",
         "fields": {
-          "summary": "Running a Storm topology on Apex.",
-          "description": "Flink streaming is compatible with Apache Storm interfaces and
therefore allows reusing code that was implemented for Storm.\r\nDetails can be found here.\r\nhttps://ci.apache.org/projects/flink/flink-docs-master/apis/storm_compatibility.html\r\nThis
jira item can contain tasks for providing similar support in Apex",
+          "summary": "Support for Mesos",
+          "description": "Today Apex has two modes of execution: Embedded mode (everything
running in a single JVM) and YARN. There has been a few questions around native support for
Mesos. A cursory look suggests that Mesos support can be added by reimplementing the YARN
specific portions in the master (AppMasterService, ContainerLauncher) and limited changes
to the streaming container driver.\r\n\r\nMesos has a different model of resource allocation:
The master offers resources to the framework while in YARN resources are requested. Apex master
needs to implement the \"framework scheduler\" that is responsible to accept the resources
and control the tasks.\r\n\r\nhttp://mesos.apache.org/documentation/latest/app-framework-development-guide/\r\n\r\nTasks
are launched through executors, command line and docker executors are provided.  \r\n\r\nApex
also requires support to deploy the dependencies to the nodes on which the streaming containers
are launched. YARN supports that through r
 esource localization. Mesos supports this through the fetcher, which can copy the resources
to the slave node.\r\n\r\nhttp://mesos.apache.org/documentation/latest/fetcher/\r\n",
           "fixVersions": [],
           "priority": {
             "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
@@ -472,9 +441,17 @@
         "projectId": 12318823
       },
       {
-        "self": "https://issues.apache.org/jira/rest/api/2/version/12333950",
-        "id": "12333950",
-        "name": "3.3.0",
+        "self": "https://issues.apache.org/jira/rest/api/2/version/12334814",
+        "id": "12334814",
+        "name": "3.3.1",
+        "archived": false,
+        "released": false,
+        "projectId": 12318823
+      },
+      {
+        "self": "https://issues.apache.org/jira/rest/api/2/version/12334813",
+        "id": "12334813",
+        "name": "3.4.0",
         "archived": false,
         "released": false,
         "projectId": 12318823
@@ -557,84 +534,6 @@
       },
       {
         "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12926158",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926158",
-        "key": "APEXMALHAR-1812",
-        "fields": {
-          "summary": "Support Anti Join",
-          "description": "Create a new operator to anti-join two datasets. An anti-join is
the opposite to the regular join. An anti-join of R with S will output a tuple from R if it
doesn't join with any tuple in S. This can also be useful if we support (NOT)EXISTS subqueries
later.",
-          "fixVersions": [
-            {
-              "self": "https://issues.apache.org/jira/rest/api/2/version/12334589",
-              "id": "12334589",
-              "name": "3.3.0",
-              "archived": false,
-              "released": false
-            }
-          ],
-          "priority": {
-            "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
-            "name": "Major",
-            "id": "3"
-          },
-          "status": {
-            "self": "https://issues.apache.org/jira/rest/api/2/status/5",
-            "description": "A resolution has been taken, and it is awaiting verification
by reporter. From here issues are either reopened, or are closed.",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/resolved.png",
-            "name": "Resolved",
-            "id": "5",
-            "statusCategory": {
-              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/3",
-              "id": 3,
-              "key": "done",
-              "colorName": "green",
-              "name": "Complete"
-            }
-          }
-        }
-      },
-      {
-        "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12926157",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926157",
-        "key": "APEXMALHAR-1813",
-        "fields": {
-          "summary": "Support Semi Join",
-          "description": "Create an new operator to SQL-like semi join two datasets. A semi
join of R with S will output a tuple from R if it joins with any tuple in S. This can also
be useful if we support IN subqueries later.",
-          "fixVersions": [
-            {
-              "self": "https://issues.apache.org/jira/rest/api/2/version/12334589",
-              "id": "12334589",
-              "name": "3.3.0",
-              "archived": false,
-              "released": false
-            }
-          ],
-          "priority": {
-            "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
-            "name": "Major",
-            "id": "3"
-          },
-          "status": {
-            "self": "https://issues.apache.org/jira/rest/api/2/status/5",
-            "description": "A resolution has been taken, and it is awaiting verification
by reporter. From here issues are either reopened, or are closed.",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/resolved.png",
-            "name": "Resolved",
-            "id": "5",
-            "statusCategory": {
-              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/3",
-              "id": 3,
-              "key": "done",
-              "colorName": "green",
-              "name": "Complete"
-            }
-          }
-        }
-      },
-      {
-        "expand": "operations,editmeta,changelog,transitions,renderedFields",
         "id": "12926152",
         "self": "https://issues.apache.org/jira/rest/api/2/issue/12926152",
         "key": "APEXMALHAR-1818",
@@ -703,7 +602,15 @@
         "fields": {
           "summary": "Create a fault-tolerant/scalable cache component backed by a persistent
store",
           "description": "We need a Scalable Cache component in Malhar. Following are some
of the key features of the cache\r\n\r\n1. Cache has limited size and is backed by a persistent
store.\r\n2. When there is a cache miss, the data is loaded from backup store.\r\n3. To minimize
misses, a range of keys can be loaded.\r\n4. Ability to purge key/values\r\n5. Writing to
the backup store should be optimized.\r\n6. It should provide support for fault-tolerance.",
-          "fixVersions": [],
+          "fixVersions": [
+            {
+              "self": "https://issues.apache.org/jira/rest/api/2/version/12334637",
+              "id": "12334637",
+              "name": "3.4.0",
+              "archived": false,
+              "released": false
+            }
+          ],
           "priority": {
             "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
             "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
@@ -711,34 +618,34 @@
             "id": "3"
           },
           "status": {
-            "self": "https://issues.apache.org/jira/rest/api/2/status/3",
-            "description": "This issue is being actively worked on at the moment by the assignee.",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/inprogress.png",
-            "name": "In Progress",
-            "id": "3",
+            "self": "https://issues.apache.org/jira/rest/api/2/status/5",
+            "description": "A resolution has been taken, and it is awaiting verification
by reporter. From here issues are either reopened, or are closed.",
+            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/resolved.png",
+            "name": "Resolved",
+            "id": "5",
             "statusCategory": {
-              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/4",
-              "id": 4,
-              "key": "indeterminate",
-              "colorName": "yellow",
-              "name": "In Progress"
+              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/3",
+              "id": 3,
+              "key": "done",
+              "colorName": "green",
+              "name": "Complete"
             }
           }
         }
       },
       {
         "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12926072",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926072",
-        "key": "APEXMALHAR-1904",
+        "id": "12926038",
+        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926038",
+        "key": "APEXMALHAR-1938",
         "fields": {
-          "summary": "Rewrite kafka input operator to use 0.9.0 new consumer",
-          "description": null,
+          "summary": "Operator checkpointing in distributed in-memory store",
+          "description": "Currently Apex engine provides operator checkpointing in Hdfs (
with Hdfs backed StorageAgents i.e. FSStorageAgent & AsyncFSStorageAgent )\r\nAs operator
check-pointing is critical functionality of Apex streaming platform to ensure fault tolerant
behavior, platform should also provide alternate StorageAgents which will work seamlessly
with large applications that requires Exactly once semantics.\r\nHDFS read/write latency is
limited and doesn't improve beyond certain point because of disk io & staging writes.
Having alternate strategy to this check-pointing in fault tolerant distributed in-memory grid
would ensure application stability and performance is not impacted by checkpointing\r\n\r\n*This
feature will add below functionalities*\r\n* A KeyValue store interface which is used by In-memory
checkpointing storage agent.\r\n* Abstract implementation of KeyValue storage agent which
can be configured with concrete implementation of KeyValue store for checkpo
 inting.\r\n* Concrete implementation of In memory storage agent for Apache Geode\r\n\r\n*This
feature depends on below APEX core feature* \r\nhttps://issues.apache.org/jira/browse/APEXCORE-283\r\n*
Interface for storage agent to provide application id\r\n* Stram client changes to pass applicationId",
           "fixVersions": [
             {
-              "self": "https://issues.apache.org/jira/rest/api/2/version/12334589",
-              "id": "12334589",
-              "name": "3.3.0",
+              "self": "https://issues.apache.org/jira/rest/api/2/version/12334637",
+              "id": "12334637",
+              "name": "3.4.0",
               "archived": false,
               "released": false
             }
@@ -750,6 +657,37 @@
             "id": "3"
           },
           "status": {
+            "self": "https://issues.apache.org/jira/rest/api/2/status/5",
+            "description": "A resolution has been taken, and it is awaiting verification
by reporter. From here issues are either reopened, or are closed.",
+            "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/resolved.png",
+            "name": "Resolved",
+            "id": "5",
+            "statusCategory": {
+              "self": "https://issues.apache.org/jira/rest/api/2/statuscategory/3",
+              "id": 3,
+              "key": "done",
+              "colorName": "green",
+              "name": "Complete"
+            }
+          }
+        }
+      },
+      {
+        "expand": "operations,editmeta,changelog,transitions,renderedFields",
+        "id": "12926037",
+        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926037",
+        "key": "APEXMALHAR-1939",
+        "fields": {
+          "summary": "Stream API",
+          "description": null,
+          "fixVersions": [],
+          "priority": {
+            "self": "https://issues.apache.org/jira/rest/api/2/priority/2",
+            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/critical.png",
+            "name": "Critical",
+            "id": "2"
+          },
+          "status": {
             "self": "https://issues.apache.org/jira/rest/api/2/status/1",
             "description": "The issue is open and ready for the assignee to start work on
it.",
             "iconUrl": "https://issues.apache.org/jira/images/icons/statuses/open.png",
@@ -767,12 +705,12 @@
       },
       {
         "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12926038",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926038",
-        "key": "APEXMALHAR-1938",
+        "id": "12926034",
+        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926034",
+        "key": "APEXMALHAR-1942",
         "fields": {
-          "summary": "Operator checkpointing in distributed in-memory store",
-          "description": "Currently Apex engine provides operator checkpointing in Hdfs (
with Hdfs backed StorageAgents i.e. FSStorageAgent & AsyncFSStorageAgent )\r\nAs operator
check-pointing is critical functionality of Apex streaming platform to ensure fault tolerant
behavior, platform should also provide alternate StorageAgents which will work seamlessly
with large applications that requires Exactly once semantics.\r\nHDFS read/write latency is
limited and doesn't improve beyond certain point because of disk io & staging writes.
Having alternate strategy to this check-pointing in fault tolerant distributed in-memory grid
would ensure application stability and performance is not impacted by checkpointing\r\n\r\n*This
feature will add below functionalities*\r\n* A KeyValue store interface which is used by In-memory
checkpointing storage agent.\r\n* Abstract implementation of KeyValue storage agent which
can be configured with concrete implementation of KeyValue store for checkpo
 inting.\r\n* Concrete implementation of In memory storage agent for Apache Geode\r\n\r\n*This
feature depends on below APEX core feature* \r\nhttps://issues.apache.org/jira/browse/APEXCORE-283\r\n*
Interface for storage agent to provide application id\r\n* Stram client changes to pass applicationId",
+          "summary": "Apex Operator for Apache Geode.",
+          "description": "We would like to contribute the Apache Geode(http://geode.incubator.apache.org/)
Operator support for Apex.\r\nIt will basically be implementation for writing to geode region.\r\nThis
is in continuation with the Operator checkpointing alternative under review (MLHR-1938)",
           "fixVersions": [],
           "priority": {
             "self": "https://issues.apache.org/jira/rest/api/2/priority/4",
@@ -798,18 +736,18 @@
       },
       {
         "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12926037",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926037",
-        "key": "APEXMALHAR-1939",
+        "id": "12924154",
+        "self": "https://issues.apache.org/jira/rest/api/2/issue/12924154",
+        "key": "APEXMALHAR-1999",
         "fields": {
-          "summary": "Stream API",
-          "description": null,
+          "summary": "Running a Storm topology on Apex.",
+          "description": "Flink streaming is compatible with Apache Storm interfaces and
therefore allows reusing code that was implemented for Storm.\r\nDetails can be found here.\r\nhttps://ci.apache.org/projects/flink/flink-docs-master/apis/storm_compatibility.html\r\nThis
jira item can contain tasks for providing similar support in Apex",
           "fixVersions": [],
           "priority": {
-            "self": "https://issues.apache.org/jira/rest/api/2/priority/2",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/critical.png",
-            "name": "Critical",
-            "id": "2"
+            "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
+            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
+            "name": "Major",
+            "id": "3"
           },
           "status": {
             "self": "https://issues.apache.org/jira/rest/api/2/status/1",
@@ -829,18 +767,18 @@
       },
       {
         "expand": "operations,editmeta,changelog,transitions,renderedFields",
-        "id": "12926034",
-        "self": "https://issues.apache.org/jira/rest/api/2/issue/12926034",
-        "key": "APEXMALHAR-1942",
+        "id": "12953482",
+        "self": "https://issues.apache.org/jira/rest/api/2/issue/12953482",
+        "key": "APEXMALHAR-2026",
         "fields": {
-          "summary": "Apex Operator for Apache Geode.",
-          "description": "We would like to contribute the Apache Geode(http://geode.incubator.apache.org/)
Operator support for Apex.\r\nIt will basically be implementation for writing to geode region.\r\nThis
is in continuation with the Operator checkpointing alternative under review (MLHR-1938)",
+          "summary": "Spill-able Datastructures",
+          "description": "Add libraryies for spooling datastructures to a key value store.
There are several customer use cases which require spooled data structures.\r\n\r\n1 - Some
operators like AbstractFileInputOperator have ever growing state. This is an issue because
eventually the state of the operator will grow larger than the memory allocated to the operator,
which will cause the operator to perpetually fail. However if the operator's datastructures
are spooled then the operator will never run out of memory.\r\n\r\n2 - Some users have requested
for the ability to maintain a map as well as a list of keys over which to iterate. Most key
value stores don't provide this functionality. However, with spooled datastructures this functionality
can be provided by maintaining a spooled map and an iterable set of keys.\r\n\r\n3 - Some
users have requested building graph databases within APEX. This would require implementing
a spooled graph data structure.\r\n\r\n4 - Another use case f
 or spooled data structures is database operators. Database operators need to write data to
a data base, but sometimes the database is down. In this case most of the database operators
repeatedly fail until the database comes back up. In order to avoid constant failures the
database operator need to writes data to a queue when the data base is down, then when the
database is up the operator need to take data from the queue and write it to the database.
In the case of a database failure this queue will grow larger than the total amount of memory
available to the operator, so the queue should be spooled in order to prevent the operator
from failing.\r\n\r\n5 - Any operator which needs to maintain a large data structure in memory
currently needs to have that data serialized and written out to HDFS with every checkpoint.
This is costly when the data structure is large. If the data structure is spooled, then only
the changes to the data structure are written out to HDFS instead of the ent
 ire data structure.\r\n\r\n6 - Also building an Apex Native database for aggregations requires
indices. These indices need to take the form of spooled data structures.\r\n\r\n7 - In the
future any operator which needs to maintain a data structure larger than the memory available
to it will need to spool the data structure.",
           "fixVersions": [],
           "priority": {
-            "self": "https://issues.apache.org/jira/rest/api/2/priority/4",
-            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/minor.png",
-            "name": "Minor",
-            "id": "4"
+            "self": "https://issues.apache.org/jira/rest/api/2/priority/3",
+            "iconUrl": "https://issues.apache.org/jira/images/icons/priorities/major.png",
+            "name": "Major",
+            "id": "3"
           },
           "status": {
             "self": "https://issues.apache.org/jira/rest/api/2/status/1",
@@ -872,14 +810,22 @@
         "self": "https://issues.apache.org/jira/rest/api/2/version/12334590",
         "id": "12334590",
         "name": "3.2.1",
+        "archived": true,
+        "released": false,
+        "projectId": 12318824
+      },
+      {
+        "self": "https://issues.apache.org/jira/rest/api/2/version/12334968",
+        "id": "12334968",
+        "name": "3.3.2",
         "archived": false,
         "released": false,
         "projectId": 12318824
       },
       {
-        "self": "https://issues.apache.org/jira/rest/api/2/version/12334589",
-        "id": "12334589",
-        "name": "3.3.0",
+        "self": "https://issues.apache.org/jira/rest/api/2/version/12334637",
+        "id": "12334637",
+        "name": "3.4.0",
         "archived": false,
         "released": false,
         "projectId": 12318824


Mime
View raw message