apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Spooled Data Structures
Date Tue, 05 Apr 2016 00:40:00 GMT
Hi All,

I'll be working on implementing Spooled Data structures. The overall
proposal is the following:

Overall design proposal:

Spooled Data structures will be designed to work ontop of a key value
store. Spooled data structures would be used as a component inside of an
operator. The way the component would function is as followed:

StateFactory:

The StateFactory is an interface. The interface will have the setup,
beginWindow, endWindow and teardown callbacks which an operator would have
to call. The StateFactory is used to create handles to spooled
DataStructures. The three types of spooled data structures it can create
are Lists, Maps, and MultivaluedMaps Like this.

StateFactory myFactory = new ManagedStateStateFactoryImpl();
List mySpooledList = myFactory.createNewList(byte[] keyPrefix, long bucket);
Map mySpooledMap = myFactory.createNewMap(byte[] keyPrefix, long bucket);
Map mySpooledMultiValuesMap = myFactory.createNewMultiValuedMap(byte[]
keyPrefix, long bucket);

The StateFactory can have specific implementations for different backing
stores, and can be set as a property on operators. More complex spooled
datastructures can simply be created from spooled lists and maps.


More details are on the ticket
https://issues.apache.org/jira/browse/APEXMALHAR-2026


Thanks,
Tim

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message