Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.incubator.apache.org
MIME-Version: 1.0
Date: Sun, 14 Feb 2016 12:47:21 +0530
Message-ID: 
 <CAEfL3KkaUz_38YTz8wF4C-HprS7AiCoxBgXCx13NwyiLQZmxVg@mail.gmail.com>
Subject: external data set support
From: Sandeep Joshi <sanjos100@gmail.com>
To: dev@asterixdb.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1143ff70f9b12e052bb5b0d9

--001a1143ff70f9b12e052bb5b0d9
Content-Type: text/plain; charset=UTF-8

Can someone describe the level of support for External data sets and the
future roadmap ?

Let me divide the question into four broad issues:

1) Schema catalog : One would have implement IMetadataProvider,
IDataSource, IDataSourceIndex and other related classes.  Is there any
functionality missing from the current schema implementation for external
data sets ?

One of the papers says that one should add comparators and hash functions
for any new data types introduced by the external data set.  Which
interface does one have to implement for that ?

2) Query optimization : There is no cost-based optimizer yet within
Algebricks, therefore there is no API to support retrieval and use of table
statistics from an external data source.

Is something planned in this regard ?

3) Data fetch and update : The VLDB'14 paper states that external data sets
are read-only, static and without indices, but the current codebase has
support for IExternalIndex and IIndexibleExternalDataSource, so presumably
I can fetch records from an external data source (base table scan as well
as index).

Can I write to an external data source ?

4) Hyracks runtime : For data retrieval, is it sufficient to implement the
interfaces within asterix.external.api or does one also have to add some
Hyracks operators which are constructed via contributeRuntimeOperator ?

-Sandeep

--001a1143ff70f9b12e052bb5b0d9--