Mailing-List: contact dev-help@asterixdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.apache.org
Subject: Re: Indexing non-ADM data.
To: dev@asterixdb.apache.org
References: <CALgBV_fibR4yW9kW=x_bQx21xLewTAcH7pWD7HzXOVTr9e+uRg@mail.gmail.com>
From: Mike Carey <dtabass@gmail.com>
Message-ID: <0d7610b3-18b0-320c-cb84-698c918df51c@gmail.com>
Date: Sun, 4 Sep 2016 10:22:09 +0530
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0)
 Gecko/20100101 Thunderbird/45.2.0
MIME-Version: 1.0
In-Reply-To: <CALgBV_fibR4yW9kW=x_bQx21xLewTAcH7pWD7HzXOVTr9e+uRg@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------31C466E0B5DCB53570F33767"
archived-at: Sun, 04 Sep 2016 04:52:24 -0000

--------------31C466E0B5DCB53570F33767
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Wail,

Great inputs/requirements!  We should definitely think about how to 
address these.  One thing that could help with the second item would be 
"functional indexes" - supporting indexing on an expression rather than 
just base data - some systems (e.g., PostgreSQL) support that - not 
rocket science - and that could make data that's convertible to spatial 
data via a function call indexable spatially.  As for the first point - 
I'm not sure I "get it" - are external indexes not good enough?  Oh - 
wait - is the issue that we should offer per-object transformations 
during load?  (E.g., the ability to put a UDF on the load pipeline, like 
we do on the feed pipeline?)

Thx!

Mike


On 9/2/16 12:50 PM, Wail Alkowaileet wrote:
> Hi Dev,
>
> In the last year or so I have been more involved in AsterixDB. However, I'm
> 90% user and 10% developer (due to the nature of my work). I want to share
> some of my (and my colleagues) experience with ADM. However, I might be too
> obvious.
>
> One of the challenges we face most of the time is Indexing non-ADM data.
> Most of the data are either in JSON or CSV format which mean all ADM
> richness are not usable.
>
> For instance in load, I usually create External (or Temporary) Dataset,
> query/transform and then insert it to my Internal Dataset, which takes more
> time compared with load, as a result of flush/merge operations.
>
> Another challenging case, The TwitterFeed example
> <https://ci.apache.org/projects/asterixdb/feeds/tutorial.html>, the
> *longitude* and *latitude* fields are not indexable and I need to ETL to
> another dataset to transform (lon,lat) to a point type*.*
>
> It would be awesome if we can bridge non-ADM to ADM types.
>
>


--------------31C466E0B5DCB53570F33767--