Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANQf6DpbgczaSGGBBXwZZau=bMPwsdbmH3_u1o1f=8xNpdFfCQ@mail.gmail.com>
References: 
 <CALgBV_e4PH7RjQmaqS1kQnOKUyYUPkVFJp_EUek3TEpbEXQORw@mail.gmail.com>
	<CANQf6DqB_RL4tB8wtw4Yhyg=hEACjNv42ZyVbb22PcLjwHzHSQ@mail.gmail.com>
	<CALgBV_eUXQOiTrGJy7=BsSh2B00Cks7y39mmEfP8Yp1DsZerzg@mail.gmail.com>
	<CANQf6Dp46MGf7rh9T3rxiG6Ot5f3uffx_7j+hgQZYOfU1RS9Aw@mail.gmail.com>
	<CALgBV_c1rQF9Ziq8Vn4x2uzP7KRMfeO4OpyzZHHYfQa1+FXY5w@mail.gmail.com>
	<CALgBV_f-L5=-Yr8pCp0Yfyeo=MvV_9wW8w7z6csAH0F4-W7weA@mail.gmail.com>
	<5B7DC174-B088-4EFB-8B5F-2138CE889E3B@apache.org>
	<CALgBV_cK_Yz2RujVrJzcQx+UO51Rrm9S5s6MCoc8jwTdCZ5uyg@mail.gmail.com>
	<1A4DF5F8-4E4E-4D09-8C56-0BFE423F9B47@apache.org>
	<CANQf6DpbgczaSGGBBXwZZau=bMPwsdbmH3_u1o1f=8xNpdFfCQ@mail.gmail.com>
Date: Wed, 20 Jan 2016 15:33:00 -0500
Message-ID: 
 <CALgBV_cdRUiLBp1mHKDj2WHMp-f17oqmqgZR3at+54KN_3xxtQ@mail.gmail.com>
Subject: Re: Asterix Schema Provider Framework
From: Wail Alkowaileet <wael.y.k@gmail.com>
To: dev@asterixdb.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11c3c5c26a76a10529c9e478

--001a11c3c5c26a76a10529c9e478
Content-Type: text/plain; charset=UTF-8

Hi Chen,

On Asterix, I think the type UNION is an Algebricks type only. The user
cannot define type UNION on their schema (except for optional which is the
equivalent of UNION(null,OTHER_TYPE).)
To represent "name" on Asterix,  the user should leave it as open. In the
storage level, "name" will be a "tagged" field and Algebricks type-computer
will infer it as type ANY.

What I'm suggesting is that if the types get inferred on ingestion, "name"
can have the type UNION(record,list[record]) instead of ANY. The question
probably is how is that useful? well .. we can help Asterix to fail at
compile time instead of failing at run-time. For example,

let's assume the following:
"x":[1, 2, 3, 4]
"x" ["hello", "world"]

The inferred type of "x" will be UNION(list[int32],list[string]) which
implies that we can apply the function count() without a problem. However,
for "name" in the previous example, count() will throw an exception.

Also, I believe knowing the schema will reduce the "code size" to handle
corner cases of the open type. For example, a bug I forgot to file:

use dataverse wosDataverse
let $c := (for $x in dataset wos
let $id := $x.id
group by $country := $x.country with $id
return {"country":$country, "id" : $id})

return count($c.id)

Throws:
Unsupported type UNION(NULL, [ null: open { id: [ ANY ] } ]) for field
access expression: function-call: asterix:field-access-by-name,
Args:[%0->$4, AString: {id}] [AlgebricksException]

I know it's a stupid query :-)) .. but I had a case which enforced me to do
that way.

As for Spark, String is their "open" type. But, with the limitation that
you cannot apply any operation such as count because it's just a string
:-)).


Thank you Chen for your feedback and engagement.

P.S @Till: I'm removing the modifications on Algebricks to have the schema
framework on Asterix.


On Mon, Jan 18, 2016 at 1:47 PM, Chen Li <chenli@gmail.com> wrote:

> Wail, Thanks for the detailed examples and documentation.  They are
> very helpful.  Just curious: for the provided example, we infer this
> "name" type as UNION of both record and a list of records.  Is it just
> a heuristic?  Is there any "principle" behind this approach, compared
> the approach by Spark's approach of inferring it as a String?
>
> Chen
>


-- 

*Regards,*
Wail Alkowaileet

--001a11c3c5c26a76a10529c9e478--