drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Oshinsky <doshin...@commvault.com>
Subject RE: [DISCUSS] Remove required type
Date Tue, 22 Mar 2016 14:31:14 GMT
Hi Jacques,
Marginally related to this, I made a small change in PR-372 (DRILL-4184) to support variable
widths for decimal quantities in Parquet.  I found the (decimal) vectoring code to be very
difficult to understand (probably because it's overly complex, but also because I'm new to
Drill code in general), so I made a small, surgical change in my pull request to support keeping
track of variable widths (lengths) and null booleans within the existing fixed width decimal
vectoring scheme.  Can my changes be reviewed/accepted, and then we discuss how to fix properly

Dave Oshinsky

-----Original Message-----
From: Jacques Nadeau [mailto:jacques@dremio.com] 
Sent: Monday, March 21, 2016 11:43 PM
To: dev
Subject: Re: [DISCUSS] Remove required type

Definitely in support of this. The required type is a huge maintenance and code complexity
nightmare that provides little to no benefit. As you point out, we can do better performance
optimizations though null count observation since most sources are nullable anyway.
On Mar 21, 2016 7:41 PM, "Steven Phillips" <steven@dremio.com> wrote:

> I have been thinking about this for a while now, and I feel it would 
> be a good idea to remove the Required vector types from Drill, and 
> only use the Nullable version of vectors. I think this will greatly simplify the code.
> It will also simplify the creation of UDFs. As is, if a function has 
> custom null handling (i.e. INTERNAL), the function has to be 
> separately implemented for each permutation of nullability of the 
> inputs. But if drill data types are always nullable, this wouldn't be a problem.
> I don't think there would be much impact on performance. In practice, 
> I think the required type is used very rarely. And there are other 
> ways we can optimize for when a column is known to have no nulls.
> Thoughts?

***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
View raw message