Mailing-List: contact dev-help@orc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@orc.apache.org
From: Dain Sundstrom <dain@iq80.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: [DISCUSS] ORC 2.0
Date: Fri, 4 Aug 2017 10:40:56 -0700
References: <CAHfHakFUQOrQ0fqRA-CnAWYQixBrrzzQ9N00jMW92FAKccytpg@mail.gmail.com>
To: dev@orc.apache.org
In-Reply-To: <CAHfHakFUQOrQ0fqRA-CnAWYQixBrrzzQ9N00jMW92FAKccytpg@mail.gmail.com>
Message-Id: <D5AE5756-BD1B-4504-BFAB-A00A2987C328@iq80.com>
archived-at: Fri, 04 Aug 2017 17:41:03 -0000

+1 to all of the ideas


If we are cool with incompatible changes=E2=80=A6
 * Allow dictionary for VARBINARY
 * Disallow old encodings in new files (e.g., no v1)
 * Fix DATE encoding epoch
 * Rearrange stripe so index is next to footer so a single IOP can get =
all data
 * Change metastore properties so there is a logical mapping from column =
names to physical column identifiers so columns can be renamed
 * New timestamp encoding with fixed size per file.. similar to decimal
 * For compression like zstd, we may want to ship a compression =
dictionary for a stream

Stuff we could do today
 * A flag that says if CHAR or VHARCHAR contain any multi byte =
characters (isAsciiOnly)
 * Max character count for CHAR or VARCHAR (so we don=E2=80=99t need to =
check length for schema changes)
 * Max length for VARBINARY (easier to estimate memory usage)
 * Truncated MIN/MAX for VARBINARY/CHAR/VARCHAR

For the new encodings, we should pick encodings that play well with =
vectorization which is coming in Java 10 (Java 9 also has vastly =
improved auto vectorization).

-dain

> On Aug 4, 2017, at 9:29 AM, Owen O'Malley <owen.omalley@gmail.com> =
wrote:
>=20
> All,
>  We've started the process of updating the encodings for ORC. These
> changes are going to extend the format in ways that aren't forward
> compatible. (eg. The ORC 1.4 readers won't be able to read the new =
format.)
>=20
> The changes that I've heard about are:
> * Decimal encoding - this will like be separated in to two categories
>   + precision <=3D 18
>   + precision > 18
>  In both cases the precision and scale will be fixed for the entire =
file
> rather than per value.
> * a new Float/Double encoding
> * a new RLE encoding
>=20
> Are there other encodings that we should consider adding?
>=20
> We haven't made forward incompatible changes in a while. Currently the =
ORC
> Writer can write either:
> * Hive 0.11 ORC files
> * Hive 0.12 ORC files
>=20
> So I'd like to propose that we add a new ORC 2.0 file version and all =
of
> these changes need to be so tagged.
>=20
> The new ORC writers will maintain the ability to write the old =
versions of
> the files (Hive 0.11 ORC and Hive 0.12 ORC) as well as the ORC 2.0 =
files.
> The new reader will automatically read all three versions.
>=20
> Thoughts?
>=20
>  Owen