Mailing-List: contact drill-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: drill-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of amir.youssefi@gmail.com
 designates 209.85.160.47 as permitted sender)
References: 
 <CACzgUQiY6iMQW6MO9GnK1x5oNXxLxc7tN-d9aeCFpUe-U72b7A@mail.gmail.com>
 <CAMHgjMpP4udZj=EWRTukbJrkb8xJtwF1qm7m-iO_S6==HjgAmw@mail.gmail.com>
 <CACzgUQge_bFyRWUXypQK_VAY=BC1V=9qjVUMfH3KufSzgnm6VA@mail.gmail.com>
In-Reply-To: 
 <CACzgUQge_bFyRWUXypQK_VAY=BC1V=9qjVUMfH3KufSzgnm6VA@mail.gmail.com>
Mime-Version: 1.0 (1.0)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <4DD0EF35-E83C-4520-BAF3-1A29E8F76D2A@gmail.com>
Cc: "drill-dev@incubator.apache.org" <drill-dev@incubator.apache.org>
From: Amir Youssefi <amir.youssefi@gmail.com>
Subject: Re: Drill native format
Date: Fri, 14 Sep 2012 14:15:28 -0700
To: "drill-dev@incubator.apache.org" <drill-dev@incubator.apache.org>

"Nested data is not yet implemented" in BigQuery (if I recall exact words co=
rrectly). Quoting speaker at the BigQuery presentation at Google Technology U=
ser Group last week in Googleplex (intentionally not citing speaker's name).=


-ay

On Sep 14, 2012, at 1:28 PM, David Gruzman <david@bigdatacraft.com> wrote:

> I assume that evolution of BigQuery reflects resolution of Dremel... If
> somebody have information on it it would be great.
> Storage system should understand that all file comprising the horizontal
> partition of the table are one logical entity, and should store them
> together / in some proximity. I agree that PAX will be much more
> convinient. The question is - is there performance penalty of PAX vs file
> per column?
> David
>=20
> On Fri, Sep 14, 2012 at 11:21 PM, Tomer Shiran <tshiran@maprtech.com> wrot=
e:
>=20
>> Is there any public information suggesting that Google moved away from
>> supporting nested data? Clearly BigQuery doesn't yet allow nested data, b=
ut
>> not sure that applies to Dremel.
>>=20
>> There are challenges with one file per column. How do you ensure that a
>> single record is located on a single machine to avoid costly record
>> reconstruction?
>>=20
>> On Fri, Sep 14, 2012 at 1:05 PM, David Gruzman <david@bigdatacraft.com
>>> wrote:
>>=20
>>> Hi All,
>>> I would like to discuss the question of what will be native format for
>>> drill. Original Google dremel paper defined their hierarchical columnar
>>> data format. Since then
>>> google shifted from hierarchical data format... So it is a question if i=
t
>>> makes sense to stick with it?
>>> If we are also moving to simple flat format we need our own format we
>> have
>>> to support "native". In case of Drill I would define that native support=

>> as
>>> "high performance".
>>> I think we can go to some kind of PAX format with comprehensive metadata=

>> in
>>> the header, so each file is completely self contained and can be
>> understood
>>> and processed without any external data.
>>> Alternative is to have single file per column. As far as I remember from=

>>> our OpenDremel work the main decision point is - if we can read one
>> column
>>> from the  file without loading into node memory unnecessary data from
>> other
>>> columns.
>>> With best regards,
>>> David
>>>=20
>>=20
>>=20
>>=20
>> --
>> Tomer Shiran
>> Director of Product Management | MapR Technologies | 650-804-8657
>>=20