Mailing-List: contact drill-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: drill-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of azuryyyu@gmail.com designates
 209.85.214.175 as permitted sender)
MIME-Version: 1.0
Date: Sat, 15 Sep 2012 20:30:41 +0800
Message-ID: 
 <CALr1C9p73w5hsGGNSz4N1zC=fr0MWouEr7sG_VNxMhi2OpLMrQ@mail.gmail.com>
Subject: Storage file format
From: Azuryy Yu <azuryyyu@gmail.com>
To: drill-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=bcaec517a4c066b41204c9bcb5cb

--bcaec517a4c066b41204c9bcb5cb
Content-Type: text/plain; charset=ISO-8859-1

Hi All,

I am interested in working on storage format. (sign up?)

I wrote a HDFS  file format, which is similar to Sequence file (row
storage, block management, compress), I provide InputFormat and
OutputFormat,

sometimes it get a great performance, sometimes not, depends on the data.

for Drill, we should implement a column-storage, this can skip some columns
during query, and skip some rows within one column file. but this
column-storage should based on the distributed file system, such as HDFS,
Mapr DFS, I like Mapr DFS because of HA.

we can implement the following column storage file format, I think it's
enough to us.

http://arxiv.org/pdf/1105.4252.pdf

--bcaec517a4c066b41204c9bcb5cb--