Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANMVRkkbizo1A19QLr9JRn=iF=fJOraK7Yq68iVA+J3muf3+1g@mail.gmail.com>
References: 
 <CANMVRkkMeMjE+=LJx0k=7pjcME5oY3MFBJMF8NkhG8LTJbiJVg@mail.gmail.com>
	<CALDYq2aU9VYWgerz=Q3tgYtwejci4UE-GtJJg=QbDKZEmVtAbg@mail.gmail.com>
	<CAD0KA=Nm2Kcb=Gvhh+8=iaQAGeu6Jx58M3tqwD2uEHLyVoniZA@mail.gmail.com>
	<CAHekGF-az4j1LZqAny6HbYTiXeJknNicm4Q0bOZFETm2veaDSA@mail.gmail.com>
	<CAHekGF8+uFQrd8b5sNjrq4yNvt_H3+Jbt3CF4JSA+7vYSd_vGg@mail.gmail.com>
	<CANMVRkkbizo1A19QLr9JRn=iF=fJOraK7Yq68iVA+J3muf3+1g@mail.gmail.com>
Date: Mon, 14 Mar 2016 13:57:53 +0530
Message-ID: 
 <CAJrpbrV0ts_-fCR7o4XbYibZ-2jXzuY-rrQQb5XuBsbBtLqq3A@mail.gmail.com>
Subject: Re: Adding ParquetReaderOperator in Malhar
From: Mohit Jotwani <mohit@datatorrent.com>
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11c1bcbc9ab2f3052dfe0e54

--001a11c1bcbc9ab2f3052dfe0e54
Content-Type: text/plain; charset=UTF-8

+1

Regards,
Mohit

On Mon, Mar 14, 2016 at 12:31 PM, Shubham Pathak <shubham@datatorrent.com>
wrote:

> @Tushar,
>
> A parquet file looks like this:
>
> 4-byte magic number "PAR1"
> <Column 1 Chunk 1 + Column Metadata>
> <Column 2 Chunk 1 + Column Metadata>
> ...
> <Column N Chunk 1 + Column Metadata>
> <Column 1 Chunk 2 + Column Metadata>
> <Column 2 Chunk 2 + Column Metadata>
> ...
> <Column N Chunk 2 + Column Metadata>
> ...
> <Column 1 Chunk M + Column Metadata>
> <Column 2 Chunk M + Column Metadata>
> ...
> <Column N Chunk M + Column Metadata>
> File Metadata
> 4-byte length in bytes of file metadata
> 4-byte magic number "PAR1"
>
> Praquet being a binary columnar storage format,  readers are expected
> to first read the file metadata to find all the column chunks they are
> interested in. The columns chunks should then be read sequentially.
>
>
>
> On Mon, Mar 14, 2016 at 11:44 AM, Yogi Devendra <yogidevendra@apache.org>
> wrote:
>
> > +1 for Parquet reader.
> >
> > ~ Yogi
> >
> > On 14 March 2016 at 11:41, Yogi Devendra <yogidevendra@apache.org>
> wrote:
> >
> > > Shubham,
> > >
> > > I feel that instead of having an operator; it should be a plugin to the
> > > input operator.
> > >
> > > So that, if someone has some other input operator for a particular file
> > > system (extending AbstractFileInputOperator) he should be able to read
> > > Parquet file from that file system using this plugin.
> > >
> > > ~ Yogi
> > >
> > > On 14 March 2016 at 11:31, Tushar Gosavi <tushar@datatorrent.com>
> wrote:
> > >
> > >> +1
> > >>
> > >> Does Parquet support partitioned read from a single file? If yes then
> > may
> > >> be we can also add support in FileSplitterInput and BlockReader to
> read
> > >> single file parallely.
> > >>
> > >> - Tushar.
> > >>
> > >>
> > >>
> > >> On Mon, Mar 14, 2016 at 11:23 AM, Devendra Tagare <
> > >> devendrat@datatorrent.com
> > >> > wrote:
> > >>
> > >> > + 1
> > >> >
> > >> > ~Dev
> > >> >
> > >> > On Mon, Mar 14, 2016 at 11:12 AM, Shubham Pathak <
> > >> shubham@datatorrent.com>
> > >> > wrote:
> > >> >
> > >> > > Hello Community,
> > >> > >
> > >> > > I am working on developing a ParquetReaderOperator which will
> allow
> > >> apex
> > >> > > users to read parquet files.
> > >> > >
> > >> > > Apache Parquet is a columnar storage format available to any
> project
> > >> in
> > >> > the
> > >> > > Hadoop ecosystem, regardless of the choice of data processing
> > >> framework,
> > >> > > data model or programming language.
> > >> > > For more information : Apache Parquet
> > >> > > <https://parquet.apache.org/documentation/latest/>
> > >> > >
> > >> > > Proposed design :
> > >> > >
> > >> > >    1. Develop  AbstractParquetFileReaderOperator that extends
> > >> > >    from AbstractFileInputOperator.
> > >> > >    2. Override openFile() method to instantiate a ParquetReader (
> > >> reader
> > >> > >    provided by parquet-mr <https://github.com/Parquet/parquet-mr>
> > >> > project
> > >> > >    that reads parquet records from a file ) with GroupReadSupport
> (
> > >> > records
> > >> > >    would be read as Group ) .
> > >> > >    3. Override  readEntity() method to read the records and call
> > >> > >    convertGroup() method.  Derived classes to override
> > convertGroup()
> > >> > > method
> > >> > >    to convert Group to any form required by downstream operators.
> > >> > >    4. Provide a concrete implementation, ParquetFilePOJOReader
> > >> operator
> > >> > >    that extends from AbstractParquetFileReaderOperator and
> > >> > >    overrides convertGroup() method to convert a given Group to
> POJO.
> > >> > >
> > >> > > Parquet schema and directory path would be inputs to the base
> > >> operator.
> > >> > For
> > >> > > ParquetFilePOJOReader, pojo class would also be required.
> > >> > >
> > >> > > Please feel free to let me know your thoughts on this.
> > >> > >
> > >> > > Thanks,
> > >> > > Shubham
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

--001a11c1bcbc9ab2f3052dfe0e54--