Mailing-List: contact dev-help@arrow.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@arrow.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <B98C99DA64CD2F47B8E0D611C9451C1C8DB5C080@szxeml524-mbx.china.huawei.com>
References: 
 <CAFy6k10JGE5nuPHnkXp24jg5tOxw=aeFZnqG+v7bwQG6zRtVcw@mail.gmail.com>
	<339404DE-D43C-4A34-89B0-A7AD408BF09A@cloudera.com>
	<B98C99DA64CD2F47B8E0D611C9451C1C8DB5C080@szxeml524-mbx.china.huawei.com>
Date: Thu, 25 Feb 2016 17:11:53 -0800
Message-ID: 
 <CAMMH1AVwmUSbjw_ZM-OrSXWE8cbUSvFu2px-PkOWjEOnZk3X=A@mail.gmail.com>
Subject: Re: Comparing with Parquet
From: Pedro Miguel Duarte <duarte.gelvez.pedromiguel@gmail.com>
To: dev@arrow.apache.org
Content-Type: multipart/alternative; boundary=94eb2c08192a062aae052ca1fc8f

--94eb2c08192a062aae052ca1fc8f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I was wondering if someone could also elaborate in the comparison with
Tachyon (now called Alluxio)
On Feb 25, 2016 5:08 PM, "Chenliang (Liang, DataSight)" <
chenliang613@huawei.com> wrote:

> In favor of Henry Robinson's points.
>
> In addition. Arrow is suitable for exchanging data high efficiently, but
> the data size may just support TB level. Parquet can support more bigger
> data, but the performance couldn't support fast query.
>
> So for PB level data and interactively query(second level), both couldn't
> solve?
>
> Regards
> Liang
> -----=E9=82=AE=E4=BB=B6=E5=8E=9F=E4=BB=B6-----
> =E5=8F=91=E4=BB=B6=E4=BA=BA: Henry Robinson [mailto:henry@cloudera.com]
> =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B42=E6=9C=8826=E6=97=A5 =
0:20
> =E6=94=B6=E4=BB=B6=E4=BA=BA: dev@arrow.apache.org
> =E4=B8=BB=E9=A2=98: Re: Comparing with Parquet
>
> Think of Parquet as a format well-suited to writing very large datasets t=
o
> disk, whereas Arrow is a format most suited to efficient storage in memor=
y.
> You might read Parquet files from disk, and then materialize them in memo=
ry
> in Arrow's format.
>
> Both formats are designed around the idiosyncrasies of the target medium:
> Parquet is not designed to support efficient random access because disks
> aren't good at that, but Arrow has fast random access  as a core design
> principle, to give just one example.
>
> Henry
>
> > On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <
> sourav.mazumder00@gmail.com> wrote:
> >
> > Hi All,
> >
> > New to this. And still trying to figure out where exactly Arrow fits
> > in the ecosystem of various Big Data technologies.
> >
> > In that respect first thing which came to my mind is how does Arrow
> > compare with parquet.
> >
> > In my understanding Parquet also supports a very efficient columnar
> > format (with support for nested structure). It is already embraced
> > (supported) by various technologies like Impala (origin), Spark, Drill
> etc.
> >
> > The only think I see missing in Parquet is support for SIMD based
> > vectorized operations.
> >
> > Am I right or am I missing many other differences between Arrow and
> > parquet ?
> >
> > Regards,
> > Sourav
>

--94eb2c08192a062aae052ca1fc8f--