Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
Content-Type: text/plain;
	charset=us-ascii
Mime-Version: 1.0 (1.0)
Subject: Re: Why dataframe can be more efficient than dataset?
From: =?utf-8?Q?J=C3=B6rn_Franke?= <jornfranke@gmail.com>
In-Reply-To: <CAOm4EMvzshLZ5koh=KODGQ9J4ChodDY-h9oHAkJ5G7kbPSsNjw@mail.gmail.com>
Date: Sat, 8 Apr 2017 21:34:50 +0200
Cc: user@spark.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <C0C778F0-BE9D-4AB7-A035-58336AC1000C@gmail.com>
References: <CAOm4EMvzshLZ5koh=KODGQ9J4ChodDY-h9oHAkJ5G7kbPSsNjw@mail.gmail.com>
To: Shiyuan <gshy2014@gmail.com>
archived-at: Sat, 08 Apr 2017 19:34:58 -0000

As far as I am aware in newer Spark versions a DataFrame is the same as Data=
set[Row].
In fact, performance depends on so many factors, so I am not sure such a com=
parison makes sense.

> On 8. Apr 2017, at 20:15, Shiyuan <gshy2014@gmail.com> wrote:
>=20
> Hi Spark-users,=20
>     I came across a few sources which mentioned DataFrame can be more effi=
cient than Dataset.  I can understand this is true because Dataset allows fu=
nctional transformation which Catalyst cannot look into and hence cannot opt=
imize well. But can DataFrame be more efficient than Dataset even if we only=
 use the relational transformation on dataset? If so, can anyone give some e=
xplanation why  it is so? Any benchmark comparing dataset vs. dataframe?   T=
hank you!
>=20
> Shiyuan=20

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org