Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: 
 <CAF+SsitX5JfAZYe_j2JZY7O7a5fBYSdBruTn1-UWK5R11o5d6A@mail.gmail.com>
References: 
 <CAF+SsitX5JfAZYe_j2JZY7O7a5fBYSdBruTn1-UWK5R11o5d6A@mail.gmail.com>
From: Michael Armbrust <michael@databricks.com>
Date: Thu, 8 Oct 2015 10:19:27 -0700
Message-ID: 
 <CAAswR-41B0q3EgrK+96p5nJxT3hvWuiuwxArBXon1N+vhTO+Ng@mail.gmail.com>
Subject: Re: Default size of a datatype in SparkSQL
To: vivek bhaskar <vivekwiz3@gmail.com>
Cc: user <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a113436fed9cc7005219b1117

--001a113436fed9cc7005219b1117
Content-Type: text/plain; charset=UTF-8

Its purely for estimation, when guessing when its safe to do a broadcast
join.  We picked a random number that we thought was larger than the common
case (its better to over estimate to avoid OOM).

On Wed, Oct 7, 2015 at 10:11 PM, vivek bhaskar <vivekwiz3@gmail.com> wrote:

> I want to understand whats use of default size for a given datatype?
>
> Following link mention that its for internal size estimation.
>
> https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/DataType.html
>
> Above behavior is also reflected in code where default value seems to be
> used for stats purpose only.
>
> But then we have default size of String datatype as 4096; why we went for
> this random number? Or will it also restrict size of data? Any further
> elaboration on how string datatype works will also help.
>
> Regards,
> Vivek
>
>
>

--001a113436fed9cc7005219b1117
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Its purely for estimation, when guessing when its safe to =
do a broadcast join.=C2=A0 We picked a random number that we thought was la=
rger than the common case (its better to over estimate to avoid OOM).</div>=
<div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Oct 7, 20=
15 at 10:11 PM, vivek bhaskar <span dir=3D"ltr">&lt;<a href=3D"mailto:vivek=
wiz3@gmail.com" target=3D"_blank">vivekwiz3@gmail.com</a>&gt;</span> wrote:=
<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-lef=
t:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">I want to understand wh=
ats use of default size for a given datatype?<div><br></div><div>Following =
link mention that its for internal size estimation.</div><div><a href=3D"ht=
tps://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/Data=
Type.html" target=3D"_blank">https://spark.apache.org/docs/latest/api/java/=
org/apache/spark/sql/types/DataType.html</a><br></div><div><br></div><div>A=
bove behavior is also reflected in code where default value seems to be use=
d for stats purpose only.=C2=A0</div><div><br></div><div>But then we have d=
efault size of String datatype as 4096; why we went for this random number?=
 Or will it also restrict size of data? Any further elaboration on how stri=
ng datatype works will also help.</div><div><br></div><div>Regards,</div><d=
iv>Vivek</div><div><br></div><div><br></div></div>
</blockquote></div><br></div>

--001a113436fed9cc7005219b1117--