Mailing-List: contact users-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@asterixdb.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <56846.141.20.24.156.1446665026.squirrel@www2.informatik.hu-berlin.de>
References: <792997416F795F469B4A87A6FEC9F9766F550879@VTTMAIL3.ad.vtt.fi>
	<563118F6.2090207@gmail.com>
	<56846.141.20.24.156.1446665026.squirrel@www2.informatik.hu-berlin.de>
Date: Wed, 4 Nov 2015 11:30:07 -0800
Message-ID: 
 <CAEZoxMcDM_GL9a=k5-98fPrDH7H8bzx=uCKNoXdFiK9PEL5C0A@mail.gmail.com>
Subject: Re: Data in AsterixDB skewing towards one node
From: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>
To: users@asterixdb.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1146ddf4b51b1a0523bc0978

--001a1146ddf4b51b1a0523bc0978
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Max,

Can you please explain this part a bit more:
"=E2=80=A6 When I load the external data it is all saved on a single node"

Are you using "external datasets" or "internal datasets, loaded from files
on HDFS".
The fact is if you are using "external datasets", then AsterixDB does not
really load any thing. It just gets the location of blocks on HDFS and
remembers them. So in this case, if there is any issue with uniform
distribution of data files, that is really related to HDFS and not
AsterixDB. But if you are 'loading' an "internal" dataset by reading
records from files on HDFS and you see issues with uniform distribution of
created on-disk components, then that is another issue and could be related
to AsterixDB.

Pouria


On Wed, Nov 4, 2015 at 11:23 AM, <schultze@informatik.hu-berlin.de> wrote:

> Hello,
>
> I have a cluster setup of AsterixDB running 4 nodes with the first being
> the master node and a node controller running on each of them. As a test =
I
> run TPC-H queries on them loading the generated TPC-H datasets from a
> hadoop distributed file system.
>
> When I load the external data it is all saved on a single node. For later
> querying that means that most of the computations are done by that single
> node which slows down the whole query (and makes the distributed
> computation idea obsolete).
>
> By now I tried to setup the system several times and interestingly enough
> two times I was able to receive a fully functional system. Unfortunatly I
> currently cannot reproduce a functional system state and whenever I try t=
o
> do a new setup I get the data skewing towards one node.
>
> Has that ever happened before? Do you know the reason for this or how to
> fix that?
>
> Regards, Max
>
>

--001a1146ddf4b51b1a0523bc0978
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Max,<div><br></div><div>Can you please explain this par=
t a bit more:</div><div>&quot;=E2=80=A6=C2=A0<span style=3D"font-size:13px"=
>When I load the external data it is all saved on a single node&quot;</span=
></div><div><span style=3D"font-size:13px"><br></span></div><div><span styl=
e=3D"font-size:13px">Are you using &quot;external datasets&quot; or &quot;i=
nternal datasets, loaded from files on HDFS&quot;.</span></div><div>The fac=
t is if you are using &quot;external datasets&quot;, then AsterixDB does no=
t really load any thing. It just gets the location of blocks on HDFS and re=
members them. So in this case, if there is any issue with uniform distribut=
ion of data files, that is really related to HDFS and not AsterixDB. But if=
 you are &#39;loading&#39; an &quot;internal&quot; dataset by reading recor=
ds from files on HDFS and you see issues with uniform distribution of creat=
ed on-disk components, then that is another issue and could be related to A=
sterixDB.</div><div><br></div><div>Pouria=C2=A0</div><div><span style=3D"fo=
nt-size:13px"><br></span></div><div><span style=3D"font-size:13px"><br></sp=
an></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On=
 Wed, Nov 4, 2015 at 11:23 AM,  <span dir=3D"ltr">&lt;<a href=3D"mailto:sch=
ultze@informatik.hu-berlin.de" target=3D"_blank">schultze@informatik.hu-ber=
lin.de</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"=
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I have a cluster setup of AsterixDB running 4 nodes with the first being<br=
>
the master node and a node controller running on each of them. As a test I<=
br>
run TPC-H queries on them loading the generated TPC-H datasets from a<br>
hadoop distributed file system.<br>
<br>
When I load the external data it is all saved on a single node. For later<b=
r>
querying that means that most of the computations are done by that single<b=
r>
node which slows down the whole query (and makes the distributed<br>
computation idea obsolete).<br>
<br>
By now I tried to setup the system several times and interestingly enough<b=
r>
two times I was able to receive a fully functional system. Unfortunatly I<b=
r>
currently cannot reproduce a functional system state and whenever I try to<=
br>
do a new setup I get the data skewing towards one node.<br>
<br>
Has that ever happened before? Do you know the reason for this or how to<br=
>
fix that?<br>
<br>
Regards, Max<br>
<br>
</blockquote></div><br></div>

--001a1146ddf4b51b1a0523bc0978--