Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of dhutchis@stevens.edu
 designates 74.125.149.84 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOiJXP7pLT9OZvZsm6eT=CfqU81Uei+sUOxEwEK0_pCK2PQz7A@mail.gmail.com>
References: 
 <CAPx=JkbY5Gh4kdhcA_xXLEnF9BsuUCxRa=67tbD7i+myUXFeXQ@mail.gmail.com>
 <54300548.2070708@gmail.com>
 <CAL5zq9ZoNR+BKCdPSuV5aHCSo6T=sidhsBPCjujMKP3enFpGig@mail.gmail.com>
 <CAPx=Jkah9XS-ptNn4osX6NQttpj6hxZUDYhG+ai48MP2bNVxdA@mail.gmail.com>
 <CAOiJXP7pLT9OZvZsm6eT=CfqU81Uei+sUOxEwEK0_pCK2PQz7A@mail.gmail.com>
From: Dylan Hutchison <dhutchis@stevens.edu>
Date: Sat, 4 Oct 2014 17:52:01 -0400
Message-ID: 
 <CAPx=JkYLSXmwZrrHSCcYvEWA7onsK0Hr2gLWMeg0K6f2tWqOUw@mail.gmail.com>
Subject: Re: Determining tablets assigned to table splits, and the number of
 rows in each tablet
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=047d7b2e549c4591bd05049fdd6e

--047d7b2e549c4591bd05049fdd6e
Content-Type: text/plain; charset=UTF-8

It should suffice to list the number of entries for a table, tablet and
tablet server.  No need to worry about number of unique rows, number of
unique column families, etc.  By entry I mean number of (key,value)s.

For load balancing, we care about how much physical data is on each tablet
/ tablet server.  This is directly proportional to the number of entries,
assuming that the key size and value size in bytes do not differ too
drastically.  If they do (say for raw documents of vastly different sizes),
the best measure is the *size of the data in bytes *for each tablet /
tablet server.  I didn't suggest it because it doesn't look like Accumulo
tracks it so it would involve a lot of new implementation and book-keeping,
which could hamper performance.

Accumulo does already track the number of entries for tables, tablets and
tablet server.  It's just hard to get to, relying on the format of the
metadata table and accessing the non-public Monitor classes.  Bringing it
to the public API just looks like a matter of reworking the API and letting
the client gather the information that the Monitor already does by
connecting to each tablet server.  Does that sound reasonable?

Regards, Dylan

On Sat, Oct 4, 2014 at 4:11 PM, David Medinets <david.medinets@gmail.com>
wrote:

> Adding this functionality into Accumulo's API would reduce it's efficiency
> for users that don't need this level of tracking. Let ingest procedures
> take the performance hit. There are synchronization issues that reduce
> degrade performance. Also what would be the appropriate level of tracking -
> at the row, column-family, or every level? Whatever answer you give,
> someone else will ask for something different. And then there are the
> aggregation questions. Not to mention the additional storage requirements.
>


-- 
www.cs.stevens.edu/~dhutchis

--047d7b2e549c4591bd05049fdd6e
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">It should suffice to list the number of entries for a tabl=
e, tablet and tablet server.=C2=A0 No need to worry about number of unique =
rows, number of unique column families, etc.=C2=A0 By entry I mean number o=
f (key,value)s.<div><br></div><div>For load balancing, we care about how mu=
ch physical data is on each tablet / tablet server.=C2=A0 This is directly =
proportional to the number of entries, assuming that the key size and value=
 size in bytes do not differ too drastically.=C2=A0 If they do (say for raw=
 documents of vastly different sizes), the best measure is the <i>size of t=
he data in bytes </i>for each tablet / tablet server.=C2=A0 I didn&#39;t su=
ggest it because it doesn&#39;t look like Accumulo tracks it so it would in=
volve a lot of new implementation and book-keeping, which could hamper perf=
ormance.</div><div><br></div><div>Accumulo does already track the number of=
 entries for tables, tablets and tablet server.=C2=A0 It&#39;s just hard to=
 get to, relying on the format of the metadata table and accessing the non-=
public Monitor classes.=C2=A0 Bringing it to the public API just looks like=
 a matter of reworking the API and letting the client gather the informatio=
n that the Monitor already does by connecting to each tablet server.=C2=A0 =
Does that sound reasonable?</div><div><br></div><div>Regards, Dylan<br><div=
 class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Sat, Oct 4, 2014 a=
t 4:11 PM, David Medinets <span dir=3D"ltr">&lt;<a href=3D"mailto:david.med=
inets@gmail.com" target=3D"_blank">david.medinets@gmail.com</a>&gt;</span> =
wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord=
er-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Adding this =
functionality into Accumulo&#39;s API would reduce it&#39;s efficiency for =
users that don&#39;t need this level of tracking. Let ingest procedures tak=
e the performance hit. There are synchronization issues that reduce degrade=
 performance. Also what would be the appropriate level of tracking - at the=
 row, column-family, or every level? Whatever answer you give, someone else=
 will ask for something different. And then there are the aggregation quest=
ions. Not to mention the additional storage requirements.</div></div></bloc=
kquote></div><br><div><br></div>-- <br><div dir=3D"ltr"><div><a href=3D"htt=
p://www.cs.stevens.edu/~dhutchis" target=3D"_blank">www.cs.stevens.edu/~dhu=
tchis</a><br></div></div>
</div></div></div>

--047d7b2e549c4591bd05049fdd6e--