Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (nike.apache.org: domain of josh.elser@gmail.com designates
 209.85.216.45 as permitted sender)
Message-ID: <54309DA0.5090008@gmail.com>
Date: Sat, 04 Oct 2014 21:23:44 -0400
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Postbox 3.0.11 (Windows/20140602)
MIME-Version: 1.0
To: user@accumulo.apache.org
Subject: Re: Determining tablets assigned to table splits, and the number
 of rows in each tablet
References: 
 <CAPx=JkbY5Gh4kdhcA_xXLEnF9BsuUCxRa=67tbD7i+myUXFeXQ@mail.gmail.com>
 <54300548.2070708@gmail.com>
 <CAL5zq9ZoNR+BKCdPSuV5aHCSo6T=sidhsBPCjujMKP3enFpGig@mail.gmail.com>
 <CAPx=Jkah9XS-ptNn4osX6NQttpj6hxZUDYhG+ai48MP2bNVxdA@mail.gmail.com>
 <CAOiJXP7pLT9OZvZsm6eT=CfqU81Uei+sUOxEwEK0_pCK2PQz7A@mail.gmail.com>
 <CAPx=JkYLSXmwZrrHSCcYvEWA7onsK0Hr2gLWMeg0K6f2tWqOUw@mail.gmail.com>
In-Reply-To: 
 <CAPx=JkYLSXmwZrrHSCcYvEWA7onsK0Hr2gLWMeg0K6f2tWqOUw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

I'll re-state it: I'd be happy to work with you to figure out some Java APIs for clients to consume for these kinds of metrics. A JIRA issue is the best way to encapsulate this. Would also love to help you provide a patch for it, too :)

The biggest concern (at least for creating an API for entries in a table -- by tablet/tabletserver/otherwise) is going to be that the number of entries is an approximation, not definitive. This is not prohibitive, though, as long as we're clear that it is an approximation and not an exact metric.

Dylan Hutchison wrote:
> It should suffice to list the number of entries for a table, tablet 
> and tablet server.  No need to worry about number of unique rows, 
> number of unique column families, etc.  By entry I mean number of 
> (key,value)s.
>
> For load balancing, we care about how much physical data is on each 
> tablet / tablet server.  This is directly proportional to the number 
> of entries, assuming that the key size and value size in b
ytes do not 
> differ too drastically.  If they do (say for raw documents of vastly 
> different sizes), the best measure is the /size of the data in bytes 
> /for each tablet / tablet server.  I didn't suggest it because it 
> doesn't look like Accumulo tracks it so it would involve a lot of new 
> implementation and book-keeping, which could hamper performance.
>
> Accumulo does already track the number of entries for tables, tablets 
> and tablet server.  It's just hard to get to, relying on the format of 
> the metadata table and accessing the non-public Monitor classes.  
> Bringing it to the public API just looks like a matter of reworking 
> the API and letting the client gather the information that the Monitor 
> already does by connecting to each tablet server.  Does that sound 
> reasonable?
>
> Regards, Dylan
>
> On Sat, Oct 4, 2014 at 4:11 PM, David Medinets 
> <david.medinets@gmail.com <mailto:david.medinets@gmail.com>> wrote:
>
>     Adding this functionality in
to Accumulo's API would reduce it's
>     efficiency for users that don't need this level of tracking. Let
>     ingest procedures take the performance hit. There are
>     synchronization issues that reduce degrade performance. Also what
>     would be the appropriate level of tracking - at the row,
>     column-family, or every level? Whatever answer you give, someone
>     else will ask for something different. And then there are the
>     aggregation questions. Not to mention the additional storage
>     requirements.
>
>
>
> -- 
> www.cs.stevens.edu/~dhutchis <http://www.cs.stevens.edu/~dhutchis>