Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (nike.apache.org: domain of roshanp@gmail.com designates
 209.85.216.53 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: Accumulo Utilities
From: roshanp@gmail.com
In-Reply-To: 
 <CAGUtCHpnLsou_AyYdAzSLh40S-n5JZ+Ex0EBe28QvuKPnAi7=A@mail.gmail.com>
Date: Thu, 28 Mar 2013 12:15:41 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <51613010-FD8C-4175-9C20-7EAF60310B63@gmail.com>
References: <5B0ECD26-9346-4463-9EB4-E2795D8839EF@gmail.com>
 <CAGUtCHpnLsou_AyYdAzSLh40S-n5JZ+Ex0EBe28QvuKPnAi7=A@mail.gmail.com>
To: user@accumulo.apache.org

Thanks! I like the idea of sending my own thread pool to the batch =
scanner, that would definitely be the better solution.

Yeah I thought about creating a batch scanner with only one thread, but =
I was not sure if that is making a separate thread (outside of the =
current one) or using the current one. At the time I did not want a new =
thread to be created at all. Though, didn't realize the Scanner was also =
spinning up a thread at all, thought that was in process.=20

To mitigate the separate RPC call per range, would it make more sense to =
do a "binRanges" based on the ranges at the tablets to reduce the number =
of ranges?

On Mar 28, 2013, at 11:55 AM, Keith Turner <keith@deenlo.com> wrote:

> I took a quick look at the code. Excluding the threading issue, a
> major conceptual difference is that BatchScannerWithScanners seems to
> do a RPC round trip for each range.   The TabletServerBatchReader
> sends all of the ranges that a tablet server needs to lookup in one
> RPC.
>=20
> Instead of creating a BatchScannerWithScanners, maybe you could create
> a batch scanner with just one thread when resources are exceeded?
> This will be similar to what you are doing now, just one thread will
> be doing work fetching data.  The client thread would just be waiting
> on this background thread.   Although this does allow the processing
> of result to happen concurrently with fetching of data.  Using
> BatchScannerWithScanners would not allow this.
>=20
> Something to be aware of, the regular scanner will spin up a read
> ahead thread if you read a lot of data through it.  It does not do
> this immediately, only after fetching a few batches of key value pairs
> from the tablet server.  If this happens you could have one thread
> fetching data while the client thread processes results.
>=20
> Do you think we should open a a ticket about giving users control over
> threads created by client code?    Maybe users could pass in their own
> thread pool to a batch scanner?
>=20
>=20
> Keith
>=20
> On Thu, Mar 28, 2013 at 11:00 AM,  <roshanp@gmail.com> wrote:
>> In some of my projects, we needed to control the number of threads =
spun up with the use of multiple batch scanners. We created a utility to =
control the number of threads, and if the max threads has been reached, =
return a batch scanner that is actually backed by Scanners. Wanted to =
get any feedback on the code. Seems like such a simple thing to do, I =
bet someone already has this. Thanks!
>>=20
>> https://github.com/calrissian/mango/tree/master/accumulo