Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <49268.72.234.129.253.1351758020.squirrel@kuehns.com>
In-Reply-To: <6714-1351740929-272712@sneakemail.com>
References: <6714-1351740929-272712@sneakemail.com>
Date: Thu, 1 Nov 2012 01:20:20 -0700 (PDT)
Subject: Re: Cost of scanner usage in a MapReduce mapper?
From: skuehn@kuehns.com
To: user@accumulo.apache.org
Cc: accumulo-user@incubator.apache.org
User-Agent: SquirrelMail/1.4.13
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Importance: Normal
Content-Transfer-Encoding: quoted-printable


For clarification, are you trying to create DAO's from the Key/Value fed
to a Mapper by AccumuloInputFormat, or are you trying to process a
different data set while simultaneously querying your DAO's?

> Today I had a really nice conversation with billie and vines on #accumu=
lo.
> This email is a followup to that conversation, and there's a little mor=
e
> context of my problem here.
>
> We have an application that we've developed independently from MapReduc=
e.
> To
> get away from the low-level keys and values of Accumulo, we quickly mad=
e a
> series of DAOs that each take in an Accumulo Instance as a constructor
> argument.  These DAOs internally create the necessary scanners and retu=
rn
> domain-specific objects.  I imagine this is a common practice.
>
> Now, we've got a feature that needs to operate on all the data, so we'r=
e
> doing
> some MapReduce.  I think I understand now the architecture of
> AccumuloInputFormat from discussions on #accumulo.  What I didn't discu=
ss
> was
> whether it was reasonable (or not reasonable because of the performance
> cost)
> to try to use one of our DAOs within a mapper.
>
> The mappers need to operate per row, and our system has potentially
> billions of
> rows.  With my DAOs, I can reuse the same Accumulo instance, but each c=
all
> will create a new scanner from my instance, so a MapReduce job using a =
DAO
> in
> the mappers will potentially create billions of scanners over the cours=
e
> of
> operation.   However, the way we've designed these DAOs, it's easy to m=
ake
> sure
> all accesses are tied to the row the mapper is tasked with (in an attem=
pt
> to
> maintain data locality).
>
> By comparison.  I feel the AccumuloInputFormat will create about as man=
y
> Accumulo scanners as there are tablet servers, so dramatically less.
>
> Our current thinking is that creating billions scanners with these DAO
> accesses
> might cost too much in performance, but we're not completely sure this =
is
> the
> case with respect to the kind of caching Accumulo does with its clients=
.
>
> If the performance cost is indeed too high, then we're going to have to
> deal
> with the abstraction challenge of trying to avoid code duplication betw=
een
> our
> DAOs and our MapReduce jobs.