Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@phoenix.apache.org
Received-SPF: pass (athena.apache.org: domain of ndimiduk@gmail.com designates
 74.125.82.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAA5C_psa1zsDwXOEsD7aYnMpt3iM7hY4=BTiMq0jU8V9K8UaqQ@mail.gmail.com>
References: 
 <CAFGo-hyzXR2CgxYCrUGp9RGRtfuzTs-NxOp7-nmQb7iD_z2WzQ@mail.gmail.com>
	<CAA5C_psa1zsDwXOEsD7aYnMpt3iM7hY4=BTiMq0jU8V9K8UaqQ@mail.gmail.com>
Date: Thu, 5 Mar 2015 07:53:18 -0800
Message-ID: 
 <CANZa=GtiroKUUVwRRU=ufd3H8HjZBmYNyPD9tSCX2oh-2FWZtA@mail.gmail.com>
Subject: Re: Bulk-loader performance
From: Nick Dimiduk <ndimiduk@gmail.com>
To: "dev@phoenix.apache.org" <dev@phoenix.apache.org>
Content-Type: multipart/alternative; boundary=001a11c3891011d29805108c91de

--001a11c3891011d29805108c91de
Content-Type: text/plain; charset=UTF-8

Also: how large is your cluster? You can make things go faster by
increasing the number of mappers. What changes did you make to the map()
method? Increased logging, performance enhancements, plugging in custom
logic, something else?

On Thursday, March 5, 2015, Gabriel Reid <gabriel.reid@gmail.com> wrote:

> Hi Tulasi,
>
> Answers (and questions) inlined below:
>
> On Thu, Mar 5, 2015 at 2:41 AM Tulasi Paradarami <
> tulasi.krishna.p@gmail.com <javascript:;>>
> wrote:
>
> > Hi,
> >
> > Here are the details of our environment:
> > Phoenix 4.3
> > HBase 0.98.6
> >
> > I'm loading data to a Phoenix table using the csv bulk-loader (after
> making
> > some changes to the map(...) method) and it is processing about 16,000 -
> > 20,000 rows/sec. I noticed that the bulk-loader spends upto 40% of the
> > execution time in the following steps.
>
>
> > //...
> > csvRecord = csvLineParser.parse(value.toString());
> > csvUpsertExecutor.execute(ImmutableList.of(csvRecord));
> > Iterator<Pair<byte[], List<KeyValue>>> uncommittedDataIterator =
> > PhoenixRuntime.getUncommittedDataIterator(conn, true);
> > //...
> >
>
> The non-code translation of those steps is:
> 1. Parse the CSV record
> 2. Convert the contents of the CSV record into KeyValues
>
> Although it may look as though data is being written over the wire to
> Phoenix, the execution of an upsert executor and retrieval of the
> uncommitted KeyValues is all local (in memory). The code is implemented in
> this way because JDBC is the general API used within Phoenix -- there isn't
> direct "convert fields to Phoenix encoding" API, although this is doing the
> equivalent operation.
>
> Could you give some more information on your performance numbers? For
> example, is this the throughput that you're getting in a single process, or
> over a number of processes? If so, how many processes? Also, how many
> columns are in the records that you're loading?
>
>
> >
> > We plan to load up-to 100TB of data and overall performance of the
> > bulk-loader is not satisfactory.
> >
>
> How many records are in that 100TB? What is the current (projected) time
> required to load the data? What is the minimum allowable ingest speed to be
> considered satisfactory?
>
> - Gabriel
>

--001a11c3891011d29805108c91de--