Mailing-List: contact dev-help@tephra.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@tephra.incubator.apache.org
From: Terence Yim <chtyim@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: TransactionCodec poor performance
Date: Wed, 31 May 2017 02:29:36 -0700
References: <b351b3c5-7311-4cd2-4a88-28f46ab82858@xpand-it.com>
To: dev@tephra.incubator.apache.org
In-Reply-To: <b351b3c5-7311-4cd2-4a88-28f46ab82858@xpand-it.com>
Message-Id: <F4F822B9-357A-4233-A457-613FBF71DF5D@gmail.com>
archived-at: Wed, 31 May 2017 09:29:50 -0000

Hi Micael,

Do you know if the invalid tx list inside the Transaction object is =
large?

Terence

> On May 31, 2017, at 1:49 AM, Micael Capit=C3=A3o =
<micael.capitao@xpand-it.com> wrote:
>=20
> Hi all,
>=20
> I've been testing Tephra 0.11.0 for a project that may need =
transactions on top of HBase and I find it's performance, for instance, =
for a bulk load, very poor. Let's not discuss why am I doing a bulk load =
with transactions.
>=20
> In my use case I am generating batches of ~10000 elements and =
inserting them with the *put(List<Put> puts)* method. There is no =
concurrent writers or readers.
> If I do the put without transactions it takes ~0.5s. If I use the =
*TransactionAwareHTable* it takes ~12s.
> I've tracked down the performance killer to be the =
*addToOperation(OperationWithAttributes op, Transaction tx)*, more =
specifically the *txCodec.encode(tx)*.
>=20
> I've created a TransactionAwareHTableFix with the =
*addToOperation(txPut, tx)* commented, and used it in my code, and each =
batch started to take ~0.5s.
>=20
> I've noticed that inside the *TransactionCodec* you were instantiating =
a new TSerializer and TDeserializer on each call to encode/decode. I =
tried instantiating the ser/deser on the constructor but even that way =
each of my batches would take the same ~12s.
>=20
> Further investigation has shown me that the Transaction instance, =
after being encoded by the TransactionCodec, has 104171 bytes of length. =
So in my 10000 elements batch, ~970MB is metadata. Is that supposed to =
happen?
>=20
>=20
> Regards,
>=20
> Micael Capit=C3=A3o