Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@phoenix.apache.org
Date: Sat, 6 Feb 2016 11:26:39 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: dev@phoenix.incubator.apache.org
Message-ID: <JIRA.12931642.1452926693000.321083.1454757999832@Atlassian.JIRA>
In-Reply-To: <JIRA.12931642.1452926693000@Atlassian.JIRA>
References: <JIRA.12931642.1452926693000@Atlassian.JIRA>
 <JIRA.12931642.1452926693312@arcas>
Subject: [jira] [Commented] (PHOENIX-2607) PhoenixMapReduceUtil Upserts with
 earlier ts (relative to latest data ts) slower by 25x after stats
 collection
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/PHOENIX-2607?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15=
135733#comment-15135733 ]=20

Hadoop QA commented on PHOENIX-2607:
------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest a=
ttachment=20
  http://issues.apache.org/jira/secure/attachment/12786655/PHOENIX-2607.pat=
ch
  against master branch at commit fa58fc5fccd11cbeb14e701a59122b3cba77d67e.
  ATTACHMENT ID: 12786655

    {color:green}+1 @author{color}.  The patch does not contain any @author=
 tags.

    {color:green}+0 tests included{color}.  The patch appears to be a docum=
entation, build,
                        or dev patch that doesn't require tests.

    {color:green}+1 javac{color}.  The applied patch does not increase the =
total number of javac compiler warnings.

    {color:red}-1 javadoc{color}.  The javadoc tool appears to have generat=
ed 20 warning messages.

    {color:green}+1 release audit{color}.  The applied patch does not incre=
ase the total number of release audit warnings.

    {color:red}-1 lineLengths{color}.  The patch introduces the following l=
ines longer than 100:
    +        return PTableImpl.makePTable(tenantId, schemaName, tableName, =
tableType, indexState, timeStamp, createdTimestamp,
+                region.mutateRowsWithLocks(tableMetadata, Collections.<byt=
e[]>emptySet(), HConstants.NO_NONCE, HConstants.NO_NONCE);
+    private boolean tableTimestampCheck(long clientTimeStamp, PTable table=
, RegionCoprocessorEnvironment env) {
+                table.getTenantId(), table.getSchemaName(), table.getTable=
Name(), table.getType(), table.getIndexState(), timeStamp, table.getCreated=
Timestamp(),
+                table.getTenantId(), table.getSchemaName(), table.getTable=
Name(), table.getType(), table.getIndexState(), table.getTimeStamp(), table=
.getCreatedTimestamp(),
+                table.getTenantId(), table.getSchemaName(), table.getTable=
Name(), table.getType(), table.getIndexState(), timeStamp, table.getCreated=
Timestamp(),
+                table.getTenantId(), table.getSchemaName(), table.getTable=
Name(), table.getType(), table.getIndexState(), timeStamp, table.getCreated=
Timestamp(),
+                table.getTenantId(), table.getSchemaName(), table.getTable=
Name(), table.getType(), table.getIndexState(), timeStamp, table.getCreated=
Timestamp(),
+                table.getTenantId(), table.getSchemaName(), table.getTable=
Name(), table.getType(), state, table.getTimeStamp(), table.getCreatedTimes=
tamp(),
+                table.getTenantId(), table.getSchemaName(), table.getTable=
Name(), table.getType(), table.getIndexState(), table.getTimeStamp(), table=
.getCreatedTimestamp(),

     {color:red}-1 core tests{color}.  The patch failed these unit tests:
    =20

     {color:red}-1 core zombie tests{color}.  There are 3 zombie test(s): =
=09at org.apache.maven.surefire.booter.ForkingRunListener.testSucceeded(For=
kingRunListener.java:127)
=09at org.apache.maven.surefire.common.junit4.JUnit4RunListener.testFinishe=
d(JUnit4RunListener.java:143)
=09at org.junit.runner.notification.SynchronizedRunListener.testFinished(Sy=
nchronizedRunListener.java:56)
=09at org.apache.phoenix.jdbc.PhoenixResultSetMetadataTest.testColumnDispla=
ySize(PhoenixResultSetMetadataTest.java:38)
=09at org.apache.phoenix.util.PhoenixRuntimeTest.testGetTenantIdExpression(=
PhoenixRuntimeTest.java:210)

Test results: https://builds.apache.org/job/PreCommit-PHOENIX-Build/245//te=
stReport/
Javadoc warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/245=
//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/245//=
console

This message is automatically generated.

> PhoenixMapReduceUtil Upserts with earlier ts (relative to latest data ts)=
 slower by 25x after stats collection
> -------------------------------------------------------------------------=
-------------------------------------
>
>                 Key: PHOENIX-2607
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2607
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.4.0
>            Reporter: Arun Thangamani
>              Labels: patch
>             Fix For: 4.4.0
>
>         Attachments: PHOENIX-2607.patch, hbase-master-fast-upload.log, hb=
ase-master-slow-upload.log, hbase-rs01-fast-upload.log, hbase-rs01-slow-upl=
oad.log, hbase-rs02-fast-upload.log, hbase-rs02-slow-upload.log, hbase-rs03=
-fast-upload.log, hbase-rs03-slow-upload.log, hbase-rs04-fast-upload.log, h=
base-rs04-slow-upload.log, phoenix_slow_map_process_jstack.txt, region_serv=
er_2_jstack.txt
>
>
> Description of the problem:
> 1) We face a 25x slow down when go back in time to load data in a table (=
when specific timestamps set on connections during upserts)
> 2) set phoenix.stats.useCurrentTime=3Dfalse (and phoenix.stats.guidepost.=
per.region 1) which at least makes the forward timestamps upserts perform c=
orrectly=20
> 3) From what I can tell from the phoenix source code, logs attached and j=
stacks from the region servers -- we continuously try to lookup the uncache=
d definition of the table when we have client timestamp earlier than the la=
st modified timestamp of the table in stats=20
> 4) To reproduce, create a table with timestamp=3D100, and load 10M rows w=
ith PhoenixMapReduceUtil and timestamps=3D1447574400000,1448092800000, wait=
 for 20 mins (15+ min, phoenix.stats.updateFrequency is 15mins)
> After 20 mins, load 10M rows with a earlier timestamp compared to the lat=
est data (timestamp=3D1447660800000) and observe the 25x slowness, after th=
is once again load a forward timestamp 1448179200000 and observe the quickn=
ess
> 5) I was not able to reproduce this issue with simple multi threaded upse=
rts from a jdbc connection, with simple multi threaded upserts the stats ta=
ble never gets populated unlike PhoenixMapReduceUtil
> We are trying to use phoenix as a cache store to do analytics with the la=
st 60 days of data, a total of about 1.5 billion rows
> The table has a composite key and the data arrives in different times fro=
m different sources, so it is easier to maintain the timestamps of the data=
 and expire the data automatically, this performance makes a difference bet=
ween inserting the data in 10 mins versus 2 hours, 2 hours for data inserts=
 blocking up the cluster that we have.
> We are even talking about our use cases in the upcoming strata conference=
 in March..  (Thanks to excellent community)
> Steps to reproduce:
> Source code is available in (https://github.com/athangamani/phoenix_mapre=
duce_timestamp_upsert) and the jar the source code produces is attached whi=
ch is readily runnable=20
> 1) We use the following params to keep the stats collection happy to isol=
ate the specific issue
>      phoenix.stats.useCurrentTime false
>      phoenix.stats.guidepost.per.region 1
> 2) Create a table in phoenix=20
>    Run the following main class from the project.. (StatPhoenixTableCreat=
ionTest).. It will create a table with timestamp=3D100 =20
> =09=09CREATE TABLE stat_table (=20
> =09=09=09pk1 VARCHAR NOT NULL,=20
> =09=09=09pk2 VARCHAR NOT NULL,=20
> =09=09=09pk3 UNSIGNED_LONG NOT NULL,=20
> =09=09=09 stat1 UNSIGNED_LONG,=20
> =09=09=09 stat2 UNSIGNED_LONG,=20
> =09=09=09 stat3 UNSIGNED_LONG,=20
> =09=09=09 CONSTRAINT pk PRIMARY KEY (pk1, pk2, pk3)=20
> =09=09) SALT_BUCKETS=3D32, COMPRESSION=3D'LZ4'
> 3) Open the code base to look at the sample for PhoenixMapReduceUtil.. Wi=
th DBWritable.. =20
> 4) Within the codebase, we get phoenix connection for the mappers using t=
he following settings in order to have a fixed client timestamp
>      conf.set(PhoenixRuntime.CURRENT_SCN_ATTRIB, ""+(timestamp));
> 5) fix the hbase-site.xml in the codebase for zookeeper quorum and hbase =
parent znode info =09=20
> 6) simply run the StatDataCreatorTest to create data for the run and load=
 it in hdfs for 10M records
> 7) to run the ready made jar attached, use the following commands,=20
>    hadoop jar phoenix_mr_ts_upsert-jar-with-dependencies.jar statPhoenixL=
oader hdfs:///user/*****/stat-data-1.txt STAT_TABLE 1447574400000
>    hadoop jar phoenix_mr_ts_upsert-jar-with-dependencies.jar statPhoenixL=
oader hdfs:///user/*****/stat-data-1.txt STAT_TABLE 1448092800000=20
>     After 20 mins=E2=80=A6=20
>    hadoop jar phoenix_mr_ts_upsert-jar-with-dependencies.jar statPhoenixL=
oader hdfs:///user/*****/stat-data-1.txt STAT_TABLE 1447660800000
>    hadoop jar phoenix_mr_ts_upsert-jar-with-dependencies.jar statPhoenixL=
oader hdfs:///user/*****/stat-data-1.txt STAT_TABLE 1449000000000
> 8) observe the 25x slowness in the 3rd run and observe the normal pace in=
 the 4th run
> 9) Attached are the region server logs for the fast process in forward pr=
ogression versus the slow progress in reverse timestamp


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)