phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-4052) Create the correct index row mutations for out-of-order data mutations
Date Mon, 31 Jul 2017 21:19:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107973#comment-16107973
] 

James Taylor commented on PHOENIX-4052:
---------------------------------------

The following test, slightly different than the description above, reproduces an issue with
an extra row with an invalid row key ending up in the index table:
{code}
    @Test
    public void testOutOfOrderDelete() throws Exception {
    	String tableName = generateUniqueName();
    	String indexName = generateUniqueName();
        Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
        long ts = 1000;
        props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
        Connection conn = DriverManager.getConnection(getUrl(), props);     
        conn.createStatement().execute("CREATE TABLE " + tableName + "(k CHAR(1) PRIMARY KEY,
v VARCHAR) COLUMN_ENCODED_BYTES = 0");
        conn.close();

        ts = 1010;
        props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
        conn = DriverManager.getConnection(getUrl(), props);
        conn.createStatement().execute("CREATE INDEX " + indexName + " ON " + tableName +
"(v)");
        conn.close();
        
        ts = 1020;
        props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
        conn = DriverManager.getConnection(getUrl(), props);        
        conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a','a')");
        conn.commit();
        conn.close();
        
        ts = 1040;
        props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
        conn = DriverManager.getConnection(getUrl(), props);
        conn.createStatement().execute("DELETE FROM " + tableName + " WHERE k='a'");
        conn.commit();
        conn.close();
        
        ts = 1030;
        props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
        conn = DriverManager.getConnection(getUrl(), props);
        conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a','bbb')");
        conn.commit();
        conn.close();
        
        TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(tableName)));
        TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(indexName)));
        
        ts = 1050;
        props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts));
        conn = DriverManager.getConnection(getUrl(), props);
        long count1 = getRowCount(conn, tableName);
        long count2 = getRowCount(conn, indexName);
        assertTrue("Expected table row count ( " + count1 + ") to match index row count ("
+ count2 + ")", count1 == count2);
        conn.close();
        
        /**
         *
			************ dumping T000001;hconnection-0x8f57e4c **************
			a/0:/1040/DeleteFamily/vlen=0/seqid=0
			a/0:V/1030/Put/vlen=3/seqid=0
			a/0:V/1020/Put/vlen=1/seqid=0
			a/0:_0/1030/Put/vlen=1/seqid=0
			a/0:_0/1020/Put/vlen=1/seqid=0
			-----------------------------------------------
			************ dumping T000002;hconnection-0x8f57e4c **************
			\x00a/0:_0/1040/Put/vlen=2/seqid=0
			a\x00a/0:/1040/DeleteFamily/vlen=0/seqid=0
			a\x00a/0:/1030/DeleteFamily/vlen=0/seqid=0
			a\x00a/0:_0/1020/Put/vlen=2/seqid=0
			-----------------------------------------------
         */
    }

    private static long getRowCount(Connection conn, String tableName) throws SQLException
{
        ResultSet rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX */ count(*)
FROM " + tableName);
        assertTrue(rs.next());
        return rs.getLong(1);
    }
{code}
Assertion that occurs is:
{code}
java.lang.AssertionError: Expected table row count ( 0) to match index row count (1)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.phoenix.end2end.index.OutOfOrderMutationsIT.testOutOfOrderDelete(OutOfOrderMutationsIT.java:135)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
{code}

> Create the correct index row mutations for out-of-order data mutations
> ----------------------------------------------------------------------
>
>                 Key: PHOENIX-4052
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4052
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>
> Out-of-order processing of data rows during index maintenance causes mutable indexes
to become out of sync with regard to the data table. Here's a simple example to illustrate
the issue:
> # Assume table T(K,V) and index X(V,K).
> # Upsert T(A, 1) at t10. Index updates: Put X(1,A) at t10.
> # Upsert T(A, 3) at t30. Index updates: Delete X(1,A) at t29, Put X(3,A) at t30.
> # Upsert T(A,2) at t20. Index updates: Delete X(1,A) at t19, Put X(2,A) at t20, Delete
X(2,A) at t29
> Ideally, we'd want to remove the Delete X(1,A) at t29 since this isn't correct in terms
of timeline consistency, but we can't do that with HBase without support for deleting/undoing
Delete markers. 
> The above is not what is occurring. Instead, when T(A,2) comes in, the Put X(2,A) will
occur at t20, but the Delete won't occur. This causes more index rows than data rows, essentially
making it invalid.
> See PHOENIX-4051 as a workaround for fixing this issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message