phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-4484) Write directly to HBase when creating an index for transactional table
Date Thu, 26 Apr 2018 00:31:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453283#comment-16453283
] 

James Taylor commented on PHOENIX-4484:
---------------------------------------

One consideration we discussed offline is to disable the Omid garbage collector while an index
is build built. One idea would be to delay the calling of the compaction hook that does the
GC if an index is being built by wrapping the coprocessor by our own Phoenix delegate coprocessor.
In the simple case, in which the hbase table maps directly to a Phoenix index table, it's
easy since we can lookup the table in SYSTEM.CATALOG and detect if it's an index being built.
However, if it's an index on a view (i.e. a shared physical table) or a local index (multiple
indexes in the same physical table as the data table but different column family), it's difficult.
You'd have to scan the entire SYSTEM.CATALOG and try to figure out if the HBase table being
compacted corresponds to any index that is in a building state. It might end up delaying the
compaction too often too, since it would delay if any index is being built (which may be a
very small portion of the overall table).

I think we should brainstorm if there are other solutions. Would it be possible, [~ohads],
to have a flag in the commit table that would be used to determine whether or not the GC happens?
Or would another alternative be to not put the index table under the control of Omid until
after the initial population is finished?

> Write directly to HBase when creating an index for transactional table
> ----------------------------------------------------------------------
>
>                 Key: PHOENIX-4484
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4484
>             Project: Phoenix
>          Issue Type: Sub-task
>            Reporter: Ohad Shacham
>            Assignee: Ohad Shacham
>            Priority: Major
>
> Today, when creating an index table for a non empty data table. The writes are performed
using the transaction api and both consumes client side memory, for storing the writeset,
and checks for conflict analysis upon commit. This is redundant and can be replaced by direct
write to HBase. For this reason, a new function in the transaction abstraction layer should
be added that writes directly to HBase at the Tephra's case and adds shadow cells with the
fence id at the Omid case. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message