phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3534) Support multi region SYSTEM.CATALOG table
Date Wed, 04 Jul 2018 23:20:03 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533114#comment-16533114
] 

ASF GitHub Bot commented on PHOENIX-3534:
-----------------------------------------

Github user JamesRTaylor commented on a diff in the pull request:

    https://github.com/apache/phoenix/pull/303#discussion_r200206862
  
    --- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
---
    @@ -1809,180 +2256,97 @@ public void createTable(RpcController controller, CreateTableRequest
request,
             } catch (Throwable t) {
                 logger.error("createTable failed", t);
                 ProtobufUtil.setControllerException(controller,
    -                    ServerUtil.createIOException(SchemaUtil.getTableName(schemaName,
tableName), t));
    +                    ServerUtil.createIOException(fullTableName, t));
             }
         }
     
    +	private void dropChildMetadata(byte[] schemaName, byte[] tableName, byte[] tenantIdBytes)
    +			throws IOException, SQLException, ClassNotFoundException {
    +		TableViewFinderResult childViewsResult = new TableViewFinderResult();
    +		findAllChildViews(tenantIdBytes, schemaName, tableName, childViewsResult);
    +		if (childViewsResult.hasViews()) {
    +			for (TableInfo viewInfo : childViewsResult.getResults()) {
    +				byte[] viewTenantId = viewInfo.getTenantId();
    +				byte[] viewSchemaName = viewInfo.getSchemaName();
    +				byte[] viewName = viewInfo.getTableName();
    +				Properties props = new Properties();
    +				if (viewTenantId != null && viewTenantId.length != 0)
    +					props.setProperty(PhoenixRuntime.TENANT_ID_ATTRIB, Bytes.toString(viewTenantId));
    +				try (PhoenixConnection connection = QueryUtil.getConnectionOnServer(env.getConfiguration())
    +						.unwrap(PhoenixConnection.class)) {
    +					MetaDataClient client = new MetaDataClient(connection);
    +					org.apache.phoenix.parse.TableName viewTableName = org.apache.phoenix.parse.TableName
    +							.create(Bytes.toString(viewSchemaName), Bytes.toString(viewName));
    +					client.dropTable(
    +							new DropTableStatement(viewTableName, PTableType.VIEW, false, true, true));
    +				}
    +			}
    +		}
    +	}
    +
         private boolean execeededIndexQuota(PTableType tableType, PTable parentTable) {
             return PTableType.INDEX == tableType && parentTable.getIndexes().size()
>= maxIndexesPerTable;
         }
    -    
    -    private void findAllChildViews(Region region, byte[] tenantId, PTable table,
    -            TableViewFinder result, long clientTimeStamp, int clientVersion) throws IOException,
SQLException {
    -        TableViewFinder currResult = findChildViews(region, tenantId, table, clientVersion,
false);
    -        result.addResult(currResult);
    -        for (ViewInfo viewInfo : currResult.getViewInfoList()) {
    -            byte[] viewtenantId = viewInfo.getTenantId();
    -            byte[] viewSchema = viewInfo.getSchemaName();
    -            byte[] viewTable = viewInfo.getViewName();
    -            byte[] tableKey = SchemaUtil.getTableKey(viewtenantId, viewSchema, viewTable);
    -            ImmutableBytesPtr cacheKey = new ImmutableBytesPtr(tableKey);
    -            PTable view = loadTable(env, tableKey, cacheKey, clientTimeStamp, clientTimeStamp,
clientVersion);
    -            if (view == null) {
    -                logger.warn("Found orphan tenant view row in SYSTEM.CATALOG with tenantId:"
    -                        + Bytes.toString(tenantId) + ", schema:"
    -                        + Bytes.toString(viewSchema) + ", table:"
    -                        + Bytes.toString(viewTable));
    -                continue;
    -            }
    -            findAllChildViews(region, viewtenantId, view, result, clientTimeStamp, clientVersion);
    -        }
    -    }
    -        
    -    // TODO use child link instead once splittable system catalog (PHOENIX-3534) is implemented
    -    // and we have a separate table for links.
    -    private TableViewFinder findChildViews_deprecated(Region region, byte[] tenantId,
PTable table, byte[] linkTypeBytes, boolean stopAfterFirst) throws IOException {
    -        byte[] schemaName = table.getSchemaName().getBytes();
    -        byte[] tableName = table.getTableName().getBytes();
    -        boolean isMultiTenant = table.isMultiTenant();
    -        Scan scan = new Scan();
    -        // If the table is multi-tenant, we need to check across all tenant_ids,
    -        // so we can't constrain the row key. Otherwise, any views would have
    -        // the same tenantId.
    -        if (!isMultiTenant) {
    -            byte[] startRow = ByteUtil.concat(tenantId, QueryConstants.SEPARATOR_BYTE_ARRAY);
    -            byte[] stopRow = ByteUtil.nextKey(startRow);
    -            scan.setStartRow(startRow);
    -            scan.setStopRow(stopRow);
    -        }
    -        SingleColumnValueFilter linkFilter = new SingleColumnValueFilter(TABLE_FAMILY_BYTES,
LINK_TYPE_BYTES, CompareOp.EQUAL, linkTypeBytes);
    -        SingleColumnValueFilter tableTypeFilter = new SingleColumnValueFilter(TABLE_FAMILY_BYTES,
TABLE_TYPE_BYTES,
    -                CompareOp.EQUAL, PTableType.VIEW.getSerializedValue().getBytes());
    -        tableTypeFilter.setFilterIfMissing(false);
    -        linkFilter.setFilterIfMissing(true);
    -        byte[] suffix = ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY, SchemaUtil
    -                .getPhysicalHBaseTableName(schemaName, tableName, table.isNamespaceMapped())
    -                .getBytes());
    -        SuffixFilter rowFilter = new SuffixFilter(suffix);
    -        List<Filter> filters = Lists.<Filter>newArrayList(linkFilter,tableTypeFilter,rowFilter);
    -        if (stopAfterFirst) {
    -            filters.add(new PageFilter(1));
    -        }
    -        FilterList filter = new FilterList(filters);
    -        scan.setFilter(filter);
    -        scan.addColumn(TABLE_FAMILY_BYTES, LINK_TYPE_BYTES);
    -        scan.addColumn(TABLE_FAMILY_BYTES, TABLE_TYPE_BYTES);
    -        scan.addColumn(TABLE_FAMILY_BYTES, TABLE_SEQ_NUM_BYTES);
    -        
    -        // Original region-only scanner modified due to PHOENIX-1208
    -        // RegionScanner scanner = region.getScanner(scan);
    -        // The following *should* work, but doesn't due to HBASE-11837
    -        // TableName systemCatalogTableName = region.getTableDesc().getTableName();
    -        // HTableInterface hTable = env.getTable(systemCatalogTableName);
    -        // These deprecated calls work around the issue
    -        try (HTableInterface hTable = ServerUtil.getHTableForCoprocessorScan(env,
    -            region.getTableDesc().getTableName().getName())) {
    -            boolean allViewsInCurrentRegion = true;
    -            int numOfChildViews = 0;
    -            List<ViewInfo> viewInfoList = Lists.newArrayList();
    -            try (ResultScanner scanner = hTable.getScanner(scan)) {
    -                for (Result result = scanner.next(); (result != null); result = scanner.next())
{
    -                    numOfChildViews++;
    -                    ImmutableBytesWritable ptr = new ImmutableBytesWritable();
    -                    ResultTuple resultTuple = new ResultTuple(result);
    -                    resultTuple.getKey(ptr);
    -                    byte[] key = ptr.copyBytes();
    -                    if (checkTableKeyInRegion(key, region) != null) {
    -                        allViewsInCurrentRegion = false;
    -                    }
    -                    byte[][] rowKeyMetaData = new byte[3][];
    -                    getVarChars(result.getRow(), 3, rowKeyMetaData);
    -                    byte[] viewTenantId = rowKeyMetaData[PhoenixDatabaseMetaData.TENANT_ID_INDEX];
    -                    byte[] viewSchemaName = rowKeyMetaData[PhoenixDatabaseMetaData.SCHEMA_NAME_INDEX];
    -                    byte[] viewName = rowKeyMetaData[PhoenixDatabaseMetaData.TABLE_NAME_INDEX];
    -                    viewInfoList.add(new ViewInfo(viewTenantId, viewSchemaName, viewName));
    -                }
    -                TableViewFinder tableViewFinderResult = new TableViewFinder(viewInfoList);
    -                if (numOfChildViews > 0 && !allViewsInCurrentRegion) {
    -                    tableViewFinderResult.setAllViewsNotInSingleRegion();
    -                }
    -                return tableViewFinderResult;
    +
    +    private void findAncestorViewsOfIndex(byte[] tenantId, byte[] schemaName, byte[]
indexName,
    +            TableViewFinderResult result, boolean isNamespaceMapped) throws IOException
{
    +        try (Table hTable =
    +                env.getTable(SchemaUtil.getPhysicalTableName(
    +                    PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES, env.getConfiguration())))
{
    +            TableViewFinderResult currentResult =
    +                    ViewFinder.findParentViewofIndex(hTable, tenantId, schemaName, indexName);
    +            if (currentResult.getResults().size() == 1) {
    +                result.addResult(currentResult);
    +                TableInfo tableInfo = currentResult.getResults().get(0);
    +                findAncestorViews(tableInfo.getTenantId(), tableInfo.getSchemaName(),
    +                    tableInfo.getTableName(), result, isNamespaceMapped);
                 }
    +            // else this is an index on a regular table and so we don't need to combine
columns
             }
         }
         
    -    private TableViewFinder findChildViews_4_11(Region region, byte[] tenantId, byte[]
schemaName, byte[] tableName, boolean stopAfterFirst) throws IOException {
    -        Scan scan = new Scan();
    -        byte[] startRow = SchemaUtil.getTableKey(tenantId, schemaName, tableName);
    -        byte[] stopRow = ByteUtil.nextKey(startRow);
    -        scan.setStartRow(startRow);
    -        scan.setStopRow(stopRow);
    -        SingleColumnValueFilter linkFilter = new SingleColumnValueFilter(TABLE_FAMILY_BYTES,
LINK_TYPE_BYTES, CompareOp.EQUAL, CHILD_TABLE_BYTES);
    -        Filter filter = linkFilter;
    -        linkFilter.setFilterIfMissing(true);
    -        if (stopAfterFirst) {
    -            filter = new FilterList(linkFilter, new PageFilter(1));
    -        }
    -        scan.setFilter(filter);
    -        scan.addColumn(TABLE_FAMILY_BYTES, LINK_TYPE_BYTES);
    -        scan.addColumn(TABLE_FAMILY_BYTES, PARENT_TENANT_ID_BYTES);
    -        
    -        // Original region-only scanner modified due to PHOENIX-1208
    -        // RegionScanner scanner = region.getScanner(scan);
    -        // The following *should* work, but doesn't due to HBASE-11837
    -        // TableName systemCatalogTableName = region.getTableDesc().getTableName();
    -        // HTableInterface hTable = env.getTable(systemCatalogTableName);
    -        // These deprecated calls work around the issue
    -        try (HTableInterface hTable = ServerUtil.getHTableForCoprocessorScan(env,
    -            region.getTableDesc().getTableName().getName())) {
    -            boolean allViewsInCurrentRegion = true;
    -            int numOfChildViews = 0;
    -            List<ViewInfo> viewInfoList = Lists.newArrayList();
    -            try (ResultScanner scanner = hTable.getScanner(scan)) {
    -                for (Result result = scanner.next(); (result != null); result = scanner.next())
{
    -                    numOfChildViews++;
    -                    ImmutableBytesWritable ptr = new ImmutableBytesWritable();
    -                    ResultTuple resultTuple = new ResultTuple(result);
    -                    resultTuple.getKey(ptr);
    -                    byte[] key = ptr.copyBytes();
    -                    if (checkTableKeyInRegion(key, region) != null) {
    -                        allViewsInCurrentRegion = false;
    -                    }
    -                    byte[][] rowViewKeyMetaData = new byte[5][];
    -                    getVarChars(result.getRow(), 5, rowViewKeyMetaData);
    -                    byte[] viewTenantId = rowViewKeyMetaData[PhoenixDatabaseMetaData.COLUMN_NAME_INDEX];
    -                    byte[] viewSchemaName = SchemaUtil.getSchemaNameFromFullName(rowViewKeyMetaData[PhoenixDatabaseMetaData.FAMILY_NAME_INDEX]).getBytes();
    -                    byte[] viewName = SchemaUtil.getTableNameFromFullName(rowViewKeyMetaData[PhoenixDatabaseMetaData.FAMILY_NAME_INDEX]).getBytes();
    -                    viewInfoList.add(new ViewInfo(viewTenantId, viewSchemaName, viewName));
    -                }
    -                TableViewFinder tableViewFinderResult = new TableViewFinder(viewInfoList);
    -                if (numOfChildViews > 0 && !allViewsInCurrentRegion) {
    -                    tableViewFinderResult.setAllViewsNotInSingleRegion();
    -                }
    -                return tableViewFinderResult;
    +    private void findAncestorViews(byte[] tenantId, byte[] schemaName, byte[] tableName,
    +            TableViewFinderResult result, boolean isNamespaceMapped) throws IOException
{
    +        try (Table hTable =
    +                env.getTable(SchemaUtil.getPhysicalTableName(
    +                    PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES, env.getConfiguration())))
{
    +            ViewFinder.findAllRelatives(hTable, tenantId, schemaName, tableName,
    +                LinkType.PARENT_TABLE, result);
    +            if (!isNamespaceMapped || schemaName.length==0) {
    +                // the child->parent link is overwritten by the child->physical
table link for first
    +                // level children of base table when namespace mapping is disabled or
if the parent
    +                // table doesn't have a schema as both the parent table name and physical
table name
    +                // are the same.
    +                // When namespace mapping is enabled the physical table name is of the
form S:T
    +                // while the table name is of the form S.T so we need to query for the
    +                // PHYSICAL_TABLE link
    --- End diff --
    
    Since the linking rows are being re-written I believe in 4.15, should we change this behavior?


> Support multi region SYSTEM.CATALOG table
> -----------------------------------------
>
>                 Key: PHOENIX-3534
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3534
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: Thomas D'Silva
>            Priority: Major
>             Fix For: 5.0.0, 4.15.0
>
>         Attachments: PHOENIX-3534.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region based on the
server-side row locks being held for operations that impact a table and all of it's views.
For example, adding/removing a column from a base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431), when a new
table is created we can do a lazy cleanup  of any rows that may be left over from a failed
DDL call (kudos to [~lhofhansl] for coming up with this idea). To implement this efficiently,
we'd need to also do PHOENIX-2051 so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on checking our
sequence numbers for each table/view before/after updating. Each table/view row would be individually
locked for their change (metadata for a view or table cannot span regions due to our split
policy), with the sequence number being incremented under lock and then returned to the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message