phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (PHOENIX-4277) Treat delete markers consistently with puts for point-in-time scans
Date Fri, 06 Oct 2017 01:21:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193400#comment-16193400
] 

James Taylor edited comment on PHOENIX-4277 at 10/6/17 1:20 AM:
----------------------------------------------------------------

[~vincentpoon] - after further discussion, another easier option surfaced. HBase has the ability
to run a scan with KEEP_DELETED_CELLS enabled. Phoenix could leverage this for any point-in-time
scan (i.e. a scan with a max time range set to a value other than LATEST_TIMESTAMP), so no
need for any new scan attribute or client-side change, since the intent with point-in-time
scans in Phoenix is to treat delete markers consistently with updates (i'm not sure of the
logic of seeing prior update version which would disappear with compaction while *not* seeing
deletes). 

Here's the new coprocessor we could add to BaseScannerRegionObserver:
{code}
  @Override
  public KeyValueScanner preStoreScannerOpen(final ObserverContext<RegionCoprocessorEnvironment>
c,
      final Store store, final Scan scan, final NavigableSet<byte[]> targetCols,
      final KeyValueScanner s) throws IOException {

    if (scan.isRaw() || scan.getTimeRange().getMax() == HConstants.LATEST_TIMESTAMP) {
      return s;
    }

    ScanInfo oldScanInfo = store.getScanInfo();

    ScanInfo scanInfo =
        new ScanInfo(oldScanInfo.getFamily(), Math.max(oldScanInfo.getMinVersions(), 1),
            oldScanInfo.getMaxVersions(), oldScanInfo.getTtl(), KeepDeletedCells.TRUE,
            oldScanInfo.getTimeToPurgeDeletes(), oldScanInfo.getComparator());
    return new StoreScanner(store, scanInfo, scan, targetCols,
        c.getEnvironment().getRegion().getReadpoint(scan.getIsolationLevel()));
  }
{code}

This will prevent us from having to replicate in Phoenix the filtering HBase does for deletes
in a user-level Filter (which the HBase community tried, but failed to do already). 

We'd need to investigate how to do this for snapshot reads, but the above should fix the current
issue with the IndexScrutinyTool.


was (Author: jamestaylor):
[~vincentpoon] - after further discussion, another easier option surfaced. HBase has the ability
to run a scan with KEEP_DELETED_CELLS enabled. Phoenix could leverage this for any point-in-time
scan (i.e. a scan with a max time range set to a value other than LATEST_TIMESTAMP), so no
need for any new scan attribute or client-side change, since the intent with point-in-time
scans in Phoenix is to treat delete markers consistently with updates (i'm not sure of the
logic of seeing prior update version which would disappear with compaction while *not* seeing
deletes). 

Here's the new coprocessor we could add to BaseScannerRegionObserver:
{code}
  @Override
  public KeyValueScanner preStoreScannerOpen(final ObserverContext<RegionCoprocessorEnvironment>
c,
      final Store store, final Scan scan, final NavigableSet<byte[]> targetCols,
      final KeyValueScanner s) throws IOException {

    if (!isSOREnabled || !scan.isRaw() || scan.getTimeRange().getMax() == HConstants.LATEST_TIMESTAMP)
{
      return s;
    }

    ScanInfo oldScanInfo = store.getScanInfo();

    ScanInfo scanInfo =
        new ScanInfo(oldScanInfo.getFamily(), Math.max(oldScanInfo.getMinVersions(), 1),
            oldScanInfo.getMaxVersions(), oldScanInfo.getTtl(), KeepDeletedCells.TRUE,
            oldScanInfo.getTimeToPurgeDeletes(), oldScanInfo.getComparator());
    return new StoreScanner(store, scanInfo, scan, targetCols,
        c.getEnvironment().getRegion().getReadpoint(scan.getIsolationLevel()));
  }
{code}

This will prevent us from having to replicate in Phoenix the filtering HBase does for deletes
in a user-level Filter (which the HBase community tried, but failed to do already). 

We'd need to investigate how to do this for snapshot reads, but the above should fix the current
issue with the IndexScrutinyTool.

> Treat delete markers consistently with puts for point-in-time scans
> -------------------------------------------------------------------
>
>                 Key: PHOENIX-4277
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4277
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: Vincent Poon
>
> The IndexScrutinyTool relies on doing point-in-time scans to determine consistency between
the index and data tables. Unfortunately, deletes to the tables cause a problem with this
approach, since delete markers take effect even if they're at a later time stamp than the
point-in-time at which the scan is being done (unless KEEP_DELETED_CELLS is true). The logic
of this is that scans should get the same results before and after a compaction take place.
> Taking snapshots does not help with this since they cannot be taken at a point-in-time
and the delete markers will act the same way - there's no way to guarantee that the index
and data table snapshots have the same "logical" set of data.
> Using raw scans would allow us to see the delete markers and do the correct point-in-time
filtering ourselves. We'd need to write the filters to do this correctly (see the Tephra TransactionVisibilityFilter
for an implementation of this that could be adapted). We'd also need to hook this into Phoenix
or potentially dip down to the HBase level  to do this.
> Thanks for brainstorming on this with me, [~lhofhansl].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message