hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
Date Thu, 07 Dec 2017 06:56:01 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281401#comment-16281401

Anoop Sam John commented on HBASE-19423:

My concern was on the general approach.. What the test is doing in the CP hook.
I have a scenario where in a user table some column families are system generated and these
families data need not to be replicated.
Because these are system generated column families it would not be appropriate to expect user
to disable replication for these families.
Well the replication enable is at CF level. One can set on HCD.   By default the replication
is disabled for any CF. ie REPLICATION_SCOPE_LOCAL 
One can enable this by setting REPLICATION_SCOPE_GLOBAL on those CFs which has to be replicated.
 So in ur case when the table is having some of the system generated CFs the replication is
OFF on them by default.. And this is what u need.  The other CFs on the user table, which
the user created, the use can opt to set replication scope on those only which he wants. 
So in general I dont see any need NOT to set the replication global scope enabled on any of
the CFs of the table and doing the selective replication of some of the CF data using the
CP hook. That is very very strange usage for me.

Hope am making it clear.  Ya seems in master branch, the place where the scope been set on
the WALKey seems different so there is no reset happening. But setting the scope on the walkey,
I really doubt whether that can be a work of the CPs. This is some thing the core code has
to do.  Even on master, the WALKey is still marked @InterfaceAudience.Private.  Means we really
dont expect CPs to use all methods in that. This is an issue that we over expose things to
CPs. I agree.. So pls consider ur solution 1st I would say.

> Replication entries are not filtered correctly when replication scope is set through
WAL Co-processor
> -----------------------------------------------------------------------------------------------------
>                 Key: HBASE-19423
>                 URL: https://issues.apache.org/jira/browse/HBASE-19423
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Mohammad Arshad
>              Labels: Replication, WAL
>             Fix For: 1.4.0, 1.3.2
>         Attachments: HBASE-19423-branch-1.3-001.patch, HBASE-19423-branch-1.4-001.patch,
> Replicaion scope set in WALObserver is getting reset in Replication.scopeWALEdits().

> Because of this problem custom implementation of WALObserver can not be used as a replication
> Suppose WALObserver implementation has logic to filter all entries from family f2
> {code}
> // Filter all family f2 rows
>   public static class ReplicationFilterWALCoprocessor extends BaseWALObserver {
>     @Override
>     public boolean preWALWrite(ObserverContext<? extends WALCoprocessorEnvironment>
>         HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
>       ArrayList<Cell> cells = logEdit.getCells();
>       for (Cell cell : cells) {
>         byte[] fam = CellUtil.cloneFamily(cell);
>         if ("f2".equals(Bytes.toString(fam))) {
>           NavigableMap<byte[], Integer> scopes = logKey.getScopes();
>           if (scopes == null) {
>             logKey.setScopes(new TreeMap<byte[], Integer>(Bytes.BYTES_COMPARATOR));
>           }
>           logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
>         }
>       }
>       return false;
>     }
>   }
> {code}
> This logic can not work as {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
recreates and populates scopes.
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not have it.
> {code}
> NavigableMap<byte[], Integer> scopes = logKey.getScopes();
>     if (scopes == null) {
>       scopes = new TreeMap<byte[], Integer>(Bytes.BYTES_COMPARATOR);
>     }
> {code}

This message was sent by Atlassian JIRA

View raw message