hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15205) Do not find the replication scope for every WAL#append()
Date Thu, 18 Feb 2016 07:18:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15151851#comment-15151851
] 

ramkrishna.s.vasudevan commented on HBASE-15205:
------------------------------------------------

bq.From above review and still prevalent in v4 of the patch: "Sometimes you use getScopes.
Other times you use this.replicationScopes." Use the method always I'd say.
Sorry for the oversight. I read it as I was using getScopes and also getReplicationScopes.
 So in the last patch I unified and made getreplicationScopes every where. Now I got what
you said.
bq.In WALKey, can this be package private 421	public NavigableMap<byte[], Integer> getReplicationScopes()
{
This was not added by me. I only renamed and made it to getReplicationScopes from getScopes.
Anyway I can remove it.
bq. FSWALEntry has repilcation scopes added by this patch. You seem to be duplicating scopes
by adding to WALEdit replication scopes again? i.e. A FSWALEntry has scopes recorded twice...
once in FSWALEntry, and then again in WALKey?
In the existing code the replicationScope was set in WALKey from the HTD. Since we had removed
that I wanted it to be passed over from FSWalEntry. Because
{code}
          for (WALActionsListener i: listeners) {
            i.visitLogEntryBeforeWrite(entry.getReplicationScope(), entry.getKey(),
              entry.getEdit());
          }
{code}
This method FSHLog.append(FSWalEntry) accepts a WALEntry. So you want me to change in such
a way that during WALKey construction itself pass the replicationScope?  ACtually can do that
but the WALKey has an explicit setter for replicationScope (already available) just because
for DLR and in region replicas case you don want the replication scope to be added to the
WALKey. Now if we use a constructor then  may be though we pass the replicationScope during
construction of WALKey again we may need to set it to null if the WALEdit says it is for region
replica or during replay. Also this way is better considering the case when there is only
bulk load replication then at that time there are chances that when there are no bulk load
entries in WAL there is nothing to be replicated and hence we need not set the replicaitonScope
to the WALKey. Hence passing on the replicationScope to the WALKey thro FSWALEntry seems fine
to me.
bq.There is convention for naming of keys used internally only. IIRC, you prefix with an '_'.
Before updating the next patch you still want me to go with the way of setting the scope on
HTD and use the same HTD to retrive the scope per family? But I fear it wont serve the purpose
of short living objects because the HTD.setValue() will convert things to Bytes. So every
time we add it to the PB serde 
{code}
        builder.addScopes(FamilyScope.newBuilder()
            .setFamily(family).setScopeType(ScopeType.valueOf(e.getValue())));
{code}
We need to get the value from that HTD map and to convert it into Integer again we need to
do Bytes.toInt(byte[]).

> Do not find the replication scope for every WAL#append()
> --------------------------------------------------------
>
>                 Key: HBASE-15205
>                 URL: https://issues.apache.org/jira/browse/HBASE-15205
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 2.0.0
>
>         Attachments: HBASE-15205.patch, HBASE-15205_1.patch, HBASE-15205_2.patch, HBASE-15205_3.patch,
HBASE-15205_4.patch, ScopeWALEdits.jpg, ScopeWALEdits_afterpatch.jpg
>
>
> After the byte[] and char[] the other top contributor for lot of GC (though it is only
2.86%) is the UTF_8.newDecoder.
> This happens because for every WAL append we try to calculate the replication scope associate
with the families associated with the TableDescriptor. I think per WAL append doing this is
very costly and creates lot of garbage. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message