hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chenglei (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-24625) AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.
Date Mon, 13 Jul 2020 09:10:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-24625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156579#comment-17156579
] 

chenglei edited comment on HBASE-24625 at 7/13/20, 9:09 AM:
------------------------------------------------------------

Open an addendum [PR#2055|https://github.com/apache/hbase/pull/2055] to  incorporate [~ndimiduk]
 's suggestions in [link PR#2034|https://github.com/apache/hbase/pull/2034]  and other minor
changes.
I did not  use the {{AtomicLong.getAndAccumulate}} to replace {{AtomicUtils.updateMax}} because

https://issues.apache.org/jira/browse/HBASE-24625?focusedCommentId=17155993&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17155993


was (Author: comnetwork):
Open an addendum PR [PR#2055|https://github.com/apache/hbase/pull/2055] to  incorporate [~ndimiduk]
 's suggestions in 
[link PR#2034|https://github.com/apache/hbase/pull/2034]  and other minor changes.
I did not  use the {{AtomicLong.getAndAccumulate}} to replace {{AtomicUtils.updateMax}} because

https://issues.apache.org/jira/browse/HBASE-24625?focusedCommentId=17155993&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17155993

> AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.
> ----------------------------------------------------------------------------------------
>
>                 Key: HBASE-24625
>                 URL: https://issues.apache.org/jira/browse/HBASE-24625
>             Project: HBase
>          Issue Type: Bug
>          Components: Replication, wal
>    Affects Versions: 2.1.0, 2.0.0, 2.2.0, 2.3.0
>            Reporter: chenglei
>            Assignee: chenglei
>            Priority: Critical
>             Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6
>
>
> By HBASE-14004, we introduce {{WALFileLengthProvider}} interface to keep the current
writing wal file length  by ourselves,  {{WALEntryStream}} used by {{ReplicationSourceWALReader}}
could only read WAL file byte size <= {{WALFileLengthProvider.getLogFileSizeIfBeingWritten}}
if the WAL file is current been writing on the same RegionServer .
> {{AsyncFSWAL}} implements {{WALFileLengthProvider}}  by {{AbstractFSWAL.getLogFileSizeIfBeingWritten}},
just as folllows :
> {code:java}
>    public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
>     rollWriterLock.lock();
>     try {
>       Path currentPath = getOldPath();
>       if (path.equals(currentPath)) {
>         W writer = this.writer;
>         return writer != null ? OptionalLong.of(writer.getLength()) : OptionalLong.empty();
>       } else {
>         return OptionalLong.empty();
>       }
>     } finally {
>       rollWriterLock.unlock();
>     }
>   }
> {code}
> For {{AsyncFSWAL}},  above {{AsyncFSWAL.writer}}  is {{AsyncProtobufLogWriter}} ,and
{{AsyncProtobufLogWriter.getLength}}  is as follows:
> {code:java}
>     public long getLength() {
>         return length.get();
>     }
> {code}
> But for {{AsyncProtobufLogWriter}}, any append method may increase the above {{AsyncProtobufLogWriter.length}},
especially for following {{AsyncFSWAL.append}}
> method just appending the {{WALEntry}} to {{FanOutOneBlockAsyncDFSOutput.buf}}:
> {code:java}
>      public void append(Entry entry) {
>           int buffered = output.buffered();
>           try {
>               entry.getKey().
>               getBuilder(compressor).setFollowingKvCount(entry.getEdit().size()).build()
>               .writeDelimitedTo(asyncOutputWrapper);
>           } catch (IOException e) {
>              throw new AssertionError("should not happen", e);
>           }
>    
>         try {
>            for (Cell cell : entry.getEdit().getCells()) {
>              cellEncoder.write(cell);
>            }
>           } catch (IOException e) {
>            throw new AssertionError("should not happen", e);
>          }
>          length.addAndGet(output.buffered() - buffered);
>      }
> {code}
> That is to say, {{AsyncFSWAL.getLogFileSizeIfBeingWritten}} could not reflect the  file
length which successfully synced to underlying HDFS, which is not as expected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message