From dev-return-80472-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Mon Aug 10 02:24:05 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 5736F18063F for ; Mon, 10 Aug 2020 04:24:05 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id D75F6124AD5 for ; Mon, 10 Aug 2020 02:24:03 +0000 (UTC) Received: (qmail 81880 invoked by uid 500); 10 Aug 2020 02:24:03 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 81840 invoked by uid 99); 10 Aug 2020 02:24:03 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2020 02:24:03 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 04DBC43D7F for ; Mon, 10 Aug 2020 02:24:02 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 8343A7809BF for ; Mon, 10 Aug 2020 02:24:00 +0000 (UTC) Date: Mon, 10 Aug 2020 02:24:00 +0000 (UTC) From: "Duo Zhang (Jira)" To: dev@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (HBASE-24625) AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-24625?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-24625. ------------------------------- Resolution: Fixed Pushed the addendum to branch-2.2+. > AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected sync= ed file length. > -------------------------------------------------------------------------= --------------- > > Key: HBASE-24625 > URL: https://issues.apache.org/jira/browse/HBASE-24625 > Project: HBase > Issue Type: Bug > Components: Replication, wal > Affects Versions: 2.1.0, 2.0.0, 2.2.0, 2.3.0 > Reporter: chenglei > Assignee: chenglei > Priority: Critical > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6 > > > By HBASE-14004, we introduce {{WALFileLengthProvider}} interface to keep = the current writing wal file length by ourselves, {{WALEntryStream}} used= by {{ReplicationSourceWALReader}} could only read WAL file byte size <=3D = {{WALFileLengthProvider.getLogFileSizeIfBeingWritten}} if the WAL file is c= urrent been writing on the same RegionServer . > {{AsyncFSWAL}} implements {{WALFileLengthProvider}} by {{AbstractFSWAL.g= etLogFileSizeIfBeingWritten}}, just as folllows : > {code:java} > public OptionalLong getLogFileSizeIfBeingWritten(Path path) { > rollWriterLock.lock(); > try { > Path currentPath =3D getOldPath(); > if (path.equals(currentPath)) { > W writer =3D this.writer; > return writer !=3D null ? OptionalLong.of(writer.getLength()) : O= ptionalLong.empty(); > } else { > return OptionalLong.empty(); > } > } finally { > rollWriterLock.unlock(); > } > } > {code} > For {{AsyncFSWAL}}, above {{AsyncFSWAL.writer}} is {{AsyncProtobufLogWr= iter}} =EF=BC=8Cand {{AsyncProtobufLogWriter.getLength}} is as follows: > {code:java} > public long getLength() { > return length.get(); > } > {code} > But for {{AsyncProtobufLogWriter}}, any append method may increase the ab= ove {{AsyncProtobufLogWriter.length}}, especially for following {{AsyncFSWA= L.append}} > method just appending the {{WALEntry}} to {{FanOutOneBlockAsyncDFSOutput.= buf}}: > {code:java} > public void append(Entry entry) { > int buffered =3D output.buffered(); > try { > entry.getKey(). > getBuilder(compressor).setFollowingKvCount(entry.getEdit().= size()).build() > .writeDelimitedTo(asyncOutputWrapper); > } catch (IOException e) { > throw new AssertionError("should not happen", e); > } > =20 > try { > for (Cell cell : entry.getEdit().getCells()) { > cellEncoder.write(cell); > } > } catch (IOException e) { > throw new AssertionError("should not happen", e); > } > length.addAndGet(output.buffered() - buffered); > } > {code} > That is to say, {{AsyncFSWAL.getLogFileSizeIfBeingWritten}} could not ref= lect the file length which successfully synced to underlying HDFS, which i= s not as expected. -- This message was sent by Atlassian Jira (v8.3.4#803005)