From hdfs-issues-return-210283-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Feb 14 20:14:28 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5C785180656 for ; Wed, 14 Feb 2018 20:14:28 +0100 (CET) Received: (qmail 97784 invoked by uid 500); 14 Feb 2018 19:14:27 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 97769 invoked by uid 99); 14 Feb 2018 19:14:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Feb 2018 19:14:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 6C45A18015C for ; Wed, 14 Feb 2018 19:14:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Frv6MaDUADzd for ; Wed, 14 Feb 2018 19:14:25 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 4C69B5F36B for ; Wed, 14 Feb 2018 19:14:25 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id DEE40E0248 for ; Wed, 14 Feb 2018 19:14:24 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 8B722247A0 for ; Wed, 14 Feb 2018 19:14:01 +0000 (UTC) Date: Wed, 14 Feb 2018 19:14:01 +0000 (UTC) From: "Gabor Bota (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HDFS-11187) Optimize disk access for last partial chunk checksum of Finalized replica MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HDFS-11187: ------------------------------ Status: Open (was: Patch Available) > Optimize disk access for last partial chunk checksum of Finalized replica > ------------------------------------------------------------------------- > > Key: HDFS-11187 > URL: https://issues.apache.org/jira/browse/HDFS-11187 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Reporter: Wei-Chiu Chuang > Assignee: Gabor Bota > Priority: Major > Fix For: 3.1.0, 3.0.2 > > Attachments: HDFS-11187-branch-2.001.patch, HDFS-11187-branch-2.002.patch, HDFS-11187-branch-2.003.patch, HDFS-11187-branch-2.004.patch, HDFS-11187.001.patch, HDFS-11187.002.patch, HDFS-11187.003.patch, HDFS-11187.004.patch, HDFS-11187.005.patch > > > The patch at HDFS-11160 ensures BlockSender reads the correct version of metafile when there are concurrent writers. > However, the implementation is not optimal, because it must always read the last partial chunk checksum from disk while holding FsDatasetImpl lock for every reader. It is possible to optimize this by keeping an up-to-date version of last partial checksum in-memory and reduce disk access. > I am separating the optimization into a new jira, because maintaining the state of in-memory checksum requires a lot more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org