From hdfs-issues-return-285583-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri Sep 27 23:53:03 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5518C180638 for ; Sat, 28 Sep 2019 01:53:03 +0200 (CEST) Received: (qmail 88219 invoked by uid 500); 27 Sep 2019 23:53:02 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 88207 invoked by uid 99); 27 Sep 2019 23:53:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Sep 2019 23:53:02 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 50E10E30FA for ; Fri, 27 Sep 2019 23:53:01 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 2D56B78074B for ; Fri, 27 Sep 2019 23:53:00 +0000 (UTC) Date: Fri, 27 Sep 2019 23:53:00 +0000 (UTC) From: "Konstantin Shvachko (Jira)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-14305?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1693= 9793#comment-16939793 ]=20 Konstantin Shvachko commented on HDFS-14305: -------------------------------------------- Attached v07 patch. This one goes after reverting the current commit (v06). In v07 patch: # Ensure we do not use negative numbers as initial serialNo. # Added an assert verifying new serialNo if always in the range. # Added unit test, which tests the range conditions. # Removed double initialization of the keys - {{generateKeys()}} was called= twice. # Added logging of the range when NN starts, this should cover HDFS-14793. If there are no objections for reverting I'll do it over the weekend. > Serial number in BlockTokenSecretManager could overlap between different = namenodes > -------------------------------------------------------------------------= --------- > > Key: HDFS-14305 > URL: https://issues.apache.org/jira/browse/HDFS-14305 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security > Reporter: Chao Sun > Assignee: Xiaoqiao He > Priority: Major > Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-14305-007.patch, HDFS-14305.001.patch, HDFS-143= 05.002.patch, HDFS-14305.003.patch, HDFS-14305.004.patch, HDFS-14305.005.pa= tch, HDFS-14305.006.patch > > > Currently, a {{BlockTokenSecretManager}} starts with a random integer as = the initial serial number, and then use this formula to rotate it: > {code:java} > this.intRange =3D Integer.MAX_VALUE / numNNs; > this.nnRangeStart =3D intRange * nnIndex; > this.serialNo =3D (this.serialNo % intRange) + (nnRangeStart); > =C2=A0{code} > while {{numNNs}} is the total number of NameNodes in the cluster, and {{n= nIndex}} is the index of the current NameNode specified in the configuratio= n {{dfs.ha.namenodes.}}. > However, with this approach, different NameNode could have overlapping ra= nges for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} = is 100, and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then = the ranges for these two are: > {code} > nn1 -> [-49, 49] > nn2 -> [1, 99] > {code} > This is because the initial serial number could be any negative integer. > Moreover, when the keys are updated, the serial number will again be upda= ted with the formula: > {code} > this.serialNo =3D (this.serialNo % intRange) + (nnRangeStart); > {code} > which means the new serial number could be updated to a range that belong= s to a different NameNode, thus increasing the chance of collision again. > When the collision happens, DataNodes could overwrite an existing key whi= ch will cause clients to fail because of {{InvalidToken}} error. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org