From common-issues-return-151600-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Apr 25 13:33:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B2EF7180676 for ; Wed, 25 Apr 2018 13:33:04 +0200 (CEST) Received: (qmail 98843 invoked by uid 500); 25 Apr 2018 11:33:03 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 98832 invoked by uid 99); 25 Apr 2018 11:33:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Apr 2018 11:33:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 3CA56C05EA for ; Wed, 25 Apr 2018 11:33:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id ihyM1kWJ4S2Q for ; Wed, 25 Apr 2018 11:33:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 2E0125FB9C for ; Wed, 25 Apr 2018 11:33:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 57905E0047 for ; Wed, 25 Apr 2018 11:33:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 17A3521208 for ; Wed, 25 Apr 2018 11:33:00 +0000 (UTC) Date: Wed, 25 Apr 2018 11:33:00 +0000 (UTC) From: "Wei-Chiu Chuang (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-15412) Hadoop KMS with HDFS keystore: No FileSystem for scheme "hdfs" MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-15412?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16= 452084#comment-16452084 ]=20 Wei-Chiu Chuang commented on HADOOP-15412: ------------------------------------------ If I understand it correctly, you wanted to implement KMS-HA using HDFS for= storing keystore? While I=C2=A0can conceive=C2=A0that=C2=A0as a simple & quick solution, it m= akes little sense to store keystore in an unencrypted HDFS cluster. It also= violates the initial design principal =E2=80=93 separation of duty. With t= he keystore in non-EZ, A hdfs admin can easily decrypt anything in the clus= ter, voiding the need of KMS. =C2=A0 KMS HA is not a trivial task. Please consult this doc for reference:=C2=A0h= ttps://hadoop.apache.org/docs/current/hadoop-kms/index.html#High_Availabili= ty > Hadoop KMS with HDFS keystore: No FileSystem for scheme "hdfs" > -------------------------------------------------------------- > > Key: HADOOP-15412 > URL: https://issues.apache.org/jira/browse/HADOOP-15412 > Project: Hadoop Common > Issue Type: Bug > Components: kms > Affects Versions: 2.7.2, 2.9.0 > Environment: RHEL 7.3 > Hadoop 2.7.2 and 2.7.9 > =C2=A0 > Reporter: Pablo San Jos=C3=A9 > Priority: Major > > I have been trying to configure the Hadoop kms to use hdfs as the key pro= vider but it seems that this functionality is failing.=C2=A0 > I followed the Hadoop docs for that matter, and I added the following fie= ld to my kms-site.xml: > {code:java} > =20 > hadoop.kms.key.provider.uri > jceks://hdfs@nn1.example.com/kms/test.jceks=20 > =20 > URI of the backing KeyProvider for the KMS.=20 > =20 > {code} > That route exists in hdfs, and I expect the kms to create the file test.j= ceks for its keystore. However, the kms failed to start due to this error: > {code:java} > ERROR: Hadoop KMS could not be started REASON: org.apache.hadoop.fs.Unsup= portedFileSystemException: No FileSystem for scheme "hdfs" Stacktrace: ----= ----------------------------------------------- org.apache.hadoop.fs.Unsupp= ortedFileSystemException: No FileSystem for scheme "hdfs" at org.apache.had= oop.fs.FileSystem.getFileSystemClass(FileSystem.java:3220) at org.apache.ha= doop.fs.FileSystem.createFileSystem(FileSystem.java:3240) at org.apache.had= oop.fs.FileSystem.access$200(FileSystem.java:121) at org.apache.hadoop.fs.F= ileSystem$Cache.getInternal(FileSystem.java:3291) at org.apache.hadoop.fs.F= ileSystem$Cache.get(FileSystem.java:3259) at org.apache.hadoop.fs.FileSyste= m.get(FileSystem.java:470) at org.apache.hadoop.fs.Path.getFileSystem(Path.= java:356) at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.(JavaK= eyStoreProvider.java:132) at org.apache.hadoop.crypto.key.JavaKeyStoreProvi= der.(JavaKeyStoreProvider.java:88) at org.apache.hadoop.crypto.key.Ja= vaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:660) at= org.apache.hadoop.crypto.key.KeyProviderFactory.get(KeyProviderFactory.jav= a:96) at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitializ= ed(KMSWebApp.java:187) at org.apache.catalina.core.StandardContext.listener= Start(StandardContext.java:4276) at org.apache.catalina.core.StandardContex= t.start(StandardContext.java:4779) at org.apache.catalina.core.ContainerBas= e.addChildInternal(ContainerBase.java:803) at org.apache.catalina.core.Cont= ainerBase.addChild(ContainerBase.java:780) at org.apache.catalina.core.Stan= dardHost.addChild(StandardHost.java:583) at org.apache.catalina.startup.Hos= tConfig.deployDirectory(HostConfig.java:1080) at org.apache.catalina.startu= p.HostConfig.deployDirectories(HostConfig.java:1003) at org.apache.catalina= .startup.HostConfig.deployApps(HostConfig.java:507) at org.apache.catalina.= startup.HostConfig.start(HostConfig.java:1322) at org.apache.catalina.start= up.HostConfig.lifecycleEvent(HostConfig.java:325) at org.apache.catalina.ut= il.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.ap= ache.catalina.core.ContainerBase.start(ContainerBase.java:1069) at org.apac= he.catalina.core.StandardHost.start(StandardHost.java:822) at org.apache.ca= talina.core.ContainerBase.start(ContainerBase.java:1061) at org.apache.cata= lina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catal= ina.core.StandardService.start(StandardService.java:525) at org.apache.cata= lina.core.StandardServer.start(StandardServer.java:761) at org.apache.catal= ina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAc= cessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.i= nvoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAcce= ssorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.= Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.sta= rt(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootst= rap.java:414){code} > =C2=A0 > For what I could manage to understand, it seems that this error is becaus= e there is no FileSystem implemented for HDFS. I have looked up this error = but it always refers to a lack of jars for the hdfs-client when upgrading, = which I have not done (it is a fresh installation). I=C2=A0have tested it u= sing Hadoop 2.7.2 and 2.9.0 > Thank you in advance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org