From issues-return-21687-archive-asf-public=cust-asf.ponee.io@kylin.apache.org Thu Nov 15 08:48:46 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5FB3C180669 for ; Thu, 15 Nov 2018 08:48:45 +0100 (CET) Received: (qmail 14286 invoked by uid 500); 15 Nov 2018 07:48:44 -0000 Mailing-List: contact issues-help@kylin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kylin.apache.org Delivered-To: mailing list issues@kylin.apache.org Received: (qmail 14276 invoked by uid 99); 15 Nov 2018 07:48:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Nov 2018 07:48:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E61CD182470 for ; Thu, 15 Nov 2018 07:48:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id wUbS9SbhghoI for ; Thu, 15 Nov 2018 07:48:42 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id C5E7A62433 for ; Thu, 15 Nov 2018 07:38:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 50DEBE0110 for ; Thu, 15 Nov 2018 07:38:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 17EC121354 for ; Thu, 15 Nov 2018 07:38:00 +0000 (UTC) Date: Thu, 15 Nov 2018 07:38:00 +0000 (UTC) From: "Jacky Woo (JIRA)" To: issues@kylin.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (KYLIN-3675) Unknown host exception when building cube of create HTable steps. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/KYLIN-3675?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1668= 7603#comment-16687603 ]=20 Jacky Woo commented on KYLIN-3675: ---------------------------------- u can update to v2.5.1 which fixes the bug. > Unknown host exception when building cube of create HTable steps. > ----------------------------------------------------------------- > > Key: KYLIN-3675 > URL: https://issues.apache.org/jira/browse/KYLIN-3675 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Storage - HBase > Affects Versions: v2.5.0 > Reporter: Jacky Woo > Priority: Major > Attachments: hbase.hdfs.xml, kylin.properties > > > Hi, all > =C2=A0 > I had "UnknownHostException" when building cube. Below is stack trace: > =C2=A0 > {panel:title=3Dstack trace:} > 2018-11-08 18:44:55,069 ERROR [Scheduler 321750220 Job 42a75dbe-4b37-bb8a= -8361-0bab7bcea106-849] common.HadoopShellExecutable:65 : error execute Had= oopShellExecutable{id=3D42a75dbe-4b37-bb8a-8361-0bab7bcea106-04, name=3DCre= ate HTable, st > ate=3DRUNNING} > org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.Do= NotRetryIOException: java.net.UnknownHostException: data-batch-hdfs Set hba= se.table.sanity.checks to false at conf or table descriptor if you want to = bypass san > ity checks > at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure= (HMaster.java:1785) > at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMa= ster.java:1646) > at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1576) > at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRp= cServices.java:469) > at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService= $2.callBlockingMethod(MasterProtos.java:55682) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2183) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:= 185) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:= 165) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct= orAccessorImpl.java:62) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC= onstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExce= ption.java:106) > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExc= eption.java:95) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(R= pcRetryingCaller.java:236) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(R= pcRetryingCaller.java:254) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcR= etryingCaller.java:150) > at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.= java:4313) > at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.= java:4305) > at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdm= in.java:768) > at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java= :689) > at org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(Cube= HTableUtil.java:107) > at org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTable= Job.java:120) > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92) > at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(Hadoop= ShellExecutable.java:63) > at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExe= cutable.java:163) > at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(Defaul= tChainedExecutable.java:69) > at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExe= cutable.java:163) > at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.r= un(DistributedScheduler.java:111) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) > at java.lang.Thread.run(Thread.java:745) > {panel} > =C2=A0 > =C2=A0 > I use kylin 2.5.0.cdh.5.7 > There are two hdfs in my deployment: > =C2=A0 =C2=A0 =C2=A0 =C2=A0 "data-batch-hdfs" : for cube building > =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0"hbase-hdfs" : for hbase > And I use=C2=A0"data-batch-hdfs" as defaultFS in core-site.xml. > =C2=A0 > {color:#FF0000}*The error occurs at "Create HTable"=C2=A0 step, but it is= =C2=A0incidental not always.*{color} > =C2=A0 > I did some research: > =C2=A0 > 1. According the error stack trace, error is thrown from "CubeHTableUtil.= createHTable" method. > So I added some log to print HBase table desc and found coprocessor jar l= ocation is in "data-batch-hdfs" which is not expected.(coprocessor should l= ocate in "hbase-hdfs") > =C2=A0 > 2. So I added more logs in "DeployCoprocessorCli.initHTableCoprocessor" m= ethod. I print which hdfs to use and all HBase configurations of all thread= s.=C2=A0Below is code : > =C2=A0 > {code:java} > private static void initHTableCoprocessor(HTableDescriptor desc) throws I= OException { > KylinConfig kylinConfig =3D KylinConfig.getInstanceFromEnv(); > Configuration hconf =3D HBaseConnection.getCurrentHBaseConfiguration(); > FileSystem fileSystem =3D FileSystem.get(hconf); > String localCoprocessorJar =3D kylinConfig.getCoprocessorLocalJar(); > Path hdfsCoprocessorJar =3D DeployCoprocessorCLI.uploadCoprocessorJar(lo= calCoprocessorJar, fileSystem, null); > if(fileSystem.getUri().toString().contains("data-batch-hdfs")){ > logger.info("use hdfs " + hconf.get(FileSystem.FS_DEFAULT_NAME_KEY)); > logger.info(String.format("use hdfs %s when deploy coprocessor, current = thread %s", > fileSystem.getUri().toString(), Thread.currentThread().getId()+"-"+Threa= d.currentThread().getName())); > =20 > //pring HBaseConnection.configThreadLocal of all threads > Map hbaseConfs =3D ThreadUtils.listThreadLocal(HB= aseConnection.configThreadLocal); > StringBuffer sb =3D new StringBuffer(); > for(Map.Entry e : hbaseConfs.entrySet()){ > sb.append("\n\t").append(e.getKey()).append(" : ") > .append(e.getValue().get(FileSystem.FS_DEFAULT_NAME_KEY)).append("\t") > .append(e.getValue().get(DFSConfigKeys.DFS_NAMESERVICES)); > } > sb.append("\n"); > logger.info("HBaseConnection configThreadLocal : " + sb.toString()); > } > logger.info("coprocessor path " + hdfsCoprocessorJar); > DeployCoprocessorCLI.addCoprocessorOnHTable(desc, hdfsCoprocessorJar); > } > {code} > =C2=A0 > 3. I startup some building jobs And got some log: > =C2=A0 > {code:java} > 2018-11-08 18:44:55,002 INFO [Scheduler 321750220 Job 42a75dbe-4b37-bb8a-= 8361-0bab7bcea106-849] util.DeployCoprocessorCLI:275 : use hdfs hdfs://data= -batch-hdfs > 2018-11-08 18:44:55,002 INFO [Scheduler 321750220 Job 42a75dbe-4b37-bb8a-= 8361-0bab7bcea106-849] util.DeployCoprocessorCLI:276 : use hdfs hdfs://data= -batch-hdfs when deploy coprocessor, current thread 849-Scheduler 321750220= Job 42a7 > 5dbe-4b37-bb8a-8361-0bab7bcea106-849 > 2018-11-08 18:44:55,012 INFO [Scheduler 321750220 Job 42a75dbe-4b37-bb8a-= 8361-0bab7bcea106-849] util.DeployCoprocessorCLI:288 : HBaseConnection conf= igThreadLocal : > 849##Scheduler 321750220 Job 42a75dbe-4b37-bb8a-8361-0bab7bcea106-849 : = hdfs://data-batch-hdfs data-batch-hdfs,hbase-hdfs > 404##pool-12-thread-9 : hdfs://hbase-hdfs data-batch-hdfs,hbase-hdfs > 1231##pool-12-thread-20 : hdfs://data-batch-hdfs data-batch-hdfs,hbase-h= dfs > 383##pool-12-thread-8 : hdfs://hbase-hdfs data-batch-hdfs,hbase-hdfs > 1073##pool-12-thread-19 : hdfs://data-batch-hdfs data-batch-hdfs,hbase-h= dfs > 538##pool-12-thread-13 : hdfs://hbase-hdfs data-batch-hdfs,hbase-hdfs > {code} > =C2=A0 > =C2=A0 > According the logic of=C2=A0 "HBaseConnection",=C2=A0 HBase configuration= of all thread will use "hbase-hdfs" but not "data-batch-hdfs". > What is the reason? > =C2=A0 > Attachment is my kylin configuration and hdfs configuration of hbase clus= ter. > =C2=A0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)