Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 27012200C65 for ; Wed, 22 Mar 2017 07:09:58 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 25AA8160B9B; Wed, 22 Mar 2017 06:09:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D1170160BA6 for ; Wed, 22 Mar 2017 07:09:55 +0100 (CET) Received: (qmail 82760 invoked by uid 500); 22 Mar 2017 06:09:55 -0000 Mailing-List: contact commits-help@bigtop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: bigtop-dev@bigtop.apache.org Delivered-To: mailing list commits@bigtop.apache.org Received: (qmail 81284 invoked by uid 99); 22 Mar 2017 06:09:52 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Mar 2017 06:09:52 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 96E4DDFF56; Wed, 22 Mar 2017 06:09:52 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: rvs@apache.org To: commits@bigtop.apache.org Date: Wed, 22 Mar 2017 06:10:14 -0000 Message-Id: <520c9fb201b34a0da5bb9cd91f6d385b@git.apache.org> In-Reply-To: <69863c16852949b9998cae3af818e0a6@git.apache.org> References: <69863c16852949b9998cae3af818e0a6@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [23/52] bigtop git commit: ODPI-193. Add Hive 1.2 to ODPi ambari reference implementation archived-at: Wed, 22 Mar 2017 06:09:58 -0000 http://git-wip-us.apache.org/repos/asf/bigtop/blob/ba8d7f50/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-interactive-site.xml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-interactive-site.xml b/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-interactive-site.xml new file mode 100755 index 0000000..27d3541 --- /dev/null +++ b/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-interactive-site.xml @@ -0,0 +1,909 @@ + + hive.server2.thrift.port + 10500 + + TCP port number to listen on, default 10500. + + HiveServer2 Port + hive-interactive-site.xml + false + + + + int + false + + + + + hive.server2.thrift.http.port + 10501 + Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'http'. + hive-interactive-site.xml + false + + + + + + + hive.server2.tez.sessions.per.default.queue + 1 + + The maximum number of queries the Hive Interactive cluster will be able to handle concurrently. + + Maximum Total Concurrent Queries + hive-interactive-site.xml + false + + + + int + 10 + 1 + 1 + + + + yarn.scheduler.capacity.root.queues + capacity-scheduler + + + hive.llap.daemon.queue.name + hive-interactive-site + + + llap_queue_capacity + hive-interactive-env + + + enable_hive_interactive + hive-interactive-env + + + + + hive.llap.daemon.num.executors + hive-interactive-site + + + llap_heap_size + hive-interactive-env + + + tez.am.resource.memory.mb + tez-interactive-site + + + hive.llap.daemon.yarn.container.mb + hive-interactive-site + + + hive.llap.io.memory.size + hive-interactive-site + + + num_llap_nodes + hive-interactive-env + + + slider_am_container_mb + hive-interactive-env + + + + hive.metastore.uris + + Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore. + hive-interactive-site.xml + false + + + + true + + + + + hive.server2.enable.doAs + false + + Setting this property to true will have HiveServer2 execute + Hive operations as the user making the calls to it. + + Run as end user instead of Hive user + hive-interactive-site.xml + false + + + + value-list + + + true + True + + + false + False + + + 1 + + + + hive_security_authorization + hive-env + + + + + hive.prewarm.enabled + false + Enables container prewarm for Tez (Hadoop 2 only) + Hold Containers to Reduce Latency + hive-interactive-site.xml + false + + + + value-list + + + true + True + + + false + False + + + 1 + + + + + hive.vectorized.execution.reduce.enabled + true + + This flag should be set to true to enable vectorized mode of the reduce-side of + query execution. + + Enable Reduce Vectorization + hive-interactive-site.xml + false + + + + value-list + + + true + True + + + false + False + + + 1 + + + + + hive.server2.tez.default.queues + default + + A list of comma separated values corresponding to YARN queues of the same name. + When HiveServer2 is launched in Tez mode, this configuration needs to be set + for multiple Tez sessions to run in parallel on the cluster. + + Default query queues + hive-interactive-site.xml + false + + + + combo + + + default + Default + + + 1+ + + + + yarn.scheduler.capacity.root.queues + capacity-scheduler + + + hive.llap.daemon.queue.name + hive-interactive-site + + + + + hive.server2.tez.initialize.default.sessions + true + + This flag is used in HiveServer2 to enable a user to use HiveServer2 without + turning on Tez for HiveServer2. The user could potentially want to run queries + over Tez without the pool of sessions. + + Start Tez session at Initialization + hive-interactive-site.xml + false + + + + value-list + + + true + True + + + false + False + + + 1 + + + + + hive.driver.parallel.compilation + true + + This flag allows HiveServer2 to compile queries in parallel. + + Compile queries in parallel + hive-interactive-site.xml + false + + + + value-list + + + true + True + + + false + False + + + 1 + + + + + hive.server2.webui.port + 10502 + Web UI port address + hive-interactive-site.xml + false + + + + + + + hive.server2.webui.use.ssl + false + Enable SSL for HiveServer2 Interactive + hive-interactive-site.xml + false + + + + + + + hive.server2.zookeeper.namespace + hiveserver2-hive2 + The parent node in ZooKeeper used by HiveServer2 when supporting dynamic service discovery. + hive-interactive-site.xml + false + + + + + + + hive.llap.daemon.queue.name + default + Choose the YARN queue in this cluster that is dedicated to interactive query. + Interactive Query Queue + hive-interactive-site.xml + false + + + + combo + + + default + Default + + + 1 + + + + yarn.scheduler.capacity.root.queues + capacity-scheduler + + + enable_hive_interactive + hive-interactive-env + + + + + hive.server2.tez.default.queues + hive-interactive-site + + + hive.llap.daemon.num.executors + hive-interactive-site + + + llap_queue_capacity + hive-interactive-env + + + llap_heap_size + hive-interactive-env + + + tez.am.resource.memory.mb + tez-interactive-site + + + hive.llap.daemon.yarn.container.mb + hive-interactive-site + + + hive.server2.tez.sessions.per.default.queue + hive-interactive-site + + + hive.llap.io.memory.size + hive-interactive-site + + + num_llap_nodes + hive-interactive-env + + + slider_am_container_mb + hive-interactive-env + + + + hive.llap.daemon.yarn.shuffle.port + 15551 + YARN shuffle port for LLAP-daemon-hosted shuffle. + hive-interactive-site.xml + false + + + + + + + hive.execution.engine + tez + hive-interactive-site.xml + false + + + + + + + hive.execution.mode + llap + Chooses whether query fragments will run in container or in llap + hive-interactive-site.xml + false + + + + + + + hive.llap.execution.mode + all + Chooses which fragments of a query will run in llap + hive-interactive-site.xml + false + + + + + + + hive.llap.io.enabled + true + Whether the LLAP IO layer is enabled. + hive-interactive-site.xml + false + + + + + + hive.llap.io.memory.size + hive-interactive-site + + + + + hive.llap.io.use.lrfu + true + Whether ORC low-level cache should use LRFU cache policy instead of default (FIFO). + hive-interactive-site.xml + false + + + + + + + hive.llap.auto.allow.uber + false + Whether or not to allow the planner to run vertices in the AM. + hive-interactive-site.xml + false + + + + + + + hive.llap.object.cache.enabled + true + Cache objects (plans, hashtables, etc) in llap + hive-interactive-site.xml + false + + + + + + + hive.tez.input.generate.consistent.splits + true + Whether to generate consistent split locations when generating splits in the AM + hive-interactive-site.xml + false + + + + + + + hive.llap.client.consistent.splits + true + + Whether to setup split locations to match nodes on which llap daemons are running, + instead of using the locations provided by the split itself. + + hive-interactive-site.xml + false + + + + + + + hive.llap.task.scheduler.locality.delay + -1 + + Amount of time to wait before allocating a request which contains location information, + to a location other than the ones requested. Set to -1 for an infinite delay, 0 + for no delay. + + hive-interactive-site.xml + false + + + + + + + hive.exec.orc.split.strategy + HYBRID + + This is not a user level config. BI strategy is used when the requirement is to spend less time in split generation + as opposed to query execution (split generation does not read or cache file footers). + ETL strategy is used when spending little more time in split generation is acceptable + (split generation reads and caches file footers). HYBRID chooses between the above strategies + based on heuristics. + + hive-interactive-site.xml + false + + + + + + + hive.llap.daemon.service.hosts + @llap0 + + Explicitly specified hosts to use for LLAP scheduling. Useful for testing. By default, + YARN registry is used. + + hive-interactive-site.xml + false + + + + + + + hive.llap.daemon.allow.permanent.fns + false + Whether LLAP daemon should localize the resources for permanent UDFs. + hive-interactive-site.xml + false + + + + + + + hive.llap.io.memory.size + 0 + The amount of memory reserved for Hive's optimized in-memory cache. + In-Memory Cache per Daemon + hive-interactive-site.xml + false + + + + int + MB + false + + + + tez.am.resource.memory.mb + tez-site + + + yarn.scheduler.capacity.root.queues + capacity-scheduler + + + hive.llap.daemon.queue.name + hive-interactive-site + + + llap_queue_capacity + hive-interactive-env + + + enable_hive_interactive + hive-interactive-env + + + hive.server2.tez.sessions.per.default.queue + hive-interactive-site + + + + + hive.llap.io.enabled + hive-interactive-site + + + + hive.llap.daemon.num.executors + 1 + The maximum number of CPUs a single LLAP daemon will use. Usually this should be equal to the number of available CPUs. + Maximum CPUs per Daemon + hive-interactive-site.xml + false + + + + int + + + + tez.am.resource.memory.mb + tez-site + + + yarn.scheduler.capacity.root.queues + capacity-scheduler + + + hive.llap.daemon.queue.name + hive-interactive-site + + + llap_queue_capacity + hive-interactive-env + + + enable_hive_interactive + hive-interactive-env + + + hive.server2.tez.sessions.per.default.queue + hive-interactive-site + + + + + hive.llap.io.threadpool.size + hive-interactive-site + + + + hive.llap.daemon.vcpus.per.instance + ${hive.llap.daemon.num.executors} + The total number of vcpus to use for the executors inside LLAP. + hive-interactive-site.xml + false + + + + + + + hive.llap.daemon.yarn.container.mb + 341 + Total memory used by individual LLAP daemons. This includes memory for the cache as well as for the query execution. + Memory per daemon + hive-interactive-site.xml + false + + + + int + MB + false + + + + yarn.scheduler.capacity.root.queues + capacity-scheduler + + + hive.llap.daemon.queue.name + hive-interactive-site + + + llap_queue_capacity + hive-interactive-env + + + enable_hive_interactive + hive-interactive-env + + + hive.server2.tez.sessions.per.default.queue + hive-interactive-site + + + + + llap.shuffle.connection-keep-alive.enable + true + hive-interactive-site.xml + false + + + + + + + llap.shuffle.connection-keep-alive.timeout + 60 + hive-interactive-site.xml + false + + + + + + + hive.llap.io.threadpool.size + 2 + Specify the number of threads to use for low-level IO thread pool. + hive-interactive-site.xml + false + + + + + + hive.llap.daemon.num.executors + hive-interactive-site + + + + + hive.llap.daemon.rpc.port + 15001 + The LLAP daemon RPC port. + hive-interactive-site.xml + false + + + + + + + hive.llap.management.rpc.port + 15004 + RPC port for LLAP daemon management service. + hive-interactive-site.xml + false + + + + + + + hive.llap.daemon.task.scheduler.enable.preemption + true + hive.llap.daemon.task.scheduler.enable.preemption + hive-interactive-site.xml + false + + + + + + + hive.tez.exec.print.summary + true + Display breakdown of execution steps, for every query executed by the shell. + hive-interactive-site.xml + false + + + + + + + hive.vectorized.execution.mapjoin.native.enabled + true + + This flag should be set to true to enable native (i.e. non-pass through) vectorization + of queries using MapJoin. + + hive-interactive-site.xml + false + + + + + + + hive.vectorized.execution.mapjoin.minmax.enabled + true + + This flag should be set to true to enable vector map join hash tables to + use max / max filtering for integer join queries using MapJoin. + + hive-interactive-site.xml + false + + + + + + + hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled + true + + This flag should be set to true to enable use of native fast vector map join hash tables in + queries using MapJoin. + + hive-interactive-site.xml + false + + + + + + + hive.mapjoin.hybridgrace.hashtable + false + Whether to use hybrid grace hash join as the join method for mapjoin. Tez only. + hive-interactive-site.xml + false + + + + + + + hive.tez.bucket.pruning + true + + When pruning is enabled, filters on bucket columns will be processed by + filtering the splits against a bitset of included buckets. This needs predicates + produced by hive.optimize.ppd and hive.optimize.index.filters. + + hive-interactive-site.xml + false + + + + + + + hive.optimize.dynamic.partition.hashjoin + true + + Whether to enable dynamically partitioned hash join optimization. + This setting is also dependent on enabling hive.auto.convert.join + + hive-interactive-site.xml + false + + + + + + + hive.llap.zk.sm.connectionString + localhost:2181 + ZooKeeper connection string for ZooKeeper SecretManager. + hive-interactive-site.xml + false + + + + + + clientPort + zoo.cfg + + + + + hive.llap.io.memory.mode + + + LLAP IO memory usage; 'cache' (the default) uses data and metadata cache with a + custom off-heap allocator, 'allocator' uses the custom allocator without the caches, + 'none' doesn't use either (this mode may result in significant performance degradation) + + hive-interactive-site.xml + false + + + + true + + + + + hive.metastore.event.listeners + + + Listeners for metastore events + + hive-interactive-site.xml + false + + + + true + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/bigtop/blob/ba8d7f50/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-log4j.xml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-log4j.xml b/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-log4j.xml new file mode 100755 index 0000000..3ecb24a --- /dev/null +++ b/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-log4j.xml @@ -0,0 +1,106 @@ + + content + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Define some default values that can be overridden by system properties +hive.log.threshold=ALL +hive.root.logger=INFO,DRFA +hive.log.dir=${java.io.tmpdir}/${user.name} +hive.log.file=hive.log + +# Define the root logger to the system property "hadoop.root.logger". +log4j.rootLogger=${hive.root.logger}, EventCounter + +# Logging Threshold +log4j.threshold=${hive.log.threshold} + +# +# Daily Rolling File Appender +# +# Use the PidDailyerRollingFileAppend class instead if you want to use separate log files +# for different CLI session. +# +# log4j.appender.DRFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender + +log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender + +log4j.appender.DRFA.File=${hive.log.dir}/${hive.log.file} + +# Rollver at midnight +log4j.appender.DRFA.DatePattern=.yyyy-MM-dd + +# 30-day backup +#log4j.appender.DRFA.MaxBackupIndex=30 +log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout + +# Pattern format: Date LogLevel LoggerName LogMessage +#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n +# Debugging Pattern format +log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n + + +# +# console +# Add "console" to rootlogger above if you want to use this +# + +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.target=System.err +log4j.appender.console.layout=org.apache.log4j.PatternLayout +log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n +log4j.appender.console.encoding=UTF-8 + +#custom logging levels +#log4j.logger.xxx=DEBUG + +# +# Event Counter Appender +# Sends counts of logging messages at different severity levels to Hadoop Metrics. +# +log4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter + + +log4j.category.DataNucleus=ERROR,DRFA +log4j.category.Datastore=ERROR,DRFA +log4j.category.Datastore.Schema=ERROR,DRFA +log4j.category.JPOX.Datastore=ERROR,DRFA +log4j.category.JPOX.Plugin=ERROR,DRFA +log4j.category.JPOX.MetaData=ERROR,DRFA +log4j.category.JPOX.Query=ERROR,DRFA +log4j.category.JPOX.General=ERROR,DRFA +log4j.category.JPOX.Enhancer=ERROR,DRFA + + +# Silence useless ZK logs +log4j.logger.org.apache.zookeeper.server.NIOServerCnxn=WARN,DRFA +log4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO=WARN,DRFA + + + Custom log4j.properties + hive-log4j template + hive-log4j.xml + false + + + + content + false + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/bigtop/blob/ba8d7f50/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-log4j2.xml ---------------------------------------------------------------------- diff --git a/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-log4j2.xml b/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-log4j2.xml new file mode 100755 index 0000000..798063b --- /dev/null +++ b/bigtop-packages/src/common/ambari/ODPi/1.0/services/HIVE/configuration/hive-log4j2.xml @@ -0,0 +1,90 @@ + + content + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +status = INFO +name = HiveLog4j2 +packages = org.apache.hadoop.hive.ql.log + +# list of properties +property.hive.log.level = INFO +property.hive.root.logger = DRFA +property.hive.log.dir = ${sys:java.io.tmpdir}/${sys:user.name} +property.hive.log.file = hive.log + +# list of all appenders +appenders = console, DRFA + +# console appender +appender.console.type = Console +appender.console.name = console +appender.console.target = SYSTEM_ERR +appender.console.layout.type = PatternLayout +appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n + +# daily rolling file appender +appender.DRFA.type = RollingFile +appender.DRFA.name = DRFA +appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file} +# Use %pid in the filePattern to append process-id@host-name to the filename if you want separate log files for different CLI session +appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}.gz +appender.DRFA.layout.type = PatternLayout +appender.DRFA.layout.pattern = %d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n +appender.DRFA.policies.type = Policies +appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy +appender.DRFA.policies.time.interval = 1 +appender.DRFA.policies.time.modulate = true +appender.DRFA.strategy.type = DefaultRolloverStrategy +appender.DRFA.strategy.max = 30 + +# list of all loggers +loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX + +logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn +logger.NIOServerCnxn.level = WARN + +logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO +logger.ClientCnxnSocketNIO.level = WARN + +logger.DataNucleus.name = DataNucleus +logger.DataNucleus.level = ERROR + +logger.Datastore.name = Datastore +logger.Datastore.level = ERROR + +logger.JPOX.name = JPOX +logger.JPOX.level = ERROR + +# root logger +rootLogger.level = ${sys:hive.log.level} +rootLogger.appenderRefs = root +rootLogger.appenderRef.root.ref = ${sys:hive.root.logger} + + Custom hive-log4j2.properties + hive-log4j2 template + hive-log4j2.xml + false + + + + content + false + + + + \ No newline at end of file