ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liym@runstone.com" <l...@runstone.com>
Subject Re: Re: about mr accelerator question.
Date Mon, 28 Mar 2016 10:30:08 GMT
There is a question .now i have 6 ignite nodes.there is a error when the mr task is running.one node is killed usually,can you tell me why.thanks a lot.
on the one node or two node,I dont find this error.

[17:42:52] Security status [authentication=off, tls/ssl=off]
[17:42:53] HADOOP_HOME is set to /home/hduser/hadoop
[17:42:55] Performance suggestions for grid  (fix if possible)
[17:42:55] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[17:42:55]   ^-- Disable grid events (remove 'includeEventTypes' from configuration)
[17:42:55] 
[17:42:55] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[17:42:55] 
[17:42:55] Ignite node started OK (id=7965370b)
[17:42:55] Topology snapshot [ver=1, servers=1, clients=0, CPUs=24, heap=32.0GB]
[17:43:12] Topology snapshot [ver=2, servers=2, clients=0, CPUs=48, heap=64.0GB]
[17:43:18] Topology snapshot [ver=3, servers=3, clients=0, CPUs=72, heap=96.0GB]
[17:43:23] Topology snapshot [ver=4, servers=4, clients=0, CPUs=96, heap=130.0GB]
[17:43:31] Topology snapshot [ver=5, servers=5, clients=0, CPUs=120, heap=160.0GB]
[17:43:38] Topology snapshot [ver=6, servers=6, clients=0, CPUs=144, heap=190.0GB]
[17:44:08] Class "o.a.i.i.processors.hadoop.counter.HadoopCountersImpl" cannot be written in binary format because it either implements Externalizable interface or have writeObject/readObject methods. Please ensure that all nodes have this class in classpath. To enable binary serialization either implement Binarylizable interface or set explicit serializer using BinaryTypeConfiguration.setSerializer() method.
[17:44:09] Class "o.a.i.i.processors.hadoop.jobtracker.HadoopJobMetadata" cannot be written in binary format because it either implements Externalizable interface or have writeObject/readObject methods. Please ensure that all nodes have this class in classpath. To enable binary serialization either implement Binarylizable interface or set explicit serializer using BinaryTypeConfiguration.setSerializer() method.
[17:44:13] Class "o.a.i.i.processors.hadoop.proto.HadoopProtocolTaskArguments" cannot be written in binary format because it either implements Externalizable interface or have writeObject/readObject methods. Please ensure that all nodes have this class in classpath. To enable binary serialization either implement Binarylizable interface or set explicit serializer using BinaryTypeConfiguration.setSerializer() method.
[17:44:34] Class "o.a.i.i.processors.hadoop.shuffle.HadoopShuffleMessage" cannot be written in binary format because it either implements Externalizable interface or have writeObject/readObject methods. Please ensure that all nodes have this class in classpath. To enable binary serialization either implement Binarylizable interface or set explicit serializer using BinaryTypeConfiguration.setSerializer() method.
[17:44:36] Class "o.a.i.i.processors.hadoop.shuffle.HadoopShuffleAck" cannot be written in binary format because it either implements Externalizable interface or have writeObject/readObject methods. Please ensure that all nodes have this class in classpath. To enable binary serialization either implement Binarylizable interface or set explicit serializer using BinaryTypeConfiguration.setSerializer() method.
[17:44:36] Class "o.a.i.i.processors.hadoop.shuffle.HadoopShuffleAck" cannot be written in binary format because it either implements Externalizable interface or have writeObject/readObject methods. Please ensure that all nodes have this class in classpath. To enable binary serialization either implement Binarylizable interface or set explicit serializer using BinaryTypeConfiguration.setSerializer() method.
./ignite.sh: line 157: 41326 Killed                  "$JAVA" ${JVM_OPTS} ${QUIET} "${RESTART_SUCCESS_OPT}" ${JMX_MON} -DIGNITE_HOME="${IGNITE_HOME}" -DIGNITE_PROG_NAME="$0" ${JVM_XOPTS} -cp "${CP}" ${MAIN_CLASS} "${CONFIG}"
hduser@rslog1-tj:~/ignite/bin$

all nodes have the same config
<?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<!--
    Ignite Spring configuration file.

    When starting a standalone Ignite node, you need to execute the following command:
    {IGNITE_HOME}/bin/ignite.{bat|sh} path-to-this-file/default-config.xml

    When starting Ignite from Java IDE, pass path to this file into Ignition:
    Ignition.start("path-to-this-file/default-config.xml");
-->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/util
       http://www.springframework.org/schema/util/spring-util.xsd">

    <!--
        Optional description.
    -->
    <description>
        Spring file for Ignite node configuration with IGFS and Apache Hadoop map-reduce support enabled.
        Ignite node will start with this configuration by default.
    </description>

    <!--
        Initialize property configurer so we can reference environment variables.
    -->
    <bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
        <property name="searchSystemEnvironment" value="true"/>
    </bean>

    <!--
        Abstract IGFS file system configuration to be used as a template.
    -->
    <bean id="igfsCfgBase" class="org.apache.ignite.configuration.FileSystemConfiguration" abstract="true">
        <!-- Must correlate with cache affinity mapper. -->
        <property name="blockSize" value="#{128 * 1024}"/>
        <property name="perNodeBatchSize" value="512"/>
        <property name="perNodeParallelBatchCount" value="16"/>

        <property name="prefetchBlocks" value="32"/>
    </bean>

    <bean class="org.apache.ignite.configuration.CacheConfiguration">
  <!-- Store cache entries on-heap. -->
  <property name="memoryMode" value="ONHEAP_TIERED"/> 
  <!-- Enable Off-Heap memory with max size of 10 Gigabytes (0 for unlimited). -->
  <property name="offHeapMaxMemory" value="#{14 * 1024L * 1024L * 1024L}"/>
  <!-- Configure eviction policy. -->
  <property name="evictionPolicy">
    <bean class="org.apache.ignite.cache.eviction.igfs.IgfsPerBlockLruEvictionPolicy">
      <!-- Evict to off-heap after cache size reaches maxSize. -->
      <property name="maxSize" value="3400000"/>
    </bean>
  </property>
  </bean>

    <!--
        Abstract cache configuration for IGFS file data to be used as a template.
    -->
    <bean id="dataCacheCfgBase" class="org.apache.ignite.configuration.CacheConfiguration" abstract="true">
        <property name="cacheMode" value="PARTITIONED"/>
        <property name="atomicityMode" value="TRANSACTIONAL"/>
        <property name="writeSynchronizationMode" value="FULL_SYNC"/>
        <property name="backups" value="0"/>
        <property name="affinityMapper">
            <bean class="org.apache.ignite.igfs.IgfsGroupDataBlocksKeyMapper">
                <!-- How many sequential blocks will be stored on the same node. -->
                <constructor-arg value="512"/>
            </bean>
        </property>
    </bean>

    <!--
        Abstract cache configuration for IGFS metadata to be used as a template.
    -->
    <bean id="metaCacheCfgBase" class="org.apache.ignite.configuration.CacheConfiguration" abstract="true">
        <property name="cacheMode" value="REPLICATED"/>
        <property name="atomicityMode" value="TRANSACTIONAL"/>
        <property name="writeSynchronizationMode" value="FULL_SYNC"/>
    </bean>

    <!--
        Configuration of Ignite node.
    -->
    <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
        <!--
            Apache Hadoop Accelerator configuration.
        -->
        <property name="hadoopConfiguration">
            <bean class="org.apache.ignite.configuration.HadoopConfiguration">
                <!-- Information about finished jobs will be kept for 30 seconds. -->
                <property name="finishedJobInfoTtl" value="300000"/>
            </bean>
        </property>

        <!--
            This port will be used by Apache Hadoop client to connect to Ignite node as if it was a job tracker.
        -->
        <property name="connectorConfiguration">
            <bean class="org.apache.ignite.configuration.ConnectorConfiguration">
                <property name="port" value="11211"/>
            </bean>
        </property>

        <!--
            Configure one IGFS file system instance named "igfs" on this node.
        -->
        <property name="fileSystemConfiguration">
            <list>
                <bean class="org.apache.ignite.configuration.FileSystemConfiguration" parent="igfsCfgBase">
                    <property name="name" value="igfs"/>

                    <!-- Caches with these names must be configured. -->
                    <property name="metaCacheName" value="igfs-meta"/>
                    <property name="dataCacheName" value="igfs-data"/>

                    <!-- Configure TCP endpoint for communication with the file system instance. -->
                    <property name="ipcEndpointConfiguration">
                        <bean class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
                            <property name="type" value="TCP" />
                            <property name="host" value="0.0.0.0" />
                            <property name="port" value="10500" />
                        </bean>
                    </property>

                    <!-- Sample secondary file system configuration.
                        'uri'      - the URI of the secondary file system.
                        'cfgPath'  - optional configuration path of the secondary file system,
                            e.g. /opt/foo/core-site.xml. Typically left to be null.
                        'userName' - optional user name to access the secondary file system on behalf of. Use it
                            if Hadoop client and the Ignite node are running on behalf of different users.
                    -->
                    <property name="secondaryFileSystem">
                        <bean class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
                            <constructor-arg name="uri" value="hdfs://202.99.96.170:9000"/>
                            <constructor-arg name="cfgPath"><null/></constructor-arg>
                            <constructor-arg name="userName" value="client-user-name"/>
                        </bean>
                    </property>
                </bean>
            </list>
        </property>

        <!--
            Caches needed by IGFS.
        -->
        <property name="cacheConfiguration">
            <list>
                <!-- File system metadata cache. -->
                <bean class="org.apache.ignite.configuration.CacheConfiguration" parent="metaCacheCfgBase">
                    <property name="name" value="igfs-meta"/>
                </bean>

                <!-- File system files data cache. -->
                <bean class="org.apache.ignite.configuration.CacheConfiguration" parent="dataCacheCfgBase">
                    <property name="name" value="igfs-data"/>
                </bean>
            </list>
        </property>

        <!--
            Disable events.
        -->
        <property name="includeEventTypes">
            <list>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_JOB_MAPPED"/>
            </list>
        </property>

        <!--
            TCP discovery SPI can be configured with list of addresses if multicast is not available.
        -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                            <list>
                                <value>202.99.96.170</value>
                                <value>202.99.69.170:47500..47509</value>
                                <value>202.99.96.174:47500..47509</value>
                                <value>202.99.96.178:47500..47509</value>
                                <value>202.99.69.174:47500..47509</value>
                                <value>202.99.69.178:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>

and the ignite.sh config is 
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#
# Grid command line loader.
#

#
# Import common functions.
#
if [ "${IGNITE_HOME}" = "" ];
    then IGNITE_HOME_TMP="$(dirname "$(cd "$(dirname "$0")"; "pwd")")";
    else IGNITE_HOME_TMP=${IGNITE_HOME};
fi

#
# Set SCRIPTS_HOME - base path to scripts.
#
SCRIPTS_HOME="${IGNITE_HOME_TMP}/bin"

source "${SCRIPTS_HOME}"/include/functions.sh

#
# Discover path to Java executable and check it's version.
#
checkJava

#
# Discover IGNITE_HOME environment variable.
#
setIgniteHome

if [ "${DEFAULT_CONFIG}" == "" ]; then
    DEFAULT_CONFIG=config/default-config.xml
fi

#
# Parse command line parameters.
#
. "${SCRIPTS_HOME}"/include/parseargs.sh

#
# Set IGNITE_LIBS.
#
. "${SCRIPTS_HOME}"/include/setenv.sh

CP="${IGNITE_LIBS}"

RANDOM_NUMBER=$("$JAVA" -cp "${CP}" org.apache.ignite.startup.cmdline.CommandLineRandomNumberGenerator)

RESTART_SUCCESS_FILE="${IGNITE_HOME}/work/ignite_success_${RANDOM_NUMBER}"
RESTART_SUCCESS_OPT="-DIGNITE_SUCCESS_FILE=${RESTART_SUCCESS_FILE}"

#
# Find available port for JMX
#
# You can specify IGNITE_JMX_PORT environment variable for overriding automatically found JMX port
#
# This is executed when -nojmx is not specified
#
if [ "${NOJMX}" == "0" ] ; then
    findAvailableJmxPort
fi

# Mac OS specific support to display correct name in the dock.
osname=`uname`

if [ "${DOCK_OPTS}" == "" ]; then
    DOCK_OPTS="-Xdock:name=Ignite Node"
fi

#
# JVM options. See http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp for more details.
#
# ADD YOUR/CHANGE ADDITIONAL OPTIONS HERE
#
if [ -z "$JVM_OPTS" ] ; then
    JVM_OPTS="-Xms32g -Xmx32g -server -XX:+AggressiveOpts -XX:MaxPermSize=16g"
fi

#
# Uncomment the following GC settings if you see spikes in your throughput due to Garbage Collection.
#
# JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:NewSize=128m -XX:MaxNewSize=128m"
# JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60"

#
# Uncomment if you get StackOverflowError.
# On 64 bit systems this value can be larger, e.g. -Xss16m
#
# JVM_OPTS="${JVM_OPTS} -Xss4m"

#
# Uncomment to set preference for IPv4 stack.
#
# JVM_OPTS="${JVM_OPTS} -Djava.net.preferIPv4Stack=true"

#
# Assertions are disabled by default since version 3.5.
# If you want to enable them - set 'ENABLE_ASSERTIONS' flag to '1'.
#
ENABLE_ASSERTIONS="0"

#
# Set '-ea' options if assertions are enabled.
#
if [ "${ENABLE_ASSERTIONS}" = "1" ]; then
    JVM_OPTS="${JVM_OPTS} -ea"
fi

#
# Set main class to start service (grid node by default).
#
if [ "${MAIN_CLASS}" = "" ]; then
    MAIN_CLASS=org.apache.ignite.startup.cmdline.CommandLineStartup
fi

#
# Remote debugging (JPDA).
# Uncomment and change if remote debugging is required.
#
# JVM_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8787 ${JVM_OPTS}"

ERRORCODE="-1"

while [ "${ERRORCODE}" -ne "130" ]
do
    if [ "${INTERACTIVE}" == "1" ] ; then
        case $osname in
            Darwin*)
                "$JAVA" ${JVM_OPTS} ${QUIET} "${DOCK_OPTS}" "${RESTART_SUCCESS_OPT}" ${JMX_MON} \
                 -DIGNITE_HOME="${IGNITE_HOME}" \
                -DIGNITE_PROG_NAME="$0" ${JVM_XOPTS} -cp "${CP}" ${MAIN_CLASS}
            ;;
            *)
                "$JAVA" ${JVM_OPTS} ${QUIET} "${RESTART_SUCCESS_OPT}" ${JMX_MON} \
                 -DIGNITE_HOME="${IGNITE_HOME}" \
                -DIGNITE_PROG_NAME="$0" ${JVM_XOPTS} -cp "${CP}" ${MAIN_CLASS}
            ;;
        esac
    else
        case $osname in
            Darwin*)
                "$JAVA" ${JVM_OPTS} ${QUIET} "${DOCK_OPTS}" "${RESTART_SUCCESS_OPT}" ${JMX_MON} \
                  -DIGNITE_HOME="${IGNITE_HOME}" \
                 -DIGNITE_PROG_NAME="$0" ${JVM_XOPTS} -cp "${CP}" ${MAIN_CLASS} "${CONFIG}"
            ;;
            *)
                "$JAVA" ${JVM_OPTS} ${QUIET} "${RESTART_SUCCESS_OPT}" ${JMX_MON} \
                  -DIGNITE_HOME="${IGNITE_HOME}" \
                 -DIGNITE_PROG_NAME="$0" ${JVM_XOPTS} -cp "${CP}" ${MAIN_CLASS} "${CONFIG}"
            ;;
        esac
    fi

    ERRORCODE="$?"

    if [ ! -f "${RESTART_SUCCESS_FILE}" ] ; then
        break
    else
        rm -f "${RESTART_SUCCESS_FILE}"
    fi
done

if [ -f "${RESTART_SUCCESS_FILE}" ] ; then
    rm -f "${RESTART_SUCCESS_FILE}"
fi

and the log info , it looks like normal.
[18:01:43,142][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-742-0-#297%null%][CodecPool] Got brand-new decompressor [.gz]
[18:01:49,041][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-741-0-#296%null%][CodecPool] Got brand-new decompressor [.gz]
[18:01:55,627][INFO ][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=7965370b, name=null]
    ^-- H/N/C [hosts=6, nodes=6, CPUs=144]
    ^-- CPU [cur=78.93%, avg=73.2%, GC=0.03%]
    ^-- Heap [used=7976MB, free=75.64%, comm=32750MB]
    ^-- Public thread pool [active=9, idle=39, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[18:02:00,667][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-743-0-#298%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:02,336][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-751-0-#306%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:02,899][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-753-0-#308%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:04,263][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-744-0-#299%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:04,293][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-758-0-#313%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:06,625][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-748-0-#303%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:07,346][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-756-0-#311%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:07,843][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-760-0-#315%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:08,437][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-757-0-#312%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:09,579][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-747-0-#302%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:09,705][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-745-0-#300%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:10,008][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-746-0-#301%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:10,567][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-761-0-#316%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:10,724][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-754-0-#309%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:11,196][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-750-0-#305%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:11,669][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-749-0-#304%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:11,753][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-752-0-#307%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:14,720][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-759-0-#314%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:18,090][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-755-0-#310%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:23,137][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-762-0-#317%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:24,544][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-740-0-#295%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:33,540][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-739-0-#294%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:35,013][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-742-0-#297%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:42,653][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-741-0-#296%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:53,009][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-751-0-#306%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:54,465][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-743-0-#298%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:55,141][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-753-0-#308%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:55,635][INFO ][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=7965370b, name=null]
    ^-- H/N/C [hosts=6, nodes=6, CPUs=144]
    ^-- CPU [cur=77.57%, avg=73.51%, GC=0%]
    ^-- Heap [used=6619MB, free=79.79%, comm=32751MB]
    ^-- Public thread pool [active=5, idle=43, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[18:02:56,101][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-758-0-#313%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:57,204][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-744-0-#299%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:59,582][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-760-0-#315%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:59,735][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-748-0-#303%null%][CodecPool] Got brand-new decompressor [.gz]
[18:02:59,862][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-756-0-#311%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:00,842][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-757-0-#312%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:02,225][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-747-0-#302%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:02,763][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-761-0-#316%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:03,297][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-746-0-#301%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:03,342][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-745-0-#300%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:04,299][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-754-0-#309%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:04,517][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-752-0-#307%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:04,530][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-750-0-#305%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:05,373][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-749-0-#304%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:07,972][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-759-0-#314%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:12,359][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-755-0-#310%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:17,763][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-762-0-#317%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:18,504][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-740-0-#295%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:29,345][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-742-0-#297%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:30,831][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-739-0-#294%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:39,843][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-741-0-#296%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:47,839][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-751-0-#306%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:51,328][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-753-0-#308%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:51,793][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-743-0-#298%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:52,013][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-758-0-#313%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:53,577][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-744-0-#299%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:55,305][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-760-0-#315%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:55,649][INFO ][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=7965370b, name=null]
    ^-- H/N/C [hosts=6, nodes=6, CPUs=144]
    ^-- CPU [cur=78.97%, avg=73.66%, GC=0.03%]
    ^-- Heap [used=11432MB, free=65.09%, comm=32752MB]
    ^-- Public thread pool [active=2, idle=46, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[18:03:56,413][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-756-0-#311%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:56,738][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-748-0-#303%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:57,375][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-757-0-#312%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:58,477][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-761-0-#316%null%][CodecPool] Got brand-new decompressor [.gz]
[18:03:59,039][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-747-0-#302%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:00,414][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-746-0-#301%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:00,758][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-754-0-#309%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:00,929][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-752-0-#307%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:01,026][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-745-0-#300%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:02,021][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-750-0-#305%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:02,494][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-749-0-#304%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:05,905][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-759-0-#314%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:09,631][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-755-0-#310%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:13,123][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-740-0-#295%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:13,795][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-762-0-#317%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:20,957][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-742-0-#297%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:25,043][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-739-0-#294%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:32,707][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-741-0-#296%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:38,130][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-751-0-#306%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:42,754][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-753-0-#308%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:43,353][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-758-0-#313%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:45,462][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-743-0-#298%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:45,802][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-744-0-#299%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:46,897][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-760-0-#315%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:47,711][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-756-0-#311%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:48,592][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-748-0-#303%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:48,780][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-757-0-#312%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:49,813][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-761-0-#316%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:51,222][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-747-0-#302%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:52,722][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-746-0-#301%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:52,756][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-752-0-#307%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:52,920][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-754-0-#309%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:53,432][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-745-0-#300%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:54,444][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-750-0-#305%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:54,752][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-749-0-#304%null%][CodecPool] Got brand-new decompressor [.gz]
[18:04:55,655][INFO ][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=7965370b, name=null]
    ^-- H/N/C [hosts=6, nodes=6, CPUs=144]
    ^-- CPU [cur=78.93%, avg=73.91%, GC=0.03%]
    ^-- Heap [used=6666MB, free=79.64%, comm=32750MB]
    ^-- Public thread pool [active=5, idle=43, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[18:04:58,114][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-759-0-#314%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:02,277][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-755-0-#310%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:04,174][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-740-0-#295%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:05,688][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-762-0-#317%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:10,720][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-742-0-#297%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:18,086][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-739-0-#294%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:25,302][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-741-0-#296%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:28,170][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-751-0-#306%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:34,389][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-753-0-#308%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:35,207][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-758-0-#313%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:38,177][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-760-0-#315%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:38,310][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-744-0-#299%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:38,703][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-743-0-#298%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:39,577][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-756-0-#311%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:40,491][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-748-0-#303%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:40,597][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-757-0-#312%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:41,248][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-761-0-#316%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:42,926][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-747-0-#302%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:45,253][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-752-0-#307%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:45,253][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-754-0-#309%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:45,330][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-746-0-#301%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:46,194][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-745-0-#300%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:47,102][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-750-0-#305%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:47,845][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-749-0-#304%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:51,186][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-759-0-#314%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:54,757][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-755-0-#310%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:55,654][INFO ][grid-timeout-worker-#97%null%][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=7965370b, name=null]
    ^-- H/N/C [hosts=6, nodes=6, CPUs=144]
    ^-- CPU [cur=80.17%, avg=74.16%, GC=0.03%]
    ^-- Heap [used=10056MB, free=69.29%, comm=32750MB]
    ^-- Public thread pool [active=3, idle=45, qSize=0]
    ^-- System thread pool [active=0, idle=48, qSize=0]
    ^-- Outbound messages queue [size=0]
[18:05:56,225][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-740-0-#295%null%][CodecPool] Got brand-new decompressor [.gz]
[18:05:58,458][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-762-0-#317%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:01,865][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-742-0-#297%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:11,590][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-739-0-#294%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:19,248][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-741-0-#296%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:19,279][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-751-0-#306%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:27,775][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-753-0-#308%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:28,494][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-758-0-#313%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:31,404][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-760-0-#315%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:32,865][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-744-0-#299%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:33,717][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-756-0-#311%null%][CodecPool] Got brand-new decompressor [.gz]
[18:06:34,421][INFO ][Hadoop-task-7965370b-6df2-4abc-ab61-8d77a2cee18a_1-MAP-743-0-#298%null%][CodecPool] Got brand-new decompressor [.gz]




 
From: Vladimir Ozerov
Date: 2016-03-24 21:00
To: user
Subject: Re: Re: about mr accelerator question.
Hi,

Possible speedup greatly depends on the nature of your task. Typically, the more MR tasks you have and the more intensively you work with actual data, the bigger improvement could be achieved. Please give more details on what kind of jobs do you run and probably I will be able to suggest something. 

One possible change you can make to your config - switch temporal file system paths used by your jobs to PRIMARY mode. This way all temp data will reside only in memory and will not hit HDFS.

Vladimir.

On Wed, Mar 23, 2016 at 8:48 AM, liym@runstone.com <liym@runstone.com> wrote:
I am so glad to tell you the problem has been solved,thanks a lot. but the peformance improve only 300%, is there other good idea for config?
there is another problem is i am not able to follow the track the job like use framework yarn.so I  cant count the jobs and view the state which I have been finished.is there good suggestion.

the ignite config is 

<?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<!--
    Ignite Spring configuration file.

    When starting a standalone Ignite node, you need to execute the following command:
    {IGNITE_HOME}/bin/ignite.{bat|sh} path-to-this-file/default-config.xml

    When starting Ignite from Java IDE, pass path to this file into Ignition:
    Ignition.start("path-to-this-file/default-config.xml");
-->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/util
       http://www.springframework.org/schema/util/spring-util.xsd">

    <!--
        Optional description.
    -->
    <description>
        Spring file for Ignite node configuration with IGFS and Apache Hadoop map-reduce support enabled.
        Ignite node will start with this configuration by default.
    </description>

    <!--
        Initialize property configurer so we can reference environment variables.
    -->
    <bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
        <property name="searchSystemEnvironment" value="true"/>
    </bean>

    <!--
        Abstract IGFS file system configuration to be used as a template.
    -->
    <bean id="igfsCfgBase" class="org.apache.ignite.configuration.FileSystemConfiguration" abstract="true">
        <!-- Must correlate with cache affinity mapper. -->
        <property name="blockSize" value="#{128 * 1024}"/>
        <property name="perNodeBatchSize" value="512"/>
        <property name="perNodeParallelBatchCount" value="16"/>

        <property name="prefetchBlocks" value="32"/>
    </bean>

    <bean class="org.apache.ignite.configuration.CacheConfiguration">
  <!-- Store cache entries on-heap. -->
  <property name="memoryMode" value="ONHEAP_TIERED"/> 
  <!-- Enable Off-Heap memory with max size of 10 Gigabytes (0 for unlimited). -->
  <property name="offHeapMaxMemory" value="#{14 * 1024L * 1024L * 1024L}"/>
  <!-- Configure eviction policy. -->
  <property name="evictionPolicy">
    <bean class="org.apache.ignite.cache.eviction.igfs.IgfsPerBlockLruEvictionPolicy">
      <!-- Evict to off-heap after cache size reaches maxSize. -->
      <property name="maxSize" value="800000"/>
    </bean>
  </property>
  </bean>

    <!--
        Abstract cache configuration for IGFS file data to be used as a template.
    -->
    <bean id="dataCacheCfgBase" class="org.apache.ignite.configuration.CacheConfiguration" abstract="true">
        <property name="cacheMode" value="PARTITIONED"/>
        <property name="atomicityMode" value="TRANSACTIONAL"/>
        <property name="writeSynchronizationMode" value="FULL_SYNC"/>
        <property name="backups" value="0"/>
        <property name="affinityMapper">
            <bean class="org.apache.ignite.igfs.IgfsGroupDataBlocksKeyMapper">
                <!-- How many sequential blocks will be stored on the same node. -->
                <constructor-arg value="512"/>
            </bean>
        </property>
    </bean>

    <!--
        Abstract cache configuration for IGFS metadata to be used as a template.
    -->
    <bean id="metaCacheCfgBase" class="org.apache.ignite.configuration.CacheConfiguration" abstract="true">
        <property name="cacheMode" value="REPLICATED"/>
        <property name="atomicityMode" value="TRANSACTIONAL"/>
        <property name="writeSynchronizationMode" value="FULL_SYNC"/>
    </bean>

    <!--
        Configuration of Ignite node.
    -->
    <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
        <!--
            Apache Hadoop Accelerator configuration.
        -->
        <property name="hadoopConfiguration">
            <bean class="org.apache.ignite.configuration.HadoopConfiguration">
                <!-- Information about finished jobs will be kept for 30 seconds. -->
                <property name="finishedJobInfoTtl" value="30000"/>
            </bean>
        </property>

        <!--
            This port will be used by Apache Hadoop client to connect to Ignite node as if it was a job tracker.
        -->
        <property name="connectorConfiguration">
            <bean class="org.apache.ignite.configuration.ConnectorConfiguration">
                <property name="port" value="11211"/>
            </bean>
        </property>

        <!--
            Configure one IGFS file system instance named "igfs" on this node.
        -->
        <property name="fileSystemConfiguration">
            <list>
                <bean class="org.apache.ignite.configuration.FileSystemConfiguration" parent="igfsCfgBase">
                    <property name="name" value="igfs"/>

                    <!-- Caches with these names must be configured. -->
                    <property name="metaCacheName" value="igfs-meta"/>
                    <property name="dataCacheName" value="igfs-data"/>

                    <!-- Configure TCP endpoint for communication with the file system instance. -->
                    <property name="ipcEndpointConfiguration">
                        <bean class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
                            <property name="type" value="TCP" />
                            <property name="host" value="0.0.0.0" />
                            <property name="port" value="10500" />
                        </bean>
                    </property>

                    <!-- Sample secondary file system configuration.
                        'uri'      - the URI of the secondary file system.
                        'cfgPath'  - optional configuration path of the secondary file system,
                            e.g. /opt/foo/core-site.xml. Typically left to be null.
                        'userName' - optional user name to access the secondary file system on behalf of. Use it
                            if Hadoop client and the Ignite node are running on behalf of different users.
                    -->
                    <property name="secondaryFileSystem">
                        <bean class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
                            <constructor-arg name="uri" value="hdfs://*.*.*.*:9000"/>
                            <constructor-arg name="cfgPath"><null/></constructor-arg>
                            <constructor-arg name="userName" value="client-user-name"/>
                        </bean>
                    </property>
                </bean>
            </list>
        </property>

        <!--
            Caches needed by IGFS.
        -->
        <property name="cacheConfiguration">
            <list>
                <!-- File system metadata cache. -->
                <bean class="org.apache.ignite.configuration.CacheConfiguration" parent="metaCacheCfgBase">
                    <property name="name" value="igfs-meta"/>
                </bean>

                <!-- File system files data cache. -->
                <bean class="org.apache.ignite.configuration.CacheConfiguration" parent="dataCacheCfgBase">
                    <property name="name" value="igfs-data"/>
                </bean>
            </list>
        </property>

        <!--
            Disable events.
        -->
        <property name="includeEventTypes">
            <list>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_JOB_MAPPED"/>
            </list>
        </property>

        <!--
            TCP discovery SPI can be configured with list of addresses if multicast is not available.
        -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                            <list>
                                <value>*.*.*.*</value>
                                <value>*.*.*.*:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>



liym@runstone.com 
北京润通丰华科技有限公司
李宜明 like wind exist
电话 13811682465
 
From: Vladimir Ozerov
Date: 2016-03-17 13:37
To: user
Subject: Re: about mr accelerator question.
Hi,
 
The fact that you can work with 29G cluster with only 8G of memory might be
caused by the following things:
1) Your job doesn't use all data form cluster and hence caches only part of
it. This is the most likely case.
2) You have eviction policy configured for IGFS data cache. 
3) Or may be you use offheap.
Please provide the full XML configuration and we will be able to understand
it.
 
Anyways, your initial question was about out-of-memory. Could you provide
exact error message? Is it about heap memory or may be permgen?
 
As per execution time, this depends on your workload. If there are lots map
tasks and very active work with data, you will see improvement in speed. If
there are lots operations on file system (e.g. mkdirs, move, etc.) and very
little amount of map jobs, chances there will be no speedup at all. Provide
more details on the job you test and type of data you use and we will be
able to give you more ideas on what to do.
 
Vladimir.
 
 
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/about-mr-accelerator-question-tp3502p3552.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message