Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Subject: Re: Creating hive external table gives GC pool 'PS MarkSweep' had
 collection(s)
To: user@hive.apache.org
References: <56D063DA.6030402@roo.ee> <56D06EDF.2030001@roo.ee>
 <56D3F2FC.6030302@roo.ee>
From: Margus Roo <margus@roo.ee>
Message-ID: <56D5434E.8010301@roo.ee>
Date: Tue, 1 Mar 2016 09:22:54 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.6.0
MIME-Version: 1.0
In-Reply-To: <56D3F2FC.6030302@roo.ee>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi

Found that in my config there was
hive.exec.dynamic.partition = true;

I turned it false and most of the times I can create table now but not 
every time.


Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780

On 29/02/16 09:27, Margus Roo wrote:
> Hi
>
> Can someone confirm that Hive checks files in destination directory 
> before creating external table?
> At the moment in Hive 1.2.1 end user can just easily kill whole Hive 
> server creating external table and pointing to directory where are 
> loads of files.
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 48 780
>
> On 26/02/16 17:27, Margus Roo wrote:
>> Basically the question is:
>> Does Hive checks files in location before creating table?
>> Because if I move files away before creating tables it works and 
>> after table is created I can move files back and all works :)
>>
>> Margus (margusja) Roo
>> http://margus.roo.ee
>> skype: margusja
>> +372 51 48 780
>>
>> On 26/02/16 16:40, Margus Roo wrote:
>>> Hi
>>>
>>> I try to create external table and in the location there are 8960 
>>> small files
>>>
>>> And I am getting every time something like that:
>>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1672ms
>>> GC pool 'PS Scavenge' had collection(s): count=1 time=45ms
>>> 2016-02-26 15:18:29,721 INFO 
>>> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@3af36922]: 
>>> util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected 
>>> pause in JVM or host machine (eg GC): pause of approximately 1995ms
>>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1936ms
>>>
>>> And all 16 cpu cores are in 100% and all 16G memory almost gone.
>>>
>>> Helps only if I restart hive server.
>>>
>>> I use Hive 1.2.1 from HDP-2.3 by packaged by Hortonworks
>>>
>>> hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true 
>>> -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB 
>>> -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
>>>
>>>
>>
>