hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Margus Roo <mar...@roo.ee>
Subject Re: Creating hive external table gives GC pool 'PS MarkSweep' had collection(s)
Date Tue, 01 Mar 2016 07:22:54 GMT
Hi

Found that in my config there was
hive.exec.dynamic.partition = true;

I turned it false and most of the times I can create table now but not 
every time.


Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780

On 29/02/16 09:27, Margus Roo wrote:
> Hi
>
> Can someone confirm that Hive checks files in destination directory 
> before creating external table?
> At the moment in Hive 1.2.1 end user can just easily kill whole Hive 
> server creating external table and pointing to directory where are 
> loads of files.
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 48 780
>
> On 26/02/16 17:27, Margus Roo wrote:
>> Basically the question is:
>> Does Hive checks files in location before creating table?
>> Because if I move files away before creating tables it works and 
>> after table is created I can move files back and all works :)
>>
>> Margus (margusja) Roo
>> http://margus.roo.ee
>> skype: margusja
>> +372 51 48 780
>>
>> On 26/02/16 16:40, Margus Roo wrote:
>>> Hi
>>>
>>> I try to create external table and in the location there are 8960 
>>> small files
>>>
>>> And I am getting every time something like that:
>>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1672ms
>>> GC pool 'PS Scavenge' had collection(s): count=1 time=45ms
>>> 2016-02-26 15:18:29,721 INFO 
>>> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@3af36922]: 
>>> util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected 
>>> pause in JVM or host machine (eg GC): pause of approximately 1995ms
>>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1936ms
>>>
>>> And all 16 cpu cores are in 100% and all 16G memory almost gone.
>>>
>>> Helps only if I restart hive server.
>>>
>>> I use Hive 1.2.1 from HDP-2.3 by packaged by Hortonworks
>>>
>>> hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true 
>>> -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB 
>>> -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
>>>
>>>
>>
>


Mime
View raw message