hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Margus Roo <mar...@roo.ee>
Subject Re: Creating hive external table gives GC pool 'PS MarkSweep' had collection(s)
Date Mon, 29 Feb 2016 07:27:56 GMT
Hi

Can someone confirm that Hive checks files in destination directory 
before creating external table?
At the moment in Hive 1.2.1 end user can just easily kill whole Hive 
server creating external table and pointing to directory where are loads 
of files.

Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780

On 26/02/16 17:27, Margus Roo wrote:
> Basically the question is:
> Does Hive checks files in location before creating table?
> Because if I move files away before creating tables it works and after 
> table is created I can move files back and all works :)
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 48 780
>
> On 26/02/16 16:40, Margus Roo wrote:
>> Hi
>>
>> I try to create external table and in the location there are 8960 
>> small files
>>
>> And I am getting every time something like that:
>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1672ms
>> GC pool 'PS Scavenge' had collection(s): count=1 time=45ms
>> 2016-02-26 15:18:29,721 INFO 
>> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@3af36922]: 
>> util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause 
>> in JVM or host machine (eg GC): pause of approximately 1995ms
>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1936ms
>>
>> And all 16 cpu cores are in 100% and all 16G memory almost gone.
>>
>> Helps only if I restart hive server.
>>
>> I use Hive 1.2.1 from HDP-2.3 by packaged by Hortonworks
>>
>> hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true 
>> -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB 
>> -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
>>
>>
>


Mime
View raw message