hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王锋 <wfeng1...@163.com>
Subject Re:Re: Re: Re:Re: hiveserver usage
Date Mon, 12 Dec 2011 08:44:58 GMT
who has test the concurrent performance of hiveserver? and how about it. before I use one hiveserver
concurrent running jobs from our schedule system, and after about 12 hours , the hiveserver
was blocked and the subsequent jobs were not executed ,no error was thrown.

At 2011-12-12 16:25:19,"alo alt" <wget.null@googlemail.com> wrote:
Ah, I see.
Take a look at the NN, hive use hdfs and if you have jobs with many small files in a table
(logfiles as example) and a large cluster the NN could be a bottleneck. 


- Alex


On Mon, Dec 12, 2011 at 9:20 AM, 王锋 <wfeng1982@163.com> wrote:
before I set -xmx 2g, but hiveserver throws many exception OOM. so I reset and at the end
I set xmx=15g, newRatio=1. Because I watch hiveserver for a long time.It use memory very large
when running job, usually it can be 8g ,10g,or 15g. so I set xmx=15g ,and newRatio=1 , the
young generation will be large enough to support concurrent running jobs and gc quickly.


At 2011-12-12 16:09:05,"alo alt" <wget.null@googlemail.com> wrote:

Hi,


see I right you set java with -xmx=15000M?  And you set minimum heap size (xms) = 15000M?

Here you give java no chance to use less than 15GB memory, because min says 15000M, and max
too. I wondering why any java-process have to need 15G of memory. Could be in large tomcat
od jboss environments. But for hive I'm quite not sure..

- Alex



2011/12/12 王锋 <wfeng1982@163.com>


I want to know why the hiveserver use so large memory,and where the memory has been used ?


在 2011-12-12 10:02:44,"王锋" <wfeng1982@163.com> 写道:




The namenode summary:




the mr summary



and hiveserver:




hiveserver jvm args:
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=1 -Xms15000m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15
-XX:+UseParallelGC -XX:ParallelGCThreads=20 -XX:+UseParall
elOldGC -XX:-UseGCOverheadLimit -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"


now we  using 3 hiveservers in the same machine.




在 2011-12-12 09:54:29,"Aaron Sun" <aaron.sun82@gmail.com> 写道:
how's the data look like? and what's the size of the cluster?


2011/12/11 王锋 <wfeng1982@163.com>

Hi,


    I'm one of engieer of sina.com.  We have used hive ,hiveserver several months. We have
our own tasks schedule system .The system can schedule tasks running with hiveserver by jdbc.


    But The hiveserver use mem very large, usally  large than 10g.   we have 5min tasks which
will be  running every 5 minutes.,and have hourly tasks .total num of tasks  is 40. And we
start 3 hiveserver in one linux server,and be cycle connected .


    so why Memory of  hiveserver  using so large and how we do or some suggestion from you
?


Thanks and Best Regards!


Royce Wang



















--

Alexander Lorenz
http://mapredit.blogspot.com


P Think of the environment: please don't print this email unless you really need to.











--

Alexander Lorenz
http://mapredit.blogspot.com


P Think of the environment: please don't print this email unless you really need to.



Mime
View raw message