hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Moore <douglas.mo...@thinkbiganalytics.com>
Subject Re: Hive 0.13 - Yarn - Tez application not shutting down
Date Thu, 12 Jun 2014 21:48:20 GMT
set hive.exec.parallel=true; will cause the Yarn application to linger
forever.

Set hive.exec.parallel=false and the Yarn application disappears after
running the Hive CLI finishes.

Is this a Hive, Yarn or Tez bug?


On Thu, Jun 12, 2014 at 4:15 PM, Douglas Moore <
douglas.moore@thinkbiganalytics.com> wrote:

> I'm on HDP 2.1 Hive 0.13/Tez
>
> I run this query:
>
> hive -v --database=ORC_db -e `select * from store_sales where
> ss_ticket_number=5741230 and ss_item_sk=4825`
>
> The yarn application runs Tez for a few seconds, generates the results and
> shuts down and goes away as reported by `yarn application -list`. All is
> well.
>
> I run the query below, Tez starts up, runs for a few seconds, generates
> the same results, Tez session shuts down and then the yarn application
> still shows up as 'RUNNING' minutes later.
> Why? How do I fix this 'leak' or re-use the YARN application?
> The problem is after running 30 or so of these type of scripts via hive,
> the jobs stop running and 'hang'
>
> Thanks!
>
> hive -v --database=ORC_db -f
> set hive.optimize.ppd=true
> set hive.exec.parallel=true
> set hive.vectorized.execution.enabled=true
> set io.sort.mb=512
> set hive.exec.reducers.bytes.per.reducer=134217728
> set hive.auto.convert.join.noconditionaltask.size=1200000000
> set mapreduce.job.reduce.slowstart.completedmaps=0.5
> set hive.mapjoin.localtask.max.memory.usage=0.99
>
> select * from store_sales where ss_ticket_number=5741230 and
> ss_item_sk=4825
>



-- 





*Douglas MoorePrincipal ConsultantThink Big
Analyticsdouglas.moore@thinkbiganalytics.com
<douglas.moore@thinkbiganalytics.com>m. 781-454-5971 o. 781-209-5565
@douglas_ma*

Mime
View raw message