hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Gao <marc7...@gmail.com>
Subject Re: Prallel running jobs
Date Thu, 04 Jun 2015 03:04:04 GMT
Hi, Jay

You can split this SQL into 3 scripts, like this:
--script1.sql
FROM EMPLOYER_STAGE
INSERT OVERWRITE TABLE  EMPLOYER  PARTITION (FISCAL_YEAR = 2015,
FISCAL_PERIOD = 01) SELECT * WHERE FISCAL_YEAR = 2014 AND
FISCAL_PERIOD = 08;

--script2.sql
FROM EMPLOYER_STAGE
INSERT OVERWRITE TABLE  EMPLOYER  PARTITION (FISCAL_YEAR = 2016,
FISCAL_PERIOD = 01) SELECT * WHERE FISCAL_YEAR = 2014 AND
FISCAL_PERIOD = 08;

--script3.sql
FROM EMPLOYER_STAGE
INSERT OVERWRITE TABLE  EMPLOYER  PARTITION (FISCAL_YEAR = 2017,
FISCAL_PERIOD = 01) SELECT * WHERE FISCAL_YEAR = 2014 AND
FISCAL_PERIOD = 08;

Then, exec hive command in parallel, like this:

hive -f script1.sql &
hive -f script2.sql &
hive -f script2.sql

There will be 3 maps RUNNING in parallel, instead of 3 tasks in one
map running sequentially.


On Thu, Jun 4, 2015 at 5:30 AM, Kumar Jayapal <kjayapal17@gmail.com> wrote:
> Hi,
>
>
> I have a question regarding jobs running in YARN.
>
> I have SET hive.exec.parallel=true;
>
> When I submit this command
>
>
> FROM EMPLOYER_STAGE
> INSERT OVERWRITE TABLE  EMPLOYER  PARTITION (FISCAL_YEAR = 2015,
> FISCAL_PERIOD = 01) SELECT * WHERE FISCAL_YEAR = 2014 AND  FISCAL_PERIOD =
> 08
> INSERT OVERWRITE TABLE  EMPLOYER  PARTITION (FISCAL_YEAR = 2016,
> FISCAL_PERIOD = 01) SELECT * WHERE FISCAL_YEAR = 2014 AND  FISCAL_PERIOD =
> 08
> INSERT OVERWRITE TABLE  EMPLOYER  PARTITION (FISCAL_YEAR = 2017,
> FISCAL_PERIOD = 01) SELECT * WHERE FISCAL_YEAR = 2014 AND  FISCAL_PERIOD =
> 08;
>
>
>
> Do the jobs submitted run parallely in YARN if so will they have different
> application id?
>
> What I see is one app_id and 3 taks (map) does it mean its running in
> parallel?
>
>
>
> Thanks
> Jay

Mime
View raw message