Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CA85E200B53 for ; Tue, 12 Jul 2016 15:39:47 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C8FD8160A56; Tue, 12 Jul 2016 13:39:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F2AE7160A53 for ; Tue, 12 Jul 2016 15:39:45 +0200 (CEST) Received: (qmail 7475 invoked by uid 500); 12 Jul 2016 13:39:44 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 7465 invoked by uid 99); 12 Jul 2016 13:39:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2016 13:39:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B914A185DC1 for ; Tue, 12 Jul 2016 13:39:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.201 X-Spam-Level: * X-Spam-Status: No, score=1.201 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, LOTS_OF_MONEY=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 1dVvXrOO87DI for ; Tue, 12 Jul 2016 13:39:38 +0000 (UTC) Received: from mail-qt0-f170.google.com (mail-qt0-f170.google.com [209.85.216.170]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B70B25FBFE for ; Tue, 12 Jul 2016 13:39:37 +0000 (UTC) Received: by mail-qt0-f170.google.com with SMTP id u25so8012355qtb.1 for ; Tue, 12 Jul 2016 06:39:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Lu8SWt7Zhs1eGoq6o/pYXpJWWkb/qWv06gwDQMVhvfg=; b=FdP3TmBJUFoQXFHPmKOvx/eTKfanSS4q732HIvhyiIhXcJJbVfu+/v8pOZg309pdbH U/Z90KHNhLNrX2ojfhDnw+Sp4wbCKENMv3woF0jg2+pcUHF/4ALVoAE9Of6BFaRNYZBy eeI5hmZPsVlI6hZuVaWCCWikJ/zSDnby1EGamCbB+34JHXUFgJFO/tm7IzQ1LLqbjnf9 dsrZewh2OANNzv42ncUVMqWsQUU2oe01MOANmJ/4WXoYjhSlpEJ2P0HxVH/lkTpUSL8y mKdPA4i0psRRfIzAFxwhX7iq7z8s57CDb5bsKc/B5NBps+jHWy/qC32Upn9XUh+lSSx1 aAog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Lu8SWt7Zhs1eGoq6o/pYXpJWWkb/qWv06gwDQMVhvfg=; b=c24Z2QcgEeNK99Gma/9Z/SwOy0H2QEKTFa2rpetjaVqkz5n6P8/X4yKrTTM3lBGJQ8 EeAavNOg2S5ZXJT/VJ4kU0sHrl7dlpQexIVv/DOi8wT8zhEiKTIjtHJaueP+FJv9w3io E3+/EXfw//TTclRaZa6WW1t78gJg2KeXwMimybdS3mqROi9y1/DO3dQYVRipjGKdU09J fxqfanUkm1BQiY8BDDXRsGBjFMAs/j++VU0k4hiz3vjhERlM74aXcosgZbpRsgDTxZ7P JSe4xCOwAmW8/O+X3hdetYAUfsF1AALopYWZdwMqFj13/ju/x7ChxNu5NFZdRSpBEmhL 2axw== X-Gm-Message-State: ALyK8tJfJX4svv17XobLr1ZM+u2kBQS3xS449LoR2tAvUf0rXHhWF9uzl7M5NvLD9lQAPuSV1Puk1G3vLOcIUQ== X-Received: by 10.237.51.162 with SMTP id v31mr350652qtd.1.1468330776415; Tue, 12 Jul 2016 06:39:36 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.143.129 with HTTP; Tue, 12 Jul 2016 06:39:34 -0700 (PDT) In-Reply-To: References: <1A0C519B-0BCA-4E9C-89B0-0546F6BFA346@gmail.com> <6402781A-E1F7-47F4-904F-7F6DA3A67CC5@gmail.com> <341CD464-B765-46B3-91BB-CCCD48B64769@gmail.com> From: Mich Talebzadeh Date: Tue, 12 Jul 2016 14:39:34 +0100 Message-ID: Subject: Re: Using Spark on Hive with Hive also using Spark as its execution engine To: user Content-Type: multipart/alternative; boundary=94eb2c124a32576ecf053770662c archived-at: Tue, 12 Jul 2016 13:39:48 -0000 --94eb2c124a32576ecf053770662c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable thanks Marcin. What Is your guesstimate on the order of "faster" please? Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianrbJd6= zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 12 July 2016 at 14:35, Marcin Tustin wrote: > Quick note - my experience (no benchmarks) is that Tez without LLAP (we'r= e > still not on hive 2) is faster than MR by some way. I haven't dug into wh= y > that might be. > > On Tue, Jul 12, 2016 at 9:19 AM, Mich Talebzadeh < > mich.talebzadeh@gmail.com> wrote: > >> sorry I completely miss your points >> >> I was NOT talking about Exadata. I was comparing Oracle 12c caching with >> that of Oracle TimesTen. no one mentioned Exadata here and neither >> storeindex etc.. >> >> >> so if Tez is not MR with DAG could you give me an example of how it >> works. No opinions but relevant to this point. I do not know much about = Tez >> as I stated it before >> >> Case in point if Tez could do the job on its own why Tez is used in >> conjunction with LLAP as Martin alluded to as well in this thread. >> >> >> Having said that , I would be interested if you provide a working exampl= e >> of Hive on Tez, compared to Hive on MR. >> >> One experiment is worth hundreds of opinions >> >> >> >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianrb= Jd6zP6AcPCCdOABUrV8Pw >> * >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damage= s >> arising from such loss, damage or destruction. >> >> >> >> On 12 July 2016 at 13:31, J=C3=B6rn Franke wrote: >> >>> >>> I think the comparison with Oracle rdbms and oracle times ten is not so >>> good. There are times when the in-memory database of Oracle is slower t= han >>> the rdbms (especially in case of Exadata) due to the issue that in-memo= ry - >>> as in Spark - means everything is in memory and everything is always >>> processed (no storage indexes , no bloom filters etc) which explains th= is >>> behavior quiet well. >>> >>> Hence, I do not agree with the statement that tez is basically mr with >>> dag (or that llap is basically in-memory which is also not correct). Th= is >>> is a wrong oversimplification and I do not think this is useful for the >>> community, but better is to understand when something can be used and w= hen >>> not. In-memory is also not the solution to everything and if you look f= or >>> example behind SAP Hana or NoSql there is much more around this, which = is >>> not even on the roadmap of Spark. >>> >>> Anyway, discovering good use case patterns should be done on >>> standardized benchmarks going beyond the select count etc >>> >>> On 12 Jul 2016, at 11:16, Mich Talebzadeh >>> wrote: >>> >>> That is only a plan not what execution engine is doing. >>> >>> As I stated before Spark uses DAG + in-memory computing. MR is serial o= n >>> disk. >>> >>> The key is the execution here or rather the execution engine. >>> >>> In general >>> >>> The standard MapReduce as I know reads the data from HDFS, apply >>> map-reduce algorithm and writes back to HDFS. If there are many iterati= ons >>> of map-reduce then, there will be many intermediate writes to HDFS. Thi= s is >>> all serial writes to disk. Each map-reduce step is completely independe= nt >>> of other steps, and the executing engine does not have any global knowl= edge >>> of what map-reduce steps are going to come after each map-reduce step. = For >>> many iterative algorithms this is inefficient as the data between each >>> map-reduce pair gets written and read from the file system. >>> >>> The equivalent to parallelism in Big Data is deploying what is known as >>> Directed Acyclic Graph (DAG >>> ) algorithm. In a >>> nutshell deploying DAG results in a fuller picture of global optimisati= on >>> by deploying parallelism, pipelining consecutive map steps into one and= not >>> writing intermediate data to HDFS. So in short this prevents writing da= ta >>> back and forth after every reduce step which for me is a significant >>> improvement, compared to the classical MapReduce algorithm. >>> >>> Now Tez is basically MR with DAG. With Spark you get DAG + in-memory >>> computing. Think of it as a comparison between a classic RDBMS like Ora= cle >>> and IMDB like Oracle TimesTen with in-memory processing. >>> >>> The outcome is that Hive using Spark as execution engine is pretty >>> impressive. You have the advantage of Hive CBO + In-memory computing. I= f >>> you use Spark for all this (say Spark SQL) but no Hive, Spark uses its = own >>> optimizer called Catalyst that does not have CBO yet plus in memory >>> computing. >>> >>> As usual your mileage varies. >>> >>> HTH >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianr= bJd6zP6AcPCCdOABUrV8Pw >>> * >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damag= es >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 12 July 2016 at 09:33, Markovitz, Dudu wrote= : >>> >>>> I don=E2=80=99t see how this explains the time differences. >>>> >>>> >>>> >>>> Dudu >>>> >>>> >>>> >>>> *From:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com] >>>> *Sent:* Tuesday, July 12, 2016 10:56 AM >>>> *To:* user >>>> *Cc:* user @spark >>>> >>>> *Subject:* Re: Using Spark on Hive with Hive also using Spark as its >>>> execution engine >>>> >>>> >>>> >>>> This the whole idea. Spark uses DAG + IM, MR is classic >>>> >>>> >>>> >>>> >>>> >>>> This is for Hive on Spark >>>> >>>> >>>> >>>> hive> explain select max(id) from dummy_parquet; >>>> OK >>>> STAGE DEPENDENCIES: >>>> Stage-1 is a root stage >>>> Stage-0 depends on stages: Stage-1 >>>> >>>> STAGE PLANS: >>>> Stage: Stage-1 >>>> Spark >>>> Edges: >>>> Reducer 2 <- Map 1 (GROUP, 1) >>>> * DagName: >>>> hduser_20160712083219_632c2749-7387-478f-972d-9eaadd9932c6:1* >>>> Vertices: >>>> Map 1 >>>> Map Operator Tree: >>>> TableScan >>>> alias: dummy_parquet >>>> Statistics: Num rows: 100000000 Data size: 700000000 >>>> Basic stats: COMPLETE Column stats: NONE >>>> Select Operator >>>> expressions: id (type: int) >>>> outputColumnNames: id >>>> Statistics: Num rows: 100000000 Data size: >>>> 700000000 Basic stats: COMPLETE Column stats: NONE >>>> Group By Operator >>>> aggregations: max(id) >>>> mode: hash >>>> outputColumnNames: _col0 >>>> Statistics: Num rows: 1 Data size: 4 Basic stats= : >>>> COMPLETE Column stats: NONE >>>> Reduce Output Operator >>>> sort order: >>>> Statistics: Num rows: 1 Data size: 4 Basic >>>> stats: COMPLETE Column stats: NONE >>>> value expressions: _col0 (type: int) >>>> Reducer 2 >>>> Reduce Operator Tree: >>>> Group By Operator >>>> aggregations: max(VALUE._col0) >>>> mode: mergepartial >>>> outputColumnNames: _col0 >>>> Statistics: Num rows: 1 Data size: 4 Basic stats: >>>> COMPLETE Column stats: NONE >>>> File Output Operator >>>> compressed: false >>>> Statistics: Num rows: 1 Data size: 4 Basic stats: >>>> COMPLETE Column stats: NONE >>>> table: >>>> input format: >>>> org.apache.hadoop.mapred.TextInputFormat >>>> output format: >>>> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat >>>> serde: >>>> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe >>>> >>>> Stage: Stage-0 >>>> Fetch Operator >>>> limit: -1 >>>> Processor Tree: >>>> ListSink >>>> >>>> Time taken: 2.801 seconds, Fetched: 50 row(s) >>>> >>>> >>>> >>>> And this is with setting the execution engine to MR >>>> >>>> >>>> >>>> hive> set hive.execution.engine=3Dmr; >>>> Hive-on-MR is deprecated in Hive 2 and may not be available in the >>>> future versions. Consider using a different execution engine (i.e. spa= rk, >>>> tez) or using Hive 1.X releases. >>>> >>>> >>>> >>>> hive> explain select max(id) from dummy_parquet; >>>> OK >>>> STAGE DEPENDENCIES: >>>> Stage-1 is a root stage >>>> Stage-0 depends on stages: Stage-1 >>>> >>>> STAGE PLANS: >>>> Stage: Stage-1 >>>> Map Reduce >>>> Map Operator Tree: >>>> TableScan >>>> alias: dummy_parquet >>>> Statistics: Num rows: 100000000 Data size: 700000000 Basic >>>> stats: COMPLETE Column stats: NONE >>>> Select Operator >>>> expressions: id (type: int) >>>> outputColumnNames: id >>>> Statistics: Num rows: 100000000 Data size: 700000000 >>>> Basic stats: COMPLETE Column stats: NONE >>>> Group By Operator >>>> aggregations: max(id) >>>> mode: hash >>>> outputColumnNames: _col0 >>>> Statistics: Num rows: 1 Data size: 4 Basic stats: >>>> COMPLETE Column stats: NONE >>>> Reduce Output Operator >>>> sort order: >>>> Statistics: Num rows: 1 Data size: 4 Basic stats: >>>> COMPLETE Column stats: NONE >>>> value expressions: _col0 (type: int) >>>> Reduce Operator Tree: >>>> Group By Operator >>>> aggregations: max(VALUE._col0) >>>> mode: mergepartial >>>> outputColumnNames: _col0 >>>> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE >>>> Column stats: NONE >>>> File Output Operator >>>> compressed: false >>>> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE >>>> Column stats: NONE >>>> table: >>>> input format: org.apache.hadoop.mapred.TextInputFormat >>>> output format: >>>> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat >>>> serde: >>>> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe >>>> >>>> Stage: Stage-0 >>>> Fetch Operator >>>> limit: -1 >>>> Processor Tree: >>>> ListSink >>>> >>>> Time taken: 0.1 seconds, Fetched: 44 row(s) >>>> >>>> >>>> >>>> >>>> >>>> HTH >>>> >>>> >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn *https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxian= rbJd6zP6AcPCCdOABUrV8Pw >>>> * >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which ma= y >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary dama= ges >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> >>>> On 12 July 2016 at 08:16, Markovitz, Dudu >>>> wrote: >>>> >>>> This is a simple task =E2=80=93 >>>> >>>> Read the files, find the local max value and combine the results (find >>>> the global max value). >>>> >>>> How do you explain the differences in the results? Spark reads the >>>> files and finds a local max 10X (+) faster than MR? >>>> >>>> Can you please attach the execution plan? >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> Dudu >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *From:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com] >>>> *Sent:* Monday, July 11, 2016 11:55 PM >>>> *To:* user ; user @spark >>>> *Subject:* Re: Using Spark on Hive with Hive also using Spark as its >>>> execution engine >>>> >>>> >>>> >>>> In my test I did like for like keeping the systematic the same namely: >>>> >>>> >>>> >>>> 1. Table was a parquet table of 100 Million rows >>>> 2. The same set up was used for both Hive on Spark and Hive on MR >>>> 3. Spark was very impressive compared to MR on this particular test= . >>>> >>>> >>>> >>>> Just to see any issues I created an ORC table in in the image of >>>> Parquet (insert/select from Parquet to ORC) with stats updated for col= umns >>>> etc >>>> >>>> >>>> >>>> These were the results of the same run using ORC table this time: >>>> >>>> >>>> >>>> hive> select max(id) from oraclehadoop.dummy; >>>> >>>> Starting Spark Job =3D b886b869-5500-4ef7-aab9-ae6fb4dad22b >>>> >>>> Query Hive on Spark job[1] stages: >>>> 2 >>>> 3 >>>> >>>> Status: Running (Hive on Spark job[1]) >>>> Job Progress Format >>>> CurrentTime StageId_StageAttemptId: >>>> SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCou= nt >>>> [StageCost] >>>> 2016-07-11 21:35:45,020 Stage-2_0: 0(+8)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:48,033 Stage-2_0: 0(+8)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:51,046 Stage-2_0: 1(+8)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:52,050 Stage-2_0: 3(+8)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:53,055 Stage-2_0: 8(+4)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:54,060 Stage-2_0: 11(+1)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:55,065 Stage-2_0: 12(+0)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:56,071 Stage-2_0: 12(+8)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:57,076 Stage-2_0: 13(+8)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:58,081 Stage-2_0: 20(+3)/23 Stage-3_0: 0/1 >>>> 2016-07-11 21:35:59,085 Stage-2_0: 23/23 Finished Stage-3_0: >>>> 0(+1)/1 >>>> 2016-07-11 21:36:00,089 Stage-2_0: 23/23 Finished Stage-3_0: 1/1 >>>> Finished >>>> Status: Finished successfully in 16.08 seconds >>>> OK >>>> 100000000 >>>> Time taken: 17.775 seconds, Fetched: 1 row(s) >>>> >>>> >>>> >>>> Repeat with MR engine >>>> >>>> >>>> >>>> hive> set hive.execution.engine=3Dmr; >>>> Hive-on-MR is deprecated in Hive 2 and may not be available in the >>>> future versions. Consider using a different execution engine (i.e. spa= rk, >>>> tez) or using Hive 1.X releases. >>>> >>>> >>>> >>>> hive> select max(id) from oraclehadoop.dummy; >>>> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available i= n >>>> the future versions. Consider using a different execution engine (i.e. >>>> spark, tez) or using Hive 1.X releases. >>>> Query ID =3D hduser_20160711213100_8dc2afae-8644-4097-ba33-c7bd3c304bf= 8 >>>> Total jobs =3D 1 >>>> Launching Job 1 out of 1 >>>> Number of reduce tasks determined at compile time: 1 >>>> In order to change the average load for a reducer (in bytes): >>>> set hive.exec.reducers.bytes.per.reducer=3D >>>> In order to limit the maximum number of reducers: >>>> set hive.exec.reducers.max=3D >>>> In order to set a constant number of reducers: >>>> set mapreduce.job.reduces=3D >>>> Starting Job =3D job_1468226887011_0008, Tracking URL =3D >>>> http://rhes564:8088/proxy/application_1468226887011_0008/ >>>> Kill Command =3D /home/hduser/hadoop-2.6.0/bin/hadoop job -kill >>>> job_1468226887011_0008 >>>> Hadoop job information for Stage-1: number of mappers: 23; number of >>>> reducers: 1 >>>> 2016-07-11 21:37:00,061 Stage-1 map =3D 0%, reduce =3D 0% >>>> 2016-07-11 21:37:06,440 Stage-1 map =3D 4%, reduce =3D 0%, Cumulative= CPU >>>> 16.48 sec >>>> 2016-07-11 21:37:14,751 Stage-1 map =3D 9%, reduce =3D 0%, Cumulative= CPU >>>> 40.63 sec >>>> 2016-07-11 21:37:22,048 Stage-1 map =3D 13%, reduce =3D 0%, Cumulativ= e CPU >>>> 58.88 sec >>>> 2016-07-11 21:37:30,412 Stage-1 map =3D 17%, reduce =3D 0%, Cumulativ= e CPU >>>> 80.72 sec >>>> 2016-07-11 21:37:37,707 Stage-1 map =3D 22%, reduce =3D 0%, Cumulativ= e CPU >>>> 103.43 sec >>>> 2016-07-11 21:37:45,999 Stage-1 map =3D 26%, reduce =3D 0%, Cumulativ= e CPU >>>> 125.93 sec >>>> 2016-07-11 21:37:54,300 Stage-1 map =3D 30%, reduce =3D 0%, Cumulativ= e CPU >>>> 147.17 sec >>>> 2016-07-11 21:38:01,538 Stage-1 map =3D 35%, reduce =3D 0%, Cumulativ= e CPU >>>> 166.56 sec >>>> 2016-07-11 21:38:08,807 Stage-1 map =3D 39%, reduce =3D 0%, Cumulativ= e CPU >>>> 189.29 sec >>>> 2016-07-11 21:38:17,115 Stage-1 map =3D 43%, reduce =3D 0%, Cumulativ= e CPU >>>> 211.03 sec >>>> 2016-07-11 21:38:24,363 Stage-1 map =3D 48%, reduce =3D 0%, Cumulativ= e CPU >>>> 235.68 sec >>>> 2016-07-11 21:38:32,638 Stage-1 map =3D 52%, reduce =3D 0%, Cumulativ= e CPU >>>> 258.27 sec >>>> 2016-07-11 21:38:40,916 Stage-1 map =3D 57%, reduce =3D 0%, Cumulativ= e CPU >>>> 278.44 sec >>>> 2016-07-11 21:38:49,206 Stage-1 map =3D 61%, reduce =3D 0%, Cumulativ= e CPU >>>> 300.35 sec >>>> 2016-07-11 21:38:58,524 Stage-1 map =3D 65%, reduce =3D 0%, Cumulativ= e CPU >>>> 322.89 sec >>>> 2016-07-11 21:39:07,889 Stage-1 map =3D 70%, reduce =3D 0%, Cumulativ= e CPU >>>> 344.8 sec >>>> 2016-07-11 21:39:16,151 Stage-1 map =3D 74%, reduce =3D 0%, Cumulativ= e CPU >>>> 367.77 sec >>>> 2016-07-11 21:39:25,456 Stage-1 map =3D 78%, reduce =3D 0%, Cumulativ= e CPU >>>> 391.82 sec >>>> 2016-07-11 21:39:33,725 Stage-1 map =3D 83%, reduce =3D 0%, Cumulativ= e CPU >>>> 415.48 sec >>>> 2016-07-11 21:39:43,037 Stage-1 map =3D 87%, reduce =3D 0%, Cumulativ= e CPU >>>> 436.09 sec >>>> 2016-07-11 21:39:51,292 Stage-1 map =3D 91%, reduce =3D 0%, Cumulativ= e CPU >>>> 459.4 sec >>>> 2016-07-11 21:39:59,563 Stage-1 map =3D 96%, reduce =3D 0%, Cumulativ= e CPU >>>> 477.92 sec >>>> 2016-07-11 21:40:05,760 Stage-1 map =3D 100%, reduce =3D 0%, Cumulati= ve >>>> CPU 491.72 sec >>>> 2016-07-11 21:40:10,921 Stage-1 map =3D 100%, reduce =3D 100%, Cumula= tive >>>> CPU 499.37 sec >>>> MapReduce Total cumulative CPU time: 8 minutes 19 seconds 370 msec >>>> Ended Job =3D job_1468226887011_0008 >>>> MapReduce Jobs Launched: >>>> Stage-Stage-1: Map: 23 Reduce: 1 Cumulative CPU: 499.37 sec HDFS >>>> Read: 403754774 HDFS Write: 10 SUCCESS >>>> Total MapReduce CPU Time Spent: 8 minutes 19 seconds 370 msec >>>> OK >>>> 100000000 >>>> Time taken: 202.333 seconds, Fetched: 1 row(s) >>>> >>>> >>>> >>>> So in summary >>>> >>>> >>>> >>>> Table MR/sec Spark/sec >>>> >>>> Parquet 239.532 14.38 >>>> >>>> ORC 202.333 17.77 >>>> >>>> >>>> >>>> Still I would use Spark if I had a choice and I agree that on VLT >>>> (very large tables), the limitation in available memory may be the >>>> overriding factor in using Spark. >>>> >>>> >>>> >>>> HTH >>>> >>>> >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn *https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxian= rbJd6zP6AcPCCdOABUrV8Pw >>>> * >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which ma= y >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary dama= ges >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> >>>> On 11 July 2016 at 19:25, Gopal Vijayaraghavan >>>> wrote: >>>> >>>> >>>> > Status: Finished successfully in 14.12 seconds >>>> > OK >>>> > 100000000 >>>> > Time taken: 14.38 seconds, Fetched: 1 row(s) >>>> >>>> That might be an improvement over MR, but that still feels far too slo= w. >>>> >>>> >>>> Parquet numbers are in general bad in Hive, but that's because the >>>> Parquet >>>> reader gets no actual love from the devs. The community, if it wants t= o >>>> keep using Parquet heavily needs a Hive dev to go over to Parquet-mr a= nd >>>> cut a significant number of memory copies out of the reader. >>>> >>>> The Spark 2.0 build for instance, has a custom Parquet reader for >>>> SparkSQL >>>> which does this. SPARK-12854 does for Spark+Parquet what Hive 2.0 does >>>> for >>>> ORC (actually, it looks more like hive's VectorizedRowBatch than >>>> Tungsten's flat layouts). >>>> >>>> But that reader cannot be used in Hive-on-Spark, because it is not a >>>> public reader impl. >>>> >>>> >>>> Not to pick an arbitrary dataset, my workhorse example is a TPC-H >>>> lineitem >>>> at 10Gb scale with a single 16 box. >>>> >>>> hive(tpch_flat_orc_10)> select max(l_discount) from lineitem; >>>> Query ID =3D gopal_20160711175917_f96371aa-2721-49c8-99a0-f7c4a1eacfda >>>> Total jobs =3D 1 >>>> Launching Job 1 out of 1 >>>> >>>> >>>> Status: Running (Executing on YARN cluster with App id >>>> application_1466700718395_0256) >>>> >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> VERTICES MODE STATUS TOTAL COMPLETED RUNNING >>>> PENDING FAILED KILLED >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> Map 1 .......... llap SUCCEEDED 13 13 0 >>>> 0 0 0 >>>> Reducer 2 ...... llap SUCCEEDED 1 1 0 >>>> 0 0 0 >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> VERTICES: 02/02 [=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D>>] 100% ELAPSED TIME: >>>> 0.71 s >>>> >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> Status: DAG finished successfully in 0.71 seconds >>>> >>>> Query Execution Summary >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> OPERATION DURATION >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> Compile Query 0.21s >>>> Prepare Plan 0.13s >>>> Submit Plan 0.34s >>>> Start DAG 0.23s >>>> Run DAG 0.71s >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> >>>> Task Execution Summary >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS >>>> OUTPUT_RECORDS >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> Map 1 604.00 0 0 59,957,438 >>>> 13 >>>> Reducer 2 105.00 0 0 13 >>>> 0 >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> >>>> LLAP IO Summary >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS >>>> ALLOCATION >>>> USED TOTAL_IO >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> Map 1 6036 0 146 0B 68.86MB >>>> 491.00MB >>>> 479.89MB 7.94s >>>> >>>> ----------------------------------------------------------------------= ----- >>>> ------------------- >>>> >>>> OK >>>> 0.1 >>>> Time taken: 1.669 seconds, Fetched: 1 row(s) >>>> hive(tpch_flat_orc_10)> >>>> >>>> >>>> This is running against a single 16 core box & I would assume it would >>>> take <1.4s to read twice as much (13 tasks is barely touching the load >>>> factors). >>>> >>>> It would probably be a bit faster if the cache had hits, but in genera= l >>>> 14s to read a 100M rows is nearly a magnitude off where Hive 2.2.0 is. >>>> >>>> Cheers, >>>> Gopal >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >> > > Want to work at Handy? Check out our culture deck and open roles > > Latest news at Handy > Handy just raised $50m > led > by Fidelity > > --94eb2c124a32576ecf053770662c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
thanks Marcin.

What Is your = guesstimate on the order of "faster" please?

=
Cheers

Dr Mich Talebzadeh

=C2=A0

LinkedIn =C2=A0https://www.linkedin.com/profile/view?id=3DAAEA= AAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

=C2=A0

http:= //talebzadehmich.wordpress.com


Disclaimer:=C2=A0Use = it=C2=A0at your own risk. Any and all responsibilit= y for any loss, damage or destruction of data or any other property which may arise from relying on this email= 9;s=C2=A0technical=C2=A0content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from = such loss, damage or destruction.

=C2=A0

<= font color=3D"#000000" face=3D"Times New Roman" size=3D"3">

On 12 July 2016 at 14:35, Marcin Tustin <mtustin@handybook.com> wrote:
Quick note - my experience (no benchmarks) is that Te= z without LLAP (we're still not on hive 2) is faster than MR by some wa= y. I haven't dug into why that might be.

On= Tue, Jul 12, 2016 at 9:19 AM, Mich Talebzadeh <mich.talebzadeh@gm= ail.com> wrote:
sor= ry I completely miss your points

I=C2=A0was NOT ta= lking about Exadata. I was comparing Oracle 12c caching with that of Oracle= TimesTen. no one mentioned Exadata here and neither storeindex etc..


so if Tez is not MR with DAG could you = give me an example of how it works. No opinions but relevant to this point.= I do not know much about Tez as I stated it before

Case in point if Tez could do the job on its own why Tez is used in conju= nction with LLAP as Martin alluded to as well in this thread.

Having said that , I would be interested if you = provide a working example of Hive on Tez, compared to Hive on MR.

One experiment is worth hundreds of opinions

=




Dr Mich Talebzadeh

=C2=A0

LinkedIn =C2=A0https://www.linkedin.com/profile/view?id=3DAAEA= AAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

=C2=A0

http:= //talebzadehmich.wordpress.com


Disclaimer:=C2=A0Use = it=C2=A0at your own risk. Any and all responsibilit= y for any loss, damage or destruction of data or any other property which may arise from relying on this email= 9;s=C2=A0technical=C2=A0content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from = such loss, damage or destruction.

=C2=A0

<= font color=3D"#000000" face=3D"Times New Roman" size=3D"3">

On 12 July 2016 at 13:31, J= =C3=B6rn Franke <jornfranke@gmail.com> wrote:

I think the comparison with = Oracle rdbms and oracle times ten is not so good. There are times when the = in-memory database of Oracle is slower than the rdbms (especially in case o= f Exadata) due to the issue that in-memory - as in Spark - means everything= is in memory and everything is always processed (no storage indexes , no b= loom filters etc) which explains this behavior quiet well.

Hence, I do not agree with the statement that tez is basically mr = with dag (or that llap is basically in-memory which is also not correct). T= his is a wrong oversimplification and I do not think this is useful for the= community, but better is to understand when something can be used and when= not. In-memory is also not the solution to everything and if you look for = example behind SAP Hana or NoSql there is much more around this, which is n= ot even on the roadmap of Spark.

Anyway, discoveri= ng good use case patterns should be done on standardized benchmarks going b= eyond the select count etc=C2=A0

On 12 Jul 2016, at= 11:16, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:

That is only a plan not wha= t execution engine is doing.

As I stated before Sp= ark uses DAG + in-memory computing. MR is serial on disk.=C2=A0
<= br>
The key is the execution here or rather the execution engine.=

In general

The standard = MapReduce=C2=A0=C2=A0as I know reads the data from HDFS, apply map-reduce a= lgorithm and writes back to HDFS. If there are many iterations of map-reduc= e then, there will be many intermediate writes to HDFS. This is all serial = writes to disk.=C2=A0Each map-reduce step is completely independent of othe= r steps, and the executing engine does not have any global knowledge of wha= t=C2=A0map-reduce steps are going to come after each=C2=A0map-reduce step. = For many iterative algorithms this is inefficient as the data between each = map-reduce pair gets written and read from the file system.

<= /div>
The equivalent to parallelism in Big Data is deploying what is kn= own as Directed Acyclic Graph (DAG) algorithm. In a nutshell deploying DAG=C2=A0results in= =C2=A0a fuller picture of global optimisation by=C2=A0deploying parallelism= , pipelining consecutive map steps into one and not writing intermediate da= ta to HDFS. So in short this prevents writing data back and forth after eve= ry reduce step which for me is a significant improvement, compared to the c= lassical MapReduce algorithm.

Now Tez is basically= MR with DAG. With Spark you get DAG + in-memory computing. Think of it as = a comparison between a classic RDBMS like Oracle and IMDB like Oracle Times= Ten with in-memory processing.

The outcome is that= Hive using Spark as execution engine is pretty impressive. You have the ad= vantage of Hive CBO + In-memory computing. If you use Spark for all this (s= ay Spark SQL) but no Hive, Spark uses its own optimizer called Catalyst=C2= =A0that does not have CBO yet plus in memory computing.

As usual your mileage varies.

HTH
=

=

Dr Mich Talebzadeh

=C2=A0

LinkedIn =C2=A0https://www.linkedin.com/profile/view?id=3DAAEA= AAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

=C2=A0

http:= //talebzadehmich.wordpress.com


Disclaimer:=C2=A0Use = it=C2=A0at your own risk. Any and all responsibilit= y for any loss, damage or destruction of data or any other property which may arise from relying on this email= 9;s=C2=A0technical=C2=A0content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from = such loss, damage or destruction.

=C2=A0

<= font color=3D"#000000" face=3D"Times New Roman" size=3D"3">

On 12 July 2016 at 09:33, Markovitz, Dudu <dmarkovitz@paypal.com> wrote:

I don=E2=80=99t see how this exp= lains the time differences.

=C2=A0

Dudu

=C2=A0

From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.= com]
Sent: Tuesday, July 12, 2016 10:56 AM
To: user <user@hive.apache.org>
Cc: user @spark <user@spark.apache.org>


Subject: Re: Using Spark on Hive with Hive also using Spark as its e= xecution engine

=C2=A0

This the whole idea.=C2=A0Spark uses DAG + IM, MR is= classic

=C2=A0

=C2=A0

This is for Hive on Spark

=C2=A0

hive> explain select max(id) from dummy_parquet;
OK
STAGE DEPENDENCIES:
=C2=A0 Stage-1 is a root stage
=C2=A0 Stage-0 depends on stages: Stage-1

STAGE PLANS:
=C2=A0 Stage: Stage-1
=C2=A0=C2=A0=C2=A0 Spark
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Edges:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Reducer 2 <- Map 1 (GROUP, 1)=
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 DagName: hduser_20160712083219_632c2749-7387-478f-972d-9eaa= dd9932c6:1
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Vertices:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Map 1
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Map Oper= ator Tree:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 TableScan
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 alias: dummy_parquet
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Statistics: Num rows: 100000000 Data size: 7000= 00000 Basic stats: COMPLETE Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Select Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expressions: id (type: int)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 outputColumnNames: id
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Statistics: Num rows: 100000000 Dat= a size: 700000000 Basic stats: COMPLETE Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Group By Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 aggregations: max(id) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mode: hash
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 outputColumnNames: _col= 0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Statistics: Num rows: 1= Data size: 4 Basic stats: COMPLETE Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Reduce Output Operator<= br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sort order:=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Statistics:= Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 value expre= ssions: _col0 (type: int)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Reducer 2
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Reduce O= perator Tree:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Group By Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 aggregations: max(VALUE._col0)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 mode: mergepartial
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 outputColumnNames: _col0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE = Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 File Output Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 compressed: false
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Statistics: Num rows: 1 Data size: 4 Basic stat= s: COMPLETE Column stats: NONE
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 table:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 input format: org.apach= e.hadoop.mapred.TextInputFormat
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 output format: org.apac= he.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 serde: org.apache.hadoo= p.hive.serde2.lazy.LazySimpleSerDe

=C2=A0 Stage: Stage-0
=C2=A0=C2=A0=C2=A0 Fetch Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 limit: -1
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Processor Tree:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ListSink

Time taken: 2.801 seconds, Fetched: 50 row(s)

=C2=A0

And this is with setting the execution engine to MR<= u>

=C2=A0

hive> set hive.e= xecution.engine=3Dmr;
Hive-on-MR is deprecated in Hive 2 and may not be available in the future v= ersions. Consider using a different execution engine (i.e. spark, tez) or u= sing Hive 1.X releases.

=C2=A0

hive> explain select max(id) from dummy_parquet;
OK
STAGE DEPENDENCIES:
=C2=A0 Stage-1 is a root stage
=C2=A0 Stage-0 depends on stages: Stage-1

STAGE PLANS:
=C2=A0 Stage: Stage-1
=C2=A0=C2=A0=C2=A0 Map Reduce
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Map Operator Tree:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 TableScan
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 alias: d= ummy_parquet
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Statisti= cs: Num rows: 100000000 Data size: 700000000 Basic stats: COMPLETE Column s= tats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Select O= perator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 expressions: id (type: int)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 outputColumnNames: id
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Statistics: Num rows: 100000000 Data size: 700000000 Basic stats: COMPL= ETE Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Group By Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 aggregations: max(id)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 mode: hash
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 outputColumnNames: _col0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE = Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Reduce Output Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 sort order:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Statistics: Num rows: 1 Data size: 4 Basic stat= s: COMPLETE Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 value expressions: _col0 (type: int)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Reduce Operator Tree:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Group By Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 aggregations: max(VA= LUE._col0)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mode: mergepartial =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 outputColumnNames: _= col0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Statistics: Num rows= : 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 File Output Operator=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 compress= ed: false
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Statisti= cs: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 table: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 input format: org.apache.hadoop.mapred.TextInputFormat
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTe= xtOutputFormat
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

=C2=A0 Stage: Stage-0
=C2=A0=C2=A0=C2=A0 Fetch Operator
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= limit: -1
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Processor Tree:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ListSink

Time taken: 0.1 seconds, Fetched: 44 row(s)

=C2=A0

=C2=A0

HTH

=C2=A0


=C2=A0

On 12 July 2016 at 08:16, Markovitz, Dudu <dmarkovitz@paypal.com<= /a>> wrote:

This is a simple task =E2=80=93<= /span>

Read the files, find the local m= ax value and combine the results (find the global max value).=

How do you explain the differenc= es in the results? Spark reads the files and finds a local max 10X (+) faster than MR?

Can you please attach the execut= ion plan?

=C2=A0

Thanks

=C2=A0

Dudu

=C2=A0

=C2=A0

=C2=A0

From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.= com]
Sent: Monday, July 11, 2016 11:55 PM
To: user <user@hive.apache.org>; user @spark <user@spark.apache.org>
Subject: Re: Using Spark on Hive with Hive also using Spark as its e= xecution engine

=C2=A0

In my test I did like for like keeping the systemati= c the same namely:

=C2=A0

  1. Table was a parquet table of 100 Million rows
  2. The same set up was used for both Hive on Spark and Hive on MR
  3. Spark was very impressive compared to MR on this particular test.=

=C2=A0

Just to see any issues I created an ORC table in in = the image of Parquet (insert/select from Parquet to ORC) with stats updated= for columns etc

=C2=A0

These were the results of the same run using ORC tab= le this time:

=C2=A0

hive> select max(id) from oraclehadoop.dummy;

Starting Spa= rk Job =3D b886b869-5500-4ef7-aab9-ae6fb4dad22b

Query Hive on Spark job[1] stages:
2
3

Status: Running (Hive on Spark job[1])
Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-= FailedTasksCount)/TotalTasksCount [StageCost]
2016-07-11 21:35:45,020 Stage-2_0: 0(+8)/23=C2=A0=C2=A0=C2=A0=C2=A0 Stage-3= _0: 0/1
2016-07-11 21:35:48,033 Stage-2_0: 0(+8)/23=C2=A0=C2=A0=C2=A0=C2=A0 Stage-3= _0: 0/1
2016-07-11 21:35:51,046 Stage-2_0: 1(+8)/23=C2=A0=C2=A0=C2=A0=C2=A0 Stage-3= _0: 0/1
2016-07-11 21:35:52,050 Stage-2_0: 3(+8)/23=C2=A0=C2=A0=C2=A0=C2=A0 Stage-3= _0: 0/1
2016-07-11 21:35:53,055 Stage-2_0: 8(+4)/23=C2=A0=C2=A0=C2=A0=C2=A0 Stage-3= _0: 0/1
2016-07-11 21:35:54,060 Stage-2_0: 11(+1)/23=C2=A0=C2=A0=C2=A0 Stage-3_0: 0= /1
2016-07-11 21:35:55,065 Stage-2_0: 12(+0)/23=C2=A0=C2=A0=C2=A0 Stage-3_0: 0= /1
2016-07-11 21:35:56,071 Stage-2_0: 12(+8)/23=C2=A0=C2=A0=C2=A0 Stage-3_0: 0= /1
2016-07-11 21:35:57,076 Stage-2_0: 13(+8)/23=C2=A0=C2=A0=C2=A0 Stage-3_0: 0= /1
2016-07-11 21:35:58,081 Stage-2_0: 20(+3)/23=C2=A0=C2=A0=C2=A0 Stage-3_0: 0= /1
2016-07-11 21:35:59,085 Stage-2_0: 23/23 Finished=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Stage-3_0: 0(+1)/1
2016-07-11 21:36:00,089 Stage-2_0: 23/23 Finished=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Stage-3_0: 1/1 Finished
Status: Finished successfully in 16.08 seconds
OK
100000000
Time taken: 17.775 seconds, Fetched: 1 row(s)

=C2=A0

Repeat with MR engine

=C2=A0

hive> set hive.execution.engine=3Dmr;
Hive-on-MR is deprecated in Hive 2 and may not be available in the future v= ersions. Consider using a different execution engine (i.e. spark, tez) or u= sing Hive 1.X releases.

=C2=A0

hive> select max(id) from oraclehadoop.dummy;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the= future versions. Consider using a different execution engine (i.e. spark, = tez) or using Hive 1.X releases.
Query ID =3D hduser_20160711213100_8dc2afae-8644-4097-ba33-c7bd3c304bf8
Total jobs =3D 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
=C2=A0 set hive.exec.reducers.bytes.per.reducer=3D<number>
In order to limit the maximum number of reducers:
=C2=A0 set hive.exec.reducers.max=3D<number>
In order to set a constant number of reducers:
=C2=A0 set mapreduce.job.reduces=3D<number>
Starting Job =3D job_1468226887011_0008, Tracking URL =3D http://rhes564:8088/proxy/application_1468226887011_0008/
Kill Command =3D /home/hduser/hadoop-2.6.0/bin/hadoop job=C2=A0 -kill job_1= 468226887011_0008
Hadoop job information for Stage-1: number of mappers: 23; number of reduce= rs: 1
2016-07-11 21:37:00,061 Stage-1 map =3D 0%,=C2=A0 reduce =3D 0%
2016-07-11 21:37:06,440 Stage-1 map =3D 4%,=C2=A0 reduce =3D 0%, Cumulative= CPU 16.48 sec
2016-07-11 21:37:14,751 Stage-1 map =3D 9%,=C2=A0 reduce =3D 0%, Cumulative= CPU 40.63 sec
2016-07-11 21:37:22,048 Stage-1 map =3D 13%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 58.88 sec
2016-07-11 21:37:30,412 Stage-1 map =3D 17%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 80.72 sec
2016-07-11 21:37:37,707 Stage-1 map =3D 22%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 103.43 sec
2016-07-11 21:37:45,999 Stage-1 map =3D 26%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 125.93 sec
2016-07-11 21:37:54,300 Stage-1 map =3D 30%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 147.17 sec
2016-07-11 21:38:01,538 Stage-1 map =3D 35%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 166.56 sec
2016-07-11 21:38:08,807 Stage-1 map =3D 39%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 189.29 sec
2016-07-11 21:38:17,115 Stage-1 map =3D 43%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 211.03 sec
2016-07-11 21:38:24,363 Stage-1 map =3D 48%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 235.68 sec
2016-07-11 21:38:32,638 Stage-1 map =3D 52%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 258.27 sec
2016-07-11 21:38:40,916 Stage-1 map =3D 57%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 278.44 sec
2016-07-11 21:38:49,206 Stage-1 map =3D 61%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 300.35 sec
2016-07-11 21:38:58,524 Stage-1 map =3D 65%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 322.89 sec
2016-07-11 21:39:07,889 Stage-1 map =3D 70%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 344.8 sec
2016-07-11 21:39:16,151 Stage-1 map =3D 74%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 367.77 sec
2016-07-11 21:39:25,456 Stage-1 map =3D 78%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 391.82 sec
2016-07-11 21:39:33,725 Stage-1 map =3D 83%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 415.48 sec
2016-07-11 21:39:43,037 Stage-1 map =3D 87%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 436.09 sec
2016-07-11 21:39:51,292 Stage-1 map =3D 91%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 459.4 sec
2016-07-11 21:39:59,563 Stage-1 map =3D 96%,=C2=A0 reduce =3D 0%, Cumulativ= e CPU 477.92 sec
2016-07-11 21:40:05,760 Stage-1 map =3D 100%,=C2=A0 reduce =3D 0%, Cumulati= ve CPU 491.72 sec
2016-07-11 21:40:10,921 Stage-1 map =3D 100%,=C2=A0 reduce =3D 100%, Cumula= tive CPU 499.37 sec
MapReduce Total cumulative CPU time: 8 minutes 19 seconds 370 msec
Ended Job =3D job_1468226887011_0008
MapReduce Jobs Launched:
Stage-Stage-1: Map: 23=C2=A0 Reduce: 1=C2=A0=C2=A0 Cumulative CPU: 499.37 s= ec=C2=A0=C2=A0 HDFS Read: 403754774 HDFS Write: 10 SUCCESS
Total MapReduce CPU Time Spent: 8 minutes 19 seconds 370 msec
OK
100000000
Time taken: 202.333 seconds, Fetched: 1 row(s)

=C2=A0

So in summary

=C2=A0

Table=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 MR/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Spark/sec

Parquet=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 239.532=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 14.38

ORC=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0202.333=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A017.77=

=C2=A0

=C2=A0Still I would use Spark if I had a choice and = I agree that on VLT (very large tables), the limitation in available memory= may be=C2=A0the overriding factor in using Spark.

=C2=A0

HTH

=C2=A0


=C2=A0

On 11 July 2016 at 19:25, Gopal Vijayaraghavan <<= a href=3D"mailto:gopalv@apache.org" target=3D"_blank">gopalv@apache.org= > wrote:


> Status: Finished successfully in 14.12 seconds
> OK
> 100000000
> Time taken: 14.38 seconds, Fetched: 1 row(s)

That might be an improvement over MR, but that still feels far too slow.

Parquet numbers are in general bad in Hive, but that's because the Parq= uet
reader gets no actual love from the devs. The community, if it wants to
keep using Parquet heavily needs a Hive dev to go over to Parquet-mr and cut a significant number of memory copies out of the reader.

The Spark 2.0 build for instance, has a custom Parquet reader for SparkSQL<= br> which does this. SPARK-12854 does for Spark+Parquet what Hive 2.0 does for<= br> ORC (actually, it looks more like hive's VectorizedRowBatch than
Tungsten's flat layouts).

But that reader cannot be used in Hive-on-Spark, because it is not a
public reader impl.


Not to pick an arbitrary dataset, my workhorse example is a TPC-H lineitem<= br> at 10Gb scale with a single 16 box.

hive(tpch_flat_orc_10)> select max(l_discount) from lineitem;
Query ID =3D gopal_20160711175917_f96371aa-2721-49c8-99a0-f7c4a1eacfda
Total jobs =3D 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id
application_1466700718395_0256)

---------------------------------------------------------------------------=
-------------------
=C2=A0 =C2=A0 =C2=A0 =C2=A0 VERTICES=C2=A0 =C2=A0 =C2=A0 MODE=C2=A0 =C2=A0 = =C2=A0 =C2=A0 STATUS=C2=A0 TOTAL=C2=A0 COMPLETED=C2=A0 RUNNING
PENDING=C2=A0 FAILED=C2=A0 KILLED
---------------------------------------------------------------------------=
-------------------
Map 1 ..........=C2=A0 =C2=A0 =C2=A0 llap=C2=A0 =C2=A0 =C2=A0SUCCEEDED=C2= =A0 =C2=A0 =C2=A013=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A013=C2=A0 =C2=A0 =C2=A0= =C2=A0 0
0=C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A00
Reducer 2 ......=C2=A0 =C2=A0 =C2=A0 llap=C2=A0 =C2=A0 =C2=A0SUCCEEDED=C2= =A0 =C2=A0 =C2=A0 1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 =C2=A0 =C2=A0= =C2=A0 0
0=C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A00
---------------------------------------------------------------------------=
-------------------
VERTICES: 02/02=C2=A0 [=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D>>] 100%=C2=A0 ELAPSED TIME: 0.71 s

---------------------------------------------------------------------------=
-------------------
Status: DAG finished successfully in 0.71 seconds

Query Execution Summary
---------------------------------------------------------------------------=
-------------------
OPERATION=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 DURATION
---------------------------------------------------------------------------=
-------------------
Compile Query=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00.21s
Prepare Plan=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0.13s
Submit Plan=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00.34s
Start DAG=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00.23s
Run DAG=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00.71s
---------------------------------------------------------------------------=
-------------------

Task Execution Summary
---------------------------------------------------------------------------=
-------------------
=C2=A0 VERTICES=C2=A0 =C2=A0DURATION(ms)=C2=A0 CPU_TIME(ms)=C2=A0 GC_TIME(m= s)=C2=A0 INPUT_RECORDS
OUTPUT_RECORDS
---------------------------------------------------------------------------=
-------------------
=C2=A0 =C2=A0 =C2=A0Map 1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0604.00=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 0=C2=A0 =C2=A0 =C2=A059,957,438
=C2=A0 =C2=A0 =C2=A0 13
=C2=A0Reducer 2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0105.00=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A013
=C2=A0 =C2=A0 =C2=A0 =C2=A00
---------------------------------------------------------------------------=
-------------------

LLAP IO Summary
---------------------------------------------------------------------------=
-------------------
=C2=A0 VERTICES ROWGROUPS=C2=A0 META_HIT=C2=A0 META_MISS=C2=A0 DATA_HIT=C2= =A0 DATA_MISS=C2=A0 ALLOCATION
=C2=A0 =C2=A0 USED=C2=A0 TOTAL_IO
---------------------------------------------------------------------------=
-------------------
=C2=A0 =C2=A0 =C2=A0Map 1=C2=A0 =C2=A0 =C2=A0 6036=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 146=C2=A0 =C2=A0 =C2=A0 =C2=A0 0B=C2= =A0 =C2=A0 68.86MB=C2=A0 =C2=A0 491.00MB
479.89MB=C2=A0 =C2=A0 =C2=A07.94s
---------------------------------------------------------------------------=
-------------------

OK
0.1
Time taken: 1.669 seconds, Fetched: 1 row(s)
hive(tpch_flat_orc_10)>


This is running against a single 16 core box & I would assume it would<= br> take <1.4s to read twice as much (13 tasks is barely touching the load factors).

It would probably be a bit faster if the cache had hits, but in general
14s to read a 100M rows is nearly a magnitude off where Hive 2.2.0 is.

Cheers,
Gopal









=C2=A0

=C2=A0





Want to work at Handy? Check out our=C2=A0culture deck and open roles
Latest=C2=A0news=C2=A0at Handy
Handy=C2=A0just raised $50= m=C2=A0led by Fidelity

=

--94eb2c124a32576ecf053770662c--