impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu, Cheng A" <cheng.a...@intel.com>
Subject RE: performance issue on big table join
Date Thu, 02 Nov 2017 02:36:09 GMT
Looks like picture failed to see here. Attach the origin files as well as the capture.

Best Regards
Ferdinand Xu

From: Xu, Cheng A [mailto:cheng.a.xu@intel.com]
Sent: Thursday, November 2, 2017 10:33 AM
To: user@impala.incubator.apache.org; dev@impala <dev@impala.incubator.apache.org>
Subject: RE: performance issue on big table join

Thanks Junjie for the profile file. Another observation is that Impala has a much lower CPU
utilization than Spark SQL (Impala is on the left and Spark SQL is on the right).

[cid:image001.jpg@01D353C5.E69BA8A0]
I think the major reason here might be that Impala is single thread based execution for non-scanner
operator (join, aggregation) in single node [1].
Do you have any suggestion that we can improve its performance especially improving its CPU
utilization? Please correct me if I miss anything here.

[1] https://issues.apache.org/jira/browse/IMPALA-3902

Best Regards
Ferdinand Xu

From: 俊杰陈 [mailto:cjjnjust@gmail.com]
Sent: Thursday, November 2, 2017 10:00 AM
To: dev@impala <dev@impala.incubator.apache.org<mailto:dev@impala.incubator.apache.org>>;
user@impala.incubator.apache.org<mailto:user@impala.incubator.apache.org>
Subject: Re: performance issue on big table join

+user list

2017-11-02 9:57 GMT+08:00 俊杰陈 <cjjnjust@gmail.com<mailto:cjjnjust@gmail.com>>:
Hi Mostafa

Cheng already put the profile in thread.

Here is another profile for impala release version. you can also see the attachment.


2017-11-02 9:30 GMT+08:00 Mostafa Mokhtar <mmokhtar@cloudera.com<mailto:mmokhtar@cloudera.com>>:
Attaching the query profile will be most helpful to investigate this issue.

If you can capture the profile from the WebUI on the coordinator node it
would be great.

On Wed, Nov 1, 2017 at 6:22 PM, 俊杰陈 <cjjnjust@gmail.com<mailto:cjjnjust@gmail.com>>
wrote:

> Thanks Hongxu,
>
> Here are configurations on my cluster,  most of them are default values.
> Which item do you think it may impact?
>
>         ABORT_ON_DEFAULT_LIMIT_EXCEEDED: [0]
>         ABORT_ON_ERROR: [0]
>         ALLOW_UNSUPPORTED_FORMATS: [0]
>         APPX_COUNT_DISTINCT: [0]
>         BATCH_SIZE: [0]
>         COMPRESSION_CODEC: [NONE]
>         DEBUG_ACTION: []
>         DEFAULT_ORDER_BY_LIMIT: [-1]
>         DISABLE_CACHED_READS: [0]
>         DISABLE_CODEGEN: [0]
>         DISABLE_OUTERMOST_TOPN: [0]
>         DISABLE_ROW_RUNTIME_FILTERING: [0]
>         DISABLE_STREAMING_PREAGGREGATIONS: [0]
>         DISABLE_UNSAFE_SPILLS: [0]
>         ENABLE_EXPR_REWRITES: [1]
>         EXEC_SINGLE_NODE_ROWS_THRESHOLD: [100]
>         EXPLAIN_LEVEL: [1]
>         HBASE_CACHE_BLOCKS: [0]
>         HBASE_CACHING: [0]
>         MAX_BLOCK_MGR_MEMORY: [0]
>         MAX_ERRORS: [100]
>         MAX_IO_BUFFERS: [0]
>         MAX_NUM_RUNTIME_FILTERS: [10]
>         MAX_SCAN_RANGE_LENGTH: [0]
>         MEM_LIMIT: [0]
>         MT_DOP: [0]
>         NUM_NODES: [0]
>         NUM_SCANNER_THREADS: [0]
>         OPTIMIZE_PARTITION_KEY_SCANS: [0]
>         PARQUET_ANNOTATE_STRINGS_UTF8: [0]
>         PARQUET_FALLBACK_SCHEMA_RESOLUTION: [0]
>         PARQUET_FILE_SIZE: [0]
>         PREFETCH_MODE: [1]
>         QUERY_TIMEOUT_S: [0]
>         REPLICA_PREFERENCE: [0]
>         REQUEST_POOL: []
>         RESERVATION_REQUEST_TIMEOUT: [0]
>         RM_INITIAL_MEM: [0]
>         RUNTIME_BLOOM_FILTER_SIZE: [1048576]
>         RUNTIME_FILTER_MAX_SIZE: [16777216]
>         RUNTIME_FILTER_MIN_SIZE: [1048576]
>         RUNTIME_FILTER_MODE: [2]
>         RUNTIME_FILTER_WAIT_TIME_MS: [0]
>         S3_SKIP_INSERT_STAGING: [1]
>         SCAN_NODE_CODEGEN_THRESHOLD: [1800000]
>         SCHEDULE_RANDOM_REPLICA: [0]
>         SCRATCH_LIMIT: [-1]
>         SEQ_COMPRESSION_MODE: [0]
>         STRICT_MODE: [0]
>         SUPPORT_START_OVER: [false]
>         SYNC_DDL: [0]
>         V_CPU_CORES: [0]
>
> 2017-10-31 15:30 GMT+08:00 Hongxu Ma <interma@outlook.com<mailto:interma@outlook.com>>:
>
> > Hi JJ
> > Consider it only takes 3mins on SparkSQL, maybe there are some mistakes
> in
> > query options.
> > Try run "set;" in impala-shell and check all query options, e.g:
> >     BATCH_SIZE: [0]
> >     DISABLE_CODEGEN: [0]
> >     RUNTIME_FILTER_MODE: GLOBAL
> >
> > Just a guess, thanks.
> >
> > 在 27/10/2017 10:25, 俊杰陈 写道:
> > The profile file is damaged. Here is a screenshot for exec summary
> > [cid:ii_j999ymep1_15f5ba563aeabb91]
> > ​
> >
> > 2017-10-27 10:04 GMT+08:00 俊杰陈 <cjjnjust@gmail.com<mailto:cjjnjust@gmail.com><mailto:cjj<mailto:cjj>
> > njust@gmail.com<mailto:njust@gmail.com>>>:
> > Hi Devs
> >
> > I met a performance issue on big table join. The query takes more than 3
> > hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
> > cluster. when running query,  the left scanner and exchange node are very
> > slow.  Did I miss some key arguments?
> >
> > you can see profile file in attachment.
> >
> > [cid:ii_j9998pph2_15f5b92f2cf47020]
> > ​
> > --
> > Thanks & Best Regards
> >
> >
> >
> > --
> > Thanks & Best Regards
> >
> >
> > --
> > Regards,
> > Hongxu.
> >
>
>
>
> --
> Thanks & Best Regards
>



--
Thanks & Best Regards



--
Thanks & Best Regards
Mime
View raw message