hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "刘奎恩(局外)" <kuien....@alibaba-inc.com>
Subject 回复:Default LOG level for "data locality ratio:" events contribute to Hawq Master log bloat
Date Fri, 01 Dec 2017 06:52:35 GMT

It makes sense to adjust its LOG level to reduce log size on master.
1. It affects master performance if log file size is huge.Query on hawq_toolkit.hawq_log_master_concise is slower as time going.
https://issues.apache.org/jira/projects/HAWQ/issues/HAWQ-1550
2. It has little value when segments are seperated from storage nodes.If storage nodes
and computing nodes are geographically distributed,it is common on Cloud service, and no distributed
cache systems are available,then we have little interest to keep them in log in such cases.

data locality ratio
-------------——
Kuien Liu/奎恩
------------------------------------------------------------------发件人:Kyle Roberts
<kroberts@pivotal.io>发送时间:2017年12月1日(星期五) 05:41收件人:dev
<dev@hawq.incubator.apache.org>主 题:Default LOG level for "data locality ratio:"
events contribute to Hawq Master log bloat
Issue:


Default LOG level for "data locality ratio:" events contribute to Hawq
Master log bloat.  Should only be needed when investigating/tuning
performance.

Case Scenario:

*"data locality ratio:"* events can report multiple times and, depending
upon the query, many times.


Sample event:


2017-11-30 20:33:39.857251
GMT,"gpadmin","testdb",p280349,th-909919968,"[local]",,2017-11-30 20:06:35
GMT,1630750,con25419,cmd28,seg-1,,,,sx2,"LOG","00000","data locality ratio:
1.000; virtual segment number: 1; different host number: 1; virtual segment
number per host(avg/min/max): (1/1/1); segment size(avg/min/max): (1016.000
B/1016 B/1016 B); segment size with penalty(avg/min/max): (1016.000 B/1016
B/1016 B); continuity(avg/min/max): (1.000/1.000/1.000).",,,,,"SQL
statement ""select count(*)::float4 from public.test_tbl as Ta where Ta.id
is null""","analyze test_tbl;",0,,"cdbdatalocality.c",3372,



For example, Hawq master logs are seen with close to *20% of the file size*
is from only the *"data locality ratio:" *events.


Product Enhancement suggestion:


Because these are only needed for performance tuning (or for investigation):


In:


https://github.com/apache/incubator-hawq/blob/master/src/backend/cdb/cdbdatalocality.c#L3372


Could we change default LOG level to something like DEBUG1, etc?




- Kyle
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message