Mailing-List: contact dev-help@hawq.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hawq.incubator.apache.org
From: dyozie <git@git.apache.org>
To: dev@hawq.incubator.apache.org
Reply-To: dev@hawq.incubator.apache.org
References: <git-pr-88-incubator-hawq-docs@git.apache.org>
In-Reply-To: <git-pr-88-incubator-hawq-docs@git.apache.org>
Subject: [GitHub] incubator-hawq-docs pull request #88: HAWQ-1261 - add discussion of HAWQ adm...
Content-Type: text/plain
Message-Id: <20170119231021.3E4F2DFB0E@git1-us-west.apache.org>
Date: Thu, 19 Jan 2017 23:10:21 +0000 (UTC)
archived-at: Thu, 19 Jan 2017 23:10:27 -0000

Github user dyozie commented on a diff in the pull request:

    https://github.com/apache/incubator-hawq-docs/pull/88#discussion_r96980622
  
    --- Diff: markdown/admin/logfiles.html.md.erb ---
    @@ -0,0 +1,272 @@
    +---
    +title: HAWQ Administrative Log Files
    +---
    +
    +Log files are files that include messages and other information about your HAWQ deployment, including the database and utilities.
    +
    +Every database instance in HAWQ \(master, standby, and segments\) runs a PostgreSQL database server with its own server log file. You generate log files when you invoke HAWQ management utilities directly, or indirectly via Ambari management operations. Additionally, other components in your HAWQ cluster (PXF, HDFS) generate log files of their own.
    +
    +These log files are distinctly located, formatted, configured, and managed.
    +
    +**Locating Log Files**
    +
    +HAWQ administrative log files reside in pre-defined or configured locations on the local file system of the HAWQ node.
    +
    +**Configuring Logging Parameters**
    +
    +Configurable logging parameters may affect what, when, and where messages are logged. You configure HAWQ administrative logging options via HAWQ server configuration parameters or command line options.
    +
    +**Managing Log Files**
    +
    +Log files are created and/or rotated at pre-defined or configured intervals.   Note that administrative log files are not automatically truncated or deleted. The administrator must implement and periodically run scripts to clean up these log files.
    +
    +## <a id="topic28"></a>HAWQ Database Server Log Files
    +
    +
    +### <a id="logfile_locate_db"></a>Locating HAWQ Log Files 
    +
    +Each HAWQ master, standby, and segment database instance has its own server log file. Daily log files are created in the `pg_log` subdirectory of the master and segment data directory on the respective HAWQ node. You can obtain the master data directory location from the `hawq_master_directory` property value set in the `$GPHOME/etc/`[`hawq-site.xml`](../reference/HAWQSampleSiteConfig.html) configuration file. Similarly, you can obtain the segment data directory location from the `hawq_segment_directory` property value from `hawq-site.xml`.
    +
    +The naming convention for HAWQ database server log files is `hawq-<date>_<time>.[csv|log]`. For example, `hawq-2017-01-02_061611.csv` or `hawq-2017-01-03_001704.log`. The number and size of log files present for a given \<date\> is dependent upon the values of certain HAWQ server configuration parameters (discussed later in this document).
    +
    +### <a id="logfile_format_db"></a>HAWQ Log Format
    +
    +The HAWQ server log files are written in text or comma-separated values \(CSV\) format.
    +
    +HAWQ log entries may include the following fields:
    +
    +<table>
    +  <tr><th>#</th><th>Field Name</th><th>Data Type</th><th>Description</th></tr>
    +  <tr><td>1</td><td>event_time</td><td>timestamp with time zone</td><td>Th time that the log entry was written to the log</td></tr>
    +  <tr><td>2</td><td>user_name</td><td>varchar(100)</td><td>The database user name</td></tr>
    +  <tr><td>3</td><td>database_name</td><td>varchar(100)</td><td>The database name</td></tr>
    +  <tr><td>4</td><td>process_id</td><td>varchar(10)</td><td>The system process ID (prefixed with "p")</td></tr>
    +  <tr><td>5</td><td>thread_id</td><td>varchar(50)</td><td>The thread count (prefixed with "th-")</td></tr>
    +  <tr><td>6</td><td>remote_host</td><td>varchar(100)</td><td>The hostname/address of the client machine (if on master node). The hostname/address of the master (if on segment node).</td></tr>
    +  <tr><td>7</td><td>remote_port</td><td>varchar(10)</td><td>The segment or master port number</td></tr>
    +  <tr><td>8</td><td>session_start_time</td><td>timestamp with time zone</td><td>The time the session connection was opened</td></tr>
    +  <tr><td>9</td><td>transaction_id</td><td>int</td><td>The top-level transaction ID on the master; this ID is the parent of any subtransactions.</td></tr>
    +  <tr><td>10</td><td>gp_session_id</td><td>text</td><td>The session identifier number (prefixed with "con")</td></tr>
    +  <tr><td>11</td><td>gp_command_count</td><td>text</td><td>The command number within a session (prefixed with "cmd")</td></tr>
    +  <tr><td>12</td><td>gp_segment</td><td>text</td><td>The segment content identifier. The master always has a content ID of -1.</td></tr>
    +  <tr><td>13</td><td>slice_id</td><td>text</td><td>The slice ID (portion of the query plan being executed)</td></tr>
    +  <tr><td>14</td><td>distr_tranx_id</td><td>text</td><td>The distributed transaction identifier</td></tr>
    +  <tr><td>15</td><td>local_tranx_id</td><td>text</td><td>The local transaction identifier</td></tr>
    +  <tr><td>16</td><td>sub_tranx_id</td><td>text</td><td>The subtransaction identifier</td></tr>
    +  <tr><td>17</td><td>event_severity</td><td>varchar(10)</td><td>The event severity; values include: LOG, ERROR, FATAL, PANIC, DEBUG1, DEBUG2</td></tr>
    +  <tr><td>18</td><td>sql_state_code</td><td>varchar(10)</td><td>The SQL state code associated with the log message</td></tr>
    +  <tr><td>19</td><td>event_message</td><td>text</td><td>The log or error message text</td></tr>
    +  <tr><td>20</td><td>event_detail</td><td>text</td><td>The detail message text associated with an error or warning message</td></tr>
    +  <tr><td>21</td><td>event_hint</td><td>text</td><td>The hint message text associated with an error or warning message</td></tr>
    +  <tr><td>22</td><td>internal_query</td><td>text</td><td>The internally-generated query text</td></tr>
    +  <tr><td>23</td><td>internal_query_pos</td><td>int</td><td>The cursor index into the internally-generated query text</td></tr>
    +  <tr><td>24</td><td>event_context</td><td>text</td><td>The context in which this message is generated</td></tr>
    +  <tr><td>25</td><td>debug_query_string</td><td>text</td><td>User-supplied query string with full detail for debugging. This string can be modified for internal use.</td></tr>
    +  <tr><td>26</td><td>error_cursor_pos</td><td>int</td><td>The cursor index into the query string</td></tr>
    +  <tr><td>27</td><td>func_name</td><td>text</td><td>The function in which this message is generated</td></tr>
    +  <tr><td>28</td><td>file_name</td><td>text</td><td>The name of the source file in which the message originated</td></tr>
    +  <tr><td>29</td><td>file_line</td><td>int</td><td>The line number of the source file in which the message originated</td></tr>
    +  <tr><td>30</td><td>stack_trace</td><td>text</td><td>The stack trace text associated with this message</td></tr>
    +</table>
    +
    +**Note**: Log entries may not include values for all log fields. For example, the `slice_id` field is populated only for log entries associated with a query worker process.
    +
    +Example HAWQ server log file entries in a `.csv` file:
    +
    +``` pre
    +2017-01-02 01:19:55.293050 PST,"gpadmin","testdb",p381034,th-1259067104,"[local]",,2017-01-02 01:19:21 PST,28083,con3595,cmd7,seg-1,,,x28083,sx1,"LOG","00000","ConnID 207. Acquired resource from resource manager, (256 MB, 0.062500 CORE) x 1.",,,,,,"INSERT INTO extblwstrings VALUES ('my string');",0,,"rmcomm_QD2RM.c",868,
    +2017-01-02 06:16:06.383827 PST,,,p360256,th-1259067104,,,,0,,,seg-10000,,,,,"LOG","00000","database system is shut down",,,,,,,0,,"xlog.c",7882,
    +```
    +
    +Example HAWQ server log file entries in a `.log` file:
    +
    +``` pre
    +2017-01-03 00:17:04.848268 PST|||706424|startup||:-LOG:  database system was shut down at 2017-01-03 00:16:58 PST
    +2017-01-03 00:17:04.988778 PST|gpadmin|template1|706437|cmd1|x31410|:-LOG:  statement: SELECT oid,rolname,rolsuper,rolresqueue FROM pg_authid
    +2017-01-03 00:17:04.990520 PST|||706432|master resource manager||:-LOG:  Resource manager successfully loaded role specifications.
    +```
    +
    +### <a id="logfile_examine_db"></a>Examining HAWQ Log Files 
    +
    +You will examine the HAWQ log files to obtain information about your HAWQ deployment, as well as diagnose problems.
    +
    +Identify log entries of related transactions using the `transaction_id`. You can identify related log entries of a particular query by the query's session identifier \(`gp_session_id`\) and command identifier \(`gp_command_count`\).
    +
    +Alternatively, you can use the [*hawq_toolkit*](../reference/toolkit/hawq_toolkit.html#topic16) administrative schema to query HAWQ log files when the [`gp_log_format`](../reference/guc/parameter_definitions.html#gp_log_format) server configuration parameter value specifies `csv` format log files.
    +
    --- End diff --
    
    This is great info, but are there any example SQL commands we could include or link to from here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---