chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (CHUKWA-267) Collect shuffling information from userlogs/syslog
Date Thu, 28 May 2009 23:49:45 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714230#action_12714230
] 

Jerome Boulon edited comment on CHUKWA-267 at 5/28/09 4:48 PM:
---------------------------------------------------------------

the Add need to be done inside this loop
if (chukwaClient == null )  { ... }

Not sure to understand, why you're writing to a file then you're calling the parent method??
+      this.qw.write(RecordConstants.escapeAllButLastRecordSeparator("\n",this.layout.format(event)));
+      super.subAppend(event);

Chukwa registration should  be extracted to a separate class in order to be reused cross-appender.

+  if (chukwaClientIsNull) {
+        synchronized (chukwaLock) {
+          chukwaClientIsNull = false;
+          String log4jFileName = getFile();
+          String recordType = getRecordType();
+          long currentLength = 0L;
+          if (chukwaClient == null) {
+            chukwaClient = new ChukwaAgentController();
+            chukwaClientIsNull = false;
+          }
+          long adaptorID = chukwaClient.add(ChukwaAgentController.CharFileTailUTF8NewLineEscaped,
+              recordType,currentLength + " " + log4jFileName, currentLength);
+
+          // Setup a shutdownHook for the controller
+          clientFinalizer = new ClientFinalizer(chukwaClient);
+          Runtime.getRuntime().addShutdownHook(clientFinalizer);
[...]

      was (Author: jboulon):
    the Add need to be done inside this loop
if (chukwaClient != null )  { ... }

Not sure to understand, why you're writing to a file then you're calling the parent method??
+      this.qw.write(RecordConstants.escapeAllButLastRecordSeparator("\n",this.layout.format(event)));
+      super.subAppend(event);

Chukwa registration should  be extracted to a separate class in order to be reused cross-appender.

+  if (chukwaClientIsNull) {
+        synchronized (chukwaLock) {
+          chukwaClientIsNull = false;
+          String log4jFileName = getFile();
+          String recordType = getRecordType();
+          long currentLength = 0L;
+          if (chukwaClient == null) {
+            chukwaClient = new ChukwaAgentController();
+            chukwaClientIsNull = false;
+          }
+          long adaptorID = chukwaClient.add(ChukwaAgentController.CharFileTailUTF8NewLineEscaped,
+              recordType,currentLength + " " + log4jFileName, currentLength);
+
+          // Setup a shutdownHook for the controller
+          clientFinalizer = new ClientFinalizer(chukwaClient);
+          Runtime.getRuntime().addShutdownHook(clientFinalizer);
[...]
  
> Collect shuffling information from userlogs/syslog
> --------------------------------------------------
>
>                 Key: CHUKWA-267
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-267
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.2.0
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.1.2, 0.2.0
>
>         Attachments: CHUKWA-267.patch
>
>
> Hadoop uses a customized version of log4j appender (org.apache.hadoop.mapred.TaskLogAppender)
for tasks.  Shuffling information is written to userlogs/syslog by TaskLogAppender.  For chukwa
to collect shuffling information, we need to write a new TaskLogAppender to do this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message