chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CHUKWA-6) Chukwa log4j appender logs corrupted data if the system is under high stress
Date Mon, 13 Jul 2009 17:43:14 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730421#action_12730421
] 

Eric Yang commented on CHUKWA-6:
--------------------------------

We implemented more proper method for locking pid file (daemon watcher) for exec system metrics.
 This shouldn't happen anymore in the normal operation.  We can close this as fixed.



> Chukwa log4j appender logs corrupted data if the system is under high stress
> ----------------------------------------------------------------------------
>
>                 Key: CHUKWA-6
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-6
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Assignee: Ari Rabkin
>
> Data from Iostat indicates that log files did not write properly when system is under
high stress.
> 2008-12-29 03:03:48,510 INFO org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec:
Linux 2.6.9-55.ELsmp (example1002)  12/29/08^D
> ^D
> avg-cpu:  %user   %nice    %sys %iowait   %idle^D
>            1.19    0.35    0.85    2.63   94.99^D
> ^D
> Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz
  await  svctm  %util^D
> sda          0.13  33.31  3.02  3.53  281.94  311.02   140.97   155.51    90.52     0.56
  86.19   2.30   1.51^D
> sdb          4.52   3.45  5.93  1.51  107.12   39.67    53.56    19.83    19.74     0.07
   9.98   3.53   2.63^D
> sdc          4.57  22.76  7.12  1.71  395.58  195.76   197.79    97.88    66.93     0.24
  27.13   3.90   3.44^D
> sdd          4.52  18.17  6.13  1.65  151.17  158.59    75.58    79.30    39.81     0.45
  57.96   3.84   2.98^D
> ^D
> avg-cpu:  %user   %nice    %sys %iowait   %idle^D
>           24.83    0.00    0.29    0.07   74.81^D
> ^D
> Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz
  await  svctm  %util^D
> sda          1.04   1.78 41.99  3.64 8706.75   45.10  4353.37    22.55   191.82     0.21
   4.50   3.70  16.89^D
> sdb          0.00   0.00  0.00  0.00    0.00    0.00     0.00   2008-12-29 03:08:48,513
INFO org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec: Linux 2.6.9-55.ELsmp (example1002)
 12/29/08^D
> ^D
> The most probable reason is that disk buffer got paged out before it is written to disk.
 Exec plugin can be configured to always flush on every output.  For hadoop logs, this need
to be fine tuned.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message