hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-5046) Chukwa log4j appender logs corrupted data if the system is under high stress
Date Thu, 15 Jan 2009 08:03:59 GMT
Chukwa log4j appender logs corrupted data if the system is under high stress
----------------------------------------------------------------------------

                 Key: HADOOP-5046
                 URL: https://issues.apache.org/jira/browse/HADOOP-5046
             Project: Hadoop Core
          Issue Type: Bug
          Components: contrib/chukwa
         Environment: Redhat EL 5.1, Java 6
            Reporter: Eric Yang


Data from Iostat indicates that log files did not write properly when system is under high
stress.

2008-12-29 03:03:48,510 INFO org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec: Linux
2.6.9-55.ELsmp (example1002)  12/29/08^D
^D
avg-cpu:  %user   %nice    %sys %iowait   %idle^D
           1.19    0.35    0.85    2.63   94.99^D
^D
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz 
 await  svctm  %util^D
sda          0.13  33.31  3.02  3.53  281.94  311.02   140.97   155.51    90.52     0.56 
 86.19   2.30   1.51^D
sdb          4.52   3.45  5.93  1.51  107.12   39.67    53.56    19.83    19.74     0.07 
  9.98   3.53   2.63^D
sdc          4.57  22.76  7.12  1.71  395.58  195.76   197.79    97.88    66.93     0.24 
 27.13   3.90   3.44^D
sdd          4.52  18.17  6.13  1.65  151.17  158.59    75.58    79.30    39.81     0.45 
 57.96   3.84   2.98^D
^D
avg-cpu:  %user   %nice    %sys %iowait   %idle^D
          24.83    0.00    0.29    0.07   74.81^D
^D
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz 
 await  svctm  %util^D
sda          1.04   1.78 41.99  3.64 8706.75   45.10  4353.37    22.55   191.82     0.21 
  4.50   3.70  16.89^D
sdb          0.00   0.00  0.00  0.00    0.00    0.00     0.00   2008-12-29 03:08:48,513 INFO
org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec: Linux 2.6.9-55.ELsmp (example1002)
 12/29/08^D
^D

The most probable reason is that disk buffer got paged out before it is written to disk. 
Exec plugin can be configured to always flush on every output.  For hadoop logs, this need
to be fine tuned.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message