hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay <vijay2...@gmail.com>
Subject File system filling with Logs - Hadoop+ Chukwa
Date Fri, 08 May 2009 18:51:54 GMT
Hi,
I am trying out Chukwa on HDFS and the filesystem is been filled up because
of the amount of logs been generated by hadoop filesystem, Can anyone help
me understand why this is happening and how can i reduce the speed atleast?

awapisvr:qawapiall:root > df -m
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sda2                19234      7987     10270  44% /
/dev/sda1                  282        24       243   9% /boot
none                      3983         0      3983   0% /dev/shm
/dev/sda3                38459       903     35604   3% /opt
/dev/sda8               338395     43645    277561  14% /spare
/dev/sda7                19234        77     18180   1% /tmp
/dev/sda5                38459      1701     34805   5% /var
64.68.126.215:/vol/vol2/logarchive
                        286720     52678    234043  19% /backup
awapisvr:qawapiall:root > df -m
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sda2                19234      7987     10270  44% /
/dev/sda1                  282        24       243   9% /boot
none                      3983         0      3983   0% /dev/shm
/dev/sda3                38459       903     35604   3% /opt
/dev/sda8               338395     43648    277559  14% /spare
/dev/sda7                19234        77     18180   1% /tmp
/dev/sda5                38459      1701     34805   5% /var
64.68.126.215:/vol/vol2/logarchive
                        286720     52678    234043  19% /backup


in matter of millisec

i see the below in the logs...

2009-05-08 18:50:38,351 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,353 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null


and

28527974-127.0.0.1-50010-1241721203027, blockid:
blk_-4466028835570444014_1290
2009-05-08 18:48:54,137 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-4466028835570444014_1290 terminating
2009-05-08 18:50:33,139 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-7272993078256778328_1291 src: /127.0.0.1:57067 dest: /127.0.0.1:50010
2009-05-08 18:50:34,222 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
127.0.0.1:57067, dest: /127.0.0.1:50010, bytes: 69046, op: HDFS_WRITE,
cliID: DFSClient_-223712422, srvID:
DS-2128527974-127.0.0.1-50010-1241721203027, blockid:
blk_-7272993078256778328_1291
2009-05-08 18:50:34,223 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-7272993078256778328_1291 terminating
2009-05-08 18:50:35,145 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-2496183264937203840_1292 src: /127.0.0.1:57136 dest: /127.0.0.1:50010
2009-05-08 18:50:41,920 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_2625289997411230424_1240




Regards,
</VJ>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message