incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramesh Natarajan <rames...@gmail.com>
Subject Re: tmp files in /var/lib/cassandra/data
Date Thu, 15 Dec 2011 03:03:33 GMT
yep, so far it looks like a file descriptor leak.  Not sure if gc or
some other event like compaction would close these files..

[root@CAP-VM-1 ~]# ls -al /proc/31134/fd   | grep MSA | wc -l
540
[root@CAP-VM-1 ~]# ls -al /proc/31134/fd   | grep MSA | wc -l
542
[root@CAP-VM-1 ~]# ls -al /proc/31134/fd   | grep MSA | wc -l
554
[root@CAP-VM-1 ~]# ls -al /proc/31134/fd   | grep MSA | wc -l
558



On Wed, Dec 14, 2011 at 8:28 PM, Bryce Godfrey
<Bryce.Godfrey@azaleos.com> wrote:
> I'm seeing this also, and my nodes have started crashing with "too many open file errors".
 Running lsof I see lots of these open tmp files.
>
> java       8185      root  911u      REG               8,32      
  38  129108266 /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268721-CompressionInfo.db
> java       8185      root  912u      REG               8,32      
   0  155320741 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1092-Data.db
> java       8185      root  913u      REG               8,32      
   0  155320742 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1097-Index.db
> java       8185      root  914u      REG               8,32      
   0  155320743 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1097-Data.db
> java       8185      root  916u      REG               8,32      
   0  155320754 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1113-Data.db
> java       8185      root  918u      REG               8,32      
   0  155320744 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1102-Index.db
> java       8185      root  919u      REG               8,32      
   0  155320745 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1102-Data.db
> java       8185      root  920u      REG               8,32      
   0  155320755 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1118-Index.db
> java       8185      root  921u      REG               8,32      
   0  129108272 /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268781-Data.db
> java       8185      root  922u      REG               8,32      
  38  129108273 /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268781-CompressionInfo.db
> java       8185      root  923u      REG               8,32      
   0  155320756 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1118-Data.db
> java       8185      root  929u      REG               8,32      
  38  129108262 /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268822-CompressionInfo.db
> java       8185      root  947u      REG               8,32      
   0  129108284 /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268854-Data.db
> java       8185      root  948u      REG               8,32      
  38  129108285 /opt/cassandra/data/MonitoringData/Properties-tmp-hc-268854-CompressionInfo.db
> java       8185      root  954u      REG               8,32      
   0  155320746 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1107-Index.db
> java       8185      root  955u      REG               8,32      
   0  155320747 /opt/cassandra/data/system/HintsColumnFamily-tmp-hc-1107-Data.db
>
> Going to try rolling back to 1.0.5 for the time being even though I was hoping to use
one of the fixes in 1.0.6
>
> -----Original Message-----
> From: Ramesh Natarajan [mailto:ramesh25@gmail.com]
> Sent: Wednesday, December 14, 2011 6:03 PM
> To: user@cassandra.apache.org
> Subject: tmp files in /var/lib/cassandra/data
>
> We are using leveled compaction running cassandra 1.0.6.  I checked the data directory
(/var/lib/cassandra/data) and i see  these 0 bytes tmp files.
> What are these files?
>
> thanks
> Ramesh
>
> -rw-r--r-- 1 root root        0 Dec 14 17:15 uid-tmp-hc-106-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:15 uid-tmp-hc-106-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:23 uid-tmp-hc-117-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:23 uid-tmp-hc-117-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 15:51 uid-tmp-hc-11-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 15:51 uid-tmp-hc-11-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:31 uid-tmp-hc-129-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:31 uid-tmp-hc-129-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:40 uid-tmp-hc-142-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:40 uid-tmp-hc-142-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:40 uid-tmp-hc-145-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:40 uid-tmp-hc-145-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:47 uid-tmp-hc-158-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:47 uid-tmp-hc-158-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:47 uid-tmp-hc-162-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:47 uid-tmp-hc-162-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:55 uid-tmp-hc-175-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:55 uid-tmp-hc-175-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:55 uid-tmp-hc-179-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:55 uid-tmp-hc-179-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:03 uid-tmp-hc-193-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:03 uid-tmp-hc-193-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:03 uid-tmp-hc-197-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:03 uid-tmp-hc-197-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 16:02 uid-tmp-hc-19-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 16:02 uid-tmp-hc-19-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:03 uid-tmp-hc-200-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:03 uid-tmp-hc-200-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:11 uid-tmp-hc-213-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:11 uid-tmp-hc-213-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:11 uid-tmp-hc-217-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:11 uid-tmp-hc-217-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:19 uid-tmp-hc-230-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:19 uid-tmp-hc-230-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:19 uid-tmp-hc-235-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:19 uid-tmp-hc-235-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:27 uid-tmp-hc-249-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:27 uid-tmp-hc-249-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:27 uid-tmp-hc-253-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:27 uid-tmp-hc-253-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:28 uid-tmp-hc-257-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:28 uid-tmp-hc-257-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:35 uid-tmp-hc-270-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:35 uid-tmp-hc-270-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:36 uid-tmp-hc-275-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:36 uid-tmp-hc-275-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:44 uid-tmp-hc-288-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:44 uid-tmp-hc-288-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 16:10 uid-tmp-hc-28-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 16:10 uid-tmp-hc-28-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:44 uid-tmp-hc-293-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:44 uid-tmp-hc-293-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:52 uid-tmp-hc-307-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:52 uid-tmp-hc-307-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:52 uid-tmp-hc-310-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:52 uid-tmp-hc-310-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 18:52 uid-tmp-hc-315-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 18:52 uid-tmp-hc-315-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:00 uid-tmp-hc-328-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:00 uid-tmp-hc-328-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:00 uid-tmp-hc-333-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:00 uid-tmp-hc-333-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:08 uid-tmp-hc-347-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:08 uid-tmp-hc-347-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:08 uid-tmp-hc-353-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:08 uid-tmp-hc-353-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:09 uid-tmp-hc-357-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:09 uid-tmp-hc-357-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:17 uid-tmp-hc-370-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:17 uid-tmp-hc-370-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:17 uid-tmp-hc-375-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:17 uid-tmp-hc-375-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:25 uid-tmp-hc-389-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:25 uid-tmp-hc-389-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 16:18 uid-tmp-hc-38-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 16:18 uid-tmp-hc-38-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:25 uid-tmp-hc-393-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:25 uid-tmp-hc-393-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:25 uid-tmp-hc-399-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:25 uid-tmp-hc-399-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:33 uid-tmp-hc-412-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:33 uid-tmp-hc-412-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:34 uid-tmp-hc-418-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:34 uid-tmp-hc-418-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:42 uid-tmp-hc-432-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:42 uid-tmp-hc-432-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:42 uid-tmp-hc-438-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:42 uid-tmp-hc-438-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:42 uid-tmp-hc-442-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:42 uid-tmp-hc-442-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:50 uid-tmp-hc-455-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:50 uid-tmp-hc-455-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 19:50 uid-tmp-hc-460-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 19:50 uid-tmp-hc-460-Index.db
> -rw-r--r-- 1 root root 10486534 Dec 14 19:58 uid-tmp-hc-470-Data.db
> -rw-r--r-- 1 root root    35176 Dec 14 19:58 uid-tmp-hc-470-Filter.db
> -rw-r--r-- 1 root root   255860 Dec 14 19:58 uid-tmp-hc-470-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 16:26 uid-tmp-hc-49-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 16:26 uid-tmp-hc-49-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 15:38 uid-tmp-hc-5-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 15:38 uid-tmp-hc-5-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 16:35 uid-tmp-hc-61-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 16:35 uid-tmp-hc-61-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 16:43 uid-tmp-hc-74-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 16:43 uid-tmp-hc-74-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 16:50 uid-tmp-hc-80-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 16:50 uid-tmp-hc-80-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 16:59 uid-tmp-hc-87-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 16:59 uid-tmp-hc-87-Index.db
> -rw-r--r-- 1 root root        0 Dec 14 17:07 uid-tmp-hc-96-Data.db
> -rw-r--r-- 1 root root        0 Dec 14 17:07 uid-tmp-hc-96-Index.db

Mime
View raw message