cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lee Parker <...@socialagency.com>
Subject Re: error during snapshot
Date Wed, 28 Apr 2010 02:01:17 GMT
The system is a ubuntu server running 8.04 LTS.  Now, I'm getting the
problem again this evening even with the addition of the swap space.

Lee Parker
On Tue, Apr 27, 2010 at 1:13 PM, Jonathan Shook <jshook@gmail.com> wrote:

> The allocation of memory may have failed depending on the available virtual
> memory, whether or not the memory would have been subsequently accessed by
> the process.  Some systems do the work of allocating physical pages only
> when they are accessed for the first time. I'm not sure if yours is one of
> them.
>
>
> On Tue, Apr 27, 2010 at 10:45 AM, Lee Parker <lee@socialagency.com> wrote:
>
>> Adding a swapfile fixed the error, but it doesn't look as though the
>> process is even using the swap file at all.
>>
>> Lee Parker
>> On Tue, Apr 27, 2010 at 9:49 AM, Eric Hauser <ewhauser@gmail.com> wrote:
>>
>>> Have you read this?
>>>
>>> http://forums.sun.com/thread.jspa?messageID=9734530
>>>
>>> <http://forums.sun.com/thread.jspa?messageID=9734530>I don't think EC2
>>> instances have any swap.
>>>
>>>
>>>
>>> On Tue, Apr 27, 2010 at 10:16 AM, Lee Parker <lee@socialagency.com>wrote:
>>>
>>>> Can anyone help with this?  It is preventing me from getting backups of
>>>> our cluster.
>>>>
>>>> Lee Parker
>>>> On Mon, Apr 26, 2010 at 10:02 PM, Lee Parker <lee@socialagency.com>wrote:
>>>>
>>>>> I was attempting to get a snapshot on our cassandra nodes.  I get the
>>>>> following error every time I run nodetool ... snapshot.
>>>>>
>>>>> Exception in thread "main" java.io.IOException: Cannot run program
>>>>> "ln": java.io.IOException: error=12, Cannot allocate memory
>>>>>  at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>>>> at
>>>>> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:221)
>>>>>  at
>>>>> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1060)
>>>>> at org.apache.cassandra.db.Table.snapshot(Table.java:256)
>>>>>  at
>>>>> org.apache.cassandra.service.StorageService.takeAllSnapshot(StorageService.java:1005)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>  at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at
>>>>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>>>>>  at
>>>>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>>>>> at
>>>>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>>>>>  at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>>>>> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>>>>>  at
>>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>>>>> at
>>>>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>>>>>  at
>>>>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426)
>>>>> at
>>>>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>>>>  at
>>>>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
>>>>> at
>>>>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359)
>>>>>  at
>>>>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>  at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>>>>>  at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>  at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>>>> at
>>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>>>>  at
>>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>>>> at
>>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:619)
>>>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>>>>> allocate memory
>>>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>>>>  at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>>>>  ... 34 more
>>>>>
>>>>> The nodes are both Amazon EC2 Large instances with 7.5G RAM (6
>>>>> allocated for Java heap) with two cores and only 70G of data in casssandra.
>>>>>  They have plenty of available RAM and HD space.  Has anyone else run
into
>>>>> this error?
>>>>>
>>>>> Lee Parker
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message