cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: error during snapshot
Date Thu, 29 Apr 2010 04:46:16 GMT
Interesting.  Googling your error turns up
http://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error12-cannot-allocate-memory-calling-runt

Why not just leave the swap on?  It's usually a Good Thing to be able
to page out unused memory, and use the ram for buffer cache instead.

On Wed, Apr 28, 2010 at 9:46 AM, Lee Parker <lee@socialagency.com> wrote:
> The thing is, that I'm not running close to being out of memory.  The data
> from nodetool info is showing that only about half of the available heap
> space is being used and running free from the command line shows that I have
> plenty of RAM available and some usage of the 1G swap space which is always
> on.
> nodetool info:
> Load             : 73.24 GB
> Generation No    : 1271626230
> Uptime (seconds) : 839414
> Heap Memory (MB) : 2584.36 / 5461.38
> free -m:
>              total       used       free     shared    buffers    
cached
> Mem:          7680       7640         39          0          8
      2364
> -/+ buffers/cache:       5266       2413
> Swap:         1023        388        635
> Lee Parker
>
> On Wed, Apr 28, 2010 at 9:18 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> If you're running so close to the edge of running out of memory that
>> creating a ln process pushes you over the edge, you should fix the
>> broader cause instead of the specific symptom. :)
>>
>> On Tue, Apr 27, 2010 at 10:09 PM, Lee Parker <lee@socialagency.com> wrote:
>> > So, after reading the thread which Eric posted earlier, I have created a
>> > workaround for the issue.  In my backup script, I add a swapfile with
>> > swapon, tell cassandra to create the snapshots, then remove the swapfile
>> > with swapoff.  Then I continue with the rest of the work the backup
>> > script
>> > needs to do in gathering up the snapshots into a tarball and pushing it
>> > to
>> > S3.
>> >
>> > Lee Parker
>> >
>> > On Tue, Apr 27, 2010 at 9:01 PM, Lee Parker <lee@socialagency.com>
>> > wrote:
>> >>
>> >> The system is a ubuntu server running 8.04 LTS.  Now, I'm getting the
>> >> problem again this evening even with the addition of the swap space.
>> >>
>> >> Lee Parker
>> >>
>> >> On Tue, Apr 27, 2010 at 1:13 PM, Jonathan Shook <jshook@gmail.com>
>> >> wrote:
>> >>>
>> >>> The allocation of memory may have failed depending on the available
>> >>> virtual memory, whether or not the memory would have been subsequently
>> >>> accessed by the process.  Some systems do the work of allocating
>> >>> physical
>> >>> pages only when they are accessed for the first time. I'm not sure if
>> >>> yours
>> >>> is one of them.
>> >>>
>> >>> On Tue, Apr 27, 2010 at 10:45 AM, Lee Parker <lee@socialagency.com>
>> >>> wrote:
>> >>>>
>> >>>> Adding a swapfile fixed the error, but it doesn't look as though
the
>> >>>> process is even using the swap file at all.
>> >>>>
>> >>>> Lee Parker
>> >>>>
>> >>>> On Tue, Apr 27, 2010 at 9:49 AM, Eric Hauser <ewhauser@gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> Have you read this?
>> >>>>> http://forums.sun.com/thread.jspa?messageID=9734530
>> >>>>> I don't think EC2 instances have any swap.
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Apr 27, 2010 at 10:16 AM, Lee Parker <lee@socialagency.com>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> Can anyone help with this?  It is preventing me from getting
>> >>>>>> backups
>> >>>>>> of our cluster.
>> >>>>>>
>> >>>>>> Lee Parker
>> >>>>>>
>> >>>>>> On Mon, Apr 26, 2010 at 10:02 PM, Lee Parker <lee@socialagency.com>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> I was attempting to get a snapshot on our cassandra
nodes.  I get
>> >>>>>>> the
>> >>>>>>> following error every time I run nodetool ... snapshot.
>> >>>>>>> Exception in thread "main" java.io.IOException: Cannot
run program
>> >>>>>>> "ln": java.io.IOException: error=12, Cannot allocate
memory
>> >>>>>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:221)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1060)
>> >>>>>>> at org.apache.cassandra.db.Table.snapshot(Table.java:256)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> org.apache.cassandra.service.StorageService.takeAllSnapshot(StorageService.java:1005)
>> >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>> >>>>>>> at
>> >>>>>>> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>> >>>>>>> at
>> >>>>>>> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>> >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>> >>>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> >>>>>>> at java.security.AccessController.doPrivileged(Native
Method)
>> >>>>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>>>> at java.lang.Thread.run(Thread.java:619)
>> >>>>>>> Caused by: java.io.IOException: java.io.IOException:
error=12,
>> >>>>>>> Cannot
>> >>>>>>> allocate memory
>> >>>>>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>> >>>>>>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>> >>>>>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>> >>>>>>> ... 34 more
>> >>>>>>> The nodes are both Amazon EC2 Large instances with 7.5G
RAM (6
>> >>>>>>> allocated for Java heap) with two cores and only 70G
of data in
>> >>>>>>> casssandra.
>> >>>>>>>  They have plenty of available RAM and HD space.  Has
anyone else
>> >>>>>>> run into
>> >>>>>>> this error?
>> >>>>>>>
>> >>>>>>> Lee Parker
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message