incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Crash when uploading large data sets
Date Sat, 14 May 2011 03:45:58 GMT
You should upgrade to the latest Sun JVM. OpenJDK is almost a year
behind in bug fixes.

On Fri, May 13, 2011 at 11:40 AM, James Cipar <jcipar@cmu.edu> wrote:
> It is a 64 bit VM.  I didn't notice the hs_err_pid.log files since I'm staring over
ssh, so they're in my home directory instead of my working directory.  I've attached one
of those below.  I don't know much about Java, so I'm not sure how to interpret this file.
>
>
>
>
>
> #
> # An unexpected error has been detected by Java Runtime Environment:
> #
> #  Internal Error (nmethod.cpp:1707), pid=10068, tid=1085823312
> #  Error: guarantee(cont_offset != 0,"unhandled implicit exception in compiled code")
> #
> # Java VM: OpenJDK 64-Bit Server VM (1.6.0_0-b11 mixed mode linux-amd64)
> # If you would like to submit a bug report, please visit:
> #   http://icedtea.classpath.org/bugzilla
> #
>
> ---------------  T H R E A D  ---------------
>
> Current thread (0x00007f7e441a9c00):  JavaThread "EXPIRING-MAP-TIMER-1" daemon [_thread_in_Java,
id=10095, stack(0x0000000040b65000,0x0000000040b86000)]
>
> Stack: [0x0000000040b65000,0x0000000040b86000],  sp=0x0000000040b83eb0,  free space=123k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> V  [libjvm.so+0x5d198f]
> V  [libjvm.so+0x5d1cf1]
> V  [libjvm.so+0x27e506]
> V  [libjvm.so+0x4970d6]
> V  [libjvm.so+0x514cd9]
> V  [libjvm.so+0x4b0b64]
>
>
> ---------------  P R O C E S S  ---------------
>
> Java Threads: ( => current thread )
>  0x0000000002913c00 JavaThread "NonPeriodicTasks:1" [_thread_blocked, id=10233, stack(0x00000000401fc000,0x000000004021d000)]
>  0x00000000026eec00 JavaThread "FILEUTILS-DELETE-POOL:1" [_thread_blocked, id=10222,
stack(0x0000000042533000,0x0000000042554000)]
>  0x00007f7e3c01c000 JavaThread "Thread-22" [_thread_in_native, id=10212, stack(0x0000000042512000,0x0000000042533000)]
>  0x000000000253c000 JavaThread "Thread-21" [_thread_in_native, id=10211, stack(0x0000000040dfc000,0x0000000040e1d000)]
>  0x000000000253a800 JavaThread "Thread-20" [_thread_in_native, id=10210, stack(0x0000000040ddb000,0x0000000040dfc000)]
>  0x0000000002537400 JavaThread "Thread-19" [_thread_in_native, id=10209, stack(0x00000000424f1000,0x0000000042512000)]
>  0x0000000002725c00 JavaThread "pool-1-thread-1" [_thread_in_native, id=10208, stack(0x000000004007d000,0x000000004009e000)]
>  0x00000000026f7400 JavaThread "Thread-18" [_thread_in_native, id=10207, stack(0x000000004034f000,0x0000000040370000)]
>  0x0000000002902800 JavaThread "Thread-17" [_thread_in_native, id=10206, stack(0x0000000040d6b000,0x0000000040d8c000)]
>  0x0000000002901400 JavaThread "Thread-16" [_thread_in_native, id=10205, stack(0x00000000424d0000,0x00000000424f1000)]
>  0x0000000002613c00 JavaThread "Thread-15" [_thread_in_native, id=10204, stack(0x000000004024e000,0x000000004026f000)]
>  0x00000000026ad800 JavaThread "Thread-14" [_thread_in_native, id=10203, stack(0x0000000040d11000,0x0000000040d32000)]
>  0x000000000276f000 JavaThread "Thread-13" [_thread_in_native, id=10202, stack(0x00000000424af000,0x00000000424d0000)]
>  0x00000000026b2c00 JavaThread "Thread-12" [_thread_in_native, id=10201, stack(0x0000000040490000,0x00000000404b1000)]
>  0x00000000026aec00 JavaThread "Thread-11" [_thread_in_native, id=10200, stack(0x000000004248e000,0x00000000424af000)]
>  0x000000000254ec00 JavaThread "Thread-10" [_thread_in_native, id=10199, stack(0x000000004246d000,0x000000004248e000)]
>  0x000000000254d000 JavaThread "Thread-9" [_thread_in_native, id=10198, stack(0x000000004244c000,0x000000004246d000)]
>  0x0000000002505000 JavaThread "Thread-8" [_thread_in_native, id=10197, stack(0x000000004242b000,0x000000004244c000)]
>  0x0000000002502400 JavaThread "Thread-7" [_thread_in_native, id=10196, stack(0x000000004240a000,0x000000004242b000)]
>  0x0000000002500400 JavaThread "WRITE-/172.19.149.80" [_thread_blocked, id=10195, stack(0x00000000407dd000,0x00000000407fe000)]
>  0x00000000024ff000 JavaThread "WRITE-/172.19.149.80" [_thread_blocked, id=10194, stack(0x0000000040c8e000,0x0000000040caf000)]
>  0x00000000024f4400 JavaThread "WRITE-/172.19.149.64" [_thread_blocked, id=10193, stack(0x00000000423e9000,0x000000004240a000)]
>  0x00000000024f3000 JavaThread "WRITE-/172.19.149.64" [_thread_blocked, id=10192, stack(0x00000000423c8000,0x00000000423e9000)]
>  0x00000000024f1400 JavaThread "WRITE-/172.19.149.71" [_thread_blocked, id=10191, stack(0x0000000040a2d000,0x0000000040a4e000)]
>  0x00000000024f0000 JavaThread "WRITE-/172.19.149.71" [_thread_blocked, id=10190, stack(0x00000000423a7000,0x00000000423c8000)]
>  0x00000000024ee400 JavaThread "WRITE-/172.19.149.62" [_thread_blocked, id=10189, stack(0x0000000042386000,0x00000000423a7000)]
>  0x00000000026bf800 JavaThread "WRITE-/172.19.149.62" [_thread_blocked, id=10188, stack(0x0000000042365000,0x0000000042386000)]
>  0x00000000026bdc00 JavaThread "WRITE-/172.19.149.72" [_thread_blocked, id=10187, stack(0x0000000040dba000,0x0000000040ddb000)]
>  0x00000000026bc400 JavaThread "WRITE-/172.19.149.72" [_thread_blocked, id=10186, stack(0x0000000042344000,0x0000000042365000)]
>  0x00000000026bac00 JavaThread "WRITE-/172.19.149.63" [_thread_blocked, id=10185, stack(0x0000000040431000,0x0000000040452000)]
>  0x00000000026b9400 JavaThread "WRITE-/172.19.149.63" [_thread_blocked, id=10184, stack(0x0000000042323000,0x0000000042344000)]
>  0x00000000026b8000 JavaThread "WRITE-/172.19.149.61" [_thread_blocked, id=10183, stack(0x0000000042302000,0x0000000042323000)]
>  0x00000000024b9c00 JavaThread "WRITE-/172.19.149.61" [_thread_blocked, id=10182, stack(0x0000000040c6d000,0x0000000040c8e000)]
>  0x00000000024b8400 JavaThread "WRITE-/172.19.149.69" [_thread_blocked, id=10181, stack(0x00000000422e1000,0x0000000042302000)]
>  0x00000000024b7400 JavaThread "WRITE-/172.19.149.69" [_thread_blocked, id=10180, stack(0x00000000422c0000,0x00000000422e1000)]
>  0x00000000024b3800 JavaThread "Thread-6" [_thread_in_native, id=10179, stack(0x0000000040e98000,0x0000000040eb9000)]
>  0x00000000026e6400 JavaThread "Thread-5" [_thread_in_native, id=10178, stack(0x0000000041222000,0x0000000041243000)]
>  0x00007f7e4420ec00 JavaThread "ReadRepairStage:8" [_thread_blocked, id=10177, stack(0x00000000407bc000,0x00000000407dd000)]
>  0x00007f7e4420d400 JavaThread "ReadRepairStage:7" [_thread_blocked, id=10176, stack(0x000000004229f000,0x00000000422c0000)]
>  0x00007f7e4420c000 JavaThread "ReadRepairStage:6" [_thread_blocked, id=10175, stack(0x000000004227e000,0x000000004229f000)]
>  0x00007f7e4420a800 JavaThread "ReadRepairStage:5" [_thread_blocked, id=10174, stack(0x00000000418ec000,0x000000004190d000)]
>  0x00007f7e44209400 JavaThread "ReadRepairStage:4" [_thread_blocked, id=10173, stack(0x0000000040a0c000,0x0000000040a2d000)]
>  0x00007f7e44208000 JavaThread "ReadRepairStage:3" [_thread_blocked, id=10172, stack(0x000000004225d000,0x000000004227e000)]
>  0x00007f7e44206800 JavaThread "ReadRepairStage:2" [_thread_blocked, id=10171, stack(0x000000004223c000,0x000000004225d000)]
>  0x00007f7e44205400 JavaThread "ReadRepairStage:1" [_thread_blocked, id=10170, stack(0x00000000418cb000,0x00000000418ec000)]
>  0x00007f7e44203c00 JavaThread "MiscStage:1" [_thread_blocked, id=10169, stack(0x00000000418aa000,0x00000000418cb000)]
>  0x00007f7e44202800 JavaThread "MigrationStage:1" [_thread_blocked, id=10168, stack(0x0000000041889000,0x00000000418aa000)]
>  0x00007f7e44201000 JavaThread "AntiEntropyStage:1" [_thread_blocked, id=10167, stack(0x0000000041868000,0x0000000041889000)]
>  0x00007f7e441ffc00 JavaThread "GossipStage:1" [_thread_blocked, id=10166, stack(0x0000000041847000,0x0000000041868000)]
>  0x00007f7e441fe400 JavaThread "StreamStage:1" [_thread_blocked, id=10165, stack(0x00000000417fe000,0x000000004181f000)]
>  0x00007f7e441fd000 JavaThread "InternalResponseStage:8" [_thread_blocked, id=10164,
stack(0x00000000417dd000,0x00000000417fe000)]
>  0x00007f7e441fb800 JavaThread "InternalResponseStage:7" [_thread_blocked, id=10163,
stack(0x0000000040c4c000,0x0000000040c6d000)]
>  0x00007f7e441fa400 JavaThread "InternalResponseStage:6" [_thread_blocked, id=10162,
stack(0x000000004177a000,0x000000004179b000)]
>  0x00007f7e441f8c00 JavaThread "InternalResponseStage:5" [_thread_blocked, id=10161,
stack(0x000000004194e000,0x000000004196f000)]
>  0x00007f7e441f7800 JavaThread "InternalResponseStage:4" [_thread_blocked, id=10160,
stack(0x000000004027f000,0x00000000402a0000)]
>  0x00007f7e441f6000 JavaThread "InternalResponseStage:3" [_thread_blocked, id=10159,
stack(0x0000000040811000,0x0000000040832000)]
>  0x00007f7e441f4c00 JavaThread "InternalResponseStage:2" [_thread_blocked, id=10158,
stack(0x000000004221b000,0x000000004223c000)]
>  0x00007f7e441f3400 JavaThread "InternalResponseStage:1" [_thread_blocked, id=10157,
stack(0x00000000409eb000,0x0000000040a0c000)]
>  0x00007f7e441f2000 JavaThread "RequestResponseStage:8" [_thread_blocked, id=10156,
stack(0x0000000041759000,0x000000004177a000)]
>  0x00007f7e441f0800 JavaThread "RequestResponseStage:7" [_thread_blocked, id=10155,
stack(0x0000000041e76000,0x0000000041e97000)]
>  0x00007f7e44315000 JavaThread "RequestResponseStage:6" [_thread_blocked, id=10154,
stack(0x0000000040bf1000,0x0000000040c12000)]
>  0x00007f7e44313800 JavaThread "RequestResponseStage:5" [_thread_blocked, id=10153,
stack(0x0000000041668000,0x0000000041689000)]
>  0x00007f7e44312400 JavaThread "RequestResponseStage:4" [_thread_blocked, id=10152,
stack(0x0000000040d99000,0x0000000040dba000)]
>  0x00007f7e44310c00 JavaThread "RequestResponseStage:3" [_thread_blocked, id=10151,
stack(0x00000000409ca000,0x00000000409eb000)]
>  0x00007f7e4430f800 JavaThread "RequestResponseStage:2" [_thread_blocked, id=10150,
stack(0x000000004079b000,0x00000000407bc000)]
>  0x00007f7e4430e000 JavaThread "RequestResponseStage:1" [_thread_blocked, id=10149,
stack(0x0000000041621000,0x0000000041642000)]
>  0x00007f7e4430cc00 JavaThread "ReadStage:8" [_thread_blocked, id=10148, stack(0x00000000421fa000,0x000000004221b000)]
>  0x00007f7e4430b400 JavaThread "ReadStage:7" [_thread_blocked, id=10147, stack(0x0000000040738000,0x0000000040759000)]
>  0x00007f7e4430a000 JavaThread "ReadStage:6" [_thread_blocked, id=10146, stack(0x0000000041264000,0x0000000041285000)]
>  0x00007f7e44308800 JavaThread "ReadStage:5" [_thread_blocked, id=10145, stack(0x0000000041e55000,0x0000000041e76000)]
>  0x00007f7e44307400 JavaThread "ReadStage:4" [_thread_blocked, id=10144, stack(0x0000000041243000,0x0000000041264000)]
>  0x00007f7e44305c00 JavaThread "ReadStage:3" [_thread_blocked, id=10143, stack(0x0000000041600000,0x0000000041621000)]
>  0x00007f7e44304800 JavaThread "ReadStage:2" [_thread_blocked, id=10142, stack(0x00000000415df000,0x0000000041600000)]
>  0x00007f7e44303000 JavaThread "ReadStage:1" [_thread_blocked, id=10141, stack(0x0000000041c20000,0x0000000041c41000)]
>  0x00007f7e44301c00 JavaThread "MutationStage:32" [_thread_blocked, id=10140, stack(0x0000000041738000,0x0000000041759000)]
>  0x00007f7e44300400 JavaThread "MutationStage:31" [_thread_blocked, id=10139, stack(0x0000000040982000,0x00000000409a3000)]
>  0x00007f7e442ff000 JavaThread "MutationStage:30" [_thread_blocked, id=10138, stack(0x00000000415be000,0x00000000415df000)]
>  0x00007f7e442fdc00 JavaThread "MutationStage:29" [_thread_blocked, id=10137, stack(0x000000004032e000,0x000000004034f000)]
>  0x00007f7e442fc400 JavaThread "MutationStage:28" [_thread_blocked, id=10136, stack(0x0000000041826000,0x0000000041847000)]
>  0x00007f7e442fb000 JavaThread "MutationStage:27" [_thread_blocked, id=10135, stack(0x0000000041bff000,0x0000000041c20000)]
>  0x00007f7e442f9800 JavaThread "MutationStage:26" [_thread_blocked, id=10134, stack(0x0000000040c2b000,0x0000000040c4c000)]
>  0x00007f7e442f8400 JavaThread "MutationStage:25" [_thread_blocked, id=10133, stack(0x0000000040117000,0x0000000040138000)]
>  0x00007f7e442f6c00 JavaThread "MutationStage:24" [_thread_blocked, id=10132, stack(0x0000000040ba7000,0x0000000040bc8000)]
>  0x00007f7e442f5800 JavaThread "MutationStage:23" [_thread_blocked, id=10131, stack(0x0000000040bd0000,0x0000000040bf1000)]
>  0x00007f7e442f4400 JavaThread "MutationStage:22" [_thread_blocked, id=10130, stack(0x0000000041717000,0x0000000041738000)]
>  0x00007f7e442f2c00 JavaThread "MutationStage:21" [_thread_blocked, id=10129, stack(0x00000000417bc000,0x00000000417dd000)]
>  0x00007f7e442f1800 JavaThread "MutationStage:20" [_thread_blocked, id=10128, stack(0x0000000040d4a000,0x0000000040d6b000)]
>  0x00007f7e442f0000 JavaThread "MutationStage:19" [_thread_blocked, id=10127, stack(0x0000000040564000,0x0000000040585000)]
>  0x00007f7e442eec00 JavaThread "MutationStage:18" [_thread_blocked, id=10126, stack(0x0000000040410000,0x0000000040431000)]
>  0x00007f7e442ed400 JavaThread "MutationStage:17" [_thread_blocked, id=10125, stack(0x0000000040717000,0x0000000040738000)]
>  0x00007f7e442ec000 JavaThread "MutationStage:16" [_thread_blocked, id=10124, stack(0x0000000040b86000,0x0000000040ba7000)]
>  0x00007f7e442eac00 JavaThread "MutationStage:15" [_thread_blocked, id=10123, stack(0x0000000041c75000,0x0000000041c96000)]
>  0x00007f7e442e9400 JavaThread "MutationStage:14" [_thread_blocked, id=10122, stack(0x00000000405ce000,0x00000000405ef000)]
>  0x00007f7e442e8000 JavaThread "MutationStage:13" [_thread_blocked, id=10121, stack(0x0000000041c41000,0x0000000041c62000)]
>  0x00007f7e442e6800 JavaThread "MutationStage:12" [_thread_blocked, id=10120, stack(0x0000000041e34000,0x0000000041e55000)]
>  0x00007f7e442e5400 JavaThread "MutationStage:11" [_thread_blocked, id=10119, stack(0x00000000421d9000,0x00000000421fa000)]
>  0x00007f7e442e3c00 JavaThread "MutationStage:10" [_thread_blocked, id=10118, stack(0x00000000416f6000,0x0000000041717000)]
>  0x00007f7e4434cc00 JavaThread "MutationStage:9" [_thread_blocked, id=10117, stack(0x00000000409a9000,0x00000000409ca000)]
>  0x0000000002799000 JavaThread "MutationStage:8" [_thread_blocked, id=10116, stack(0x0000000041bde000,0x0000000041bff000)]
>  0x0000000002797800 JavaThread "MutationStage:7" [_thread_blocked, id=10115, stack(0x00000000405ad000,0x00000000405ce000)]
>  0x0000000002796000 JavaThread "MutationStage:6" [_thread_blocked, id=10114, stack(0x0000000041e13000,0x0000000041e34000)]
>  0x0000000002794800 JavaThread "MutationStage:5" [_thread_blocked, id=10113, stack(0x0000000041df2000,0x0000000041e13000)]
>  0x0000000002792c00 JavaThread "MutationStage:4" [_thread_blocked, id=10112, stack(0x0000000040cf0000,0x0000000040d11000)]
>  0x0000000002791c00 JavaThread "MutationStage:3" [_thread_blocked, id=10111, stack(0x0000000041dd1000,0x0000000041df2000)]
>  0x0000000002790c00 JavaThread "MutationStage:2" [_thread_blocked, id=10110, stack(0x000000004077a000,0x000000004079b000)]
>  0x0000000002790400 JavaThread "MutationStage:1" [_thread_blocked, id=10109, stack(0x000000004159d000,0x00000000415be000)]
>  0x00007f7e4434c000 JavaThread "DestroyJavaVM" [_thread_blocked, id=10069, stack(0x0000000040543000,0x0000000040564000)]
>  0x00007f7e4434c000 JavaThread "DestroyJavaVM" [_thread_blocked, id=10069, stack(0x0000000040543000,0x0000000040564000)]
>  0x00007f7e44151c00 JavaThread "Thread-3" [_thread_in_native, id=10107, stack(0x0000000040759000,0x000000004077a000)]
>  0x00007f7e441e4000 JavaThread "HintedHandoff:1" [_thread_blocked, id=10106, stack(0x0000000040178000,0x0000000040199000)]
>  0x00000000026e7c00 JavaThread "WRITE-/172.19.149.83" [_thread_blocked, id=10105, stack(0x00000000402f8000,0x0000000040319000)]
>  0x00000000026e7000 JavaThread "WRITE-/172.19.149.83" [_thread_blocked, id=10104, stack(0x000000004179b000,0x00000000417bc000)]
>  0x0000000002700c00 JavaThread "ACCEPT-172-19-149-76/172.19.149.76" [_thread_in_native,
id=10103, stack(0x0000000041647000,0x0000000041668000)]
>  0x00007f7e44162000 JavaThread "SSTABLE-DELETER" [_thread_blocked, id=10102, stack(0x0000000041201000,0x0000000041222000)]
>  0x00007f7e441ccc00 JavaThread "CompactionExecutor:1" [_thread_blocked, id=10101, stack(0x00000000416d5000,0x00000000416f6000)]
>  0x0000000002676c00 JavaThread "PERIODIC-COMMIT-LOG-SYNCER" [_thread_blocked, id=10100,
stack(0x000000004046f000,0x0000000040490000)]
>  0x00000000026b7000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked, id=10099, stack(0x00000000406f6000,0x0000000040717000)]
>  0x00007f7e441bdc00 JavaThread "MemtablePostFlusher:1" [_thread_blocked, id=10098, stack(0x000000004095a000,0x000000004097b000)]
>  0x00007f7e4414c400 JavaThread "FlushWriter:1" [_thread_blocked, id=10097, stack(0x0000000040ccf000,0x0000000040cf0000)]
>  0x00007f7e4414b800 JavaThread "FlushSorter:1" [_thread_blocked, id=10096, stack(0x0000000041aa6000,0x0000000041ac7000)]
> =>0x00007f7e441a9c00 JavaThread "EXPIRING-MAP-TIMER-1" daemon [_thread_in_Java, id=10095,
stack(0x0000000040b65000,0x0000000040b86000)]
>  0x00007f7e441c3400 JavaThread "ScheduledTasks:1" [_thread_blocked, id=10094, stack(0x0000000040157000,0x0000000040178000)]
>  0x00007f7e44152c00 JavaThread "FileWatchdog" daemon [_thread_blocked, id=10093, stack(0x0000000040939000,0x000000004095a000)]
>  0x00007f7e44146000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=10091,
stack(0x0000000041a85000,0x0000000041aa6000)]
>  0x00007f7e44127c00 JavaThread "RMI TCP Accept-8080" daemon [_thread_in_native, id=10090,
stack(0x00000000401db000,0x00000000401fc000)]
>  0x00007f7e44126800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=10089,
stack(0x00000000403ef000,0x0000000040410000)]
>  0x00007f7e4402f000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=10088,
stack(0x0000000041699000,0x00000000416ba000)]
>  0x00007f7e4402d000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=10087,
stack(0x000000004149c000,0x000000004159d000)]
>  0x00007f7e4402b800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=10086,
stack(0x00000000420d8000,0x00000000421d9000)]
>  0x00007f7e4402a000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10085,
stack(0x00000000401ba000,0x00000000401db000)]
>  0x00007f7e44028c00 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked,
id=10084, stack(0x000000004191b000,0x000000004193c000)]
>  0x0000000002512c00 JavaThread "Finalizer" daemon [_thread_blocked, id=10083, stack(0x0000000041da9000,0x0000000041dca000)]
>  0x0000000002511400 JavaThread "Reference Handler" daemon [_thread_blocked, id=10082,
stack(0x00000000420b7000,0x00000000420d8000)]
>
> Other Threads:
>  0x000000000250c000 VMThread [stack: 0x0000000040a64000,0x0000000040b65000] [id=10081]
>  0x00007f7e44147c00 WatcherThread [stack: 0x00000000405f5000,0x00000000406f6000] [id=10092]
>
> VM state:not at safepoint (normal execution)
>
> VM Mutex/Monitor currently owned by a thread: None
>
> Heap
>  par new generation   total 737280K, used 216364K [0x00007f7e548a0000, 0x00007f7e868a0000,
0x00007f7e868a0000)
>  eden space 655360K,  20% used [0x00007f7e548a0000, 0x00007f7e5cbeb350, 0x00007f7e7c8a0000)
>  from space 81920K, 100% used [0x00007f7e7c8a0000, 0x00007f7e818a0000, 0x00007f7e818a0000)
>  to   space 81920K,   0% used [0x00007f7e818a0000, 0x00007f7e818a0000, 0x00007f7e868a0000)
>  concurrent mark-sweep generation total 6718464K, used 1264492K [0x00007f7e868a0000,
0x00007f80209a0000, 0x00007f8020aa0000)
>  concurrent-mark-sweep perm gen total 30432K, used 18222K [0x00007f8020aa0000, 0x00007f8022858000,
0x00007f8025ea0000)
>
> Dynamic libraries:
> 00400000-00408000 r-xp 00000000 03:01 8576                            
  /usr/lib/jvm/java-6-openjdk/jre/bin/java
> 00608000-00609000 rw-p 00008000 03:01 8576                            
  /usr/lib/jvm/java-6-openjdk/jre/bin/java
> 023ba000-0416e000 rw-p 023ba000 00:00 0                              
   [heap]
> ...
> ...
> [ Not sure how important this stuff is.  There was a lot of it, if it's important I
can send it.
> ...
> 7f802b6f9000-7f802b6fc000 rw-p 7f802b6f9000 00:00 0
> 7f802b6fc000-7f802b6fe000 rw-p 0001b000 03:01 82264                      /lib/ld-2.7.so
> 7fff7210c000-7fff72120000 rwxp 7ffffffe9000 00:00 0                      [stack]
> 7fff72121000-7fff72121000 rw-p 7fffffffe000 00:00 0
> 7fff721ff000-7fff72200000 r-xp 7fff721ff000 00:00 0                      [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
>
> VM Arguments:
> jvm_args: -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms7361M -Xmx7361M
-Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=8080
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true
> java_command: org.apache.cassandra.thrift.CassandraDaemon
> Launcher Type: SUN_STANDARD
>
> Environment Variables:
> PATH=/h/jcipar/SOFTWARE/ROOTS/Linux/x86_64/bin:/h/jcipar/bin:/h/jcipar/SOFTWARE/ROOTS/All/bin:/h/jcipar/SOFTWARE/ant/apache-ant-1.8.1/bin/:~mabdelm/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
> LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64
> SHELL=/bin/bash
>
> Signal Handlers:
> SIGSEGV: [libjvm.so+0x5d2630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGBUS: [libjvm.so+0x5d2630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGFPE: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGPIPE: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGXFSZ: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGILL: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
> SIGUSR2: [libjvm.so+0x4ab380], sa_mask[0]=0x00000000, sa_flags=0x10000004
> SIGHUP: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
> SIGTERM: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGQUIT: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>
>
> ---------------  S Y S T E M  ---------------
>
> OS:5.0.6
>
> uname:Linux 2.6.26-2-amd64 #1 SMP Thu Sep 16 15:56:38 UTC 2010 x86_64
> libc:glibc 2.7 NPTL 2.7
> rlimit: STACK 8192k, CORE 0k, NPROC 124096, NOFILE 1024, AS infinity
> load average:3.09 3.56 3.72
>
> CPU:total 8 (1 cores per cpu, 1 threads per core) family 6 model 2 stepping 3, cmov,
cx8, fxsr, mmx, sse, sse2, sse3
>
> Memory: 4k page, physical 15075756k(6082384k free), swap 0k(0k free)
>
> vm_info: OpenJDK 64-Bit Server VM (1.6.0_0-b11) for linux-amd64 JRE (1.6.0_0-b11), built
on Apr  9 2009 19:35:18 by "pbuilder" with gcc 4.3.2
>
> time: Tue May 10 13:01:39 2011
> elapsed time: 2175 seconds
>
>
>
>
>
>
>
>
>
>
> On May 12, 2011, at 9:30 PM, Jeffrey Kesselman wrote:
>
>> If this a 64bit VM?
>>
>> A 32bit Java VM with default c-heap settings can only actually use
>> about 2GB of Java Heap.
>>
>> On Thu, May 12, 2011 at 8:08 PM, James Cipar <jcipar@cmu.edu> wrote:
>>> Oh, forgot this detail:  I have no swap configured, so swapping is not the cause
of the crash.  Could it be that I'm running out of memory on a 15GB machine?  That seems
unlikely.  I grepped dmesg for "oom" and didn't see anything from the oom killer, and I used
the instructions from the following web page and didn't see that the oom killer had killed
anything.
>>>
>>> http://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer
>>>
>>> jcipar@172-19-149-62:~$ sudo cat /var/log/messages | grep --ignore-case "killed
process"
>>> jcipar@172-19-149-62:~$
>>>
>>>
>>>
>>> Also, this is pretty subjective, so I can't say for sure until it finishes, but
this seems to be running *much* slower after setting the heap size and setting up JNA.
>>>
>>>
>>>
>>> On May 12, 2011, at 7:52 PM, James Cipar wrote:
>>>
>>>> It looks like MAX_HEAP_SIZE is set in cassandra-env.sh to be half of my physical
memory.  These are 15GB VMs, so that's 7.5GB for Cassandra.  I would have expected that
to work, but I will override to 13 GB just to see what happens.
>>>>
>>>> I've also got the JNA thing set up.  Do you think this would cause the crashes,
or is it just a performance improvement?
>>>>
>>>>
>>>>
>>>> On May 12, 2011, at 7:27 PM, Sameer Farooqui wrote:
>>>>
>>>>> The key JVM options for Cassandra are in cassandra.in.sh.
>>>>>
>>>>> What is your min and max heap size?
>>>>>
>>>>> The default setting of max heap size is 1GB. How much RAM do your nodes
have? You may want to increase this setting. You can also set the -Xmx and -Xms options to
the same value to keep Java from having to manage heap growth. On a 32-bit machine, you can
get a max of about 1.6 GB of heap; you can get a lot more on 64-bit.
>>>>>
>>>>> Try messing with some of the other settings in the cassandra.in.sh file.
>>>>>
>>>>> You may not have DEBUG mode turned on for Cassandra and therefore may
not be getting the full details of what's going on when the server crashes. In the <cassandra-home>/conf/log4j-server.properties
file, set this line from the default of INFO to DEBUG:
>>>>>
>>>>> log4j.rootLogger=INFO,stdout,R
>>>>>
>>>>>
>>>>> Also, you haven't configured JNA on this server. Here's some info about
it and how to configure it:
>>>>>
>>>>> JNA provides Java programs easy access to native shared libraries without
writing anything but Java code.
>>>>>
>>>>> Note from Cassandra developers for why JNA is needed:
>>>>> "Linux aggressively swaps out infrequently used memory to make more room
for its file system buffer cache. Unfortunately, modern generational garbage collectors like
the JVM's leave parts of its heap un-touched for relatively large amounts of time, leading
Linux to swap it out. When the JVM finally goes to use or GC that memory, swap hell ensues.
>>>>>
>>>>> Setting swappiness to zero can mitigate this behavior but does not eliminate
it entirely. Turning off swap entirely is effective. But to avoid surprising people who don't
know about this behavior, the best solution is to tell Linux not to swap out the JVM, and
that is what we do now with mlockall via JNA.
>>>>>
>>>>> Because of licensing issues, we can't distribute JNA with Cassandra,
so you must manually add it to the Cassandra lib/ directory or otherwise place it on the classpath.
If the JNA jar is not present, Cassandra will continue as before."
>>>>>
>>>>> Get JNA with:
>>>>> cd ~
>>>>> wget http://debian.riptano.com/debian/pool/libjna-java_3.2.7-0~nmu.2_amd64.deb
>>>>>
>>>>> To install:
>>>>> techlabs@cassandraN1:~$ sudo dpkg -i libjna-java_3.2.7-0~nmu.2_amd64.deb
>>>>> (Reading database ... 44334 files and directories currently installed.)
>>>>> Preparing to replace libjna-java 3.2.4-2 (using libjna-java_3.2.7-0~nmu.2_amd64.deb)
...
>>>>> Unpacking replacement libjna-java ...
>>>>> Setting up libjna-java (3.2.7-0~nmu.2) ...
>>>>>
>>>>>
>>>>> The deb package will install the JNA jar file to /usr/share/java/jna.jar,
but Cassandra only loads it if its in the class path. The easy way to do this is just create
a symlink into your Cassandra lib directory (note: replace /home/techlabs with your home dir
location):
>>>>> ln -s /usr/share/java/jna.jar /home/techlabs/apache-cassandra-0.7.0/lib
>>>>>
>>>>> Research:
>>>>> http://journal.paul.querna.org/articles/2010/11/11/enabling-jna-in-cassandra/
>>>>>
>>>>>
>>>>> - Sameer
>>>>>
>>>>>
>>>>> On Thu, May 12, 2011 at 4:15 PM, James Cipar <jcipar@cmu.edu> wrote:
>>>>> I'm using Cassandra 0.7.5, and uploading about 200 GB of data total (20
GB unique data), to a cluster of 10 servers.  I'm using batch_mutate, and breaking the data
up into chunks of about 10k records.  Each record is about 5KB, so a total of about 50MB
per batch.  When I upload a smaller 2 GB data set, everything works fine.  When I upload
the 20 GB data set, servers will occasionally crash.  Currently I have my client code automatically
detect this and restart the server, but that is less than ideal.
>>>>>
>>>>> I'm not sure what information to gather to determine what's going on
here.  Here is a sample of a log file from when a crash occurred.  The crash was immediately
after the log entry tagged "2011-05-12 19:02:19,377".  Any idea what's going on here?  Any
other info I can gather to try to debug this?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:07,855 GCInspector.java (line
128) GC for ParNew: 375 ms, 576641232 reclaimed leaving 5471432144 used; max is 7774142464
>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:08,857 GCInspector.java (line
128) GC for ParNew: 450 ms, -63738232 reclaimed leaving 5546942544 used; max is 7774142464
>>>>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:10,652 CommitLogSegment.java
(line 50) Creating new commitlog segment /mnt/scratch/jcipar/cassandra/commitlog/CommitLog-1305241330652.log
>>>>> INFO [MutationStage:24] 2011-05-12 19:02:10,680 ColumnFamilyStore.java
(line 1070) Enqueuing flush of Memtable-Standard1@1256245282(51921529 bytes, 1115783 operations)
>>>>> INFO [FlushWriter:1] 2011-05-12 19:02:10,680 Memtable.java (line 158)
Writing Memtable-Standard1@1256245282(51921529 bytes, 1115783 operations)
>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:12,932 GCInspector.java (line
128) GC for ParNew: 249 ms, 571827736 reclaimed leaving 3165899760 used; max is 7774142464
>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:15,253 GCInspector.java (line
128) GC for ParNew: 341 ms, 561823592 reclaimed leaving 1764208800 used; max is 7774142464
>>>>> INFO [FlushWriter:1] 2011-05-12 19:02:16,743 Memtable.java (line 165)
Completed flushing /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-74-Data.db (53646223
bytes)
>>>>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:16,745 CommitLog.java (line
440) Discarding obsolete commit log:CommitLogSegment(/mnt/scratch/jcipar/cassandra/commitlog/CommitLog-1305241306438.log)
>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:18,256 GCInspector.java (line
128) GC for ParNew: 305 ms, 544491840 reclaimed leaving 865198712 used; max is 7774142464
>>>>> INFO [MutationStage:19] 2011-05-12 19:02:19,000 ColumnFamilyStore.java
(line 1070) Enqueuing flush of Memtable-Standard1@479849353(51941121 bytes, 1115783 operations)
>>>>> INFO [FlushWriter:1] 2011-05-12 19:02:19,000 Memtable.java (line 158)
Writing Memtable-Standard1@479849353(51941121 bytes, 1115783 operations)
>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,310 SSTable.java (line
147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-51
>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,324 SSTable.java (line
147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-55
>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,339 SSTable.java (line
147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-58
>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,357 SSTable.java (line
147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-67
>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,377 SSTable.java (line
147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-61
>>>>> INFO [main] 2011-05-12 19:02:21,026 AbstractCassandraDaemon.java (line
78) Logging initialized
>>>>> INFO [main] 2011-05-12 19:02:21,040 AbstractCassandraDaemon.java (line
96) Heap size: 7634681856/7635730432
>>>>> INFO [main] 2011-05-12 19:02:21,042 CLibrary.java (line 61) JNA not found.
Native methods will be disabled.
>>>>> INFO [main] 2011-05-12 19:02:21,052 DatabaseDescriptor.java (line 121)
Loading settings from file:/h/jcipar/Projects/HP/OtherDBs/Cassandra/apache-cassandra-0.7.5/conf/cassandra.yaml
>>>>> INFO [main] 2011-05-12 19:02:21,178 DatabaseDescriptor.java (line 181)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
>>>>> INFO [main] 2011-05-12 19:02:21,310 SSTableReader.java (line 154) Opening
/mnt/scratch/jcipar/cassandra/data/system/Schema-f-1
>>>>> INFO [main] 2011-05-12 19:02:21,327 SSTableReader.java (line 154) Opening
/mnt/scratch/jcipar/cassandra/data/system/Schema-f-2
>>>>> INFO [main] 2011-05-12 19:02:21,336 SSTableReader.java (line 154) Opening
/mnt/scratch/jcipar/cassandra/data/system/Migrations-f-1
>>>>> INFO [main] 2011-05-12 19:02:21,337 SSTableReader.java (line 154) Opening
/mnt/scratch/jcipar/cassandra/data/system/Migrations-f-2
>>>>> INFO [main] 2011-05-12 19:02:21,342 SSTableReader.java (line 154) Opening
/mnt/scratch/jcipar/cassandra/data/system/LocationInfo-f-2
>>>>> INFO [main] 2011-05-12 19:02:21,344 SSTableReader.java (line 154) Opening
/mnt/scratch/jcipar/cassandra/data/system/LocationInfo-f-1
>>>>> INFO [main] 2011-05-12 19:02:21,379 DatabaseDescriptor.java (line 461)
Loading schema version 9467ffe0-7cea-11e0-8ddc-f74ef74e382f
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> It's always darkest just before you are eaten by a grue.
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message