cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sris <srisatish.amb...@gmail.com>
Subject Re: Crash when uploading large data sets
Date Sat, 14 May 2011 10:01:18 GMT
> vm_info: OpenJDK 64-Bit Server VM (1.6.0_0-b11) for linux-amd64 JRE  
> (1.6.0_0-b11), built on Apr  9 2009 19:35:18 by "pbuilder" with gcc  
> 4.3.2


Indeed this is an old jvm.
There were (are?) a few of the nmethod crashes lurking around that  
time such as,
http://bugs.sun.com/view_bug.do?bug_id=6782820

The last report of this has been in _20.

Upgrade to the latest sun 1.6.0_25 and let us know if it still  
reproduces for you,

Thanks,
Sri

On May 13, 2011, at 8:45 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> You should upgrade to the latest Sun JVM. OpenJDK is almost a year
> behind in bug fixes.
>
> On Fri, May 13, 2011 at 11:40 AM, James Cipar <jcipar@cmu.edu> wrote:
>> It is a 64 bit VM.  I didn't notice the hs_err_pid.log files since  
>> I'm staring over ssh, so they're in my home directory instead of my  
>> working directory.  I've attached one of those below.  I don't know  
>> much about Java, so I'm not sure how to interpret this file.
>>
>>
>>
>>
>>
>> #
>> # An unexpected error has been detected by Java Runtime Environment:
>> #
>> #  Internal Error (nmethod.cpp:1707), pid=10068, tid=1085823312
>> #  Error: guarantee(cont_offset != 0,"unhandled implicit exception  
>> in compiled code")
>> #
>> # Java VM: OpenJDK 64-Bit Server VM (1.6.0_0-b11 mixed mode linux- 
>> amd64)
>> # If you would like to submit a bug report, please visit:
>> #   http://icedtea.classpath.org/bugzilla
>> #
>>
>> ---------------  T H R E A D  ---------------
>>
>> Current thread (0x00007f7e441a9c00):  JavaThread "EXPIRING-MAP- 
>> TIMER-1" daemon [_thread_in_Java, id=10095, stack 
>> (0x0000000040b65000,0x0000000040b86000)]
>>
>> Stack: [0x0000000040b65000,0x0000000040b86000],   
>> sp=0x0000000040b83eb0,  free space=123k
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,  
>> C=native code)
>> V  [libjvm.so+0x5d198f]
>> V  [libjvm.so+0x5d1cf1]
>> V  [libjvm.so+0x27e506]
>> V  [libjvm.so+0x4970d6]
>> V  [libjvm.so+0x514cd9]
>> V  [libjvm.so+0x4b0b64]
>>
>>
>> ---------------  P R O C E S S  ---------------
>>
>> Java Threads: ( => current thread )
>>  0x0000000002913c00 JavaThread "NonPeriodicTasks: 
>> 1" [_thread_blocked, id=10233, stack 
>> (0x00000000401fc000,0x000000004021d000)]
>>  0x00000000026eec00 JavaThread "FILEUTILS-DELETE-POOL: 
>> 1" [_thread_blocked, id=10222, stack 
>> (0x0000000042533000,0x0000000042554000)]
>>  0x00007f7e3c01c000 JavaThread "Thread-22" [_thread_in_native,  
>> id=10212, stack(0x0000000042512000,0x0000000042533000)]
>>  0x000000000253c000 JavaThread "Thread-21" [_thread_in_native,  
>> id=10211, stack(0x0000000040dfc000,0x0000000040e1d000)]
>>  0x000000000253a800 JavaThread "Thread-20" [_thread_in_native,  
>> id=10210, stack(0x0000000040ddb000,0x0000000040dfc000)]
>>  0x0000000002537400 JavaThread "Thread-19" [_thread_in_native,  
>> id=10209, stack(0x00000000424f1000,0x0000000042512000)]
>>  0x0000000002725c00 JavaThread "pool-1- 
>> thread-1" [_thread_in_native, id=10208, stack 
>> (0x000000004007d000,0x000000004009e000)]
>>  0x00000000026f7400 JavaThread "Thread-18" [_thread_in_native,  
>> id=10207, stack(0x000000004034f000,0x0000000040370000)]
>>  0x0000000002902800 JavaThread "Thread-17" [_thread_in_native,  
>> id=10206, stack(0x0000000040d6b000,0x0000000040d8c000)]
>>  0x0000000002901400 JavaThread "Thread-16" [_thread_in_native,  
>> id=10205, stack(0x00000000424d0000,0x00000000424f1000)]
>>  0x0000000002613c00 JavaThread "Thread-15" [_thread_in_native,  
>> id=10204, stack(0x000000004024e000,0x000000004026f000)]
>>  0x00000000026ad800 JavaThread "Thread-14" [_thread_in_native,  
>> id=10203, stack(0x0000000040d11000,0x0000000040d32000)]
>>  0x000000000276f000 JavaThread "Thread-13" [_thread_in_native,  
>> id=10202, stack(0x00000000424af000,0x00000000424d0000)]
>>  0x00000000026b2c00 JavaThread "Thread-12" [_thread_in_native,  
>> id=10201, stack(0x0000000040490000,0x00000000404b1000)]
>>  0x00000000026aec00 JavaThread "Thread-11" [_thread_in_native,  
>> id=10200, stack(0x000000004248e000,0x00000000424af000)]
>>  0x000000000254ec00 JavaThread "Thread-10" [_thread_in_native,  
>> id=10199, stack(0x000000004246d000,0x000000004248e000)]
>>  0x000000000254d000 JavaThread "Thread-9" [_thread_in_native,  
>> id=10198, stack(0x000000004244c000,0x000000004246d000)]
>>  0x0000000002505000 JavaThread "Thread-8" [_thread_in_native,  
>> id=10197, stack(0x000000004242b000,0x000000004244c000)]
>>  0x0000000002502400 JavaThread "Thread-7" [_thread_in_native,  
>> id=10196, stack(0x000000004240a000,0x000000004242b000)]
>>  0x0000000002500400 JavaThread "WRITE-/ 
>> 172.19.149.80" [_thread_blocked, id=10195, stack 
>> (0x00000000407dd000,0x00000000407fe000)]
>>  0x00000000024ff000 JavaThread "WRITE-/ 
>> 172.19.149.80" [_thread_blocked, id=10194, stack 
>> (0x0000000040c8e000,0x0000000040caf000)]
>>  0x00000000024f4400 JavaThread "WRITE-/ 
>> 172.19.149.64" [_thread_blocked, id=10193, stack 
>> (0x00000000423e9000,0x000000004240a000)]
>>  0x00000000024f3000 JavaThread "WRITE-/ 
>> 172.19.149.64" [_thread_blocked, id=10192, stack 
>> (0x00000000423c8000,0x00000000423e9000)]
>>  0x00000000024f1400 JavaThread "WRITE-/ 
>> 172.19.149.71" [_thread_blocked, id=10191, stack 
>> (0x0000000040a2d000,0x0000000040a4e000)]
>>  0x00000000024f0000 JavaThread "WRITE-/ 
>> 172.19.149.71" [_thread_blocked, id=10190, stack 
>> (0x00000000423a7000,0x00000000423c8000)]
>>  0x00000000024ee400 JavaThread "WRITE-/ 
>> 172.19.149.62" [_thread_blocked, id=10189, stack 
>> (0x0000000042386000,0x00000000423a7000)]
>>  0x00000000026bf800 JavaThread "WRITE-/ 
>> 172.19.149.62" [_thread_blocked, id=10188, stack 
>> (0x0000000042365000,0x0000000042386000)]
>>  0x00000000026bdc00 JavaThread "WRITE-/ 
>> 172.19.149.72" [_thread_blocked, id=10187, stack 
>> (0x0000000040dba000,0x0000000040ddb000)]
>>  0x00000000026bc400 JavaThread "WRITE-/ 
>> 172.19.149.72" [_thread_blocked, id=10186, stack 
>> (0x0000000042344000,0x0000000042365000)]
>>  0x00000000026bac00 JavaThread "WRITE-/ 
>> 172.19.149.63" [_thread_blocked, id=10185, stack 
>> (0x0000000040431000,0x0000000040452000)]
>>  0x00000000026b9400 JavaThread "WRITE-/ 
>> 172.19.149.63" [_thread_blocked, id=10184, stack 
>> (0x0000000042323000,0x0000000042344000)]
>>  0x00000000026b8000 JavaThread "WRITE-/ 
>> 172.19.149.61" [_thread_blocked, id=10183, stack 
>> (0x0000000042302000,0x0000000042323000)]
>>  0x00000000024b9c00 JavaThread "WRITE-/ 
>> 172.19.149.61" [_thread_blocked, id=10182, stack 
>> (0x0000000040c6d000,0x0000000040c8e000)]
>>  0x00000000024b8400 JavaThread "WRITE-/ 
>> 172.19.149.69" [_thread_blocked, id=10181, stack 
>> (0x00000000422e1000,0x0000000042302000)]
>>  0x00000000024b7400 JavaThread "WRITE-/ 
>> 172.19.149.69" [_thread_blocked, id=10180, stack 
>> (0x00000000422c0000,0x00000000422e1000)]
>>  0x00000000024b3800 JavaThread "Thread-6" [_thread_in_native,  
>> id=10179, stack(0x0000000040e98000,0x0000000040eb9000)]
>>  0x00000000026e6400 JavaThread "Thread-5" [_thread_in_native,  
>> id=10178, stack(0x0000000041222000,0x0000000041243000)]
>>  0x00007f7e4420ec00 JavaThread "ReadRepairStage: 
>> 8" [_thread_blocked, id=10177, stack 
>> (0x00000000407bc000,0x00000000407dd000)]
>>  0x00007f7e4420d400 JavaThread "ReadRepairStage: 
>> 7" [_thread_blocked, id=10176, stack 
>> (0x000000004229f000,0x00000000422c0000)]
>>  0x00007f7e4420c000 JavaThread "ReadRepairStage: 
>> 6" [_thread_blocked, id=10175, stack 
>> (0x000000004227e000,0x000000004229f000)]
>>  0x00007f7e4420a800 JavaThread "ReadRepairStage: 
>> 5" [_thread_blocked, id=10174, stack 
>> (0x00000000418ec000,0x000000004190d000)]
>>  0x00007f7e44209400 JavaThread "ReadRepairStage: 
>> 4" [_thread_blocked, id=10173, stack 
>> (0x0000000040a0c000,0x0000000040a2d000)]
>>  0x00007f7e44208000 JavaThread "ReadRepairStage: 
>> 3" [_thread_blocked, id=10172, stack 
>> (0x000000004225d000,0x000000004227e000)]
>>  0x00007f7e44206800 JavaThread "ReadRepairStage: 
>> 2" [_thread_blocked, id=10171, stack 
>> (0x000000004223c000,0x000000004225d000)]
>>  0x00007f7e44205400 JavaThread "ReadRepairStage: 
>> 1" [_thread_blocked, id=10170, stack 
>> (0x00000000418cb000,0x00000000418ec000)]
>>  0x00007f7e44203c00 JavaThread "MiscStage:1" [_thread_blocked,  
>> id=10169, stack(0x00000000418aa000,0x00000000418cb000)]
>>  0x00007f7e44202800 JavaThread "MigrationStage:1" [_thread_blocked,  
>> id=10168, stack(0x0000000041889000,0x00000000418aa000)]
>>  0x00007f7e44201000 JavaThread "AntiEntropyStage: 
>> 1" [_thread_blocked, id=10167, stack 
>> (0x0000000041868000,0x0000000041889000)]
>>  0x00007f7e441ffc00 JavaThread "GossipStage:1" [_thread_blocked,  
>> id=10166, stack(0x0000000041847000,0x0000000041868000)]
>>  0x00007f7e441fe400 JavaThread "StreamStage:1" [_thread_blocked,  
>> id=10165, stack(0x00000000417fe000,0x000000004181f000)]
>>  0x00007f7e441fd000 JavaThread "InternalResponseStage: 
>> 8" [_thread_blocked, id=10164, stack 
>> (0x00000000417dd000,0x00000000417fe000)]
>>  0x00007f7e441fb800 JavaThread "InternalResponseStage: 
>> 7" [_thread_blocked, id=10163, stack 
>> (0x0000000040c4c000,0x0000000040c6d000)]
>>  0x00007f7e441fa400 JavaThread "InternalResponseStage: 
>> 6" [_thread_blocked, id=10162, stack 
>> (0x000000004177a000,0x000000004179b000)]
>>  0x00007f7e441f8c00 JavaThread "InternalResponseStage: 
>> 5" [_thread_blocked, id=10161, stack 
>> (0x000000004194e000,0x000000004196f000)]
>>  0x00007f7e441f7800 JavaThread "InternalResponseStage: 
>> 4" [_thread_blocked, id=10160, stack 
>> (0x000000004027f000,0x00000000402a0000)]
>>  0x00007f7e441f6000 JavaThread "InternalResponseStage: 
>> 3" [_thread_blocked, id=10159, stack 
>> (0x0000000040811000,0x0000000040832000)]
>>  0x00007f7e441f4c00 JavaThread "InternalResponseStage: 
>> 2" [_thread_blocked, id=10158, stack 
>> (0x000000004221b000,0x000000004223c000)]
>>  0x00007f7e441f3400 JavaThread "InternalResponseStage: 
>> 1" [_thread_blocked, id=10157, stack 
>> (0x00000000409eb000,0x0000000040a0c000)]
>>  0x00007f7e441f2000 JavaThread "RequestResponseStage: 
>> 8" [_thread_blocked, id=10156, stack 
>> (0x0000000041759000,0x000000004177a000)]
>>  0x00007f7e441f0800 JavaThread "RequestResponseStage: 
>> 7" [_thread_blocked, id=10155, stack 
>> (0x0000000041e76000,0x0000000041e97000)]
>>  0x00007f7e44315000 JavaThread "RequestResponseStage: 
>> 6" [_thread_blocked, id=10154, stack 
>> (0x0000000040bf1000,0x0000000040c12000)]
>>  0x00007f7e44313800 JavaThread "RequestResponseStage: 
>> 5" [_thread_blocked, id=10153, stack 
>> (0x0000000041668000,0x0000000041689000)]
>>  0x00007f7e44312400 JavaThread "RequestResponseStage: 
>> 4" [_thread_blocked, id=10152, stack 
>> (0x0000000040d99000,0x0000000040dba000)]
>>  0x00007f7e44310c00 JavaThread "RequestResponseStage: 
>> 3" [_thread_blocked, id=10151, stack 
>> (0x00000000409ca000,0x00000000409eb000)]
>>  0x00007f7e4430f800 JavaThread "RequestResponseStage: 
>> 2" [_thread_blocked, id=10150, stack 
>> (0x000000004079b000,0x00000000407bc000)]
>>  0x00007f7e4430e000 JavaThread "RequestResponseStage: 
>> 1" [_thread_blocked, id=10149, stack 
>> (0x0000000041621000,0x0000000041642000)]
>>  0x00007f7e4430cc00 JavaThread "ReadStage:8" [_thread_blocked,  
>> id=10148, stack(0x00000000421fa000,0x000000004221b000)]
>>  0x00007f7e4430b400 JavaThread "ReadStage:7" [_thread_blocked,  
>> id=10147, stack(0x0000000040738000,0x0000000040759000)]
>>  0x00007f7e4430a000 JavaThread "ReadStage:6" [_thread_blocked,  
>> id=10146, stack(0x0000000041264000,0x0000000041285000)]
>>  0x00007f7e44308800 JavaThread "ReadStage:5" [_thread_blocked,  
>> id=10145, stack(0x0000000041e55000,0x0000000041e76000)]
>>  0x00007f7e44307400 JavaThread "ReadStage:4" [_thread_blocked,  
>> id=10144, stack(0x0000000041243000,0x0000000041264000)]
>>  0x00007f7e44305c00 JavaThread "ReadStage:3" [_thread_blocked,  
>> id=10143, stack(0x0000000041600000,0x0000000041621000)]
>>  0x00007f7e44304800 JavaThread "ReadStage:2" [_thread_blocked,  
>> id=10142, stack(0x00000000415df000,0x0000000041600000)]
>>  0x00007f7e44303000 JavaThread "ReadStage:1" [_thread_blocked,  
>> id=10141, stack(0x0000000041c20000,0x0000000041c41000)]
>>  0x00007f7e44301c00 JavaThread "MutationStage:32" [_thread_blocked,  
>> id=10140, stack(0x0000000041738000,0x0000000041759000)]
>>  0x00007f7e44300400 JavaThread "MutationStage:31" [_thread_blocked,  
>> id=10139, stack(0x0000000040982000,0x00000000409a3000)]
>>  0x00007f7e442ff000 JavaThread "MutationStage:30" [_thread_blocked,  
>> id=10138, stack(0x00000000415be000,0x00000000415df000)]
>>  0x00007f7e442fdc00 JavaThread "MutationStage:29" [_thread_blocked,  
>> id=10137, stack(0x000000004032e000,0x000000004034f000)]
>>  0x00007f7e442fc400 JavaThread "MutationStage:28" [_thread_blocked,  
>> id=10136, stack(0x0000000041826000,0x0000000041847000)]
>>  0x00007f7e442fb000 JavaThread "MutationStage:27" [_thread_blocked,  
>> id=10135, stack(0x0000000041bff000,0x0000000041c20000)]
>>  0x00007f7e442f9800 JavaThread "MutationStage:26" [_thread_blocked,  
>> id=10134, stack(0x0000000040c2b000,0x0000000040c4c000)]
>>  0x00007f7e442f8400 JavaThread "MutationStage:25" [_thread_blocked,  
>> id=10133, stack(0x0000000040117000,0x0000000040138000)]
>>  0x00007f7e442f6c00 JavaThread "MutationStage:24" [_thread_blocked,  
>> id=10132, stack(0x0000000040ba7000,0x0000000040bc8000)]
>>  0x00007f7e442f5800 JavaThread "MutationStage:23" [_thread_blocked,  
>> id=10131, stack(0x0000000040bd0000,0x0000000040bf1000)]
>>  0x00007f7e442f4400 JavaThread "MutationStage:22" [_thread_blocked,  
>> id=10130, stack(0x0000000041717000,0x0000000041738000)]
>>  0x00007f7e442f2c00 JavaThread "MutationStage:21" [_thread_blocked,  
>> id=10129, stack(0x00000000417bc000,0x00000000417dd000)]
>>  0x00007f7e442f1800 JavaThread "MutationStage:20" [_thread_blocked,  
>> id=10128, stack(0x0000000040d4a000,0x0000000040d6b000)]
>>  0x00007f7e442f0000 JavaThread "MutationStage:19" [_thread_blocked,  
>> id=10127, stack(0x0000000040564000,0x0000000040585000)]
>>  0x00007f7e442eec00 JavaThread "MutationStage:18" [_thread_blocked,  
>> id=10126, stack(0x0000000040410000,0x0000000040431000)]
>>  0x00007f7e442ed400 JavaThread "MutationStage:17" [_thread_blocked,  
>> id=10125, stack(0x0000000040717000,0x0000000040738000)]
>>  0x00007f7e442ec000 JavaThread "MutationStage:16" [_thread_blocked,  
>> id=10124, stack(0x0000000040b86000,0x0000000040ba7000)]
>>  0x00007f7e442eac00 JavaThread "MutationStage:15" [_thread_blocked,  
>> id=10123, stack(0x0000000041c75000,0x0000000041c96000)]
>>  0x00007f7e442e9400 JavaThread "MutationStage:14" [_thread_blocked,  
>> id=10122, stack(0x00000000405ce000,0x00000000405ef000)]
>>  0x00007f7e442e8000 JavaThread "MutationStage:13" [_thread_blocked,  
>> id=10121, stack(0x0000000041c41000,0x0000000041c62000)]
>>  0x00007f7e442e6800 JavaThread "MutationStage:12" [_thread_blocked,  
>> id=10120, stack(0x0000000041e34000,0x0000000041e55000)]
>>  0x00007f7e442e5400 JavaThread "MutationStage:11" [_thread_blocked,  
>> id=10119, stack(0x00000000421d9000,0x00000000421fa000)]
>>  0x00007f7e442e3c00 JavaThread "MutationStage:10" [_thread_blocked,  
>> id=10118, stack(0x00000000416f6000,0x0000000041717000)]
>>  0x00007f7e4434cc00 JavaThread "MutationStage:9" [_thread_blocked,  
>> id=10117, stack(0x00000000409a9000,0x00000000409ca000)]
>>  0x0000000002799000 JavaThread "MutationStage:8" [_thread_blocked,  
>> id=10116, stack(0x0000000041bde000,0x0000000041bff000)]
>>  0x0000000002797800 JavaThread "MutationStage:7" [_thread_blocked,  
>> id=10115, stack(0x00000000405ad000,0x00000000405ce000)]
>>  0x0000000002796000 JavaThread "MutationStage:6" [_thread_blocked,  
>> id=10114, stack(0x0000000041e13000,0x0000000041e34000)]
>>  0x0000000002794800 JavaThread "MutationStage:5" [_thread_blocked,  
>> id=10113, stack(0x0000000041df2000,0x0000000041e13000)]
>>  0x0000000002792c00 JavaThread "MutationStage:4" [_thread_blocked,  
>> id=10112, stack(0x0000000040cf0000,0x0000000040d11000)]
>>  0x0000000002791c00 JavaThread "MutationStage:3" [_thread_blocked,  
>> id=10111, stack(0x0000000041dd1000,0x0000000041df2000)]
>>  0x0000000002790c00 JavaThread "MutationStage:2" [_thread_blocked,  
>> id=10110, stack(0x000000004077a000,0x000000004079b000)]
>>  0x0000000002790400 JavaThread "MutationStage:1" [_thread_blocked,  
>> id=10109, stack(0x000000004159d000,0x00000000415be000)]
>>  0x00007f7e4434c000 JavaThread "DestroyJavaVM" [_thread_blocked,  
>> id=10069, stack(0x0000000040543000,0x0000000040564000)]
>>  0x00007f7e4434c000 JavaThread "DestroyJavaVM" [_thread_blocked,  
>> id=10069, stack(0x0000000040543000,0x0000000040564000)]
>>  0x00007f7e44151c00 JavaThread "Thread-3" [_thread_in_native,  
>> id=10107, stack(0x0000000040759000,0x000000004077a000)]
>>  0x00007f7e441e4000 JavaThread "HintedHandoff:1" [_thread_blocked,  
>> id=10106, stack(0x0000000040178000,0x0000000040199000)]
>>  0x00000000026e7c00 JavaThread "WRITE-/ 
>> 172.19.149.83" [_thread_blocked, id=10105, stack 
>> (0x00000000402f8000,0x0000000040319000)]
>>  0x00000000026e7000 JavaThread "WRITE-/ 
>> 172.19.149.83" [_thread_blocked, id=10104, stack 
>> (0x000000004179b000,0x00000000417bc000)]
>>  0x0000000002700c00 JavaThread  
>> "ACCEPT-172-19-149-76/172.19.149.76" [_thread_in_native, id=10103,  
>> stack(0x0000000041647000,0x0000000041668000)]
>>  0x00007f7e44162000 JavaThread "SSTABLE-DELETER" [_thread_blocked,  
>> id=10102, stack(0x0000000041201000,0x0000000041222000)]
>>  0x00007f7e441ccc00 JavaThread "CompactionExecutor: 
>> 1" [_thread_blocked, id=10101, stack 
>> (0x00000000416d5000,0x00000000416f6000)]
>>  0x0000000002676c00 JavaThread "PERIODIC-COMMIT-LOG- 
>> SYNCER" [_thread_blocked, id=10100, stack 
>> (0x000000004046f000,0x0000000040490000)]
>>  0x00000000026b7000 JavaThread "COMMIT-LOG- 
>> WRITER" [_thread_blocked, id=10099, stack 
>> (0x00000000406f6000,0x0000000040717000)]
>>  0x00007f7e441bdc00 JavaThread "MemtablePostFlusher: 
>> 1" [_thread_blocked, id=10098, stack 
>> (0x000000004095a000,0x000000004097b000)]
>>  0x00007f7e4414c400 JavaThread "FlushWriter:1" [_thread_blocked,  
>> id=10097, stack(0x0000000040ccf000,0x0000000040cf0000)]
>>  0x00007f7e4414b800 JavaThread "FlushSorter:1" [_thread_blocked,  
>> id=10096, stack(0x0000000041aa6000,0x0000000041ac7000)]
>> =>0x00007f7e441a9c00 JavaThread "EXPIRING-MAP-TIMER-1" daemon  
>> [_thread_in_Java, id=10095, stack 
>> (0x0000000040b65000,0x0000000040b86000)]
>>  0x00007f7e441c3400 JavaThread "ScheduledTasks:1" [_thread_blocked,  
>> id=10094, stack(0x0000000040157000,0x0000000040178000)]
>>  0x00007f7e44152c00 JavaThread "FileWatchdog" daemon  
>> [_thread_blocked, id=10093, stack 
>> (0x0000000040939000,0x000000004095a000)]
>>  0x00007f7e44146000 JavaThread "RMI TCP Accept-0" daemon  
>> [_thread_in_native, id=10091, stack 
>> (0x0000000041a85000,0x0000000041aa6000)]
>>  0x00007f7e44127c00 JavaThread "RMI TCP Accept-8080" daemon  
>> [_thread_in_native, id=10090, stack 
>> (0x00000000401db000,0x00000000401fc000)]
>>  0x00007f7e44126800 JavaThread "RMI TCP Accept-0" daemon  
>> [_thread_in_native, id=10089, stack 
>> (0x00000000403ef000,0x0000000040410000)]
>>  0x00007f7e4402f000 JavaThread "Low Memory Detector" daemon  
>> [_thread_blocked, id=10088, stack 
>> (0x0000000041699000,0x00000000416ba000)]
>>  0x00007f7e4402d000 JavaThread "CompilerThread1" daemon  
>> [_thread_blocked, id=10087, stack 
>> (0x000000004149c000,0x000000004159d000)]
>>  0x00007f7e4402b800 JavaThread "CompilerThread0" daemon  
>> [_thread_blocked, id=10086, stack 
>> (0x00000000420d8000,0x00000000421d9000)]
>>  0x00007f7e4402a000 JavaThread "Signal Dispatcher" daemon  
>> [_thread_blocked, id=10085, stack 
>> (0x00000000401ba000,0x00000000401db000)]
>>  0x00007f7e44028c00 JavaThread "Surrogate Locker Thread (CMS)"  
>> daemon [_thread_blocked, id=10084, stack 
>> (0x000000004191b000,0x000000004193c000)]
>>  0x0000000002512c00 JavaThread "Finalizer" daemon [_thread_blocked,  
>> id=10083, stack(0x0000000041da9000,0x0000000041dca000)]
>>  0x0000000002511400 JavaThread "Reference Handler" daemon  
>> [_thread_blocked, id=10082, stack 
>> (0x00000000420b7000,0x00000000420d8000)]
>>
>> Other Threads:
>>  0x000000000250c000 VMThread [stack:  
>> 0x0000000040a64000,0x0000000040b65000] [id=10081]
>>  0x00007f7e44147c00 WatcherThread [stack:  
>> 0x00000000405f5000,0x00000000406f6000] [id=10092]
>>
>> VM state:not at safepoint (normal execution)
>>
>> VM Mutex/Monitor currently owned by a thread: None
>>
>> Heap
>>  par new generation   total 737280K, used 216364K  
>> [0x00007f7e548a0000, 0x00007f7e868a0000, 0x00007f7e868a0000)
>>  eden space 655360K,  20% used [0x00007f7e548a0000,  
>> 0x00007f7e5cbeb350, 0x00007f7e7c8a0000)
>>  from space 81920K, 100% used [0x00007f7e7c8a0000,  
>> 0x00007f7e818a0000, 0x00007f7e818a0000)
>>  to   space 81920K,   0% used [0x00007f7e818a0000,  
>> 0x00007f7e818a0000, 0x00007f7e868a0000)
>>  concurrent mark-sweep generation total 6718464K, used 1264492K  
>> [0x00007f7e868a0000, 0x00007f80209a0000, 0x00007f8020aa0000)
>>  concurrent-mark-sweep perm gen total 30432K, used 18222K  
>> [0x00007f8020aa0000, 0x00007f8022858000, 0x00007f8025ea0000)
>>
>> Dynamic libraries:
>> 00400000-00408000 r-xp 00000000 03:01  
>> 8576                               /usr/lib/jvm/java-6-openjdk/jre/ 
>> bin/java
>> 00608000-00609000 rw-p 00008000 03:01  
>> 8576                               /usr/lib/jvm/java-6-openjdk/jre/ 
>> bin/java
>> 023ba000-0416e000 rw-p 023ba000 00:00  
>> 0                                  [heap]
>> ...
>> ...
>> [ Not sure how important this stuff is.  There was a lot of it, if  
>> it's important I can send it.
>> ...
>> 7f802b6f9000-7f802b6fc000 rw-p 7f802b6f9000 00:00 0
>> 7f802b6fc000-7f802b6fe000 rw-p 0001b000 03:01  
>> 82264                      /lib/ld-2.7.so
>> 7fff7210c000-7fff72120000 rwxp 7ffffffe9000 00:00  
>> 0                      [stack]
>> 7fff72121000-7fff72121000 rw-p 7fffffffe000 00:00 0
>> 7fff721ff000-7fff72200000 r-xp 7fff721ff000 00:00  
>> 0                      [vdso]
>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00  
>> 0                  [vsyscall]
>>
>> VM Arguments:
>> jvm_args: -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 - 
>> Xms7361M -Xmx7361M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError - 
>> Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: 
>> +CMSParallelRemarkEnabled -XX:SurvivorRatio=8 - 
>> XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX: 
>> +UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true - 
>> Dcom.sun.management.jmxremote.port=8080 - 
>> Dcom.sun.management.jmxremote.ssl=false - 
>> Dcom.sun.management.jmxremote.authenticate=false - 
>> Dlog4j.configuration=log4j-server.properties - 
>> Dlog4j.defaultInitOverride=true
>> java_command: org.apache.cassandra.thrift.CassandraDaemon
>> Launcher Type: SUN_STANDARD
>>
>> Environment Variables:
>> PATH=/h/jcipar/SOFTWARE/ROOTS/Linux/x86_64/bin:/h/jcipar/bin:/h/ 
>> jcipar/SOFTWARE/ROOTS/All/bin:/h/jcipar/SOFTWARE/ant/apache- 
>> ant-1.8.1/bin/:~mabdelm/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/ 
>> X11:/usr/games
>> LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/ 
>> usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6- 
>> openjdk/jre/../lib/amd64
>> SHELL=/bin/bash
>>
>> Signal Handlers:
>> SIGSEGV: [libjvm.so+0x5d2630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> SIGBUS: [libjvm.so+0x5d2630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> SIGFPE: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> SIGPIPE: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> SIGXFSZ: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> SIGILL: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
>> SIGUSR2: [libjvm.so+0x4ab380], sa_mask[0]=0x00000000, sa_flags=0x10000004
>> SIGHUP: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
>> SIGTERM: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> SIGQUIT: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>>
>>
>> ---------------  S Y S T E M  ---------------
>>
>> OS:5.0.6
>>
>> uname:Linux 2.6.26-2-amd64 #1 SMP Thu Sep 16 15:56:38 UTC 2010 x86_64
>> libc:glibc 2.7 NPTL 2.7
>> rlimit: STACK 8192k, CORE 0k, NPROC 124096, NOFILE 1024, AS infinity
>> load average:3.09 3.56 3.72
>>
>> CPU:total 8 (1 cores per cpu, 1 threads per core) family 6 model 2  
>> stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3
>>
>> Memory: 4k page, physical 15075756k(6082384k free), swap 0k(0k free)
>>
>> vm_info: OpenJDK 64-Bit Server VM (1.6.0_0-b11) for linux-amd64 JRE  
>> (1.6.0_0-b11), built on Apr  9 2009 19:35:18 by "pbuilder" with gcc  
>> 4.3.2
>>
>> time: Tue May 10 13:01:39 2011
>> elapsed time: 2175 seconds
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On May 12, 2011, at 9:30 PM, Jeffrey Kesselman wrote:
>>
>>> If this a 64bit VM?
>>>
>>> A 32bit Java VM with default c-heap settings can only actually use
>>> about 2GB of Java Heap.
>>>
>>> On Thu, May 12, 2011 at 8:08 PM, James Cipar <jcipar@cmu.edu> wrote:
>>>> Oh, forgot this detail:  I have no swap configured, so swapping  
>>>> is not the cause of the crash.  Could it be that I'm running out  
>>>> of memory on a 15GB machine?  That seems unlikely.  I grepped  
>>>> dmesg for "oom" and didn't see anything from the oom killer, and  
>>>> I used the instructions from the following web page and didn't  
>>>> see that the oom killer had killed anything.
>>>>
>>>> http://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer
>>>>
>>>> jcipar@172-19-149-62:~$ sudo cat /var/log/messages | grep -- 
>>>> ignore-case "killed process"
>>>> jcipar@172-19-149-62:~$
>>>>
>>>>
>>>>
>>>> Also, this is pretty subjective, so I can't say for sure until it  
>>>> finishes, but this seems to be running *much* slower after  
>>>> setting the heap size and setting up JNA.
>>>>
>>>>
>>>>
>>>> On May 12, 2011, at 7:52 PM, James Cipar wrote:
>>>>
>>>>> It looks like MAX_HEAP_SIZE is set in cassandra-env.sh to be  
>>>>> half of my physical memory.  These are 15GB VMs, so that's 7.5GB  
>>>>> for Cassandra.  I would have expected that to work, but I will  
>>>>> override to 13 GB just to see what happens.
>>>>>
>>>>> I've also got the JNA thing set up.  Do you think this would  
>>>>> cause the crashes, or is it just a performance improvement?
>>>>>
>>>>>
>>>>>
>>>>> On May 12, 2011, at 7:27 PM, Sameer Farooqui wrote:
>>>>>
>>>>>> The key JVM options for Cassandra are in cassandra.in.sh.
>>>>>>
>>>>>> What is your min and max heap size?
>>>>>>
>>>>>> The default setting of max heap size is 1GB. How much RAM do  
>>>>>> your nodes have? You may want to increase this setting. You can 

>>>>>> also set the -Xmx and -Xms options to the same value to keep  
>>>>>> Java from having to manage heap growth. On a 32-bit machine,  
>>>>>> you can get a max of about 1.6 GB of heap; you can get a lot  
>>>>>> more on 64-bit.
>>>>>>
>>>>>> Try messing with some of the other settings in the  
>>>>>> cassandra.in.sh file.
>>>>>>
>>>>>> You may not have DEBUG mode turned on for Cassandra and  
>>>>>> therefore may not be getting the full details of what's going  
>>>>>> on when the server crashes. In the <cassandra-home>/conf/log4j-

>>>>>> server.properties file, set this line from the default of INFO  
>>>>>> to DEBUG:
>>>>>>
>>>>>> log4j.rootLogger=INFO,stdout,R
>>>>>>
>>>>>>
>>>>>> Also, you haven't configured JNA on this server. Here's some  
>>>>>> info about it and how to configure it:
>>>>>>
>>>>>> JNA provides Java programs easy access to native shared  
>>>>>> libraries without writing anything but Java code.
>>>>>>
>>>>>> Note from Cassandra developers for why JNA is needed:
>>>>>> "Linux aggressively swaps out infrequently used memory to make  
>>>>>> more room for its file system buffer cache. Unfortunately,  
>>>>>> modern generational garbage collectors like the JVM's leave  
>>>>>> parts of its heap un-touched for relatively large amounts of  
>>>>>> time, leading Linux to swap it out. When the JVM finally goes  
>>>>>> to use or GC that memory, swap hell ensues.
>>>>>>
>>>>>> Setting swappiness to zero can mitigate this behavior but does  
>>>>>> not eliminate it entirely. Turning off swap entirely is  
>>>>>> effective. But to avoid surprising people who don't know about  
>>>>>> this behavior, the best solution is to tell Linux not to swap  
>>>>>> out the JVM, and that is what we do now with mlockall via JNA.
>>>>>>
>>>>>> Because of licensing issues, we can't distribute JNA with  
>>>>>> Cassandra, so you must manually add it to the Cassandra lib/  
>>>>>> directory or otherwise place it on the classpath. If the JNA  
>>>>>> jar is not present, Cassandra will continue as before."
>>>>>>
>>>>>> Get JNA with:
>>>>>> cd ~
>>>>>> wget http://debian.riptano.com/debian/pool/libjna-java_3.2.7-0~nmu.2_amd64.deb
>>>>>>
>>>>>> To install:
>>>>>> techlabs@cassandraN1:~$ sudo dpkg -i libjna-java_3.2.7-0~nmu. 
>>>>>> 2_amd64.deb
>>>>>> (Reading database ... 44334 files and directories currently  
>>>>>> installed.)
>>>>>> Preparing to replace libjna-java 3.2.4-2 (using libjna- 
>>>>>> java_3.2.7-0~nmu.2_amd64.deb) ...
>>>>>> Unpacking replacement libjna-java ...
>>>>>> Setting up libjna-java (3.2.7-0~nmu.2) ...
>>>>>>
>>>>>>
>>>>>> The deb package will install the JNA jar file to /usr/share/ 
>>>>>> java/jna.jar, but Cassandra only loads it if its in the class  
>>>>>> path. The easy way to do this is just create a symlink into  
>>>>>> your Cassandra lib directory (note: replace /home/techlabs with 

>>>>>> your home dir location):
>>>>>> ln -s /usr/share/java/jna.jar /home/techlabs/apache- 
>>>>>> cassandra-0.7.0/lib
>>>>>>
>>>>>> Research:
>>>>>> http://journal.paul.querna.org/articles/2010/11/11/enabling-jna-in-cassandra/
>>>>>>
>>>>>>
>>>>>> - Sameer
>>>>>>
>>>>>>
>>>>>> On Thu, May 12, 2011 at 4:15 PM, James Cipar <jcipar@cmu.edu>
 
>>>>>> wrote:
>>>>>> I'm using Cassandra 0.7.5, and uploading about 200 GB of data  
>>>>>> total (20 GB unique data), to a cluster of 10 servers.  I'm  
>>>>>> using batch_mutate, and breaking the data up into chunks of  
>>>>>> about 10k records.  Each record is about 5KB, so a total of  
>>>>>> about 50MB per batch.  When I upload a smaller 2 GB data set,  
>>>>>> everything works fine.  When I upload the 20 GB data set,  
>>>>>> servers will occasionally crash.  Currently I have my client  
>>>>>> code automatically detect this and restart the server, but that 

>>>>>> is less than ideal.
>>>>>>
>>>>>> I'm not sure what information to gather to determine what's  
>>>>>> going on here.  Here is a sample of a log file from when a  
>>>>>> crash occurred.  The crash was immediately after the log entry  
>>>>>> tagged "2011-05-12 19:02:19,377".  Any idea what's going on  
>>>>>> here?  Any other info I can gather to try to debug this?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:07,855  
>>>>>> GCInspector.java (line 128) GC for ParNew: 375 ms, 576641232  
>>>>>> reclaimed leaving 5471432144 used; max is 7774142464
>>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:08,857  
>>>>>> GCInspector.java (line 128) GC for ParNew: 450 ms, -63738232  
>>>>>> reclaimed leaving 5546942544 used; max is 7774142464
>>>>>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:10,652  
>>>>>> CommitLogSegment.java (line 50) Creating new commitlog segment /

>>>>>> mnt/scratch/jcipar/cassandra/commitlog/ 
>>>>>> CommitLog-1305241330652.log
>>>>>> INFO [MutationStage:24] 2011-05-12 19:02:10,680  
>>>>>> ColumnFamilyStore.java (line 1070) Enqueuing flush of Memtable- 
>>>>>> Standard1@1256245282(51921529 bytes, 1115783 operations)
>>>>>> INFO [FlushWriter:1] 2011-05-12 19:02:10,680 Memtable.java  
>>>>>> (line 158) Writing Memtable-Standard1@1256245282(51921529  
>>>>>> bytes, 1115783 operations)
>>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:12,932  
>>>>>> GCInspector.java (line 128) GC for ParNew: 249 ms, 571827736  
>>>>>> reclaimed leaving 3165899760 used; max is 7774142464
>>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:15,253  
>>>>>> GCInspector.java (line 128) GC for ParNew: 341 ms, 561823592  
>>>>>> reclaimed leaving 1764208800 used; max is 7774142464
>>>>>> INFO [FlushWriter:1] 2011-05-12 19:02:16,743 Memtable.java  
>>>>>> (line 165) Completed flushing /mnt/scratch/jcipar/cassandra/ 
>>>>>> data/Keyspace1/Standard1-f-74-Data.db (53646223 bytes)
>>>>>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:16,745 CommitLog.java 

>>>>>> (line 440) Discarding obsolete commit log:CommitLogSegment(/mnt/

>>>>>> scratch/jcipar/cassandra/commitlog/CommitLog-1305241306438.log)
>>>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:18,256  
>>>>>> GCInspector.java (line 128) GC for ParNew: 305 ms, 544491840  
>>>>>> reclaimed leaving 865198712 used; max is 7774142464
>>>>>> INFO [MutationStage:19] 2011-05-12 19:02:19,000  
>>>>>> ColumnFamilyStore.java (line 1070) Enqueuing flush of Memtable- 
>>>>>> Standard1@479849353(51941121 bytes, 1115783 operations)
>>>>>> INFO [FlushWriter:1] 2011-05-12 19:02:19,000 Memtable.java  
>>>>>> (line 158) Writing Memtable-Standard1@479849353(51941121 bytes, 1115783

>>>>>>  operations)
>>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,310 SSTable.java  
>>>>>> (line 147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/

>>>>>> Standard1-f-51
>>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,324 SSTable.java  
>>>>>> (line 147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/

>>>>>> Standard1-f-55
>>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,339 SSTable.java  
>>>>>> (line 147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/

>>>>>> Standard1-f-58
>>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,357 SSTable.java  
>>>>>> (line 147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/

>>>>>> Standard1-f-67
>>>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,377 SSTable.java  
>>>>>> (line 147) Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/

>>>>>> Standard1-f-61
>>>>>> INFO [main] 2011-05-12 19:02:21,026  
>>>>>> AbstractCassandraDaemon.java (line 78) Logging initialized
>>>>>> INFO [main] 2011-05-12 19:02:21,040  
>>>>>> AbstractCassandraDaemon.java (line 96) Heap size: 7634681856/7635730432
>>>>>> INFO [main] 2011-05-12 19:02:21,042 CLibrary.java (line 61) JNA 

>>>>>> not found. Native methods will be disabled.
>>>>>> INFO [main] 2011-05-12 19:02:21,052 DatabaseDescriptor.java  
>>>>>> (line 121) Loading settings from file:/h/jcipar/Projects/HP/ 
>>>>>> OtherDBs/Cassandra/apache-cassandra-0.7.5/conf/cassandra.yaml
>>>>>> INFO [main] 2011-05-12 19:02:21,178 DatabaseDescriptor.java  
>>>>>> (line 181) DiskAccessMode 'auto' determined to be mmap,  
>>>>>> indexAccessMode is mmap
>>>>>> INFO [main] 2011-05-12 19:02:21,310 SSTableReader.java (line  
>>>>>> 154) Opening /mnt/scratch/jcipar/cassandra/data/system/Schema-f-1
>>>>>> INFO [main] 2011-05-12 19:02:21,327 SSTableReader.java (line  
>>>>>> 154) Opening /mnt/scratch/jcipar/cassandra/data/system/Schema-f-2
>>>>>> INFO [main] 2011-05-12 19:02:21,336 SSTableReader.java (line  
>>>>>> 154) Opening /mnt/scratch/jcipar/cassandra/data/system/ 
>>>>>> Migrations-f-1
>>>>>> INFO [main] 2011-05-12 19:02:21,337 SSTableReader.java (line  
>>>>>> 154) Opening /mnt/scratch/jcipar/cassandra/data/system/ 
>>>>>> Migrations-f-2
>>>>>> INFO [main] 2011-05-12 19:02:21,342 SSTableReader.java (line  
>>>>>> 154) Opening /mnt/scratch/jcipar/cassandra/data/system/ 
>>>>>> LocationInfo-f-2
>>>>>> INFO [main] 2011-05-12 19:02:21,344 SSTableReader.java (line  
>>>>>> 154) Opening /mnt/scratch/jcipar/cassandra/data/system/ 
>>>>>> LocationInfo-f-1
>>>>>> INFO [main] 2011-05-12 19:02:21,379 DatabaseDescriptor.java  
>>>>>> (line 461) Loading schema version 9467ffe0-7cea-11e0-8ddc- 
>>>>>> f74ef74e382f
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> It's always darkest just before you are eaten by a grue.
>>>
>>
>>
>
>
>
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com

Mime
View raw message