hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Kellerman <...@powerset.com>
Subject RE: Concurrent Modification Exception
Date Wed, 12 Mar 2008 17:12:35 GMT
Thanks for the information. That is really helpful. We'll look into it.

---
Jim Kellerman, Senior Engineer; Powerset


> -----Original Message-----
> From: Goel, Ankur [mailto:Ankur.Goel@corp.aol.com]
> Sent: Tuesday, March 11, 2008 11:36 PM
> To: hbase-user@hadoop.apache.org
> Subject: RE: Concurrent Modification Exception
>
> Hi,
>    I compile Hadoop and Hbase locally on my machine, so
> Hadoop version is '0.17.0-dev' and Hbase version is '0.2.0-dev'.
>
> Also note that my machine is a windows XP machine and I use
> cygwin for both compiling and running Hadoop/Hbase on my machine.
>
> The stack trace from Region server logs is given below.
>
> 008-03-11 15:49:17,652 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: region
> TestTable,0000097920,1205226364049 available
> 2008-03-11 15:49:17,652 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN :
> TestTable,0029434692,1205229579914
> 2008-03-11 15:49:17,683 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: region
> TestTable,0029434692,1205229579914 available
> 2008-03-11 15:49:17,683 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN :
> TestTable,0006355562,1205229162173
> 2008-03-11 15:49:17,730 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: region
> TestTable,0006355562,1205229162173 available
> 2008-03-11 15:49:17,730 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN :
> TestTable,,1205226328794
> 2008-03-11 15:49:17,777 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: region
> TestTable,,1205226328794 available
> 2008-03-11 15:55:38,395 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> java.io.IOException: java.io.IOException:
> java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
>
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCo
> nstructorA
> ccessorImpl.java:39)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Dele
> gatingCons
> tructorAccessorImpl.java:27)
>         at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExc
> eption(Rem
> oteExceptionHandler.java:82)
>         at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOExceptio
> n(RemoteEx
> ceptionHandler.java:48)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegion
> Server.jav
> a:386)
>         at java.lang.Thread.run(Thread.java:619)
> 2008-03-11 15:55:38,395 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 1)
> java.io.IOException: java.io.IOException:
> java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
>
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCo
> nstructorA
> ccessorImpl.java:39)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Dele
> gatingCons
> tructorAccessorImpl.java:27)
>         at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExc
> eption(Rem
> oteExceptionHandler.java:82)
>         at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOExceptio
> n(RemoteEx
> ceptionHandler.java:48)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegion
> Server.jav
> a:386)
>         at java.lang.Thread.run(Thread.java:619)
> 2008-03-11 15:55:38,395 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 2)
> java.io.IOException: java.io.IOException:
> java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
>
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCo
> nstructorA
> ccessorImpl.java:39)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Dele
> gatingCons
> tructorAccessorImpl.java:27)
>         at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExc
> eption(Rem
> oteExceptionHandler.java:82)
>         at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOExceptio
> n(RemoteEx
> ceptionHandler.java:48)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegion
> Server.jav
> a:386)
>         at java.lang.Thread.run(Thread.java:619)
>
>
> Here are the logs from Hmaster
>
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> 10.146.165.80:60020 serving TestTable,0018969406,1205229954640
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> 10.146.165.80:60020 serving TestTable,0023087066,1205229579914
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> 10.146.165.80:60020 serving TestTable,0000097920,1205226364049
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> 10.146.165.80:60020 serving TestTable,0029434692,1205229579914
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> 10.146.165.80:60020 serving TestTable,0006355562,1205229162173
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> 10.146.165.80:60020 serving TestTable,,1205226328794
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0002123598,1205226744066, startKey: <0002123598>, endKey:
> <0004206876>, encodedName: 309209042, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0002123598,1205226744066 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,591 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0005281155,1205229224041, startKey: <0005281155>, endKey:
> <0006355562>, encodedName: 1633815830, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0005281155,1205229224041 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0009448095,1205229850239, startKey: <0009448095>, endKey:
> <0012621780>, encodedName: 1014866630, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0009448095,1205229850239 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0012621780,1205229850239, startKey: <0012621780>, endKey:
> <0015795593>, encodedName: 165593217, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0012621780,1205229850239 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0007429841,1205229162173, startKey: <0007429841>, endKey:
> <0009448095>, encodedName: 1851690796, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0007429841,1205229162173 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0015795593,1205229954640, startKey: <0015795593>, endKey:
> <0018969406>, encodedName: 477232303, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation:
> updating row TestTable,0015795593,1205229954640 in table
> .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0004206876,1205229224041, startKey: <0004206876>, endKey:
> <0005281155>, encodedName: 247834599, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0004206876,1205229224041 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0018969406,1205229954640, startKey: <0018969406>, endKey:
> <0023087066>, encodedName: 1686119456, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation:
> updating row TestTable,0018969406,1205229954640 in table
> .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0023087066,1205229579914, startKey: <0023087066>, endKey:
> <0029434692>, encodedName: 1844401047, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0023087066,1205229579914 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0000097920,1205226364049, startKey: <0000097920>, endKey:
> <0002123598>, encodedName: 57245683, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0000097920,1205226364049 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0029434692,1205229579914, startKey: <0029434692>,
> endKey: <>,
> encodedName: 1168116125, tableDesc: {name: TestTable, families:
> {info:={name: info, max versions: 3, compression: NONE, in memory:
> false, block cache enabled: false, max length: 2147483647,
> bloom filter:
> none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0029434692,1205229579914 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,0006355562,1205229162173, startKey: <0006355562>, endKey:
> <0007429841>, encodedName: 1182565503, tableDesc: {name: TestTable,
> families: {info:={name: info, max versions: 3, compression: NONE, in
> memory: false, block cache enabled: false, max length:
> 2147483647, bloom
> filter: none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,606 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,0006355562,1205229162173 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:46:19,622 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: regionname:
> TestTable,,1205226328794, startKey: <>, endKey: <0000097920>,
> encodedName: 2134019504, tableDesc: {name: TestTable, families:
> {info:={name: info, max versions: 3, compression: NONE, in memory:
> false, block cache enabled: false, max length: 2147483647,
> bloom filter:
> none}}} open on 10.146.165.80:60020
> 2008-03-11 15:46:19,622 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,,1205226328794 in table .META.,,1 with startcode
> 1205230563513 and server 10.146.165.80:60020
> 2008-03-11 15:47:01,325 INFO org.apache.hadoop.hbase.master.HMaster:
> deleted table: TestTable
> 2008-03-11 15:47:01,638 INFO org.apache.hadoop.ipc.Server:
> IPC Server handler 5 on 60000, call regionServerReport(address:
> 10.146.165.80:60020, startcode: 1205230563513, load: (requests: 21
> regions: 16), [Lorg.apache.hadoop.hbase.HMsg;@7e9ce2) from
> 10.146.165.80:2099: error: java.io.IOException:
> java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 2008-03-11 15:47:01,653 INFO org.apache.hadoop.ipc.Server:
> IPC Server handler 6 on 60000, call regionServerReport(address:
> 10.146.165.80:60020, startcode: 1205230563513, load: (requests: 0
> regions: 16), [Lorg.apache.hadoop.hbase.HMsg;@39c8c1) from
> 10.146.165.80:2099: error: java.io.IOException:
> java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 2008-03-11 15:47:01,653 INFO org.apache.hadoop.ipc.Server:
> IPC Server handler 8 on 60000, call regionServerReport(address:
> 10.146.165.80:60020, startcode: 1205230563513, load: (requests: 0
> regions: 16), [Lorg.apache.hadoop.hbase.HMsg;@111ded2) from
> 10.146.165.80:2099: error: java.io.IOException:
> java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 2008-03-11 15:47:01,653 INFO org.apache.hadoop.ipc.Server:
> IPC Server handler 7 on 60000, call regionServerReport(address:
> 10.146.165.80:60020, startcode: 1205230563513, load: (requests: 0
> regions: 16), [Lorg.apache.hadoop.hbase.HMsg;@3bc20e) from
> 10.146.165.80:2099: error: java.io.IOException:
> java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 2008-03-11 15:47:01,653 INFO org.apache.hadoop.ipc.Server:
> IPC Server handler 9 on 60000, call regionServerReport(address:
> 10.146.165.80:60020, startcode: 1205230563513, load: (requests: 0
> regions: 16), [Lorg.apache.hadoop.hbase.HMsg;@1415056) from
> 10.146.165.80:2099: error: java.io.IOException:
> java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 2008-03-11 15:47:01,653 INFO org.apache.hadoop.ipc.Server:
> IPC Server handler 0 on 60000, call regionServerReport(address:
> 10.146.165.80:60020, startcode: 1205230563513, load: (requests: 0
> regions: 16), [Lorg.apache.hadoop.hbase.HMsg;@1014e21) from
> 10.146.165.80:2099: error: java.io.IOException:
> java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$ValueIterator.next(HashMap.java:822)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(Serve
> rManager.j
> ava:366)
>         at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServ
> erAllsWell
> (ServerManager.java:268)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerRepor
> t(ServerMa
> nager.java:187)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMas
> ter.java:5
> 53)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessor
> Impl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> 2008-03-11 15:47:03,763 INFO
> org.apache.hadoop.hbase.master.ServerManager: Cancelling
> lease for 10.146.165.80:60020
> 2008-03-11 15:47:03,763 INFO
> org.apache.hadoop.hbase.master.ServerManager: Region server
> 10.146.165.80:60020: MSG_REPORT_EXITING -- lease cancelled
>
>
> Thanks
> -Ankur
>
> -----Original Message-----
> From: stack [mailto:stack@duboce.net]
> Sent: Wednesday, March 12, 2008 1:44 AM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Concurrent Modification Exception
>
> Can we have the stack trace and version number?
> Thanks,
> St.Ack
>
> Goel, Ankur wrote:
> > Hi ,
> >      I have a single machine setup running Hadoop and HBase.
> > I coupled HBase with Hadoop in order to run the HBase performance
> > evaluation test plan.
> >
> > The test plan ran for about 15-20 min and then terminated abruptly
> > taking down the region server along with it.
> >
> > Now everytime I try to delete the "TestTable" (created in the failed
> > test)
> > through the my Java code,  the region server dies and I get a
> > Concurrent
> >
> > modification exception in the logs.
> >
> > how can I restore the sanity of HBase?
> >
> > Thanks
> > -Ankur
> >
> >
>
>
> No virus found in this incoming message.
> Checked by AVG.
> Version: 7.5.518 / Virus Database: 269.21.7/1327 - Release
> Date: 3/12/2008 1:27 PM
>
>

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.518 / Virus Database: 269.21.7/1327 - Release Date: 3/12/2008 1:27 PM


Mime
View raw message