hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: 答复: 答复: how to Dynamic load of Coprocessors
Date Sat, 23 Jul 2016 16:04:44 GMT
Did you run your program as user 'hbase' ?

>From the region server log snippet it seems you were running as '
shengchen.ma'

Cheers


On Sat, Jul 23, 2016 at 7:25 AM, Ma, Sheng-Chen (Aven) <
shengchen.ma@esgyn.cn> wrote:

> yes, it’s the master log.
>
> 2016-07-23 14:17:53,175 INFO
> [aven01.novalocal,16088,1469275959674_ChoreService_1]
> regionserver.HRegionServer:
> aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting flush
> of hbase:meta,,1.1588230740 because info has an old edit so flush to free
> WALs after random delay 126800ms
> 2016-07-23 14:17:57,520 WARN  [group-cache-0]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups
> for user shengchen.ma
> ExitCodeException exitCode=1: id: shengchen.ma: No such user
>
>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>         at org.apache.hadoop.util.Shell.run(Shell.java:455)
>         at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>         at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
>         at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
>         at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:84)
>         at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
>         at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
>         at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
>         at
> org.apache.hadoop.hbase.security.UserProvider$1.getGroupStrings(UserProvider.java:93)
>         at
> org.apache.hadoop.hbase.security.UserProvider$1.access$000(UserProvider.java:81)
>         at
> org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:108)
>         at
> org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:104)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 2016-07-23 14:18:03,207 INFO
> [aven01.novalocal,16088,1469275959674_ChoreService_1]
> regionserver.HRegionServer:
> aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting flush
> of hbase:meta,,1.1588230740 because info has an old edit so flush to free
> WALs after random delay 71020ms
>
> Got the error. But how to make the test.jar accessed for all user?
>
> [root@aven01 bin]# ./hadoop fs -ls /
> 16/07/23 14:15:58 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 9 items
> ...
> -rwxrwxrwx   1 root      supergroup    1771201 2016-07-23 14:13 /test.jar
> ...
>
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2016年7月23日 0:37
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: how to Dynamic load of Coprocessors
>
> The pasted log was from master, right ?
>
> Can you take a look at region server involved ?
>
> Cheers
>
> On Fri, Jul 22, 2016 at 9:23 AM, Ma, Sheng-Chen (Aven) <
> shengchen.ma@esgyn.cn> wrote:
>
> > Hi Ted:
> > Following is part of log. I have changed log level to TRACE. But still
> > can’t find any clue.
> >
> > 2016-07-22 12:03:38,261 DEBUG [ProcedureExecutor-1]
> > procedure2.ProcedureExecutor: Procedure completed in 1.6310sec:
> > DisableTableProcedure (table=testTbl) id=109 owner=shengchen.ma
> > state=F INISHED
> > 2016-07-22 12:03:38,866 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 14 method_name: "IsMasterRunning" request_param:
> > true totalRequestSize: 23 bytes
> > 2016-07-22 12:03:38,866 TRACE
> > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> > 14 service: MasterService methodName: IsMasterRunning size: 23
> connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,866 TRACE
> > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> > 14 service: MasterService methodName: IsMasterRunning size: 23
> connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunnin
> > gRequest
> > connection: 192.168.3.10:49752, response is_master_running: true
> > queueTime: 0 processingTime: 0 totalTime: 0
> > 2016-07-22 12:03:38,878 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 15 method_name: "getProcedureResult"
> request_param:
> > true totalRequestSize: 28 bytes
> > 2016-07-22 12:03:38,878 TRACE
> > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> > callId: 15
> > service: MasterService methodName: getProcedureResult size: 28
> connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,878 DEBUG
> > [B.defaultRpcServer.handler=7,queue=1,port=39479]
> master.MasterRpcServices:
> > Checking to see if procedure is done procId=109
> > 2016-07-22 12:03:38,878 TRACE
> > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> > callId: 15
> > service: MasterService methodName: getProcedureResult size: 28
> connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetProcedureRe
> > sultRequest
> > connection: 192.168.3.10:49752, response state: FINISHED start_time:
> > 1469189016526 last_update: 1469189018157 queueTime: 0 processingTime:
> > 0
> > totalTime: 0
> > 2016-07-22 12:03:38,888 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 16 method_name: "IsMasterRunning" request_param:
> > true totalRequestSize: 23 bytes
> > 2016-07-22 12:03:38,888 TRACE
> > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> > callId: 16
> > service: MasterService methodName: IsMasterRunning size: 23 connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,888 TRACE
> > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> > callId: 16
> > service: MasterService methodName: IsMasterRunning size: 23 connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunnin
> > gRequest
> > connection: 192.168.3.10:49752, response is_master_running: true
> > queueTime: 0 processingTime: 0 totalTime: 0
> > 2016-07-22 12:03:39,007 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 17 method_name: "ModifyTable" request_param:
> > true
> > totalRequestSize: 505 bytes
> > 2016-07-22 12:03:39,007 TRACE
> > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer: callId:
> > 17 service: MasterService methodName: ModifyTable size: 505 connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:39,047 DEBUG
> > [B.defaultRpcServer.handler=17,queue=2,port=39479]
> > util.CoprocessorClassLoader: Skipping exempt class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint -
> > delegating directly to parent
> > 2016-07-22 12:03:39,048 DEBUG
> > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer:
> > B.defaultRpcServer.handler=17,queue=2,port=39479: callId: 17 service:
> > MasterService methodName: ModifyTable size: 505 connection:
> > 192.168.3.10:49752
> > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > cannot be loaded Set hbase.table.sanity.checks to false at conf or
> > table descriptor if you want to bypass sanity checks
> >         at
> >
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1680)
> >         at
> >
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1541)
> >         at
> > org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2028)
> >         at
> >
> org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1170)
> >         at
> >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55680)
> >         at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> >         at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> >         at
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> >         at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> >         at java.lang.Thread.run(Thread.java:745)
> > 2016-07-22 12:03:39,622 DEBUG
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RpcServer.listener,port=39479: Caught exception while
> > reading:Connection reset by peer
> > 2016-07-22 12:03:39,622 DEBUG
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RpcServer.listener,port=39479: DISCONNECTING client 192.168.3.10:49752
> > because read count=-1. Number of active connections: 3
> > 2016-07-22 12:03:39,623 WARN
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> > server.NIOServerCnxn: Exception causing close of session
> > 0x1561182c97e0009 due to java.io.IOException: Connection reset by peer
> > 2016-07-22 12:03:39,623 INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> > server.NIOServerCnxn: Closed socket connection for client /
> > 192.168.3.10:49751 which had sessionid 0x1561182c97e0009
> > 2016-07-22 12:03:40,127 TRACE
> > [RpcServer.reader=1,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 5466 method_name: "RegionServerReport"
> > request_param: true totalRequestSize: 5748 bytes
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] ipc.RpcServer: callId:
> > 5466 service: RegionServerStatusService methodName: RegionServerReport
> > size: 5.6 K connection: 192.168.0.17:49483 executing as traf
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, family=mt_, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, family=tddlcf, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 3052710e85ab46a686a22e23bff937e2, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 3052710e85ab46a686a22e23bff937e2, family=cpf, existingValue=-1,
> > completeSequenceId=-1
> >
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2016年7月22日 19:48
> > 收件人: user@hbase.apache.org
> > 主题: Re: how to Dynamic load of Coprocessors
> >
> > w.r.t. the DoNotRetryIOException, can you take a look at region server
> > log where testTbl region(s) was hosted ?
> >
> > See if there is some clue why the sanity check failed.
> >
> > Thanks
> >
> > On Fri, Jul 22, 2016 at 1:12 AM, Ma, Sheng-Chen (Aven) <
> > shengchen.ma@esgyn.cn> wrote:
> >
> > > Hi all:
> > > I want to dynamic add coprocessor in order to not restart hbase.
> > >
> > > Following is my code:
> > >         Path path = new Path("/coprocessor_jars");
> > >         FileSystem fs = FileSystem.get(conf);
> > >         FileStatus[] status = fs.listStatus(path);
> > >         Path[] listedPaths = FileUtil.stat2Paths(status);
> > >         for (Path p : listedPaths) {
> > >             if(p.getName().contains("test.jar")){
> > >                 hdfsPath = p;
> > >             }
> > >         }
> > >         HBaseAdmin hadmin = new HBaseAdmin(conf);
> > >         HTableDescriptor tableDesc =
> > > hadmin.getTableDescriptor("testTbl".getBytes());
> > >
> > > tableDesc.addCoprocessor("org.apache.hadoop.hbase.coprocessor.transa
> > > ct
> > > ional.TestRegionEndpoint",
> > > hdfsPath,
> > >                 Coprocessor.PRIORITY_USER, null);
> > >         //
> > >
> > tableDesc.removeCoprocessor("org.apache.hadoop.hbase.coprocessor.trans
> > actional.TrxRegionEndpoint");
> > >         for (Entry<ImmutableBytesWritable, ImmutableBytesWritable>
> > > entry
> > :
> > > tableDesc.getValues().entrySet()) {
> > >             System.out.println(Bytes.toString(entry.getKey().get())
> > > + " = " + Bytes.toString(entry.getValue().get()));
> > >         }
> > >         hadmin.disableTable("testTbl".getBytes());
> > >         hadmin.modifyTable("testTbl", tableDesc);
> > >         hadmin.enableTable("testTbl");
> > >
> > > the syso print : coprocessor$1 = hdfs://
> > > 192.168.0.17:17400/coprocessor_jars/test.jar|org.apache.hadoop.hbase
> > > .c oprocessor.transactional.TestRegionEndpoint|1073741823|
> > >
> > > and the remote side return Exception:
> > > org.apache.hadoop.hbase.DoNotRetryIOException:
> > > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > > cannot be loaded Set hbase.table.sanity.checks to false at conf or
> > > table descriptor if you want to bypass sanity checks
> > >
> > > I use hbase 1.2 and the test.jar is not under hbase/lib, I just put
> > > the test.jar in hdfs.
> > >
> > > If I add test.jar to hbase/lib but not restart hbase, the upon code
> > > still throw same exception.
> > > If I add test.jar to hbase/lib and restart hbase, the upon code will
> > > exec successful.
> > >
> > > But my requirement is not restart hbase.
> > >
> > > Is there someone can give me a favor.
> > >
> > > Thanks
> > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message