hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Sela <am...@infolinks.com>
Subject Re: HBase 0.94 security configurations
Date Sat, 02 Jun 2012 16:26:46 GMT
I don't know about a local zookeeper running.
I'm trying to run a test that extends HBaseTestingUtils.
It worked fine with the old versions when it extended HBaseClusterTestCase.
Since it is deprecated (also same for HBaseTestCase) I adjusted the test a
little bit (things like cong, fs, dfs etc.. became private and use getters
now) - maybe what i'm missing is here ?

On Sat, Jun 2, 2012 at 2:43 PM, Andrew Purtell <andrew.purtell@gmail.com>wrote:

> Do you have a local zookeeper running? Telnet localhost 2181 connect to
> anything?
>
> Obviously we run 0.94 with no security setup with no problem. Hence
> looking for basic setup problems.
>
>    - Andy
>
> On Jun 2, 2012, at 1:20 PM, Amit Sela <amits@infolinks.com> wrote:
>
> > I still get the same error.
> >
> > This is the contents of the configuration as it is set right before
> > calling "new
> > HBaseAdmin(getConfiguration())":
> >
> > key :hbase.auth.token.max.lifetime
> > value :604800000
> > key :hbase.thrift.maxQueuedRequests
> > value :1000
> > key :io.seqfile.compress.blocksize
> > value :1000000
> > key :hbase.hstore.compactionThreshold
> > value :3
> > key :hbase.coprocessor.abortonerror
> > value :false
> > key :hadoop.log.dir
> > value :/tmp
> > key :hbase.master.port
> > value :60000
> > key :webinterface.private.actions
> > value :false
> > key :dfs.support.append
> > value :true
> > key :hbase.rpc.engine
> > value :org.apache.hadoop.hbase.ipc.WritableRpcEngine
> > key :hbase.auth.key.update.interval
> > value :86400000
> > key :fs.s3.impl
> > value :org.apache.hadoop.fs.s3.S3FileSystem
> > key :hbase.zookeeper.leaderport
> > value :3888
> > key :hadoop.native.lib
> > value :true
> > key :fs.checkpoint.edits.dir
> > value :${fs.checkpoint.dir}
> > key :ipc.server.listen.queue.size
> > value :128
> > key :hbase.regionserver.hlog.reader.impl
> > value :org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader
> > key :hbase.regionserver.info.bindAddress
> > value :0.0.0.0
> > key :hadoop.security.authorization
> > value :false
> > key :hbase.mapreduce.hfileoutputformat.blocksize
> > value :65536
> > key :hbase.regionserver.hlog.writer.impl
> > value :org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter
> > key :hbase.regionserver.logroll.errors.tolerated
> > value :2
> > key :hbase.regionserver.nbreservationblocks
> > value :4
> > key :hbase.tmp.dir
> > value :/tmp/hbase-${user.name}
> > key :hbase.zookeeper.dns.nameserver
> > value :default
> > key :hbase.hregion.memstore.mslab.enabled
> > value :true
> > key :io.file.buffer.size
> > value :4096
> > key :hbase.zookeeper.property.dataDir
> > value :${hbase.tmp.dir}/zookeeper
> > key :hbase.data.umask.enable
> > value :false
> > key :hadoop.logfile.size
> > value :10000000
> > key :hbase.client.retries.number
> > value :10
> > key :fs.webhdfs.impl
> > value :org.apache.hadoop.hdfs.web.WebHdfsFileSystem
> > key :ipc.client.kill.max
> > value :10
> > key :hbase.regionserver.lease.period
> > value :60000
> > key :hbase.defaults.for.version.skip
> > value :false
> > key :hbase.zookeeper.property.clientPort
> > value :2181
> > key :zookeeper.znode.acl.parent
> > value :acl
> > key :hbase.regionserver.dns.nameserver
> > value :default
> > key :ipc.server.tcpnodelay
> > value :false
> > key :hbase.balancer.period
> > value :300000
> > key :hbase.rest.readonly
> > value :false
> > key :hbase.master.info.bindAddress
> > value :0.0.0.0
> > key :hbase.regionserver.global.memstore.upperLimit
> > value :0.4
> > key :hadoop.logfile.count
> > value :10
> > key :hbase.hregion.majorcompaction
> > value :86400000
> > key :hbase.client.keyvalue.maxsize
> > value :10485760
> > key :hadoop.security.uid.cache.secs
> > value :14400
> > key :fs.ftp.impl
> > value :org.apache.hadoop.fs.ftp.FTPFileSystem
> > key :hbase.cluster.distributed
> > value :false
> > key :hbase.client.pause
> > value :1000
> > key :hbase.hregion.preclose.flush.size
> > value :5242880
> > key :fs.file.impl
> > value :org.apache.hadoop.fs.LocalFileSystem
> > key :hbase.regionserver.global.memstore.lowerLimit
> > value :0.35
> > key :hbase.regionserver.handler.count
> > value :10
> > key :ipc.client.connection.maxidletime
> > value :10000
> > key :hbase.online.schema.update.enable
> > value :false
> > key :hbase.hash.type
> > value :murmur
> > key :hbase.hregion.max.filesize
> > value :10737418240
> > key :hbase.hregion.memstore.block.multiplier
> > value :2
> > key :hadoop.policy.file
> > value :hbase-policy.xml
> > key :hbase.hstore.blockingWaitTime
> > value :90000
> > key :hbase.zookeeper.quorum
> > value :localhost
> > key :hbase.hregion.memstore.flush.size
> > value :134217728
> > key :hbase.zookeeper.property.syncLimit
> > value :5
> > key :fs.checkpoint.size
> > value :67108864
> > key :io.skip.checksum.errors
> > value :false
> > key :fs.s3n.impl
> > value :org.apache.hadoop.fs.s3native.NativeS3FileSystem
> > key :hbase.zookeeper.dns.interface
> > value :default
> > key :fs.s3.maxRetries
> > value :4
> > key :hbase.regionserver.logroll.period
> > value :3600000
> > key :hbase.metrics.showTableName
> > value :true
> > key :hbase.offheapcache.percentage
> > value :0
> > key :hbase.client.scanner.caching
> > value :1
> > key :hfile.format.version
> > value :2
> > key :hbase.regionserver.port
> > value :60020
> > key :fs.default.name
> > value :file:///
> > key :ipc.client.idlethreshold
> > value :4000
> > key :fs.hsftp.impl
> > value :org.apache.hadoop.hdfs.HsftpFileSystem
> > key :hadoop.tmp.dir
> > value :/tmp/hadoop-${user.name}
> > key :fs.checkpoint.dir
> > value :${hadoop.tmp.dir}/dfs/namesecondary
> > key :fs.s3.block.size
> > value :67108864
> > key :hbase.rs.cacheblocksonwrite
> > value :false
> > key :hbase.rootdir
> > value :file:///tmp/hbase-${user.name}/hbase
> > key :hbase.regionserver.class
> > value :org.apache.hadoop.hbase.ipc.HRegionInterface
> > key :hbase.regionserver.info.port
> > value :60030
> > key :io.serializations
> > value :org.apache.hadoop.io.serializer.WritableSerialization
> > key :hbase.regionserver.msginterval
> > value :3000
> > key :hbase.regionserver.dns.interface
> > value :default
> > key :hadoop.util.hash.type
> > value :murmur
> > key :io.seqfile.lazydecompress
> > value :true
> > key :hbase.rest.port
> > value :8080
> > key :hbase.defaults.for.version
> > value :0.94.0
> > key :hbase.zookeeper.peerport
> > value :2888
> > key :zookeeper.znode.rootserver
> > value :root-region-server
> > key :io.mapfile.bloom.size
> > value :1048576
> > key :io.storefile.bloom.block.size
> > value :131072
> > key :fs.s3.buffer.dir
> > value :${hadoop.tmp.dir}/s3
> > key :hbase.zookeeper.property.maxClientCnxns
> > value :300
> > key :hbase.master.dns.interface
> > value :default
> > key :hbase.server.versionfile.writeattempts
> > value :3
> > key :hbase.thrift.minWorkerThreads
> > value :16
> > key :io.compression.codecs
> > value
> >
> :org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec
> > key :topology.script.number.args
> > value :100
> > key :fs.har.impl
> > value :org.apache.hadoop.fs.HarFileSystem
> > key :io.seqfile.sorter.recordlimit
> > value :1000000
> > key :zookeeper.session.timeout
> > value :1800000
> > key :fs.trash.interval
> > value :0
> > key :local.cache.size
> > value :10737418240
> > key :hadoop.security.authentication
> > value :simple
> > key :hadoop.security.group.mapping
> > value :org.apache.hadoop.security.ShellBasedUnixGroupsMapping
> > key :hbase.regions.slop
> > value :0.2
> > key :hadoop.security.token.service.use_ip
> > value :true
> > key :ipc.client.connect.max.retries
> > value :10
> > key :fs.ramfs.impl
> > value :org.apache.hadoop.fs.InMemoryFileSystem
> > key :hadoop.rpc.socket.factory.class.default
> > value :org.apache.hadoop.net.StandardSocketFactory
> > key :fs.kfs.impl
> > value :org.apache.hadoop.fs.kfs.KosmosFileSystem
> > key :hfile.block.index.cacheonwrite
> > value :false
> > key :hbase.master.dns.nameserver
> > value :default
> > key :hbase.bulkload.retries.number
> > value :0
> > key :hbase.hstore.compaction.max
> > value :10
> > key :fs.checkpoint.period
> > value :3600
> > key :topology.node.switch.mapping.impl
> > value :org.apache.hadoop.net.ScriptBasedMapping
> > key :zookeeper.znode.parent
> > value :/hbase
> > key :mapred.output.dir
> > value :/tmp/hadoop-amits
> > key :hbase.server.thread.wakefrequency
> > value :10000
> > key :hbase.master.info.port
> > value :60010
> > key :hfile.index.block.max.size
> > value :131072
> > key :hbase.regionserver.optionallogflushinterval
> > value :1000
> > key :fs.hdfs.impl
> > value :org.apache.hadoop.hdfs.DistributedFileSystem
> > key :hbase.thrift.maxWorkerThreads
> > value :1000
> > key :io.storefile.bloom.cacheonwrite
> > value :false
> > key :hbase.hstore.blockingStoreFiles
> > value :7
> > key :hfile.block.cache.size
> > value :0.25
> > key :io.mapfile.bloom.error.rate
> > value :0.005
> > key :io.bytes.per.checksum
> > value :512
> > key :hbase.zookeeper.property.initLimit
> > value :10
> > key :fs.har.impl.disable.cache
> > value :true
> > key :ipc.client.tcpnodelay
> > value :false
> > key :fs.hftp.impl
> > value :org.apache.hadoop.hdfs.HftpFileSystem
> > key :hbase.data.umask
> > value :000
> > key :hbase.master.logcleaner.plugins
> > value :org.apache.hadoop.hbase.master.TimeToLiveLogCleaner
> > key :hbase.master.logcleaner.ttl
> > value :600000
> > key :hbase.regionserver.regionSplitLimit
> > value :2147483647
> > key :fs.s3.sleepTimeSeconds
> > value :10
> > key :hbase.client.write.buffer
> > value :2097152
> > key :hbase.regionserver.info.port.auto
> > value :false
> >
> > Am I missing something ? because it looks OK to me.
> >
> > On Thu, May 31, 2012 at 11:05 PM, Andrew Purtell
> > <andrew.purtell@gmail.com>wrote:
> >
> >> Great, now remove any security related Zookeeper properties that you
> added
> >> in hbase-site.xml. Only keep hbase.zookeeper.quorum.
> >>
> >> On May 31, 2012, at 9:52 PM, Amit Sela <amits@infolinks.com> wrote:
> >>
> >>> I did some debug and the code does calls HBaseConfiguration.create() -
> >>> since my test extends HBaseTestingUtility.
> >>> and conf.properties.get("hbase.zookeeper.quorum") returns "localhost".
> >>>
> >>> Is that properly set, or should it be something else ? keep in my it's
> a
> >>> test running on my laptop, so it seems OK to me.
> >>>
> >>>
> >>> On Thu, May 31, 2012 at 10:01 PM, Andrew Purtell
> >>> <andrew.purtell@gmail.com>wrote:
> >>>
> >>>> I mean of course server null means that hbase.zookeeper.quorum config
> >>>> property is unset. And the two most common reasons are:
> >>>>
> >>>> 1. Not defined in the site file
> >>>>
> >>>> 2. Configuration object not created with HBaseConfiguration.create()
> >>>>
> >>>> I hope this is clearer.
> >>>>
> >>>> On May 31, 2012, at 8:59 PM, Andrew Purtell <andrew.purtell@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>>> Server null usually means you haven't configured
> hbase.zookeeper.quorum
> >>>> in your client's hbase-site.xml file. And that is usually because you
> >> are
> >>>> using a Configuration not created by HBaseConfiguration.create()
> >>>>>
> >>>>> If so the JAAS warning is a red herring.
> >>>>>
> >>>>> On May 31, 2012, at 8:52 PM, Amit Sela <amits@infolinks.com>
wrote:
> >>>>>
> >>>>>> I'm trying to run a test for HBase (some think we wrote, internal)
> on
> >> my
> >>>>>> laptop - runs perfectly with the old versions of Hadoop, HBase
and
> >>>>>> ZooKeeeper.
> >>>>>>
> >>>>>> After deploying the new versions and re-compiling our code,
I run
> the
> >>>> test.
> >>>>>>
> >>>>>> When I try to instantiate  "new HBaseAdmin(getConfiguration())"
-
> >> where
> >>>> the
> >>>>>> configuration is from HBaseTestingUtility, I get the following
on
> the
> >>>>>> console output:
> >>>>>>
> >>>>>> 2012-05-31 21:36:51.728
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.client.ZooKeeperSaslClient            
 -
> >>>>>> SecurityException: java.lang.SecurityException: Unable to locate
a
> >> login
> >>>>>> configuration occurred when trying to find JAAS configuration.
> >>>>>> 2012-05-31 21:36:51.741
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.ClientCnxn                            
 -
> Session
> >>>> 0x0
> >>>>>> for server null, unexpected error, closing socket connection
and
> >>>> attempting
> >>>>>> reconnect
> >>>>>> java.net.ConnectException: Connection refused
> >>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >>>>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>>>>
> >>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> 2012-05-31 21:36:51.852 [main]      WARN
> >>>>>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper     
 -
> >> Possibly
> >>>>>> transient ZooKeeper exception:
> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
> >>>>>> 2012-05-31 21:36:52.847
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.client.ZooKeeperSaslClient            
 -
> >>>>>> SecurityException: java.lang.SecurityException: Unable to locate
a
> >> login
> >>>>>> configuration occurred when trying to find JAAS configuration.
> >>>>>> 2012-05-31 21:36:52.848
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.ClientCnxn                            
 -
> Session
> >>>> 0x0
> >>>>>> for server null, unexpected error, closing socket connection
and
> >>>> attempting
> >>>>>> reconnect
> >>>>>> java.net.ConnectException: Connection refused
> >>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >>>>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>>>>
> >>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> 2012-05-31 21:36:53.949
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.client.ZooKeeperSaslClient            
 -
> >>>>>> SecurityException: java.lang.SecurityException: Unable to locate
a
> >> login
> >>>>>> configuration occurred when trying to find JAAS configuration.
> >>>>>> 2012-05-31 21:36:53.951
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.ClientCnxn                            
 -
> Session
> >>>> 0x0
> >>>>>> for server null, unexpected error, closing socket connection
and
> >>>> attempting
> >>>>>> reconnect
> >>>>>> java.net.ConnectException: Connection refused
> >>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >>>>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>>>>
> >>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> 2012-05-31 21:36:54.052 [main]      WARN
> >>>>>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper     
 -
> >> Possibly
> >>>>>> transient ZooKeeper exception:
> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
> >>>>>> 2012-05-31 21:36:55.052
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.client.ZooKeeperSaslClient            
 -
> >>>>>> SecurityException: java.lang.SecurityException: Unable to locate
a
> >> login
> >>>>>> configuration occurred when trying to find JAAS configuration.
> >>>>>> 2012-05-31 21:36:55.053
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.ClientCnxn                            
 -
> Session
> >>>> 0x0
> >>>>>> for server null, unexpected error, closing socket connection
and
> >>>> attempting
> >>>>>> reconnect
> >>>>>> java.net.ConnectException: Connection refused
> >>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >>>>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>>>>
> >>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> 2012-05-31 21:36:56.155
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.client.ZooKeeperSaslClient            
 -
> >>>>>> SecurityException: java.lang.SecurityException: Unable to locate
a
> >> login
> >>>>>> configuration occurred when trying to find JAAS configuration.
> >>>>>> 2012-05-31 21:36:56.156
> [main-SendThread(localhost.localdomain:2181)]
> >>>> WARN
> >>>>>> org.apache.zookeeper.ClientCnxn                            
 -
> Session
> >>>> 0x0
> >>>>>> for server null, unexpected error, closing socket connection
and
> >>>> attempting
> >>>>>> reconnect
> >>>>>> java.net.ConnectException: Connection refused
> >>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >>>>>> ~[na:1.6.0_31]
> >>>>>> at
> >>>>>>
> >>>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> >>>>>> ~[zookeeper-3.4.3.jar:3.4.3-1240972]
> >>>>>> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> >>>>>> ~[zookeeper-3.4.3.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message