Return-Path: Delivered-To: apmail-hadoop-hbase-commits-archive@minotaur.apache.org Received: (qmail 23534 invoked from network); 27 Feb 2010 00:29:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 27 Feb 2010 00:29:54 -0000 Received: (qmail 56847 invoked by uid 500); 27 Feb 2010 00:29:54 -0000 Delivered-To: apmail-hadoop-hbase-commits-archive@hadoop.apache.org Received: (qmail 56823 invoked by uid 500); 27 Feb 2010 00:29:54 -0000 Mailing-List: contact hbase-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-commits@hadoop.apache.org Received: (qmail 56816 invoked by uid 99); 27 Feb 2010 00:29:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Feb 2010 00:29:53 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Feb 2010 00:29:50 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id D195423888DD; Sat, 27 Feb 2010 00:29:28 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r916882 - in /hadoop/hbase/trunk: ./ contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/ contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metr... Date: Sat, 27 Feb 2010 00:29:28 -0000 To: hbase-commits@hadoop.apache.org From: apurtell@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20100227002928.D195423888DD@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: apurtell Date: Sat Feb 27 00:29:27 2010 New Revision: 916882 URL: http://svn.apache.org/viewvc?rev=916882&view=rev Log: HBASE-2273 [stargate] export metrics via Hadoop metrics, JMX, and zookeeper Added: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateMetrics.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateStatistics.java Modified: hadoop/hbase/trunk/CHANGES.txt hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/Constants.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/Main.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RESTServlet.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RegionsResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RootResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RowResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerInstanceResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/SchemaResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterStatusResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterVersionResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/TableResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/VersionResource.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/JDBCAuthenticator.java hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/ZooKeeperAuthenticator.java hadoop/hbase/trunk/contrib/stargate/src/test/java/org/apache/hadoop/hbase/stargate/auth/TestZooKeeperAuthenticator.java Modified: hadoop/hbase/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/CHANGES.txt?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/CHANGES.txt (original) +++ hadoop/hbase/trunk/CHANGES.txt Sat Feb 27 00:29:27 2010 @@ -361,12 +361,15 @@ (bulk loading) HBASE-2153 Publish generated HTML documentation for Thrift on the website (Lars Francke via Stack) - HBASE-1373 Update Thrift to use compact/framed protocol (Lars Francke via Stack) + HBASE-1373 Update Thrift to use compact/framed protocol (Lars Francke via + Stack) HBASE-2172 Add constructor to Put for row key and timestamp (Lars Francke via Stack) HBASE-2178 Hooks for replication - HBASE-2180 Bad random read performance from synchronizing hfile.fddatainputstream - HBASE-2194 HTable - put(Put) , put(List e: connectors) { + status.object() + .key("host").value(e.getFirst()) + .key("port").value(e.getSecond()) + .endObject(); + } + status.endArray(); + status.endObject(); + updateNode(wrapper, znode, CreateMode.EPHEMERAL, + Bytes.toBytes(status.toString())); + } catch (Exception e) { + LOG.error(StringUtils.stringifyException(e)); + } + } + + } + + final String znode = INSTANCE_ZNODE_ROOT + "/" + System.currentTimeMillis(); transient final Configuration conf; transient final HTablePool pool; + transient volatile ZooKeeperWrapper wrapper; + transient Chore statusReporter; + transient Authenticator authenticator; + AtomicBoolean stopping = new AtomicBoolean(false); + boolean multiuser; Map maxAgeMap = Collections.synchronizedMap(new HashMap()); - boolean multiuser; - Authenticator authenticator; + List> connectors = + Collections.synchronizedList(new ArrayList>()); + StargateMetrics metrics = new StargateMetrics(); /** * @return the RESTServlet singleton instance @@ -65,6 +130,55 @@ return instance; } + static boolean ensureExists(final ZooKeeperWrapper zkw, final String znode, + final CreateMode mode) throws IOException { + ZooKeeper zk = zkw.getZooKeeper(); + try { + Stat stat = zk.exists(znode, false); + if (stat != null) { + return true; + } + zk.create(znode, new byte[0], Ids.OPEN_ACL_UNSAFE, mode); + LOG.debug("Created ZNode " + znode); + return true; + } catch (KeeperException.NodeExistsException e) { + return true; // ok, move on. + } catch (KeeperException.NoNodeException e) { + return ensureParentExists(zkw, znode, mode) && + ensureExists(zkw, znode, mode); + } catch (KeeperException e) { + throw new IOException(e); + } catch (InterruptedException e) { + throw new IOException(e); + } + } + + static boolean ensureParentExists(final ZooKeeperWrapper zkw, + final String znode, final CreateMode mode) throws IOException { + int index = znode.lastIndexOf("/"); + if (index <= 0) { // Parent is root, which always exists. + return true; + } + return ensureExists(zkw, znode.substring(0, index), mode); + } + + static void updateNode(final ZooKeeperWrapper zkw, final String znode, + final CreateMode mode, final byte[] data) throws IOException { + ensureExists(zkw, znode, mode); + ZooKeeper zk = zkw.getZooKeeper(); + try { + zk.setData(znode, data, -1); + } catch (KeeperException e) { + throw new IOException(e); + } catch (InterruptedException e) { + throw new IOException(e); + } + } + + ZooKeeperWrapper initZooKeeperWrapper() throws IOException { + return new ZooKeeperWrapper(conf, this); + } + /** * Constructor * @throws IOException @@ -72,23 +186,54 @@ public RESTServlet() throws IOException { this.conf = HBaseConfiguration.create(); this.pool = new HTablePool(conf, 10); + this.wrapper = initZooKeeperWrapper(); + this.statusReporter = new StatusReporter( + conf.getInt(STATUS_REPORT_PERIOD_KEY, 1000 * 60), stopping); } - /** - * Get a table pool for the given table. - * @return the table pool - */ - protected HTablePool getTablePool() { + @Override + public void process(WatchedEvent event) { + LOG.debug(("ZooKeeper.Watcher event " + event.getType() + " with path " + + event.getPath())); + // handle disconnection (or manual delete to test disconnection scenario) + if (event.getState() == KeeperState.Expired || + (event.getType().equals(EventType.NodeDeleted) && + event.getPath().equals(znode))) { + wrapper.close(); + wrapper = null; + while (!stopping.get()) try { + wrapper = initZooKeeperWrapper(); + break; + } catch (IOException e) { + LOG.error(StringUtils.stringifyException(e)); + try { + Thread.sleep(10 * 1000); + } catch (InterruptedException ex) { + } + } + } + } + + HTablePool getTablePool() { return pool; } - /** - * @return the servlet's global HBase configuration - */ - protected Configuration getConfiguration() { + ZooKeeperWrapper getZooKeeperWrapper() { + return wrapper; + } + + Configuration getConfiguration() { return conf; } + StargateMetrics getMetrics() { + return metrics; + } + + void addConnectorAddress(String host, int port) { + connectors.add(new Pair(host, port)); + } + /** * @param tableName the table name * @return the maximum cache age suitable for use with this table, in @@ -148,10 +293,24 @@ */ public Authenticator getAuthenticator() { if (authenticator == null) { - String className = conf.get("stargate.auth.authenticator"); - if (className != null) try { + String className = conf.get(AUTHENTICATOR_KEY, + HBCAuthenticator.class.getCanonicalName()); + try { Class c = getClass().getClassLoader().loadClass(className); - authenticator = (Authenticator)c.newInstance(); + if (className.endsWith(HBCAuthenticator.class.getName()) || + className.endsWith(HTableAuthenticator.class.getName()) || + className.endsWith(JDBCAuthenticator.class.getName())) { + Constructor cons = c.getConstructor(Configuration.class); + authenticator = (Authenticator) + cons.newInstance(new Object[] { conf }); + } else if (className.endsWith(ZooKeeperAuthenticator.class.getName())) { + Constructor cons = c.getConstructor(Configuration.class, + ZooKeeperWrapper.class); + authenticator = (Authenticator) + cons.newInstance(new Object[] { conf, wrapper }); + } else { + authenticator = (Authenticator)c.newInstance(); + } } catch (Exception e) { LOG.error(StringUtils.stringifyException(e)); } Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RegionsResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RegionsResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RegionsResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RegionsResource.java Sat Feb 27 00:29:27 2010 @@ -83,6 +83,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); try { TableInfoModel model = new TableInfoModel(table); Map regions = getTableRegions(); Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RootResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RootResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RootResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RootResource.java Sat Feb 27 00:29:27 2010 @@ -91,6 +91,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); if (servlet.isMultiUser()) { throw new WebApplicationException(Response.Status.BAD_REQUEST); } @@ -114,7 +115,7 @@ } @Path("version") - public VersionResource getVersionResource() { + public VersionResource getVersionResource() throws IOException { return new VersionResource(); } Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RowResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RowResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RowResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/RowResource.java Sat Feb 27 00:29:27 2010 @@ -87,6 +87,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); try { ResultGenerator generator = ResultGenerator.fromRowSpec(actualTableName, rowspec); @@ -124,6 +125,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath() + " as "+ MIMETYPE_BINARY); } + servlet.getMetrics().incrementRequests(1); // doesn't make sense to use a non specific coordinate as this can only // return a single cell if (!rowspec.hasColumns() || rowspec.getColumns().length > 1) { @@ -146,7 +148,8 @@ } } - private Response update(CellSetModel model, boolean replace) { + Response update(CellSetModel model, boolean replace) { + servlet.getMetrics().incrementRequests(1); HTablePool pool = servlet.getTablePool(); HTableInterface table = null; try { @@ -180,8 +183,9 @@ } } - private Response updateBinary(byte[] message, HttpHeaders headers, + Response updateBinary(byte[] message, HttpHeaders headers, boolean replace) { + servlet.getMetrics().incrementRequests(1); HTablePool pool = servlet.getTablePool(); HTableInterface table = null; try { @@ -276,7 +280,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("DELETE " + uriInfo.getAbsolutePath()); } - + servlet.getMetrics().incrementRequests(1); Delete delete = null; if (rowspec.hasTimestamp()) delete = new Delete(rowspec.getRow(), rowspec.getTimestamp(), null); Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerInstanceResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerInstanceResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerInstanceResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerInstanceResource.java Sat Feb 27 00:29:27 2010 @@ -47,16 +47,18 @@ private static final Log LOG = LogFactory.getLog(ScannerInstanceResource.class); - protected ResultGenerator generator; - private String id; - private int batch; - private CacheControl cacheControl; + ResultGenerator generator; + String id; + int batch; + RESTServlet servlet; + CacheControl cacheControl; public ScannerInstanceResource(String table, String id, ResultGenerator generator, int batch) throws IOException { this.id = id; this.generator = generator; this.batch = batch; + servlet = RESTServlet.getInstance(); cacheControl = new CacheControl(); cacheControl.setNoCache(true); cacheControl.setNoTransform(false); @@ -68,6 +70,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); CellSetModel model = new CellSetModel(); RowModel rowModel = null; byte[] rowKey = null; @@ -115,6 +118,7 @@ LOG.debug("GET " + uriInfo.getAbsolutePath() + " as " + MIMETYPE_BINARY); } + servlet.getMetrics().incrementRequests(1); try { KeyValue value = generator.next(); if (value == null) { @@ -140,6 +144,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("DELETE " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); ScannerResource.delete(id); return Response.ok().build(); } Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/ScannerResource.java Sat Feb 27 00:29:27 2010 @@ -54,8 +54,9 @@ User user; String tableName; String actualTableName; + RESTServlet servlet; - public ScannerResource(User user, String table) { + public ScannerResource(User user, String table) throws IOException { if (user != null) { this.user = user; this.actualTableName = @@ -64,14 +65,24 @@ this.actualTableName = table; } this.tableName = table; + servlet = RESTServlet.getInstance(); } - private Response update(ScannerModel model, boolean replace, - UriInfo uriInfo) { + static void delete(String id) { + synchronized (scanners) { + ScannerInstanceResource instance = scanners.remove(id); + if (instance != null) { + instance.generator.close(); + } + } + } + + Response update(ScannerModel model, boolean replace, UriInfo uriInfo) { + servlet.getMetrics().incrementRequests(1); + byte[] endRow = model.hasEndRow() ? model.getEndRow() : null; + RowSpec spec = new RowSpec(model.getStartRow(), endRow, + model.getColumns(), model.getStartTime(), model.getEndTime(), 1); try { - byte[] endRow = model.hasEndRow() ? model.getEndRow() : null; - RowSpec spec = new RowSpec(model.getStartRow(), endRow, - model.getColumns(), model.getStartTime(), model.getEndTime(), 1); ScannerResultGenerator gen = new ScannerResultGenerator(actualTableName, spec); String id = gen.getID(); ScannerInstanceResource instance = @@ -123,12 +134,4 @@ } } - static void delete(String id) { - synchronized (scanners) { - ScannerInstanceResource instance = scanners.remove(id); - if (instance != null) { - instance.generator.close(); - } - } - } } Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/SchemaResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/SchemaResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/SchemaResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/SchemaResource.java Sat Feb 27 00:29:27 2010 @@ -70,10 +70,10 @@ this.actualTableName = table; } this.tableName = table; + servlet = RESTServlet.getInstance(); cacheControl = new CacheControl(); cacheControl.setNoCache(true); cacheControl.setNoTransform(false); - servlet = RESTServlet.getInstance(); } private HTableDescriptor getTableSchema() throws IOException, @@ -93,6 +93,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); try { HTableDescriptor htd = getTableSchema(); TableSchemaModel model = new TableSchemaModel(); @@ -207,6 +208,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("PUT " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); // use the name given in the path, but warn if the name on the path and // the name in the schema are different if (model.getName() != tableName) { @@ -222,6 +224,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("PUT " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); // use the name given in the path, but warn if the name on the path and // the name in the schema are different if (model.getName() != tableName) { @@ -236,6 +239,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("DELETE " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); try { HBaseAdmin admin = new HBaseAdmin(servlet.getConfiguration()); admin.disableTable(actualTableName); Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterStatusResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterStatusResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterStatusResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterStatusResource.java Sat Feb 27 00:29:27 2010 @@ -48,10 +48,10 @@ private RESTServlet servlet; public StorageClusterStatusResource() throws IOException { + servlet = RESTServlet.getInstance(); cacheControl = new CacheControl(); cacheControl.setNoCache(true); cacheControl.setNoTransform(false); - servlet = RESTServlet.getInstance(); } @GET @@ -60,6 +60,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); try { HBaseAdmin admin = new HBaseAdmin(servlet.getConfiguration()); ClusterStatus status = admin.getClusterStatus(); Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterVersionResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterVersionResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterVersionResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/StorageClusterVersionResource.java Sat Feb 27 00:29:27 2010 @@ -42,8 +42,10 @@ LogFactory.getLog(StorageClusterVersionResource.class); private CacheControl cacheControl; + private RESTServlet servlet; - public StorageClusterVersionResource() { + public StorageClusterVersionResource() throws IOException { + servlet = RESTServlet.getInstance(); cacheControl = new CacheControl(); cacheControl.setNoCache(true); cacheControl.setNoTransform(false); @@ -55,10 +57,10 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); + Configuration conf = servlet.getConfiguration(); try { - RESTServlet server = RESTServlet.getInstance(); - Configuration hconf = server.getConfiguration(); - HBaseAdmin admin = new HBaseAdmin(hconf); + HBaseAdmin admin = new HBaseAdmin(conf); StorageClusterVersionModel model = new StorageClusterVersionModel(); model.setVersion(admin.getClusterStatus().getHBaseVersion()); ResponseBuilder response = Response.ok(model); Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/TableResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/TableResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/TableResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/TableResource.java Sat Feb 27 00:29:27 2010 @@ -46,7 +46,7 @@ } @Path("scanner") - public ScannerResource getScannerResource() { + public ScannerResource getScannerResource() throws IOException { return new ScannerResource(user, table); } Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/VersionResource.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/VersionResource.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/VersionResource.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/VersionResource.java Sat Feb 27 00:29:27 2010 @@ -20,6 +20,8 @@ package org.apache.hadoop.hbase.stargate; +import java.io.IOException; + import javax.servlet.ServletContext; import javax.ws.rs.GET; import javax.ws.rs.Path; @@ -45,8 +47,10 @@ private static final Log LOG = LogFactory.getLog(VersionResource.class); private CacheControl cacheControl; + private RESTServlet servlet; - public VersionResource() { + public VersionResource() throws IOException { + servlet = RESTServlet.getInstance(); cacheControl = new CacheControl(); cacheControl.setNoCache(true); cacheControl.setNoTransform(false); @@ -65,6 +69,7 @@ if (LOG.isDebugEnabled()) { LOG.debug("GET " + uriInfo.getAbsolutePath()); } + servlet.getMetrics().incrementRequests(1); ResponseBuilder response = Response.ok(new VersionModel(context)); response.cacheControl(cacheControl); return response.build(); @@ -74,7 +79,8 @@ * Dispatch to StorageClusterVersionResource */ @Path("cluster") - public StorageClusterVersionResource getClusterVersionResource() { + public StorageClusterVersionResource getClusterVersionResource() + throws IOException { return new StorageClusterVersionResource(); } Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/JDBCAuthenticator.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/JDBCAuthenticator.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/JDBCAuthenticator.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/JDBCAuthenticator.java Sat Feb 27 00:29:27 2010 @@ -7,6 +7,7 @@ import java.sql.ResultSet; import java.sql.SQLException; +import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.util.StringUtils; import org.mortbay.log.Log; @@ -24,12 +25,13 @@ /** * Constructor - * @param url - * @param user - * @param password + * @param conf */ - public JDBCAuthenticator(String url, String user, String password) { - this(url, "users", user, password); + public JDBCAuthenticator(HBaseConfiguration conf) { + this(conf.get("stargate.auth.jdbc.url"), + conf.get("stargate.auth.jdbc.table"), + conf.get("stargate.auth.jdbc.user"), + conf.get("stargate.auth.jdbc.password")); } /** Modified: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/ZooKeeperAuthenticator.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/ZooKeeperAuthenticator.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/ZooKeeperAuthenticator.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/auth/ZooKeeperAuthenticator.java Sat Feb 27 00:29:27 2010 @@ -23,6 +23,7 @@ import java.io.IOException; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.stargate.Constants; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper; import org.apache.zookeeper.CreateMode; @@ -45,10 +46,10 @@ * 'name' (String, required), 'token' (String, optional), 'admin' (boolean, * optional), and 'disabled' (boolean, optional). */ -public class ZooKeeperAuthenticator extends Authenticator { - - static final String USERS_ROOT_ZNODE = "/stargate/users"; +public class ZooKeeperAuthenticator extends Authenticator + implements Constants { + final String usersZNode; ZooKeeperWrapper wrapper; private boolean ensureParentExists(final String znode) { @@ -85,17 +86,20 @@ * @throws IOException */ public ZooKeeperAuthenticator(Configuration conf) throws IOException { - this(new ZooKeeperWrapper(conf, new Watcher() { + this(conf, new ZooKeeperWrapper(conf, new Watcher() { public void process(WatchedEvent event) { } })); - ensureExists(USERS_ROOT_ZNODE); + ensureExists(USERS_ZNODE_ROOT); } /** * Constructor + * @param conf * @param wrapper */ - public ZooKeeperAuthenticator(ZooKeeperWrapper wrapper) { + public ZooKeeperAuthenticator(Configuration conf, + ZooKeeperWrapper wrapper) { + this.usersZNode = conf.get("stargate.auth.zk.users", USERS_ZNODE_ROOT); this.wrapper = wrapper; } @@ -103,7 +107,7 @@ public User getUserForToken(String token) throws IOException { ZooKeeper zk = wrapper.getZooKeeper(); try { - byte[] data = zk.getData(USERS_ROOT_ZNODE + "/" + token, null, null); + byte[] data = zk.getData(usersZNode + "/" + token, null, null); if (data == null) { return null; } Added: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateMetrics.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateMetrics.java?rev=916882&view=auto ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateMetrics.java (added) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateMetrics.java Sat Feb 27 00:29:27 2010 @@ -0,0 +1,87 @@ +/* + * Copyright 2010 The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.stargate.metrics; + +import org.apache.hadoop.hbase.metrics.MetricsRate; + +import org.apache.hadoop.metrics.MetricsContext; +import org.apache.hadoop.metrics.MetricsRecord; +import org.apache.hadoop.metrics.MetricsUtil; +import org.apache.hadoop.metrics.Updater; +import org.apache.hadoop.metrics.jvm.JvmMetrics; +import org.apache.hadoop.metrics.util.MetricsRegistry; + +public class StargateMetrics implements Updater { + private final MetricsRecord metricsRecord; + private final MetricsRegistry registry = new MetricsRegistry(); + private final StargateStatistics stargateStatistics; + + private MetricsRate requests = new MetricsRate("requests", registry); + + public StargateMetrics() { + MetricsContext context = MetricsUtil.getContext("stargate"); + metricsRecord = MetricsUtil.createRecord(context, "stargate"); + String name = Thread.currentThread().getName(); + metricsRecord.setTag("Master", name); + context.registerUpdater(this); + JvmMetrics.init("Stargate", name); + // expose the MBean for metrics + stargateStatistics = new StargateStatistics(registry); + + } + + public void shutdown() { + if (stargateStatistics != null) { + stargateStatistics.shutdown(); + } + } + + /** + * Since this object is a registered updater, this method will be called + * periodically, e.g. every 5 seconds. + * @param unused + */ + public void doUpdates(MetricsContext unused) { + synchronized (this) { + requests.pushMetric(metricsRecord); + } + this.metricsRecord.update(); + } + + public void resetAllMinMax() { + // Nothing to do + } + + /** + * @return Count of requests. + */ + public float getRequests() { + return requests.getPreviousIntervalValue(); + } + + /** + * @param inc How much to add to requests. + */ + public void incrementRequests(final int inc) { + requests.inc(inc); + } + +} Added: hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateStatistics.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateStatistics.java?rev=916882&view=auto ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateStatistics.java (added) +++ hadoop/hbase/trunk/contrib/stargate/src/main/java/org/apache/hadoop/hbase/stargate/metrics/StargateStatistics.java Sat Feb 27 00:29:27 2010 @@ -0,0 +1,45 @@ +/* + * Copyright 2010 The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hbase.stargate.metrics; + +import javax.management.ObjectName; + +import org.apache.hadoop.hbase.metrics.MetricsMBeanBase; + +import org.apache.hadoop.metrics.util.MBeanUtil; +import org.apache.hadoop.metrics.util.MetricsRegistry; + +public class StargateStatistics extends MetricsMBeanBase { + private final ObjectName mbeanName; + + public StargateStatistics(MetricsRegistry registry) { + super(registry, "StargateStatistics"); + mbeanName = MBeanUtil.registerMBean("Stargate", + "StargateStatistics", this); + } + + public void shutdown() { + if (mbeanName != null) { + MBeanUtil.unregisterMBean(mbeanName); + } + } + +} Modified: hadoop/hbase/trunk/contrib/stargate/src/test/java/org/apache/hadoop/hbase/stargate/auth/TestZooKeeperAuthenticator.java URL: http://svn.apache.org/viewvc/hadoop/hbase/trunk/contrib/stargate/src/test/java/org/apache/hadoop/hbase/stargate/auth/TestZooKeeperAuthenticator.java?rev=916882&r1=916881&r2=916882&view=diff ============================================================================== --- hadoop/hbase/trunk/contrib/stargate/src/test/java/org/apache/hadoop/hbase/stargate/auth/TestZooKeeperAuthenticator.java (original) +++ hadoop/hbase/trunk/contrib/stargate/src/test/java/org/apache/hadoop/hbase/stargate/auth/TestZooKeeperAuthenticator.java Sat Feb 27 00:29:27 2010 @@ -45,9 +45,9 @@ public void setUp() throws Exception { authenticator = new ZooKeeperAuthenticator(conf); ZooKeeper zk = authenticator.wrapper.getZooKeeper(); - if (zk.exists(ZooKeeperAuthenticator.USERS_ROOT_ZNODE + "/" + + if (zk.exists(ZooKeeperAuthenticator.USERS_ZNODE_ROOT + "/" + ADMIN_TOKEN, null) == null) { - zk.create(ZooKeeperAuthenticator.USERS_ROOT_ZNODE + "/" + ADMIN_TOKEN, + zk.create(ZooKeeperAuthenticator.USERS_ZNODE_ROOT + "/" + ADMIN_TOKEN, Bytes.toBytes(new JSONStringer() .object() .key("name").value(ADMIN_USERNAME) @@ -55,9 +55,9 @@ .endObject().toString()), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); } - if (zk.exists(ZooKeeperAuthenticator.USERS_ROOT_ZNODE + "/" + + if (zk.exists(ZooKeeperAuthenticator.USERS_ZNODE_ROOT + "/" + USER_TOKEN, null) == null) { - zk.create(ZooKeeperAuthenticator.USERS_ROOT_ZNODE + "/" + USER_TOKEN, + zk.create(ZooKeeperAuthenticator.USERS_ZNODE_ROOT + "/" + USER_TOKEN, Bytes.toBytes(new JSONStringer() .object() .key("name").value(USER_USERNAME) @@ -65,9 +65,9 @@ .endObject().toString()), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); } - if (zk.exists(ZooKeeperAuthenticator.USERS_ROOT_ZNODE + "/" + + if (zk.exists(ZooKeeperAuthenticator.USERS_ZNODE_ROOT + "/" + DISABLED_TOKEN, null) == null) { - zk.create(ZooKeeperAuthenticator.USERS_ROOT_ZNODE + "/" +DISABLED_TOKEN, + zk.create(ZooKeeperAuthenticator.USERS_ZNODE_ROOT + "/" +DISABLED_TOKEN, Bytes.toBytes(new JSONStringer() .object() .key("name").value(DISABLED_USERNAME)