Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8759C200D61 for ; Tue, 19 Dec 2017 22:31:03 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 85C81160C1B; Tue, 19 Dec 2017 21:31:03 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0D8E5160BFF for ; Tue, 19 Dec 2017 22:31:00 +0100 (CET) Received: (qmail 21874 invoked by uid 500); 19 Dec 2017 21:30:55 -0000 Mailing-List: contact announce-help@apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list announce@apache.org Delivered-To: moderator for announce@apache.org Received: (qmail 71987 invoked by uid 99); 19 Dec 2017 19:33:21 -0000 X-Gm-Message-State: AKGB3mInzCSwxEdgH3sixDtMf0ShN+2UN0vjVhLh581RbaMuJWC8jC4L qcmBNtwoVEiDWzOzRcV4hqyqj/RYeY6LDrKDLVg= X-Google-Smtp-Source: ACJfBouB9E6owLqejh6JkX0uIReG+uKLWQa4PRJRIozP+Xrfi7lQhJSoZ0vzdliNgCFQzVVylG24Kgfd5VoUsd2koZA= X-Received: by 10.202.48.195 with SMTP id w186mr2881677oiw.341.1513711996927; Tue, 19 Dec 2017 11:33:16 -0800 (PST) MIME-Version: 1.0 From: Andrew Purtell Date: Tue, 19 Dec 2017 11:32:36 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: [ANNOUNCE] Apache HBase 1.4.0 is now available for download To: announce@apache.org, "dev@hbase.apache.org" , "user@hbase.apache.org" Content-Type: multipart/alternative; boundary="001a113ce526de84520560b68979" archived-at: Tue, 19 Dec 2017 21:31:03 -0000 --001a113ce526de84520560b68979 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The HBase team is happy to announce the immediate availability of Apache =E2=80=8BHBase=E2=80=8B 1.4.0! Apache HBase is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn more about HBase, see https://hbase.apache.org/. Download through an ASF mirror: https://www.apache.org/dyn/closer.lua/hbase/1.4.0 HBase 1.4.0 is the first release in the new HBase 1.4 line, continuing on the theme of earlier 1.x releases of bringing a stable, reliable database to the Apache Big Data ecosystem and beyond. As a minor release, 1.4.0 contains a number of new features and improvements that won't appear in maintenance releases of older code lines. However complete compatibility with data formats and interoperability with older clients is assured. There are no special considerations for upgrade or rollback except as noted in this release announcement. Maintenance releases of the 1.4.0 code line will occur at roughly a monthly cadence. For instructions on verifying ASF release downloads, please see https://www.apache.org/dyn/closer.cgi#verify Project member signature keys can be found at https://www.apache.org/dist/hbase/KEYS Thanks to all the contributors who made this release possible! The complete list of the 660 issues resolved in this release can be found at https://s.apache.org/OErT . New developer and user-facing incompatibilities, important issues, features, and major improvements include: Critical - HBASE-15484 Correct the semantic of batch and partial Now setBatch doesn't mean setAllowPartialResult(true) Scan#setBatch is helpful in paging queries, if you just want to prevent OOM at client, use setAllowPartialResults(true) is better. We deprecated isPartial and use mayHaveMoreCellsInRow. If it returns false, current Result must be the last one of this row. - HBASE-17287 Master becomes a zombie if filesystem object closes If filesystem is not available during log split, abort master server. - HBASE-17471 Region Seqid will be out of order in WAL if using mvccPreAssign MVCCPreAssign is added by HBASE-16698, but pre-assign mvcc is only used in put/delete path. Other write paths like increment/append still assig= n mvcc in ringbuffer's consumer thread. If put and increment are used parallel. Then seqid in WAL may not increase monotonically. Disorder in wals will lead to data loss. This patch bring all mvcc/seqid event in wal.append, and synchronize wal append and mvcc acquirement. No disorde= r in wal will happen. Performance test shows no regression. - HBASE-17595 Add partial result support for small/limited scan Now small scan and limited scan can also return partial results. - HBASE-17717 Incorrect ZK ACL set for HBase superuser In previous versions of HBase, the system intended to set a ZooKeeper ACL on all "sensitive" ZNodes for the user specified in the hbase.superuser configuration property. Unfortunately, the ACL was malformed which resulted in the hbase.superuser being unable to access the sensitive ZNodes that HBase creates. HBase will automatically correct the ACLs on start so users do not need to manually correct the ACLs. - HBASE-17887 Row-level consistency is broken for read Now we pass on list of memstoreScanners to the StoreScanner along with the new files to ensure that the StoreScanner sees the latest memstore after flush. - HBASE-17931 Assign system tables to servers with highest version We usually keep compatibility between old client and new server so we can do rolling upgrade, HBase cluster first, then HBase client. But we don't guarantee new client can access old server. In an HBase cluster, we have system tables and region servers will access these tables so fo= r servers they are also an HBase client. So if the system tables are in region servers with lower version we may get trouble because region servers with higher version may can not access them. After this patch, we will move all system regions to region servers with highest version. So when we do a rolling upgrade across two major or minor versions, we should ALWAYS UPGRADE MASTER FIRST and then upgrade region =E2=80=8B =E2=80=8B servers. =E2=80=8B =E2=80=8B The new master will handle system tables correctly. - HBASE-18035 Meta replica does not give any primaryOperationTimeout to primary meta region When a client is configured to use meta replica, it sends scan request to all meta replicas almost at the same time. Since meta replica contains stale data, if result from one of replica comes back first, th= e client may get wrong region locations. To fix this, "hbase.client.meta.replica.scan.timeout" is introduced, a client will always send to primary meta region first, wait the configured timeout for reply. If no result is received, it will send request to replica meta regions. The unit for "hbase.client.meta.replica.scan.timeout" is microsecond, the default value is 1000000 (1 second). - HBASE-18137 Replication gets stuck for empty WALs 0-length WAL files can potentially cause the replication queue to get stuck. A new config "replication.source.eof.autorecovery" has been added: if set to true (default is false), the 0-length WAL file will be skipped after 1) the max number of retries has been hit, and 2) there are more WAL files in the queue. The risk of enabling this is that there is a chance the 0-length WAL file actually has some data (e.g. block went missing and will come back once a datanode is recovered). - HBASE-18164 Much faster locality cost function and candidate generator New locality cost function and candidate generator that use caching and incremental computation to allow the stochastic load balancer to consider ~20x more cluster configurations for big clusters. - HBASE-18192 Replication drops recovered queues on region server shutdown If a region server that is processing recovered queue for another previously dead region server is gracefully shut down, it can drop the recovered queue under certain conditions. Running without this fix on a 1.2+ release means possibility of continuing data loss in replication, irrespective of which WALProvider is used. If a single WAL group (or DefaultWALProvider) is used, running without this fix will always cause dataloss in replication whenever a region server processing recovered queues is gracefully shutdown. - HBASE-18233 We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock This patch plus the sort of mutations done in HBASE-17924 fixes a performance regression doing increments and/or checkAndPut-style operations. - HBASE-18255 Time-Delayed HBase Performance Degradation with Java 7 This change sets the JVM property ReservedCodeCacheSize to 256MB in the provided hbase-env.sh example file. The specific value for this propert= y attempts to prevent performance issues seen when HBase using Java 7. Th= e value set is the same as the default when using Java 8. - HBASE-18469 Correct RegionServer metric of totalRequestCount We introduced a new RegionServer metrics in name of "totalRowActionRequestCount" which counts in all row actions and equals to the sum of "readRequestCount" and "writeRequestCount". Meantime, we have changed "totalRequestCount" to count only once for multi request, while previously we will count in action number of the request. As a result, existing monitoring system on totalRequestCount will still work but see a smaller value, and we strongly recommend to change to use the new metrics to monitor server load. - HBASE-18577 shaded client includes several non-relocated third party dependencies The HBase shaded artifacts (hbase-shaded-client and hbase-shaded-server= ) no longer contain several non-relocated third party dependency classes that were mistakenly included. Downstream users who relied on these classes being present will need to add a runtime dependency onto an appropriate third party artifact. Previously, we erroneously packaged several third party libs without relocating them. In some cases these libraries have now been relocated; in some cases they are no longer included at all. Includes: * jaxb * jetty * jersey * codahale metrics (HBase 1.4+ only) * commons-crypto * jets3t * junit * curator (HBase 1.4+) * netty 3 (HBase 1.1) * mokito-junit4 (HBase 1.1) - HBASE-18665 ReversedScannerCallable invokes getRegionLocations incorrectly Performing reverse scan on tables used the meta cache incorrectly and fetched data from meta table every time. This fix solves this issue and which results in performance improvement for reverse scans. - HBASE-19285 Add per-table latency histograms Per-RegionServer table latency histograms have been returned to HBase (after being removed due to impacting performance). These metrics are exposed via a new JMX bean "TableLatencies" with the typical naming conventions: namespace, table, and histogram component. Major - HBASE-7621 REST client doesn't support binary row keys RemoteHTable now supports binary row keys with any character or byte by properly encoding request URLs. This is a both a behavioral change from earlier versions and an important fix for protocol correctness. - HBASE-11013 Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions While creating a snapshot, it will save permissions of the original table into .snapshotinfo file(Backward compatibility), which is in the snapshot root directory. For clone_snapshot/restore_snapshot command, we provide an additional option (RESTORE_ACL) to decide whether we will grant permissions of the origin table to the newly created table. - HBASE-14548 Expand how table coprocessor jar and dependency path can be specified Allows a directory containing the jars or some wildcards to be specified, such as: hdfs://namenode:port/user/hadoop-user/ or hdfs://namenode:port/user/hadoop-user/*.jar Please note that if a directory is specified, all jar files directly in the directory are added, but it does not search files in the subtree rooted in the directory. Do not use a wildcard if you would like to specify a directory. - HBASE-14925 HBase shell command/tool to list table's region info through command line Added a shell command 'list_regions' for displaying the table's region info through command line. List all regions for a particular table as a= n array and also filter them by server name (optional) as prefix and maximum locality (optional). By default, it will return all the regions for the table with any locality. The command displays server name, region name, start key, end key, size of the region in MB, number of requests and the locality. The information can be projected out via an array as third parameter. By default all these information is displayed= . Possible array values are SERVER_NAME, REGION_NAME, START_KEY, END_KEY, SIZE, REQ and LOCALITY. Values are not case sensitive. If you don't want to filter by server name, pass an empty hash or string. - HBASE-15187 Integrate CSRF prevention filter to REST gateway Protection against CSRF attack can be turned on with config parameter, hbase.rest.csrf.enabled - default value is false. The custom header to be sent can be changed via config parameter, hbase.rest.csrf.custom.header whose default value is "X-XSRF-HEADER". The configuration parameter hbase.rest.csrf.methods.to.ignore controls which HTTP methods are not associated with custom header check. The config parameter hbase.rest-csrf.browser-useragents-regex is a comma-separated list of regular expressions used to match against an HTTP request's User-Agent header when protection against cross-site request forgery (CSRF) is enabled for REST server by setting hbase.rest.csrf.enabled to true. - HBASE-15236 Inconsistent cell reads over multiple bulk-loaded HFiles During bulkloading, if there are multiple hfiles corresponding to the same region, and if they have same timestamps (which may have been set using importtsv.timestamp) and duplicate keys across them, then get and scan may return values coming from different hfiles. - HBASE-15243 Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT When all filters in a MUST_PASS_ONE FilterList return a SEEK_USING_NEXT_HINT code, we return SEEK_NEXT_USING_HINT from FilterList#filterKeyValue() to utilize the lowest seek value. - HBASE-15386 PREFETCH_BLOCKS_ON_OPEN in HColumnDescriptor is ignored Changes the prefetch TRACE-level loggings to include the word 'Prefetch' in them so you know what they are about. Changes the cryptic logging of the CacheConfig#toString to have some preamble saying why and what column family is responsible. - HBASE-15576 Scanning cursor to prevent blocking long time on ResultScanner.next() If you don't like scanning being blocked too long because of heartbeat and partial result, you can use Scan#setNeedCursorResult(true) to get a special result within scanning timeout setting time which will tell you where row the server is scanning. See its javadoc for more details. - HBASE-15633 Backport HBASE-15507 to branch-1 Adds update_peer_config to the HBase shell and ReplicationAdmin, and provides a callback for custom replication endpoints to be notified of changes to configuration and peer data - HBASE-15686 Add override mechanism for the exempt classes when dynamically loading table coprocessor The =E2=80=8Bn=E2=80=8B ew coprocessor table descriptor attribute, hbase.coprocessor.classloader.included.classes, is added. A user can specify class name prefixes (semicolon separated) which should be loaded by CoprocessorClassLoader. - HBASE-15711 Add client side property to allow logging details for batch errors A new client side property hbase.client.log.batcherrors.details is introduced to allow logging the full stacktrace of exceptions for batch errors. It is disabled by default. - HBASE-15816 Provide client with ability to set priority on Operations Added setPriority(int priority) API to Put, Delete, Increment, Append, Get and Scan. For all these ops, the user can provide a custom RPC priority level. - HBASE-15924 Enhance hbase services autorestart capability to hbase-daemon.sh Now one can start hbase services with enabled "autostart/autorestart" feature in controlled fashion with the help of "--autostart-window-size= " to define the window period and the "--autostart-window-retry-limit" to define the number of times the hbase services have to be restarted upon being killed/terminated abnormally within the provided window perioid. The following cases are supported with "autostart/autorestart": a) --autostart-window-size=3D0 and --autostart-window-retry-limit=3D0, indicates infinite window size and no retry limit b) not providing the args, will default to a) c) --autostart-window-size=3D0 and --autostart-window-retry-limit=3D indicates the autostart process to bail out if the retry limit exceeds irrespective of window period d) --autostart-window-size=3D and --autostart-window-retry-limit=3D<= y> indicates the autostart process to bail out if the retry limit "y" is exceeded for the last window period "x". - HBASE-15941 HBCK repair should not unsplit healthy splitted region A new option -removeParents is now available that will remove an old parent when two valid daughters for that parent exist and -fixHdfsOverlaps is used. If there is an issue trying to remove the parent from META or sidelining the parent from HDFS we will fallback to do a regular merge. For now this option only works when the overlap group consists only of 3 regions (a parent, daughter A and daughter B) - HBASE-15950 Fix memstore size estimates to be tighter The estimates of heap usage by the memstore objects (KeyValue, object and array header sizes, etc) have been made more accurate for heap sizes up to 32G (using CompressedOops), resulting in them dropping by 10-50% in practice. This also results in less number of flushes and compactions due to "fatter" flushes. As a result, the actual heap usage of the memstore before being flushed may increase by up to 100%. If configured memory limits for the region server had been tuned based on observed usage, this change could result in worse GC behavior or even OutOfMemory errors. Set the environment property (not hbase-site.xml) "hbase.memorylayout.use.unsafe" to false to disable. - HBASE-15994 Allow selection of RpcSchedulers Adds a FifoRpcSchedulerFactory so you can try the FifoRpcScheduler by setting "hbase.region.server.rpc.scheduler.factory.class" - HBASE-16052 Improve HBaseFsck Scalability Improves the performance and scalability of HBaseFsck, especially for large clusters with a small number of large tables. Searching for lingering reference files is now a multi-threaded operation. Loading HDFS region directory information is now multi- threaded at the region-level instead of the table-level to maximize concurrency. A performance bug in HBaseFsck that resulted in redundant I/O and RPCs was fixed by introducing a FileStatusFilter that filters FileStatus objects directly. - HBASE-16213 A new HFileBlock structure for fast random get Introduces a new DataBlockEncoding in name of ROW_INDEX_V1, which could improve random read (get) performance especially when the average record size (key-value size per row) is small. To use this feature, please set DATA_BLOCK_ENCODING to ROW_INDEX_V1 for column family of newly created table, or change existing CF with below shell command: alter 'table_name',{NAME =3D> 'cf', DATA_BLOCK_ENCODING =3D> 'ROW_INDEX_V1'}. Please note that if we turn this DBE on, HFile block will be bigger than NONE encoding because it adds some metadata for binary search. Seek in row when random reading is one of the main consumers of CPU. This helps. - HBASE-16244 LocalHBaseCluster start timeout should be configurable When LocalHBaseCluster is started from the command line the Master would give up after 30 seconds due to a hardcoded timeout meant for unit tests. This change allows the timeout to be configured via hbase-site as well as sets it to 5 minutes when LocalHBaseCluster is started from the command line. - HBASE-16336 Removing peers seems to be leaving spare queues Add a ReplicationZKNodeCleaner periodic check and delete any useless replication queue belonging to a peer which does not exist. - HBASE-16388 Prevent client threads being blocked by only one slow region server Adds a new configuration, hbase.client.perserver.requests.threshold, to limit the max number of concurrent request to one region server. If the user still create new request after reaching the limit, client will throw ServerTooBusyException and do not send the request to the server. This is a client side feature and can prevent client's threads being blocked by one slow region server resulting in the availability of client is much lower than the availability of region servers. - HBASE-16540 Scan should do additional validation on start and stop row Scan#setStartRow() and Scan#setStopRow() now validate the argument passed for each row key. If the length of the parameter passed exceeds Short.MAX_VALUE, an IllegalArgumentException will be thrown. - HBASE-16584 Backport the new ipc implementation in HBASE-16432 to branch-1 The netty dependency is upgraded to 4.1.1.Final. And also some configurations of the old AsyncRpcClient is gone. Such as "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". - HBASE-16653 Backport HBASE-11393 to all branches which support namespace During HBASE-11393, we have done two things: 1. unify tableCFs with peerConfig 2. Fix ns not support issue for replication. This issue is to backport it to branch-1 How to rolling update if the replication peer have old table-cfs config? Due to we modify proto object of ReplicationPeerConfig (add tableCFs field), so when we do rolling update, we have to update original ReplicationPeerConfig data on ZK firstly. 1. Make sure the master have the permission to modify replication peer znode. 2. Disable the replication peer. 3. Rolling update master first. The master will copy the table-cfs config from old table-cfs znode and add it to the new proto object of ReplicationPeerConfig. 4. Rolling update regionservers. 5. Enable the replication peer. If you can't change the replication peer znode permission, you can use the TableCFsUpdater tool to copy the table-cfs config. 1. Disable the replication peer. 2. bin/hbase org.apache.hadoop.hbase.replication.master.TableCFsUpdater update 3. Rolling update master and regionservers. 4. Enable the replication peer. - HBASE-16672 Add option for bulk load to always copy hfile(s) instead of renaming This issue adds a config, always.copy.files, to LoadIncrementalHFiles. When set to true, source hfiles would be copied. Meaning source hfiles would be kept after bulk load is done. Default value is false. - HBASE-16698] Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload Assign sequenceid to an edit before we go on the ringbuffer; undoes contention on WALKey latch. Adds a new config "hbase.hregion.mvcc.preassign" which defaults to true: i.e. this speedu= p is enabled. - HBASE-16755 Honor flush policy under global memstore pressure Prior to this change, when the memstore low water mark is exceeded on a regionserver, the regionserver will force flush all stores on the regions selected for flushing until we drop below the low water mark. With this change, the regionserver will continue to force flush regions when above the memstore low water mark, but will only flush the stores returned by the configured FlushPolicy. - HBASE-16993 BucketCache throw java.io.IOException: Invalid HFile block magic when configuring hbase.bucketcache.bucket.sizes Any value for hbase.bucketcache.bucket.sizes configuration must be a multiple of 256. If that is not the case, instantiation of L2 Bucket cache itself will fail throwing IllegalArgumentException. - HBASE-17112 Prevent setting timestamp of delta operations the same as previous value Before this issue, two concurrent Increments/Appends done in same millisecond or RS's clock going back will result in two results having same TS, which is not friendly to versioning and will get wrong result in sink cluster if the replication is disordered. After this issue, the result of Increment/Append will always have an incremental TS. There is no longer any inconsistency in replication for these operations. - HBASE-17178 Add region balance throttling Add region balance throttling. Master execute every region balance plan per balance interval, which is equals to divide max balancing time by the size of region balance plan. And Introduce a new config hbase.master.balancer.maxRitPercent to protect availability. If config this to 0.01, then the max percent of regions in transition is 1% when balancing. Then the cluster's availability is at least 99% when balancing. - HBASE-17280 Add mechanism to control cleaner chore behavior The HBase cleaner chore process cleans up old WAL files and archived HFiles. Cleaner operation can affect query performance when running heavy workloads, so disable the cleaner during peak hours. The cleaner has the following HBase shell commands: - cleaner_chore_enabled: Queries whether cleaner chore is enabled/ disabled. - cleaner_chore_run: Manually runs the cleaner to remove files. - cleaner_chore_switch: enables or disables the cleaner and returns the previous state of the cleaner. For example, cleaner-switch true enables the cleaner. Following APIs are added in Admin: - setCleanerChoreRunning(boolean on): Enable/Disable the cleaner chore - runCleanerChore(): Ask for cleaner chore to run - isCleanerChoreEnabled(): Query whether cleaner chore is enabled/ disabled. - HBASE-17296 Provide per peer throttling for replication Provide per peer throttling for replication. Add the bandwidth upper limit to ReplicationPeerConfig and a new shell cmd set_peer_bandwidth to update the bandwidth as needed. - HBASE-17426 Inconsistent environment variable names for enabling JMX In bin/hbase-config.sh, if value for HBASE_JMX_BASE is empty, keep current behavior. If HBASE_JMX_OPTS is not empty, keep current behavior. Otherwise use the value of HBASE_JMX_BASE - HBASE-17437 Support specifying a WAL directory outside of the root directory This patch adds support for specifying a WAL directory outside of the HBase root directory. Multiple configuration variables were added to accomplish this: hbase.wal.dir: used to configure where the root WAL directory is located. Could be on a different FileSystem than the root directory. WA= L directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset. hbase.rootdir.perms: Configures FileSystem permissions to set on the root directory. This is '700' by default. hbase.wal.dir.perms: Configures FileSystem permissions to set on the WA= L directory FileSystem. This is '700' by default. - HBASE-17472 Correct the semantic of permission grant Before this patch, later granted permissions will override previous granted permissions, and previous granted permissions will be lost. Thi= s issue re-defines the grant semantic: for master branch, later granted permissions will merge with previous granted permissions. For branch-1.4, grant keep override behavior for compatibility purpose, and a grant with mergeExistingPermission flag provided. - HBASE-17508 Unify the implementation of small scan and regular scan for sync client Now the scan.setSmall method is deprecated. Consider using scan.setLimi= t and scan.setReadType in the future. And we will open scanner lazily whe= n you call scanner.next. This is an incompatible change which delays the table existence check and permission check. - HBASE-17578 Thrift per-method metrics should still update in the case of exceptions In prior versions, the HBase Thrift handlers failed to increment per- method metrics when an exception was encountered. These metrics will now always be incremented, whether an exception is encountered or not. This change also adds exception-type metrics, similar to those exposed in regionservers, for individual exceptions which are received by the Thrift handlers. - HBASE-17583 Add inclusive/exclusive support for startRow and endRow of scan for sync client Now you can include or exclude the startRow and stopRow for a scan. The new methods to specify startRow and stopRow are withStartRow and withStopRow. The old methods to specify startRow and Row(include constructors) are marked as deprecated as in the old time if startRow and stopRow are equal then we will consider it as a get scan and includ= e the stopRow implicitly. This is strange after we can set inclusiveness explicitly so we add new methods and depredate the old methods. The deprecated methods will be removed in the future. - HBASE-17584 Expose ScanMetrics with ResultScanner rather than Scan Now you can use ResultScanner.getScanMetrics to get the scan metrics at any time during the scan operation. The old Scan.getScanMetrics is deprecated and still work, but if you use ResultScanner.getScanMetrics to get the scan metrics and reset it, then the metrics published to the Scan instaince will be messed up. - HBASE-17599 Use mayHaveMoreCellsInRow instead of isPartial The word 'isPartial' is ambiguous so we introduce a new method 'mayHaveMoreCellsInRow' to replace it. And the old meaning of 'isPartial' is not the same with 'mayHaveMoreCellsInRow' as for batched scan, if the number of returned cells equals to the batch, isPartial will be false. After this change the meaning of 'isPartial' will be sam= e with 'mayHaveMoreCellsInRow'. This is an incompatible change but it is not likely to break a lot of things as for batched scan the old 'isPartial' is just a redundant information, i.e, if the number of returned cells reaches the batch limit. You have already know the numbe= r of returned cells and the value of batch. - HBASE-17737 Thrift2 proxy should support scan timeRange per column family Thrift2 proxy now supports scan timeRange per column family. - HBASE-17757 Unify blocksize after encoding to decrease memory fragment Blocksize is set in columnfamily's atrributes. It is used to control block sizes when generating blocks. But, it doesn't take encoding into count. If you set encoding to blocks, after encoding, the block size varies. Since blocks will be cached in memory after encoding (default), it will cause memory fragment if using blockcache, or decrease the pool efficiency if using bucketCache. This issue introduced a new config named 'hbase.writer.unified.encoded.blocksize.ratio'. The default value of this config is 1, meaning doing nothing. If this value is set to a smaller value like 0.5, and the blocksize is set to 64KB (default value of blocksize). It will unify the blocksize after encoding to 64KB * 0.5 =3D 32KB. Unified blocksize will relieve the memory problems mentioned above. - HBASE-17817 Make Regionservers log which tables it removed coprocessors from when aborting Adds table name to exception logging when a coprocessor is removed from a table by the region server. - HBASE-17861 Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3 Some object store does not support unix style permission. This fixes th= e permission check issue when specify staging dir in different file system. Currently it covers s3, wasb, swift. - HBASE-17877 Improve HBase's byte[] comparator Updated the lexicographic byte array comparator to use a slightly more optimized version similar to the one available in the guava library tha= t compares only the first index where left[index] !=3D right[index]. The comparator also returns the diff directly instead of mapping it to -1, 0, +1 range as was being done in the earlier version. We have seen significant performance gains, calculated in terms of throughput (ops/ ms) with these changes ranging from approx 20% for smaller byte arrays up to 200 bytes and almost 100% for large byte array sizes that are in few KBs. We benchmarked with up to 16KB arrays and the general trend indicates that the performance improvement increases as the size of the byte array increases. - HBASE-17956 Raw scan should ignore TTL Now raw scan can also read expired cells. - HBASE-18023 Log multi-requests for more than threshold number of rows Introduces a warning message in the RegionServer log when an RPC is received from a client that has more than 5000 "actions" (where an "action" is a collection of mutations for a specific row) in a single RPC. Misbehaving clients who send large RPCs to RegionServers can be malicious, causing temporary pauses via garbage collection or denial of service via crashes. The threshold of 5000 actions per RPC is defined b= y the property "hbase.rpc.rows.warning.threshold" in hbase-site.xml. - HBASE-18090 Improve TableSnapshotInputFormat to allow multiple mappers per region In this task, we make it possible to run multiple mappers per region in the table snapshot. - HBASE-18122 Scanner id should include ServerName of region server The first 32 bits are MurmurHash32 of ServerName string "host,port,ts". The ServerName contains both host, port, and start timestamp so it can prevent collision. The lowest 32bit is generated by atomic int. - HBASE-18149 The setting rules for table-scope attributes and family-scope attributes should keep consistent If the table-scope attributes value is false, you need not to enclose 'false' in single quotation. Both COMPACTION_ENABLED =3D> false and COMPACTION_ENABLED =3D> 'false' will take effect. - HBASE-18226 Disable reverse DNS lookup at HMaster and use the hostname provided by RegionServer The following config is added: =E2=80=8B =E2=80=8B hbase.regionserver.hostname.disable.master.reversedns This config is for experts: don't set its value unless you really know what you are doing. When set to true, regionserver will use the current node hostname for the servername and HMaster will skip reverse DNS lookup and use the hostname sent by regionserver instead. Note that thi= s config and hbase.regionserver.hostname are mutually exclusive. See https://issues.apache.org/jira/browse/HBASE-18226 for more details. Caution: please make sure rolling upgrade succeeds before turning on this feature. - HBASE-18247 Hbck to fix the case that replica region shows as key in the meta table The hbck tool can now correct the meta table should it get an entry for a read replica region. - HBASE-18374 RegionServer Metrics improvements This change adds the latency metrics checkAndPut, checkAndDelete, putBatch and deleteBatch . Also the previous regionserver "mutate" latency metrics are renamed to "put" metrics. Batch metrics capture the latency of the entire batch containing put/delete whereas put/delete metrics capture latency per operation. Note this change will break existing monitoring based on regionserver "mutate" latency metric. - HBASE-18520 Add jmx value to determine true Master Start time Adds a JMX value to track when the Master has finished initializing. The jmx config is 'masterFinishedInitializationTime' and details the time in millis that the Master is fully usable and ready to serve requests. - HBASE-18533 Expose BucketCache values to be configured This patch exposes configuration for Bucketcache. These configs are ver= y similar to those for the LRU cache, but are described below: "hbase.bucketcache.single.factor"; /** Single access bucket size */ "hbase.bucketcache.multi.factor"; /** Multiple access bucket size */ "hbase.bucketcache.memory.factor"; /** In-memory bucket size */ "hbase.bucketcache.extrafreefactor"; /** Free this floating point factor of extra blocks when evicting. For example free the number of blocks requested * (1 + extraFreeFactor) */ "hbase.bucketcache.acceptfactor"; /** Acceptable size of cache (no evictions if size < acceptable) */ "hbase.bucketcache.minfactor"; /** Minimum threshold of cache (when evicting, evict until size < min) */ - HBASE-18675 Making {max,min}SessionTimeout configurable for MiniZooKeeperCluster Standalone clusters and minicluster instances can now configure the session timeout for our embedded ZooKeeper quorum using "hbase.zookeeper.property.minSessionTimeout" and "hbase.zookeeper.property.maxSessionTimeout". - HBASE-18786 FileNotFoundException should not be silently handled for primary region replicas FileNotFoundException opening a StoreFile in a primary replica now causes a RegionServer to crash out where before it would be ignored (or optionally handled via close/reopen). - HBASE-18993 Backport patches in HBASE-18410 to branch-1.x branches This change fixes bugs in FilterList, and also does a code refactor which ensures interface compatibility. The primary bug fixes are : 1. For sub-filter in FilterList with MUST_PASS_ONE, if previous filterKeyValue() of sub-filter returns NEXT_COL, we cannot make sure that the next cell will be the first cell in next column, because FilterList choose the minimal forward step among sub-filters, and it may return a SKIP. so here we add an extra check to ensure that the next cell will match previous return code for sub-filters. 2. Previous logic about transforming cell of FilterList is incorrect, w= e should set the previous transform result (rather than the given cell in question) as the initial value of transform cell before call filterKeyValue() of FilterList. 3. Handle the ReturnCodes which the previous code did not handle. About code refactor, we divided the FilterList into two separated sub- classes: FilterListWithOR and FilterListWithAND, The FilterListWithOR has been optimized to choose the next minimal step to seek cell rather than SKIP cell one by one, and the FilterListWithAND has been optimized to choose the next maximal key to seek among sub-filters in filter list= . All in all, The code in FilterList is clean and easier to follow now. Note that ReturnCode NEXT_ROW has been redefined as skipping to next row in current family, not to next row in all family. it=E2=80=99s more reasonable, because ReturnCode is a concept in store level, not in region level. - HBASE-19035 Miss metrics when coprocessor use region scanner to read data Move read requests count to region level. Because RegionScanner is exposed to CP. Update write requests count in processRowsWithLocks. Remove requestRowActionCount in RSRpcServices. This metric can be computed by region's readRequestsCount and writeRequestsCount. - HBASE-19051 Add new split algorithm for num string Add new split algorithm DecimalStringSplit=EF=BC=8Crow are decimal-enco= ded long values in the range "00000000" =3D> "99999999". - HBASE-19131 Add the ClusterStatus hook and cleanup other hooks which can be replaced by ClusterStatus hook** 1) Add preGetClusterStatus() and postGetClusterStatus() hooks 2) add preGetClusterStatus() to access control check - an admin action - HBASE-19144 Retry assignments in FAILED_OPEN state when servers (re)join the cluster When regionserver placement groups (RSGroups) is active, as servers join the cluster the Master will attempt to reassign regions in FAILED_OPEN state. - HBASE-19419 Remove hbase-native-client from branch-1 Removed the hbase-native-client module from branch-1 (it is still in Master). It is not complete. Look for a finished C++ client in the near future which may be backported to branch-1 at that point. --- Cheers, The HBase Dev Team --001a113ce526de84520560b68979 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The HBase team is happy to announce the immediate availabi= lity of Apache
=E2=80=8BHBase=E2=80=8B 1.4.0!

Apache HBase is an = open-source, distributed, versioned, non-relational
database. Apache HBa= se gives you low latency random access to billions of
rows with millions= of columns atop non-specialized hardware. To learn more
about HBase, se= e https://hbase.apache.org/.
<= br>Download through an ASF mirror:

=C2=A0=C2=A0=C2=A0 https://www.apache.org/dyn/= closer.lua/hbase/1.4.0

HBase 1.4.0 is the first release in the n= ew HBase 1.4 line, continuing on
the theme of earlier 1.x releases of br= inging a stable, reliable database
to the Apache Big Data ecosystem and = beyond. As a minor release, 1.4.0
contains a number of new features and = improvements that won't appear in
maintenance releases of older code= lines. However complete compatibility
with data formats and interoperab= ility with older clients is assured.
There are no special considerations= for upgrade or rollback except as
noted in this release announcement.
Maintenance releases of the 1.4.0 code line will occur at roughly amonthly cadence.

For instructions on verifying ASF release downloa= ds, please see

=C2=A0=C2=A0=C2=A0 https://www.apache.org/dyn/closer.cgi#verify
=
Project member signature keys can be found at

=C2=A0=C2=A0=C2=A0= https://www.apache.org/= dist/hbase/KEYS

Thanks to all the contributors who made this rel= ease possible!

The complete list of the 660 issues resolved in this = release can be found
at https://s.= apache.org/OErT . New developer and user-facing
incompatibilities, i= mportant issues, features, and major improvements
include:

Critic= al

- HBASE-15484 Correct the semantic of batch and partial

= =C2=A0=C2=A0=C2=A0 Now setBatch doesn't mean setAllowPartialResult(true= )

=C2=A0=C2=A0=C2=A0 Scan#setBatch is helpful in paging queries, if = you just want to prevent
=C2=A0=C2=A0=C2=A0 OOM at client, use setAllowP= artialResults(true) is better.

=C2=A0=C2=A0=C2=A0 We deprecated isPa= rtial and use mayHaveMoreCellsInRow. If it returns
=C2=A0=C2=A0=C2=A0 fa= lse, current Result must be the last one of this row.

- HBASE-17287 = Master becomes a zombie if filesystem object closes

=C2=A0=C2=A0=C2= =A0 If filesystem is not available during log split, abort master server.
- HBASE-17471 Region Seqid will be out of order in WAL if using
= =C2=A0 mvccPreAssign

=C2=A0=C2=A0=C2=A0 MVCCPreAssign is added by HB= ASE-16698, but pre-assign mvcc is only used
=C2=A0=C2=A0=C2=A0 in put/de= lete path. Other write paths like increment/append still assign
=C2=A0= =C2=A0=C2=A0 mvcc in ringbuffer's consumer thread. If put and increment= are used
=C2=A0=C2=A0=C2=A0 parallel. Then seqid in WAL may not increas= e monotonically. Disorder in
=C2=A0=C2=A0=C2=A0 wals will lead to data l= oss. This patch bring all mvcc/seqid event in
=C2=A0=C2=A0=C2=A0 wal.app= end, and synchronize wal append and mvcc acquirement. No disorder
=C2=A0= =C2=A0=C2=A0 in wal will happen. Performance test shows no regression.
<= br>- HBASE-17595 Add partial result support for small/limited scan

= =C2=A0=C2=A0=C2=A0 Now small scan and limited scan can also return partial = results.

- HBASE-17717 Incorrect ZK ACL set for HBase superuser
<= br>=C2=A0=C2=A0=C2=A0 In previous versions of HBase, the system intended to= set a ZooKeeper
=C2=A0=C2=A0=C2=A0 ACL on all "sensitive" ZNo= des for the user specified in the
=C2=A0=C2=A0=C2=A0 hbase.superuser con= figuration property. Unfortunately, the ACL was
=C2=A0=C2=A0=C2=A0 malfo= rmed which resulted in the hbase.superuser being unable to access
=C2=A0= =C2=A0=C2=A0 the sensitive ZNodes that HBase creates. HBase will automatica= lly
=C2=A0=C2=A0=C2=A0 correct the ACLs on start so users do not need to= manually correct the
=C2=A0=C2=A0=C2=A0 ACLs.

- HBASE-17887 Row-= level consistency is broken for read

=C2=A0=C2=A0=C2=A0 Now we pass = on list of memstoreScanners to the StoreScanner along with
=C2=A0=C2=A0= =C2=A0 the new files to ensure that the StoreScanner sees the latest memsto= re
=C2=A0=C2=A0=C2=A0 after flush.

- HBASE-17931 Assign system ta= bles to servers with highest version

=C2=A0=C2=A0=C2=A0 We usually k= eep compatibility between old client and new server so we
=C2=A0=C2=A0= =C2=A0 can do rolling upgrade, HBase cluster first, then HBase client. But = we
=C2=A0=C2=A0=C2=A0 don't guarantee new client can access old serv= er. In an HBase cluster,
=C2=A0=C2=A0=C2=A0 we have system tables and re= gion servers will access these tables so for
=C2=A0=C2=A0=C2=A0 servers = they are also an HBase client. So if the system tables are in
=C2=A0=C2= =A0=C2=A0 region servers with lower version we may get trouble because regi= on
=C2=A0=C2=A0=C2=A0 servers with higher version may can not access the= m. After this patch,
=C2=A0=C2=A0=C2=A0 we will move all system regions = to region servers with highest version.
=C2=A0=C2=A0=C2=A0 So when we do= a rolling upgrade across two major or minor versions, we
=C2=A0=C2=A0= =C2=A0 should ALWAYS UPGRADE MASTER FIRST and then upgrade region
=E2=80=8B =E2=80=8B
=C2=A0=C2=A0 servers.<= div style=3D"font-family:arial,helvetica,sans-serif;font-size:small;display= :inline" class=3D"gmail_default">=E2=80=8B =E2=80=8B
The new master wi= ll handle system tables correctly.

- HBASE-18035 Meta replica does n= ot give any primaryOperationTimeout to
=C2=A0 primary meta region
=C2=A0=C2=A0=C2=A0 When a client is configured to use meta replica, it sen= ds scan request
=C2=A0=C2=A0=C2=A0 to all meta replicas almost at the sa= me time. Since meta replica
=C2=A0=C2=A0=C2=A0 contains stale data, if r= esult from one of replica comes back first, the
=C2=A0=C2=A0=C2=A0 clien= t may get wrong region locations. To fix this,
=C2=A0=C2=A0=C2=A0 "= hbase.client.meta.replica.scan.timeout" is introduced, a client will=C2=A0=C2=A0=C2=A0 always send to primary meta region first, wait the con= figured timeout
=C2=A0=C2=A0=C2=A0 for reply. If no result is received, = it will send request to replica
=C2=A0=C2=A0=C2=A0 meta regions. The uni= t for "hbase.client.meta.replica.scan.timeout" is
=C2=A0=C2=A0= =C2=A0 microsecond, the default value is 1000000 (1 second).

- HBASE= -18137 Replication gets stuck for empty WALs

=C2=A0=C2=A0=C2=A0 0-le= ngth WAL files can potentially cause the replication queue to get
=C2=A0= =C2=A0=C2=A0 stuck.=C2=A0 A new config "replication.source.eof.autorec= overy" has been
=C2=A0=C2=A0=C2=A0 added: if set to true (default i= s false), the 0-length WAL file will be
=C2=A0=C2=A0=C2=A0 skipped after= 1) the max number of retries has been hit, and 2) there
=C2=A0=C2=A0=C2= =A0 are more WAL files in the queue.=C2=A0 The risk of enabling this is tha= t
=C2=A0=C2=A0=C2=A0 there is a chance the 0-length WAL file actually ha= s some data (e.g.
=C2=A0=C2=A0=C2=A0 block went missing and will come ba= ck once a datanode is recovered).

- HBASE-18164 Much faster locality= cost function and candidate generator

=C2=A0=C2=A0=C2=A0 New locali= ty cost function and candidate generator that use caching and
=C2=A0=C2= =A0=C2=A0 incremental computation to allow the stochastic load balancer to<= br>=C2=A0=C2=A0=C2=A0 consider ~20x more cluster configurations for big clu= sters.

- HBASE-18192 Replication drops recovered queues on region se= rver shutdown

=C2=A0=C2=A0=C2=A0 If a region server that is processi= ng recovered queue for another
=C2=A0=C2=A0=C2=A0 previously dead region= server is gracefully shut down, it can drop the
=C2=A0=C2=A0=C2=A0 reco= vered queue under certain conditions. Running without this fix on a
=C2= =A0=C2=A0=C2=A0 1.2+ release means possibility of continuing data loss in r= eplication,
=C2=A0=C2=A0=C2=A0 irrespective of which WALProvider is used= . If a single WAL group (or
=C2=A0=C2=A0=C2=A0 DefaultWALProvider) is us= ed, running without this fix will always cause
=C2=A0=C2=A0=C2=A0 datalo= ss in replication whenever a region server processing recovered
=C2=A0= =C2=A0=C2=A0 queues is gracefully shutdown.

- HBASE-18233 We shouldn= 't wait for readlock in doMiniBatchMutation in case
=C2=A0 of deadlo= ck

=C2=A0=C2=A0=C2=A0 This patch plus the sort of mutations done in = HBASE-17924 fixes a
=C2=A0=C2=A0=C2=A0 performance regression doing incr= ements and/or checkAndPut-style
=C2=A0=C2=A0=C2=A0 operations.

- = HBASE-18255 Time-Delayed HBase Performance Degradation with Java 7

= =C2=A0=C2=A0=C2=A0 This change sets the JVM property ReservedCodeCacheSize = to 256MB in the
=C2=A0=C2=A0=C2=A0 provided hbase-env.sh example file. T= he specific value for this property
=C2=A0=C2=A0=C2=A0 attempts to preve= nt performance issues seen when HBase using Java 7. The
=C2=A0=C2=A0=C2= =A0 value set is the same as the default when using Java 8.

- HBASE-= 18469 Correct RegionServer metric of=C2=A0 totalRequestCount

=C2=A0= =C2=A0=C2=A0 We introduced a new RegionServer metrics in name of
=C2=A0= =C2=A0=C2=A0 "totalRowActionRequestCount" which counts in all row= actions and equals
=C2=A0=C2=A0=C2=A0 to the sum of "readRequestCo= unt" and "writeRequestCount". Meantime, we
=C2=A0=C2=A0= =C2=A0 have changed "totalRequestCount" to count only once for mu= lti request,
=C2=A0=C2=A0=C2=A0 while previously we will count in action= number of the request. As a
=C2=A0=C2=A0=C2=A0 result, existing monitor= ing system on totalRequestCount will still work
=C2=A0=C2=A0=C2=A0 but s= ee a smaller value, and we strongly recommend to change to use the
=C2= =A0=C2=A0=C2=A0 new metrics to monitor server load.

- HBASE-18577 sh= aded client includes several non-relocated third party
=C2=A0 dependenci= es

=C2=A0=C2=A0=C2=A0 The HBase shaded artifacts (hbase-shaded-clien= t and hbase-shaded-server)
=C2=A0=C2=A0=C2=A0 no longer contain several = non-relocated third party dependency classes
=C2=A0=C2=A0=C2=A0 that wer= e mistakenly included. Downstream users who relied on these
=C2=A0=C2=A0= =C2=A0 classes being present will need to add a runtime dependency onto an<= br>=C2=A0=C2=A0=C2=A0 appropriate third party artifact. Previously, we erro= neously packaged
=C2=A0=C2=A0=C2=A0 several third party libs without rel= ocating them. In some cases these
=C2=A0=C2=A0=C2=A0 libraries have now = been relocated; in some cases they are no longer
=C2=A0=C2=A0=C2=A0 incl= uded at all.

=C2=A0=C2=A0=C2=A0 Includes:

=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 * jaxb
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * jetty
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 * jersey
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * codahale= metrics (HBase 1.4+ only)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * commons-cryp= to
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * jets3t
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 * junit
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * curator (HBase 1.4+)
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * netty 3 (HBase 1.1)
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 * mokito-junit4 (HBase 1.1)

- HBASE-18665 ReversedScann= erCallable invokes getRegionLocations
=C2=A0 incorrectly

=C2=A0= =C2=A0=C2=A0 Performing reverse scan on tables used the meta cache incorrec= tly and
=C2=A0=C2=A0=C2=A0 fetched data from meta table every time. This= fix solves this issue and
=C2=A0=C2=A0=C2=A0 which results in performan= ce improvement for reverse scans.

- HBASE-19285 Add per-table latenc= y histograms

=C2=A0=C2=A0=C2=A0 Per-RegionServer table latency histo= grams have been returned to HBase
=C2=A0=C2=A0=C2=A0 (after being remove= d due to impacting performance). These metrics are
=C2=A0=C2=A0=C2=A0 ex= posed via a new JMX bean "TableLatencies" with the typical naming=
=C2=A0=C2=A0=C2=A0 conventions: namespace, table, and histogram compone= nt.

Major

- HBASE-7621 REST client doesn't support binary= row keys

=C2=A0=C2=A0=C2=A0 RemoteHTable now supports binary row ke= ys with any character or byte
=C2=A0=C2=A0=C2=A0 by properly encoding re= quest URLs. This is a both a behavioral change
=C2=A0=C2=A0=C2=A0 from e= arlier versions and an important fix for protocol correctness.

- HBA= SE-11013 Clone Snapshots on Secure Cluster Should provide option to
=C2= =A0 apply Retained User Permissions

=C2=A0=C2=A0=C2=A0 While creatin= g a snapshot, it will save permissions of the original
=C2=A0=C2=A0=C2= =A0 table into .snapshotinfo file(Backward compatibility), which is in the<= br>=C2=A0=C2=A0=C2=A0 snapshot root directory.=C2=A0 For clone_snapshot/res= tore_snapshot command,
=C2=A0=C2=A0=C2=A0 we provide an additional optio= n (RESTORE_ACL) to decide whether we will
=C2=A0=C2=A0=C2=A0 grant permi= ssions of the origin table to the newly created table.

- HBASE-14548= Expand how table coprocessor jar and dependency path can
=C2=A0 be spec= ified

=C2=A0=C2=A0=C2=A0 Allows a directory containing the jars or s= ome wildcards to be
=C2=A0=C2=A0=C2=A0 specified, such as:

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 hdfs://namenode:port/user/hadoop-user/

=C2= =A0=C2=A0=C2=A0 or

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hdfs://namenode:po= rt/user/hadoop-user/*.jar

=C2=A0=C2=A0=C2=A0 Please note that if a d= irectory is specified, all jar files directly
=C2=A0=C2=A0=C2=A0 in the = directory are added, but it does not search files in the
=C2=A0=C2=A0=C2= =A0 subtree rooted in the directory. Do not use a wildcard if you would
= =C2=A0=C2=A0=C2=A0 like to specify a directory.

- HBASE-14925 HBase = shell command/tool to list table's region info through
=C2=A0 comman= d line

=C2=A0=C2=A0=C2=A0 Added a shell command 'list_regions= 9; for displaying the table's region
=C2=A0=C2=A0=C2=A0 info through= command line. List all regions for a particular table as an
=C2=A0=C2= =A0=C2=A0 array and also filter them by server name (optional) as prefix an= d
=C2=A0=C2=A0=C2=A0 maximum locality (optional). By default, it will re= turn all the regions
=C2=A0=C2=A0=C2=A0 for the table with any locality.= The command displays server name,
=C2=A0=C2=A0=C2=A0 region name, start= key, end key, size of the region in MB, number of
=C2=A0=C2=A0=C2=A0 re= quests and the locality. The information can be projected out via an
=C2= =A0=C2=A0=C2=A0 array as third parameter. By default all these information = is displayed.
=C2=A0=C2=A0=C2=A0 Possible array values are SERVER_NAME, = REGION_NAME, START_KEY,
=C2=A0=C2=A0=C2=A0 END_KEY, SIZE, REQ and LOCALI= TY. Values are not case sensitive. If you
=C2=A0=C2=A0=C2=A0 don't w= ant to filter by server name, pass an empty hash or string.

- HBASE-= 15187 Integrate CSRF prevention filter to REST gateway

=C2=A0=C2=A0= =C2=A0 Protection against CSRF attack can be turned on with config
=C2= =A0=C2=A0=C2=A0 parameter, hbase.rest.csrf.enabled - default value is false= .

=C2=A0=C2=A0=C2=A0 The custom header to be sent can be changed via= config parameter,
=C2=A0=C2=A0=C2=A0 hbase.rest.csrf.custom.header whos= e default value is
=C2=A0=C2=A0=C2=A0 "X-XSRF-HEADER".

= =C2=A0=C2=A0=C2=A0 The configuration parameter hbase.rest.csrf.methods.to.i= gnore
=C2=A0=C2=A0=C2=A0 controls which HTTP methods are not associated = with custom header
=C2=A0=C2=A0=C2=A0 check.

=C2=A0=C2=A0=C2=A0 T= he config parameter hbase.rest-csrf.browser-useragents-regex is a
=C2=A0= =C2=A0=C2=A0 comma-separated list of regular expressions used to match agai= nst an
=C2=A0=C2=A0=C2=A0 HTTP request's User-Agent header when prot= ection against cross-site
=C2=A0=C2=A0=C2=A0 request forgery (CSRF) is e= nabled for REST server by setting
=C2=A0=C2=A0=C2=A0 hbase.rest.csrf.ena= bled to true.

- HBASE-15236 Inconsistent cell reads over multiple bu= lk-loaded HFiles

=C2=A0=C2=A0=C2=A0 During bulkloading, if there are= multiple hfiles corresponding to
=C2=A0=C2=A0=C2=A0 the same region, an= d if they have same timestamps (which may have
=C2=A0=C2=A0=C2=A0 been s= et using importtsv.timestamp) and duplicate keys across them,
=C2=A0=C2= =A0=C2=A0 then get and scan may return values coming from different hfiles.=

- HBASE-15243 Utilize the lowest seek value when all Filters in
= =C2=A0 MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT

=C2=A0= =C2=A0=C2=A0 When all filters in a MUST_PASS_ONE FilterList return a
=C2= =A0=C2=A0=C2=A0 SEEK_USING_NEXT_HINT code, we return SEEK_NEXT_USING_HINT f= rom
=C2=A0=C2=A0=C2=A0 FilterList#filterKeyValue() to utilize the lowest= seek value.

- HBASE-15386 PREFETCH_BLOCKS_ON_OPEN in HColumnDescrip= tor is ignored

=C2=A0=C2=A0=C2=A0 Changes the prefetch TRACE-level l= oggings to include the word
=C2=A0=C2=A0=C2=A0 'Prefetch' in the= m so you know what they are about.

=C2=A0=C2=A0=C2=A0 Changes the cr= yptic logging of the CacheConfig#toString to have
=C2=A0=C2=A0=C2=A0 som= e preamble saying why and what column family is responsible.

- HBASE= -15576 Scanning cursor to prevent blocking long time on
=C2=A0 ResultSca= nner.next()

=C2=A0=C2=A0=C2=A0 If you don't like scanning being = blocked too long because of heartbeat
=C2=A0=C2=A0=C2=A0 and partial res= ult, you can use Scan#setNeedCursorResult(true) to get a
=C2=A0=C2=A0=C2= =A0 special result within scanning timeout setting time which will tell you=
=C2=A0=C2=A0=C2=A0 where row the server is scanning. See its javadoc fo= r more details.

- HBASE-15633 Backport HBASE-15507 to branch-1
=C2=A0=C2=A0=C2=A0 Adds update_peer_config to the HBase shell and Replica= tionAdmin, and
=C2=A0=C2=A0=C2=A0 provides a callback for custom replica= tion endpoints to be notified
=C2=A0=C2=A0=C2=A0 of changes to configura= tion and peer data

- HBASE-15686 Add override mechanism for the exem= pt classes when
=C2=A0 dynamically loading table coprocessor

=C2= =A0=C2=A0=C2=A0 The
=E2=80=8Bn=E2=80=8Bew coprocessor table descriptor attribute,
=C2=A0=C2=A0=C2=A0 hbase.= coprocessor.classloader.included.classes, is added. A user can
=C2=A0=C2= =A0=C2=A0 specify class name prefixes (semicolon separated) which should be=
=C2=A0=C2=A0=C2=A0 loaded by CoprocessorClassLoader.

- HBASE-157= 11 Add client side property to allow logging details for
=C2=A0 batch er= rors

=C2=A0=C2=A0=C2=A0 A new client side property hbase.client.log.= batcherrors.details is
=C2=A0=C2=A0=C2=A0 introduced to allow logging th= e full stacktrace of exceptions for
=C2=A0=C2=A0=C2=A0 batch errors. It = is disabled by default.

- HBASE-15816 Provide client with ability to= set priority on Operations

=C2=A0=C2=A0=C2=A0 Added setPriority(int= priority) API to Put, Delete, Increment, Append,
=C2=A0=C2=A0=C2=A0 Get= and Scan. For all these ops, the user can provide a custom RPC
=C2=A0= =C2=A0=C2=A0 priority level.

- HBASE-15924 Enhance hbase services au= torestart capability to
=C2=A0 hbase-daemon.sh

=C2=A0=C2=A0=C2=A0= Now one can start hbase services with enabled "autostart/autorestart&= quot;
=C2=A0=C2=A0=C2=A0 feature in controlled fashion with the help of = "--autostart-window-size"
=C2=A0=C2=A0=C2=A0 to define the win= dow period and the "--autostart-window-retry-limit" to
=C2=A0= =C2=A0=C2=A0 define the number of times the hbase services have to be resta= rted upon
=C2=A0=C2=A0=C2=A0 being killed/terminated abnormally within t= he provided window perioid.

=C2=A0=C2=A0=C2=A0 The following cases a= re supported with "autostart/autorestart":

=C2=A0=C2=A0=C2= =A0 a) --autostart-window-size=3D0 and --autostart-window-retry-limit=3D0,<= br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 indicates infinite window size and = no retry limit
=C2=A0=C2=A0=C2=A0 b) not providing the args, will defaul= t to a)
=C2=A0=C2=A0=C2=A0 c) --autostart-window-size=3D0 and --autostar= t-window-retry-limit=3D
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <positiv= e value> indicates the autostart process to bail out if the
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 retry limit exceeds irrespective of window peri= od
=C2=A0=C2=A0=C2=A0 d) --autostart-window-size=3D<x> and --autos= tart-window-retry-limit=3D<y>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= indicates the autostart process to bail out if the retry limit "y&quo= t;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 is exceeded for the last window = period "x".

- HBASE-15941 HBCK repair should not unsplit h= ealthy splitted region

=C2=A0=C2=A0=C2=A0 A new option -removeParent= s is now available that will remove an old
=C2=A0=C2=A0=C2=A0 parent whe= n two valid daughters for that parent exist and
=C2=A0=C2=A0=C2=A0 -fixH= dfsOverlaps is used. If there is an issue trying to remove the
=C2=A0=C2= =A0=C2=A0 parent from META or sidelining the parent from HDFS we will fallb= ack to
=C2=A0=C2=A0=C2=A0 do a regular merge. For now this option only w= orks when the overlap
=C2=A0=C2=A0=C2=A0 group consists only of 3 region= s (a parent, daughter A and daughter B)

- HBASE-15950 Fix memstore s= ize estimates to be tighter

=C2=A0=C2=A0=C2=A0 The estimates of heap= usage by the memstore objects (KeyValue, object
=C2=A0=C2=A0=C2=A0 and = array header sizes, etc) have been made more accurate for heap
=C2=A0=C2= =A0=C2=A0 sizes up to 32G (using CompressedOops), resulting in them droppin= g by
=C2=A0=C2=A0=C2=A0 10-50% in practice. This also results in less nu= mber of flushes and
=C2=A0=C2=A0=C2=A0 compactions due to "fatter&q= uot; flushes. As a result, the actual heap usage
=C2=A0=C2=A0=C2=A0 of t= he memstore before being flushed may increase by up to 100%. If
=C2=A0= =C2=A0=C2=A0 configured memory limits for the region server had been tuned = based on
=C2=A0=C2=A0=C2=A0 observed usage, this change could result in = worse GC behavior or even
=C2=A0=C2=A0=C2=A0 OutOfMemory errors. Set the= environment property (not hbase-site.xml)
=C2=A0=C2=A0=C2=A0 "hbas= e.memorylayout.use.unsafe" to false to disable.

- HBASE-15994 A= llow selection of RpcSchedulers

=C2=A0=C2=A0=C2=A0 Adds a FifoRpcSch= edulerFactory so you can try the FifoRpcScheduler by
=C2=A0=C2=A0=C2=A0 = setting "hbase.region.server.rpc.scheduler.factory.class"

= - HBASE-16052 Improve HBaseFsck Scalability

=C2=A0=C2=A0=C2=A0 Impro= ves the performance and scalability of HBaseFsck, especially for
=C2=A0= =C2=A0=C2=A0 large clusters with a small number of large tables.

=C2= =A0=C2=A0=C2=A0 Searching for lingering reference files is now a multi-thre= aded
=C2=A0=C2=A0=C2=A0 operation.=C2=A0 Loading HDFS region directory i= nformation is now multi-
=C2=A0=C2=A0=C2=A0 threaded at the region-level= instead of the table-level to maximize
=C2=A0=C2=A0=C2=A0 concurrency.= =C2=A0 A performance bug in HBaseFsck that resulted in
=C2=A0=C2=A0=C2= =A0 redundant I/O and RPCs was fixed by introducing a FileStatusFilter
= =C2=A0=C2=A0=C2=A0 that filters FileStatus objects directly.

- HBASE= -16213 A new HFileBlock structure for fast random get

=C2=A0=C2=A0= =C2=A0 Introduces a new DataBlockEncoding in name of ROW_INDEX_V1, which=C2=A0=C2=A0=C2=A0 could improve random read (get) performance especially = when the
=C2=A0=C2=A0=C2=A0 average record size (key-value size per row)= is small. To use this
=C2=A0=C2=A0=C2=A0 feature, please set DATA_BLOCK= _ENCODING to ROW_INDEX_V1 for column
=C2=A0=C2=A0=C2=A0 family of newly = created table, or change existing CF with below shell
=C2=A0=C2=A0=C2=A0= command:

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 alter 'table_name',= {NAME =3D> 'cf', DATA_BLOCK_ENCODING =3D>
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 'ROW_INDEX_V1'}.

=C2=A0=C2= =A0=C2=A0 Please note that if we turn this DBE on, HFile block will be bigg= er
=C2=A0=C2=A0=C2=A0 than NONE encoding because it adds some metadata f= or binary search.

=C2=A0=C2=A0=C2=A0 Seek in row when random reading= is one of the main consumers of CPU.
=C2=A0=C2=A0=C2=A0 This helps.
=
- HBASE-16244 LocalHBaseCluster start timeout should be configurable
=C2=A0=C2=A0=C2=A0 When LocalHBaseCluster is started from the command = line the Master
=C2=A0=C2=A0=C2=A0 would give up after 30 seconds due to= a hardcoded timeout meant for
=C2=A0=C2=A0=C2=A0 unit tests. This chang= e allows the timeout to be configured via
=C2=A0=C2=A0=C2=A0 hbase-site = as well as sets it to 5 minutes when LocalHBaseCluster
=C2=A0=C2=A0=C2= =A0 is started from the command line.

- HBASE-16336 Removing peers s= eems to be leaving spare queues

=C2=A0=C2=A0=C2=A0 Add a Replication= ZKNodeCleaner periodic check and delete any useless
=C2=A0=C2=A0=C2=A0 r= eplication queue belonging to a peer which does not exist.

- HBASE-1= 6388 Prevent client threads being blocked by only one slow
=C2=A0 region= server

=C2=A0=C2=A0=C2=A0 Adds a new configuration, hbase.client.pe= rserver.requests.threshold,
=C2=A0=C2=A0=C2=A0 to limit the max number o= f concurrent request to one region server.
=C2=A0=C2=A0=C2=A0 If the use= r still create new request after reaching the limit,
=C2=A0=C2=A0=C2=A0 = client will throw ServerTooBusyException and do not send the request
=C2= =A0=C2=A0=C2=A0 to the server. This is a client side feature and can preven= t client's
=C2=A0=C2=A0=C2=A0 threads being blocked by one slow regi= on server resulting in the
=C2=A0=C2=A0=C2=A0 availability of client is = much lower than the availability of region
=C2=A0=C2=A0=C2=A0 servers.
- HBASE-16540 Scan should do additional validation on start and stop = row

=C2=A0=C2=A0=C2=A0 Scan#setStartRow() and Scan#setStopRow() now = validate the argument
=C2=A0=C2=A0=C2=A0 passed for each row key.=C2=A0 = If the length of the parameter passed
=C2=A0=C2=A0=C2=A0 exceeds Short.M= AX_VALUE, an IllegalArgumentException will be thrown.

- HBASE-16584 = Backport the new ipc implementation in HBASE-16432 to
=C2=A0 branch-1
=C2=A0=C2=A0=C2=A0 The netty dependency is upgraded to 4.1.1.Final. An= d also some
=C2=A0=C2=A0=C2=A0 configurations of the old AsyncRpcClient = is gone. Such as
=C2=A0=C2=A0=C2=A0 "hbase.rpc.client.threads.max&q= uot; and "hbase.rpc.client.nativetransport".

- HBASE-16653= Backport HBASE-11393 to all branches which support namespace

=C2=A0= =C2=A0=C2=A0 During HBASE-11393, we have done two things:

=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 1.=C2=A0 unify tableCFs with peerConfig
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 2.=C2=A0 Fix ns not support issue for replication.
=C2=A0=C2=A0=C2=A0 This issue is to backport it to branch-1

=C2= =A0=C2=A0=C2=A0 How to rolling update if the replication peer have old tabl= e-cfs
=C2=A0=C2=A0=C2=A0 config? Due to we modify proto object of Replic= ationPeerConfig (add
=C2=A0=C2=A0=C2=A0 tableCFs field), so when we do r= olling update, we have to update
=C2=A0=C2=A0=C2=A0 original Replication= PeerConfig data on ZK firstly.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1. Mak= e sure the master have the permission to modify replication
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 peer znode.
=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 2. Disable the replication peer.
=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 3. Rolling update master first. The master will copy the table-cfs
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 config from old table-cfs = znode and add it to the new proto
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 object of ReplicationPeerConfig.
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 4. Rolling update regionservers.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 5= . Enable the replication peer.

=C2=A0=C2=A0=C2=A0 If you can't c= hange the replication peer znode permission, you can
=C2=A0=C2=A0=C2=A0 = use the TableCFsUpdater tool to copy the table-cfs config.

=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 1. Disable the replication peer.
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 2. bin/hbase org.apache.hadoop.hbase.replication.master.Tab= leCFsUpdater
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 update
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3. Rolling update master and regionservers.<= br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4. Enable the replication peer.

- = HBASE-16672 Add option for bulk load to always copy hfile(s) instead of
= =C2=A0 renaming

=C2=A0=C2=A0=C2=A0 This issue adds a config, always.= copy.files, to LoadIncrementalHFiles.
=C2=A0=C2=A0=C2=A0 When set to tru= e, source hfiles would be copied. Meaning source hfiles
=C2=A0=C2=A0=C2= =A0 would be kept after bulk load is done. Default value is false.

-= HBASE-16698] Performance issue: handlers stuck waiting for CountDownLatch<= br>=C2=A0 inside WALKey#getWriteEntry under high writing workload

= =C2=A0=C2=A0=C2=A0 Assign sequenceid to an edit before we go on the ringbuf= fer; undoes
=C2=A0=C2=A0=C2=A0 contention on WALKey latch. Adds a new co= nfig
=C2=A0=C2=A0=C2=A0 "hbase.hregion.mvcc.preassign" which d= efaults to true: i.e. this speedup
=C2=A0=C2=A0=C2=A0 is enabled.
- HBASE-16755 Honor flush policy under global memstore pressure

=C2= =A0=C2=A0=C2=A0 Prior to this change, when the memstore low water mark is e= xceeded on a
=C2=A0=C2=A0=C2=A0 regionserver, the regionserver will forc= e flush all stores on the
=C2=A0=C2=A0=C2=A0 regions selected for flushi= ng until we drop below the low water mark.
=C2=A0=C2=A0=C2=A0 With this = change, the regionserver will continue to force flush regions
=C2=A0=C2= =A0=C2=A0 when above the memstore low water mark, but will only flush the s= tores
=C2=A0=C2=A0=C2=A0 returned by the configured FlushPolicy.

= - HBASE-16993 BucketCache throw java.io.IOException: Invalid HFile block=C2=A0 magic when configuring hbase.bucketcache.bucket.sizes

=C2=A0= =C2=A0=C2=A0 Any value for hbase.bucketcache.bucket.sizes configuration mus= t be a
=C2=A0=C2=A0=C2=A0 multiple of 256.=C2=A0 If that is not the case= , instantiation of L2 Bucket
=C2=A0=C2=A0=C2=A0 cache itself will fail t= hrowing IllegalArgumentException.

- HBASE-17112 Prevent setting time= stamp of delta operations the same as
=C2=A0 previous value

=C2= =A0=C2=A0=C2=A0 Before this issue, two concurrent Increments/Appends done i= n same
=C2=A0=C2=A0=C2=A0 millisecond or RS's clock going back will = result in two results having
=C2=A0=C2=A0=C2=A0 same TS, which is not fr= iendly to versioning and will get wrong result
=C2=A0=C2=A0=C2=A0 in sin= k cluster if the replication is disordered. After this issue, the
=C2=A0= =C2=A0=C2=A0 result of Increment/Append will always have an incremental TS.= There is
=C2=A0=C2=A0=C2=A0 no longer any inconsistency in replication = for these operations.

- HBASE-17178 Add region balance throttling
=C2=A0=C2=A0=C2=A0 Add region balance throttling. Master execute every= region balance plan
=C2=A0=C2=A0=C2=A0 per balance interval, which is e= quals to divide max balancing time by
=C2=A0=C2=A0=C2=A0 the size of reg= ion balance plan. And Introduce a new config
=C2=A0=C2=A0=C2=A0 hbase.ma= ster.balancer.maxRitPercent to protect availability. If config
=C2=A0=C2= =A0=C2=A0 this to 0.01, then the max percent of regions in transition is 1%= when
=C2=A0=C2=A0=C2=A0 balancing. Then the cluster's availability = is at least 99% when
=C2=A0=C2=A0=C2=A0 balancing.

- HBASE-17280 = Add mechanism to control cleaner chore behavior

=C2=A0=C2=A0=C2=A0 T= he HBase cleaner chore process cleans up old WAL files and archived
=C2= =A0=C2=A0=C2=A0 HFiles. Cleaner operation can affect query performance when= running
=C2=A0=C2=A0=C2=A0 heavy workloads, so disable the cleaner duri= ng peak hours. The cleaner
=C2=A0=C2=A0=C2=A0 has the following HBase sh= ell commands:

=C2=A0=C2=A0=C2=A0 - cleaner_chore_enabled: Queries wh= ether cleaner chore is enabled/
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 disabled.=
=C2=A0=C2=A0=C2=A0 - cleaner_chore_run: Manually runs the cleaner to re= move files.
=C2=A0=C2=A0=C2=A0 - cleaner_chore_switch: enables or disabl= es the cleaner and returns
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 the previous s= tate of the cleaner. For example, cleaner-switch true
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 enables the cleaner.

=C2=A0=C2=A0=C2=A0 Following APIs = are added in Admin:

=C2=A0=C2=A0=C2=A0 - setCleanerChoreRunning(bool= ean on): Enable/Disable the cleaner chore
=C2=A0=C2=A0=C2=A0 - runCleane= rChore(): Ask for cleaner chore to run
=C2=A0=C2=A0=C2=A0 - isCleanerCho= reEnabled(): Query whether cleaner chore is enabled/
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 disabled.

- HBASE-17296 Provide per peer throttling for= replication

=C2=A0=C2=A0=C2=A0 Provide per peer throttling for repl= ication. Add the bandwidth upper
=C2=A0=C2=A0=C2=A0 limit to Replication= PeerConfig and a new shell cmd set_peer_bandwidth
=C2=A0=C2=A0=C2=A0 to = update the bandwidth as needed.

- HBASE-17426 Inconsistent environme= nt variable names for enabling JMX

=C2=A0=C2=A0=C2=A0 In bin/hbase-c= onfig.sh, if value for HBASE_JMX_BASE is empty, keep
=C2=A0=C2=A0=C2=A0 = current behavior. If HBASE_JMX_OPTS is not empty, keep current
=C2=A0=C2= =A0=C2=A0 behavior. Otherwise use the value of HBASE_JMX_BASE

- HBAS= E-17437 Support specifying a WAL directory outside of the root
=C2=A0 di= rectory

=C2=A0=C2=A0=C2=A0 This patch adds support for specifying a = WAL directory outside of the
=C2=A0=C2=A0=C2=A0 HBase root directory.
=C2=A0=C2=A0=C2=A0 Multiple configuration variables were added to acco= mplish this:

=C2=A0=C2=A0=C2=A0 hbase.wal.dir: used to configure whe= re the root WAL directory is
=C2=A0=C2=A0=C2=A0 located. Could be on a d= ifferent FileSystem than the root directory. WAL
=C2=A0=C2=A0=C2=A0 dire= ctory can not be set to a subdirectory of the root directory. The
=C2=A0= =C2=A0=C2=A0 default value of this is the root directory if unset.

= =C2=A0=C2=A0=C2=A0 hbase.rootdir.perms: Configures FileSystem permissions t= o set on the
=C2=A0=C2=A0=C2=A0 root directory. This is '700' by= default.

=C2=A0=C2=A0=C2=A0 hbase.wal.dir.perms: Configures FileSys= tem permissions to set on the WAL
=C2=A0=C2=A0=C2=A0 directory FileSyste= m. This is '700' by default.

- HBASE-17472 Correct the seman= tic of permission grant

=C2=A0=C2=A0=C2=A0 Before this patch, later = granted permissions will override previous
=C2=A0=C2=A0=C2=A0 granted pe= rmissions, and previous granted permissions will be lost. This
=C2=A0=C2= =A0=C2=A0 issue re-defines the grant semantic: for master branch, later gra= nted
=C2=A0=C2=A0=C2=A0 permissions will merge with previous granted per= missions. For
=C2=A0=C2=A0=C2=A0 branch-1.4, grant keep override behavio= r for compatibility purpose, and
=C2=A0=C2=A0=C2=A0 a grant with mergeEx= istingPermission flag provided.

- HBASE-17508 Unify the implementati= on of small scan and regular scan for
=C2=A0 sync client

=C2=A0= =C2=A0=C2=A0 Now the scan.setSmall method is deprecated. Consider using sca= n.setLimit
=C2=A0=C2=A0=C2=A0 and scan.setReadType in the future. And we= will open scanner lazily when
=C2=A0=C2=A0=C2=A0 you call scanner.next.= This is an incompatible change which delays the
=C2=A0=C2=A0=C2=A0 tabl= e existence check and permission check.

- HBASE-17578 Thrift per-met= hod metrics should still update in the case of
=C2=A0 exceptions

= =C2=A0=C2=A0=C2=A0 In prior versions, the HBase Thrift handlers failed to i= ncrement per-
=C2=A0=C2=A0=C2=A0 method metrics when an exception was en= countered.=C2=A0 These metrics will
=C2=A0=C2=A0=C2=A0 now always be inc= remented, whether an exception is encountered or not.
=C2=A0=C2=A0=C2=A0= This change also adds exception-type metrics, similar to those exposed
= =C2=A0=C2=A0=C2=A0 in regionservers, for individual exceptions which are re= ceived by the
=C2=A0=C2=A0=C2=A0 Thrift handlers.

- HBASE-17583 A= dd inclusive/exclusive support for startRow and endRow of
=C2=A0 scan fo= r sync client

=C2=A0=C2=A0=C2=A0 Now you can include or exclude the = startRow and stopRow for a scan. The
=C2=A0=C2=A0=C2=A0 new methods to s= pecify startRow and stopRow are withStartRow and
=C2=A0=C2=A0=C2=A0 with= StopRow. The old methods to specify startRow and Row(include
=C2=A0=C2= =A0=C2=A0 constructors) are marked as deprecated as in the old time if star= tRow
=C2=A0=C2=A0=C2=A0 and stopRow are equal then we will consider it a= s a get scan and include
=C2=A0=C2=A0=C2=A0 the stopRow implicitly. This= is strange after we can set inclusiveness
=C2=A0=C2=A0=C2=A0 explicitly= so we add new methods and depredate the old methods. The
=C2=A0=C2=A0= =C2=A0 deprecated methods will be removed in the future.

- HBASE-175= 84 Expose ScanMetrics with ResultScanner rather than Scan

=C2=A0=C2= =A0=C2=A0 Now you can use ResultScanner.getScanMetrics to get the scan metr= ics at
=C2=A0=C2=A0=C2=A0 any time during the scan operation. The old Sc= an.getScanMetrics is
=C2=A0=C2=A0=C2=A0 deprecated and still work, but i= f you use ResultScanner.getScanMetrics
=C2=A0=C2=A0=C2=A0 to get the sca= n metrics and reset it, then the metrics published to the
=C2=A0=C2=A0= =C2=A0 Scan instaince will be messed up.

- HBASE-17599 Use mayHaveMo= reCellsInRow instead of isPartial

=C2=A0=C2=A0=C2=A0 The word 'i= sPartial' is ambiguous so we introduce a new method
=C2=A0=C2=A0=C2= =A0 'mayHaveMoreCellsInRow' to replace it. And the old meaning of=C2=A0=C2=A0=C2=A0 'isPartial' is not the same with 'mayHaveM= oreCellsInRow' as for batched
=C2=A0=C2=A0=C2=A0 scan, if the number= of returned cells equals to the batch, isPartial
=C2=A0=C2=A0=C2=A0 wil= l be false. After this change the meaning of 'isPartial' will be sa= me
=C2=A0=C2=A0=C2=A0 with 'mayHaveMoreCellsInRow'. This is an i= ncompatible change but it is
=C2=A0=C2=A0=C2=A0 not likely to break a lo= t of things as for batched scan the old
=C2=A0=C2=A0=C2=A0 'isPartia= l' is just a redundant information, i.e, if the number of
=C2=A0=C2= =A0=C2=A0 returned cells reaches the batch limit. You have already know the= number
=C2=A0=C2=A0=C2=A0 of returned cells and the value of batch.
=
- HBASE-17737 Thrift2 proxy should support scan timeRange per column=C2=A0 family

=C2=A0=C2=A0=C2=A0 Thrift2 proxy now supports scan ti= meRange per column family.

- HBASE-17757 Unify blocksize after encod= ing to decrease memory fragment

=C2=A0=C2=A0=C2=A0 Blocksize is set = in columnfamily's atrributes. It is used to control
=C2=A0=C2=A0=C2= =A0 block sizes when generating blocks. But, it doesn't take encoding i= nto
=C2=A0=C2=A0=C2=A0 count. If you set encoding to blocks, after encod= ing, the block size
=C2=A0=C2=A0=C2=A0 varies. Since blocks will be cach= ed in memory after encoding (default),
=C2=A0=C2=A0=C2=A0 it will cause = memory fragment if using blockcache, or decrease the pool
=C2=A0=C2=A0= =C2=A0 efficiency if using bucketCache. This issue introduced a new config<= br>=C2=A0=C2=A0=C2=A0 named 'hbase.writer.unified.encoded.blocksize.rat= io'. The default value
=C2=A0=C2=A0=C2=A0 of this config is 1, meani= ng doing nothing. If this value is set to a
=C2=A0=C2=A0=C2=A0 smaller v= alue like 0.5, and the blocksize is set to 64KB (default value
=C2=A0=C2= =A0=C2=A0 of blocksize). It will unify the blocksize after encoding to 64KB= * 0.5
=C2=A0=C2=A0=C2=A0 =3D 32KB. Unified blocksize will relieve the m= emory problems mentioned
=C2=A0=C2=A0=C2=A0 above.

- HBASE-17817 = Make Regionservers log which tables it removed coprocessors
=C2=A0 from = when aborting

=C2=A0=C2=A0=C2=A0 Adds table name to exception loggin= g when a coprocessor is removed from
=C2=A0=C2=A0=C2=A0 a table by the r= egion server.

- HBASE-17861 Regionserver down when checking the perm= ission of staging dir
=C2=A0 if hbase.rootdir is on S3

=C2=A0=C2= =A0=C2=A0 Some object store does not support unix style permission. This fi= xes the
=C2=A0=C2=A0=C2=A0 permission check issue when specify staging d= ir in different file
=C2=A0=C2=A0=C2=A0 system. Currently it covers s3, = wasb, swift.

- HBASE-17877 Improve HBase's byte[] comparator
=
=C2=A0=C2=A0=C2=A0 Updated the lexicographic byte array comparator to u= se a slightly more
=C2=A0=C2=A0=C2=A0 optimized version similar to the o= ne available in the guava library that
=C2=A0=C2=A0=C2=A0 compares only = the first index where left[index] !=3D right[index]. The
=C2=A0=C2=A0=C2= =A0 comparator also returns the diff directly instead of mapping it to -1,<= br>=C2=A0=C2=A0=C2=A0 0, +1 range as was being done in the earlier version.= We have seen
=C2=A0=C2=A0=C2=A0 significant performance gains, calculat= ed in terms of throughput (ops/
=C2=A0=C2=A0=C2=A0 ms) with these change= s ranging from approx 20% for smaller byte arrays
=C2=A0=C2=A0=C2=A0 up = to 200 bytes and almost 100% for large byte array sizes that are in
=C2= =A0=C2=A0=C2=A0 few KBs. We benchmarked with up to 16KB arrays and the gene= ral trend
=C2=A0=C2=A0=C2=A0 indicates that the performance improvement = increases as the size of the
=C2=A0=C2=A0=C2=A0 byte array increases.
- HBASE-17956 Raw scan should ignore TTL

=C2=A0=C2=A0=C2=A0 Now= raw scan can also read expired cells.

- HBASE-18023 Log multi-reque= sts for more than threshold number of rows

=C2=A0=C2=A0=C2=A0 Introd= uces a warning message in the RegionServer log when an RPC is
=C2=A0=C2= =A0=C2=A0 received from a client that has more than 5000 "actions"= ; (where an
=C2=A0=C2=A0=C2=A0 "action" is a collection of mut= ations for a specific row) in a single
=C2=A0=C2=A0=C2=A0 RPC. Misbehavi= ng clients who send large RPCs to RegionServers can be
=C2=A0=C2=A0=C2= =A0 malicious, causing temporary pauses via garbage collection or denial of=
=C2=A0=C2=A0=C2=A0 service via crashes. The threshold of 5000 actions p= er RPC is defined by
=C2=A0=C2=A0=C2=A0 the property "hbase.rpc.row= s.warning.threshold" in hbase-site.xml.

- HBASE-18090 Improve T= ableSnapshotInputFormat to allow multiple mappers
=C2=A0 per region
<= br>=C2=A0=C2=A0=C2=A0 In this task, we make it possible to run multiple map= pers per region in
=C2=A0=C2=A0=C2=A0 the table snapshot.

- HBASE= -18122 Scanner id should include ServerName of region server

=C2=A0= =C2=A0=C2=A0 The first 32 bits are MurmurHash32 of ServerName string "= host,port,ts".
=C2=A0=C2=A0=C2=A0 The ServerName contains both host= , port, and start timestamp so it can
=C2=A0=C2=A0=C2=A0 prevent collisi= on. The lowest 32bit is generated by atomic int.

- HBASE-18149 The s= etting rules for table-scope attributes and family-scope
=C2=A0 attribut= es should keep consistent

=C2=A0=C2=A0=C2=A0 If the table-scope attr= ibutes value is false, you need not to enclose
=C2=A0=C2=A0=C2=A0 'f= alse' in single quotation. Both COMPACTION_ENABLED =3D> false and=C2=A0=C2=A0=C2=A0 COMPACTION_ENABLED =3D> 'false' will take ef= fect.

- HBASE-18226 Disable reverse DNS lookup at HMaster and use th= e hostname
=C2=A0 provided by RegionServer

=C2=A0=C2=A0=C2=A0 The= following config is added:

=E2=80= =8B=C2=A0=C2=A0=C2=A0 =E2=80=8B
=C2=A0=C2=A0=C2=A0 hbase.regionserver.= hostname.disable.master.reversedns

=C2=A0=C2=A0=C2=A0 This config is= for experts: don't set its value unless you really know
=C2=A0=C2= =A0=C2=A0 what you are doing. When set to true, regionserver will use the c= urrent
=C2=A0=C2=A0=C2=A0 node hostname for the servername and HMaster w= ill skip reverse DNS
=C2=A0=C2=A0=C2=A0 lookup and use the hostname sent= by regionserver instead. Note that this
=C2=A0=C2=A0=C2=A0 config and h= base.regionserver.hostname are mutually exclusive. See
=C2=A0=C2=A0=C2= =A0 https://i= ssues.apache.org/jira/browse/HBASE-18226 for more details.

=C2= =A0=C2=A0=C2=A0 Caution: please make sure rolling upgrade succeeds before t= urning on
=C2=A0=C2=A0=C2=A0 this feature.

- HBASE-18247 Hbck to = fix the case that replica region shows as key in the
=C2=A0 meta table
=C2=A0=C2=A0=C2=A0 The hbck tool can now correct the meta table shoul= d it get an entry for
=C2=A0=C2=A0=C2=A0 a read replica region.

-= HBASE-18374 RegionServer Metrics improvements

=C2=A0=C2=A0=C2=A0 Th= is change adds the latency metrics checkAndPut, checkAndDelete,
=C2=A0= =C2=A0=C2=A0 putBatch and deleteBatch . Also the previous regionserver &quo= t;mutate"
=C2=A0=C2=A0=C2=A0 latency metrics are renamed to "p= ut" metrics. Batch metrics capture the
=C2=A0=C2=A0=C2=A0 latency o= f the entire batch containing put/delete whereas put/delete
=C2=A0=C2=A0= =C2=A0 metrics capture latency per operation. Note this change will break=C2=A0=C2=A0=C2=A0 existing monitoring based on regionserver "mutate= " latency metric.

- HBASE-18520 Add jmx value to determine true= Master Start time

=C2=A0=C2=A0=C2=A0 Adds a JMX value to track when= the Master has finished initializing.
=C2=A0=C2=A0=C2=A0 The jmx config= is 'masterFinishedInitializationTime' and details the
=C2=A0=C2= =A0=C2=A0 time in millis that the Master is fully usable and ready to serve=
=C2=A0=C2=A0=C2=A0 requests.

- HBASE-18533 Expose BucketCache va= lues to be configured

=C2=A0=C2=A0=C2=A0 This patch exposes configur= ation for Bucketcache. These configs are very
=C2=A0=C2=A0=C2=A0 similar= to those for the LRU cache, but are described below:

=C2=A0=C2=A0= =C2=A0 "hbase.bucketcache.single.factor"; /** Single access bucke= t size */
=C2=A0=C2=A0=C2=A0 "hbase.bucketcache.multi.factor";= /** Multiple access bucket size */
=C2=A0=C2=A0=C2=A0 "hbase.bucke= tcache.memory.factor"; /** In-memory bucket size */
=C2=A0=C2=A0=C2= =A0 "hbase.bucketcache.extrafreefactor"; /** Free this floating p= oint
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 factor of extra blocks when evicting= . For example free the number of
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 blocks r= equested * (1 + extraFreeFactor) */
=C2=A0=C2=A0=C2=A0 "hbase.bucke= tcache.acceptfactor"; /** Acceptable size of cache (no
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 evictions if size < acceptable) */
=C2=A0=C2= =A0=C2=A0 "hbase.bucketcache.minfactor"; /** Minimum threshold of= cache (when
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 evicting, evict until = size < min) */

- HBASE-18675 Making {max,min}SessionTimeout confi= gurable for
=C2=A0 MiniZooKeeperCluster

=C2=A0=C2=A0=C2=A0 Standa= lone clusters and minicluster instances can now configure the
=C2=A0=C2= =A0=C2=A0 session timeout for our embedded ZooKeeper quorum using
=C2=A0= =C2=A0=C2=A0 "hbase.zookeeper.property.minSessionTimeout" and
= =C2=A0=C2=A0=C2=A0 "hbase.zookeeper.property.maxSessionTimeout".<= br>
- HBASE-18786 FileNotFoundException should not be silently handled f= or
=C2=A0 primary region replicas

=C2=A0=C2=A0=C2=A0 FileNotFound= Exception opening a StoreFile in a primary replica now
=C2=A0=C2=A0=C2= =A0 causes a RegionServer to crash out where before it would be ignored (or=
=C2=A0=C2=A0=C2=A0 optionally handled via close/reopen).

- HBASE= -18993 Backport patches in HBASE-18410 to branch-1.x branches

=C2=A0= =C2=A0=C2=A0 This change fixes bugs in FilterList, and also does a code ref= actor
=C2=A0=C2=A0=C2=A0=C2=A0 which ensures interface compatibility.
=C2=A0=C2=A0=C2=A0 The primary bug fixes are :

=C2=A0=C2=A0=C2= =A0 1. For sub-filter in FilterList with MUST_PASS_ONE, if previous
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 filterKeyValue() of sub-filter returns NE= XT_COL, we cannot make sure
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 that th= e next cell will be the first cell in next column, because
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 FilterList choose the minimal forward step among s= ub-filters, and it
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 may return a SKI= P. so here we add an extra check to ensure that the
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 next cell will match previous return code for sub-filter= s.

=C2=A0=C2=A0=C2=A0 2. Previous logic about transforming cell of F= ilterList is incorrect, we
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 should s= et the previous transform result (rather than the given cell
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 in question) as the initial value of transform = cell before call
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 filterKeyValue() o= f FilterList.

=C2=A0=C2=A0=C2=A0 3. Handle the ReturnCodes which the= previous code did not handle.

=C2=A0=C2=A0=C2=A0 About code refacto= r, we divided the FilterList into two separated sub-
=C2=A0=C2=A0=C2=A0 = classes: FilterListWithOR and FilterListWithAND, The FilterListWithOR
= =C2=A0=C2=A0=C2=A0 has been optimized to choose the next minimal step to se= ek cell rather
=C2=A0=C2=A0=C2=A0 than SKIP cell one by one, and the Fil= terListWithAND has been optimized
=C2=A0=C2=A0=C2=A0 to choose the next = maximal key to seek among sub-filters in filter list.
=C2=A0=C2=A0=C2=A0= All in all, The code in FilterList is clean and easier to follow now.
<= br>=C2=A0=C2=A0=C2=A0 Note that ReturnCode NEXT_ROW has been redefined as s= kipping to next
=C2=A0=C2=A0=C2=A0 row in current family, not to next ro= w in all family. it=E2=80=99s more
=C2=A0=C2=A0=C2=A0 reasonable, becaus= e ReturnCode is a concept in store level, not in
=C2=A0=C2=A0=C2=A0 regi= on level.

- HBASE-19035 Miss metrics when coprocessor use region sca= nner to read
=C2=A0 data

=C2=A0=C2=A0=C2=A0 Move read requests co= unt to region level. Because RegionScanner is
=C2=A0=C2=A0=C2=A0 exposed= to CP. Update write requests count in processRowsWithLocks.
=C2=A0=C2= =A0=C2=A0 Remove requestRowActionCount in RSRpcServices. This metric can be=
=C2=A0=C2=A0=C2=A0 computed by region's readRequestsCount and write= RequestsCount.

- HBASE-19051 Add new split algorithm for num string<= br>
=C2=A0=C2=A0=C2=A0 Add new split algorithm DecimalStringSplit=EF=BC= =8Crow are decimal-encoded long
=C2=A0=C2=A0=C2=A0 values in the range &= quot;00000000" =3D> "99999999".

- HBASE-19131 Add = the ClusterStatus hook and cleanup other hooks which can
=C2=A0 be repla= ced by ClusterStatus hook**

=C2=A0=C2=A0=C2=A0 1) Add preGetClusterS= tatus() and postGetClusterStatus() hooks
=C2=A0=C2=A0=C2=A0 2) add preGe= tClusterStatus() to access control check - an admin action

- HBASE-1= 9144 Retry assignments in FAILED_OPEN state when servers (re)join
=C2=A0= the cluster

=C2=A0=C2=A0=C2=A0=C2=A0 When regionserver placement gr= oups (RSGroups) is active, as servers
=C2=A0=C2=A0=C2=A0=C2=A0 join the = cluster the Master will attempt to reassign regions in
=C2=A0=C2=A0=C2= =A0=C2=A0 FAILED_OPEN state.

- HBASE-19419 Remove hbase-native-clien= t from branch-1

=C2=A0=C2=A0=C2=A0 Removed the hbase-native-client m= odule from branch-1 (it is still in
=C2=A0=C2=A0=C2=A0 Master). It is no= t complete. Look for a finished C++ client in the near
=C2=A0=C2=A0=C2= =A0 future which may be backported to branch-1 at that point.

---Cheers,
The HBase Dev Team
--001a113ce526de84520560b68979--