tune-cs-messaging

Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 377CE200C1B for ; Tue, 14 Feb 2017 20:40:02 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 363C5160B73; Tue, 14 Feb 2017 19:40:02 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 645BD160B82 for ; Tue, 14 Feb 2017 20:39:48 +0100 (CET) Received: (qmail 75742 invoked by uid 500); 14 Feb 2017 19:39:47 -0000 Mailing-List: contact commits-help@geode.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@geode.apache.org Delivered-To: mailing list commits@geode.apache.org Received: (qmail 74190 invoked by uid 99); 14 Feb 2017 19:39:46 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Feb 2017 19:39:46 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 34B19DFA3D; Tue, 14 Feb 2017 19:39:46 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: abaker@apache.org To: commits@geode.apache.org Date: Tue, 14 Feb 2017 19:40:06 -0000 Message-Id: <23cfbfae2d374bb8b62a8f82604a04ee@git.apache.org> In-Reply-To: <158ab04503fe4d86bd018fa4a2e277af@git.apache.org> References: <158ab04503fe4d86bd018fa4a2e277af@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [22/74] [abbrv] [partial] geode-native git commit: GEODE-1964: native client documentation (note: contains references to images in the geode-docs directories) archived-at: Tue, 14 Feb 2017 19:40:02 -0000 http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/images_svg/tune_cs_event_messaging.svg ---------------------------------------------------------------------- diff --git a/geode-docs/images_svg/tune_cs_event_messaging.svg b/geode-docs/images_svg/tune_cs_event_messaging.svg deleted file mode 100644 index 5af9f53..0000000 --- a/geode-docs/images_svg/tune_cs_event_messaging.svg +++ /dev/null @@ -1,3 +0,0 @@ - - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/images_svg/unbalanced_network_capacity_probs.svg ---------------------------------------------------------------------- diff --git a/geode-docs/images_svg/unbalanced_network_capacity_probs.svg b/geode-docs/images_svg/unbalanced_network_capacity_probs.svg deleted file mode 100644 index c24c082..0000000 --- a/geode-docs/images_svg/unbalanced_network_capacity_probs.svg +++ /dev/null @@ -1,3 +0,0 @@ - - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/autoreconnect/member-reconnect.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/autoreconnect/member-reconnect.html.md.erb b/geode-docs/managing/autoreconnect/member-reconnect.html.md.erb deleted file mode 100644 index 7303800..0000000 --- a/geode-docs/managing/autoreconnect/member-reconnect.html.md.erb +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: Handling Forced Cache Disconnection Using Autoreconnect ---- - -A Geode member may be forcibly disconnected from a Geode distributed system if the member is unresponsive for a period of time, or if a network partition separates one or more members into a group that is too small to act as the distributed system. - -## How the Autoreconnection Process Works - -After being disconnected from a distributed system a Geode member shuts down and then automatically restarts into a "reconnecting" state, while periodically attempting to rejoin the distributed system by contacting a list of known locators. If the member succeeds in reconnecting to a known locator, the member rebuilds its view of the distributed system from existing members and receives a new distributed system ID. - -If the member cannot connect to a known locator, the member will then check to see if it itself is a locator (or hosting an embedded locator process). If the member is a locator, then the member does a quorum-based reconnect; it will attempt to contact a quorum of the members that were in the membership view just before it became disconnected. If a quorum of members can be contacted, then startup of the distributed system is allowed to begin. Since the reconnecting member does not know which members survived the network partition event, all members that are in a reconnecting state will keep their UDP unicast ports open and respond to ping requests. - -Membership quorum is determined using the same member weighting system used in network partition detection. See [Membership Coordinators, Lead Members and Member Weighting](../network_partitioning/membership_coordinators_lead_members_and_weighting.html#concept_23C2606D59754106AFBFE17515DF4330). - -Note that when a locator is in the reconnecting state, it provides no discovery services for the distributed system. - -After the cache has reconnected, applications must fetch a reference to the new Cache, Regions, DistributedSystem and other artifacts. Old references will continue to throw cancellation exceptions like `CacheClosedException(cause=ForcedDisconnectException)`. - -See the Geode `DistributedSystem` and `Cache` Java API documentation for more information. - -## Managing the Autoreconnection Process - -By default a Geode member will try to reconnect until it is told to stop by using the `DistributedSystem.stopReconnecting()` or `Cache.stopReconnecting()` method. You can disable automatic reconnection entirely by setting `disable-auto-reconnect` Geode property to "true." - -You can use `DistributedSystem` and `Cache` callback methods to perform actions during the reconnect process, or to cancel the reconnect process if necessary. - -The `DistributedSystem` and `Cache` API provide several methods you can use to take actions while a member is reconnecting to the distributed system: - -- `DistributedSystem.isReconnecting()` returns true if the member is in the process of reconnecting and recreating the cache after having been removed from the system by other members. -- `DistributedSystem.waitUntilReconnected(long, TimeUnit)` waits for a period of time, and then returns a boolean value to indicate whether the member has reconnected to the DistributedSystem. Use a value of -1 seconds to wait indefinitely until the reconnect completes or the member shuts down. Use a value of 0 seconds as a quick probe to determine if the member has reconnected. -- `DistributedSystem.getReconnectedSystem()` returns the reconnected DistributedSystem. -- `DistributedSystem.stopReconnecting()` stops the reconnection process and ensures that the DistributedSystem stays in a disconnected state. -- `Cache.isReconnecting()` returns true if the cache is attempting to reconnect to a distributed system. -- `Cache.waitForReconnect(long, TimeUnit)` waits for a period of time, and then returns a boolean value to indicate whether the DistributedSystem has reconnected. Use a value of -1 seconds to wait indefinitely until the reconnect completes or the cache shuts down. Use a value of 0 seconds as a quick probe to determine if the member has reconnected. -- `Cache.getReconnectedCache()` returns the reconnected Cache. -- `Cache.stopReconnecting()` stops the reconnection process and ensures that the DistributedSystem stays in a disconnected state. - -## Operator Intervention - -You may need to intervene in the autoreconnection process if processes or hardware have crashed or are otherwise shut down before the network connection is healed. In this case the members in a "reconnecting" state will not be able to find the lost processes through UDP probes and will not rejoin the system until they are able to contact a locator. - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/book_intro.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/book_intro.html.md.erb b/geode-docs/managing/book_intro.html.md.erb deleted file mode 100644 index 568203a..0000000 --- a/geode-docs/managing/book_intro.html.md.erb +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: Managing Apache Geode ---- - -*Managing Apache Geode* describes how to plan and implement tasks associated with managing, monitoring, and troubleshooting Apache Geode. - -- **[Apache Geode Management and Monitoring](../managing/management/management_and_monitoring.html)** - - Apache Geode provides APIs and tools for managing your distributed system and monitoring the health of your distributed system members. - -- **[Managing Heap and Off-heap Memory](../managing/heap_use/heap_management.html)** - - By default, Apache Geode uses the JVM heap. Apache Geode also offers an option to store data off heap. This section describes how to manage heap and off-heap memory to best support your application. - -- **[Disk Storage](../managing/disk_storage/chapter_overview.html)** - - With Apache Geode disk stores, you can persist data to disk as a backup to your in-memory copy and overflow data to disk when memory use gets too high. - -- **[Cache and Region Snapshots](../managing/cache_snapshots/chapter_overview.html)** - - Snapshots allow you to save region data and reload it later. A typical use case is loading data from one environment into another, such as capturing data from a production system and moving it into a smaller QA or development system. - -- **[Region Compression](../managing/region_compression/region_compression.html)** - - This section describes region compression, its benefits and usage. - -- **[Network Partitioning](../managing/network_partitioning/chapter_overview.html)** - - Apache Geode architecture and management features help detect and resolve network partition problems. - -- **[Security](../managing/security/chapter_overview.html)** - - The security framework establishes trust by authenticating components - and members upon connection. It facilitates the authorization of operations. - -- **[Performance Tuning and Configuration](../managing/monitor_tune/chapter_overview.html)** - - A collection of tools and controls allow you to monitor and adjust Apache Geode performance. - -- **[Logging](../managing/logging/logging.html)** - - Comprehensive logging messages help you confirm system configuration and debug problems in configuration and code. - -- **[Statistics](../managing/statistics/chapter_overview.html)** - - Every application and server in a distributed system can access statistical data about Apache Geode operations. You can configure the gathering of statistics by using the `alter runtime` command of `gfsh` or in the `gemfire.properties` file to facilitate system analysis and troubleshooting. - -- **[Troubleshooting and System Recovery](../managing/troubleshooting/chapter_overview.html)** - - This section provides strategies for handling common errors and failure situations. - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/cache_snapshots/chapter_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/cache_snapshots/chapter_overview.html.md.erb b/geode-docs/managing/cache_snapshots/chapter_overview.html.md.erb deleted file mode 100644 index 38b4251..0000000 --- a/geode-docs/managing/cache_snapshots/chapter_overview.html.md.erb +++ /dev/null @@ -1,34 +0,0 @@ ---- -title: Cache and Region Snapshots ---- - -Snapshots allow you to save region data and reload it later. A typical use case is loading data from one environment into another, such as capturing data from a production system and moving it into a smaller QA or development system. - -In effect, you can load data from one distributed system into another distributed system. Administrators export a snapshot of a region or an entire cache (multiple regions) and later import the snapshot into another region or distributed system by using the `RegionSnapshotService` or `CacheSnapshotService` interface and the `Region.getSnapshotService` or `Cache.getSnapshotService` method. - -The snapshot file is a binary file that contains all data from a particular region. The binary format contains serialized key/value pairs and supports PDX type registry to allow the deserialization of PDX data. The snapshot can be directly imported into a region or read entry-by-entry for further processing or transformation into other formats. - -**Note:** -The previous `Region.loadSnapshot` and `Region.saveSnapshot` APIs have been deprecated. Data written in this format is not compatible with the new APIs. - -- **[Usage and Performance Notes](../../managing/cache_snapshots/using_cache_and_region_snapshots.html)** - - Optimize the cache and region snapshot feature by understanding how it performs. - -- **[Exporting Cache and Region Snapshots](../../managing/cache_snapshots/exporting_a_snapshot.html)** - - To save Geode cache or region data to a snapshot that you can later load into another distributed system or region, use the `cache.getSnapshotService.save` API, `region.getSnapshotService.save` API, or the `gfsh` command-line interface (`export data`). - -- **[Importing Cache and Region Snapshots](../../managing/cache_snapshots/importing_a_snapshot.html)** - - To import a Geode cache or region data snapshot that you previously exported into another distributed system or region, use the `cache.getSnapshotService.load` API, `region.getSnapshotService.load` API, or the `gfsh` command-line interface (`import data`). - -- **[Filtering Entries During Import or Export](../../managing/cache_snapshots/filtering_snapshot_entries.html)** - - You can customize your snapshot by filtering entries during the import or export of a region or a cache. - -- **[Reading Snapshots Programmatically](../../managing/cache_snapshots/read_snapshots_programmatically.html)** - - You can read a snapshot entry-by-entry for further processing or transformation into other formats. - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/cache_snapshots/exporting_a_snapshot.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/cache_snapshots/exporting_a_snapshot.html.md.erb b/geode-docs/managing/cache_snapshots/exporting_a_snapshot.html.md.erb deleted file mode 100644 index 1cd2065..0000000 --- a/geode-docs/managing/cache_snapshots/exporting_a_snapshot.html.md.erb +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Exporting Cache and Region Snapshots ---- - -To save Geode cache or region data to a snapshot that you can later load into another distributed system or region, use the `cache.getSnapshotService.save` API, `region.getSnapshotService.save` API, or the `gfsh` command-line interface (`export data`). - -If an error occurs during export, the export halts and the snapshot operation is canceled. Typical errors that halt an export include scenarios such as full disk, problems with file permissions, and network partitioning. - -## Exporting Cache Snapshots - -When you export an entire cache, it exports all regions in the cache as individual snapshot files into a directory. If no directory is specified, the default is the current directory. A snapshot file is created for each region, and the export operation automatically names each snapshot filename using the following convention: - -`snapshot-[-]*` - -When the export operation writes the snapshot filename, it replaces each forward slash ('/') in the region path with a dash ('-'). - -**Using Java API:** - -``` pre -File mySnapshotDir = ... -Cache cache = ... - -cache.getSnapshotService().save(mySnapshotDir, SnapshotFormat.GEMFIRE); -``` - -Optionally, you can set a filter on the snapshot entries during the export. See [Filtering Entries During Import or Export](filtering_snapshot_entries.html#concept_7E1F89E70A25465EBEBA584FADFEF353) for an example. - -## Exporting a Region Snapshot - -You can also export a specific region using the following API or gfsh command: - -**Java API:** - -``` pre -File mySnapshot = ... -Region region = ... - -region.getSnapshotService().save(mySnapshot, SnapshotFormat.GEMFIRE); -``` - -**gfsh:** - -Open a gfsh prompt. After connecting to a Geode distributed system, at the prompt type: - -``` pre -gfsh>export data --region=Region --file=filename.gfd ---member=membername -``` - -where *Region* corresponds to the name of the region that you want to export, *filename* (must end in .gfd) corresponds to the name of the export file and *membername* corresponds to a member where the region to export is hosted. For example: - -``` pre -gfsh>export data --region=region1 --file=region1_2012_10_10.gfd ---member=server1 -``` - -The snapshot file will be written on the remote member at the location specified by the `--file` argument. For example, in the example command above, the `region1_2012_10_10.gfd` file will be written in the working directory of `server1`. For more information on this command, see [export data](../../tools_modules/gfsh/command-pages/export.html#topic_263B70069BFC4A7185F86B3272011734). http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/cache_snapshots/filtering_snapshot_entries.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/cache_snapshots/filtering_snapshot_entries.html.md.erb b/geode-docs/managing/cache_snapshots/filtering_snapshot_entries.html.md.erb deleted file mode 100644 index 27b87c7..0000000 --- a/geode-docs/managing/cache_snapshots/filtering_snapshot_entries.html.md.erb +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: Filtering Entries During Import or Export ---- - -You can customize your snapshot by filtering entries during the import or export of a region or a cache. - -For example, use filters to limit the export of data to a certain date range. If you set up a filter on the import or export of a cache, the filter is applied to every single region in the cache. - -The following example filters snapshot data by even numbered keys. - -``` pre -File mySnapshot = ... -Region region = ... - -SnapshotFilter even = new SnapshotFilter() { - @Override - public boolean accept(Entry entry) { - return entry.getKey() % 2 == 0; - } -}; - -RegionSnapshotService snapsrv = region.getSnapshotService(); -SnapshotOptions options = rss.createOptions().setFilter(even); - -// only save cache entries with an even key -snapsrv.save(mySnapshot, SnapshotFormat.GEMFIRE, options); -``` - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/cache_snapshots/importing_a_snapshot.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/cache_snapshots/importing_a_snapshot.html.md.erb b/geode-docs/managing/cache_snapshots/importing_a_snapshot.html.md.erb deleted file mode 100644 index 626e0df..0000000 --- a/geode-docs/managing/cache_snapshots/importing_a_snapshot.html.md.erb +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: Importing Cache and Region Snapshots ---- - -To import a Geode cache or region data snapshot that you previously exported into another distributed system or region, use the `cache.getSnapshotService.load` API, `region.getSnapshotService.load` API, or the `gfsh` command-line interface (`import data`). - -## Import Requirements - -Before you import a region snapshot: - -- Make sure the cache is configured correctly. Configure all registered PdxSerializers, DataSerializers, and Instantiators; create regions; and ensure the classpath contains any required classes. -- When you import a snapshot containing PDX types, you must wait until the exported type definitions are imported into the cache before inserting data that causes type conflicts. It is recommended that you wait for the import to complete before inserting data. - -## Import Limitations - -During an import, the `CacheWriter` and `CacheListener` callbacks are not invoked. - -If an error occurs during import, the import is halted and the region will contain some but not all snapshot data. - -The state of a cache client is indeterminate after an import. It is likely that the data in the client's cache is inconsistent with the imported data. Take the client offline during the import and restart it after the import completes. - -## Importing Cache Snapshots - -When you import a cache snapshot, the snapshot file is imported into the same region (match determined by name) that was used during snapshot export. When you import a cache, you import all snapshot files located within a directory into the cache. The API attempts to load all files in the specified directory. - -**Java API:** - -``` pre -File mySnapshotDir = ... -Cache cache = ... - -cache.getSnapshotService().load(mySnapshotDir, SnapshotFormat.GEMFIRE); -``` - -## Importing a Region Snapshot - -**Java API:** - -``` pre -File mySnapshot = ... -Region region = ... - -region.getSnapshotService().load(mySnapshot, SnapshotFormat.GEMFIRE); -``` - -**gfsh:** - -Open a gfsh prompt. After connecting to a Geode distributed system, at the prompt type: - -``` pre -gfsh>import data --region=Region --file=filename.gfd ---member=membername -``` - -where *Region* corresponds to the name of the region that you want to import data into; *filename* (must end in .gfd) corresponds to the name of the file to be imported; and *membername* corresponds to the member where the region to be imported is hosted. For example: - -``` pre -gfsh>import data --region=region1 --file=region1_2012_10_10.gfd ---member=server2 -``` - -The snapshot file must already reside on the specified member at the location specified in the `--file` argument before import. - -For more information on this command, see [import data](../../tools_modules/gfsh/command-pages/import.html#topic_jw2_2ld_2l). http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/cache_snapshots/read_snapshots_programmatically.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/cache_snapshots/read_snapshots_programmatically.html.md.erb b/geode-docs/managing/cache_snapshots/read_snapshots_programmatically.html.md.erb deleted file mode 100644 index 6d0d0ca..0000000 --- a/geode-docs/managing/cache_snapshots/read_snapshots_programmatically.html.md.erb +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: Reading Snapshots Programmatically ---- - -You can read a snapshot entry-by-entry for further processing or transformation into other formats. - -The following is an example of a snapshot reader that processes entries from a previously generated snapshot file. - -``` pre -File mySnapshot = ... -SnapshotIterator iter = SnapshotReader.read(mySnapshot); -try { - while (iter.hasNext()) { - Entry entry = iter.next(); - - String key = entry.getKey(); - MyObject value = entry.getValue(); - - System.out.println(key + " = " + value); - } -} finally { - iter.close(); -} -``` - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/cache_snapshots/using_cache_and_region_snapshots.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/cache_snapshots/using_cache_and_region_snapshots.html.md.erb b/geode-docs/managing/cache_snapshots/using_cache_and_region_snapshots.html.md.erb deleted file mode 100644 index ccf5382..0000000 --- a/geode-docs/managing/cache_snapshots/using_cache_and_region_snapshots.html.md.erb +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: Usage and Performance Notes ---- - -Optimize the cache and region snapshot feature by understanding how it performs. - -## Cache Consistency and Concurrent Operations - -Importing and exporting region data is an administrative operation, and certain simultaneous runtime conditions can cause the import or export operation to fail such as when you are rebalancing partitioned region buckets or experience a network partition event. This behavior is expected, and you should retry the operation. Redoing an export overwrites an incomplete snapshot file, and redoing an import updates partially imported data. - -The snapshot feature does not guarantee consistency. Concurrent cache operations during a snapshot import or export can cause data consistency issues. If snapshot consistency is important, we recommend that you take your application offline before export and import, to provide a quiet period ensures data consistency in your snapshot. - -For example, modifications to region entries during an export can result in a snapshot that contains some but not all updates. If entries { A, B } are updated to { A', B'} during the export, the snapshot can contain { A, B' } depending on the write order. Also, modifications to region entries during an import can cause lost updates in the cache. If the region contains entries { A, B } and the snapshot contains { A', B' }, concurrent updates { A\*, B\* } can result in the region containing { A\*, B' } after the import completes. - -The default behavior is to perform all I/O operations on the node where the snapshot operations are invoked. This will involve either collecting or dispersing data over the network if the region is a partitioned region. - -## Performance Considerations - -When using the data snapshot feature, be aware of the following performance considerations: - -- Importing and exporting cache or region snapshots causes additional CPU and network load. You may need to increase CPU capacity or network bandwidth depending on your applications and infrastructure. In addition, if you export regions that have been configured to overflow to disk, you may require additional disk I/O to perform the export. -- When exporting partitioned region data, allocate additional heap memory so the member performing the export can buffer data gathered from other cache members. Allocate at least 10MB per member to your heap in addition to whatever configuration is necessary to support your application or cache. - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/disk_storage/backup_restore_disk_store.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/disk_storage/backup_restore_disk_store.html.md.erb b/geode-docs/managing/disk_storage/backup_restore_disk_store.html.md.erb deleted file mode 100644 index 173cdfe..0000000 --- a/geode-docs/managing/disk_storage/backup_restore_disk_store.html.md.erb +++ /dev/null @@ -1,172 +0,0 @@ ---- -title: Creating Backups for System Recovery and Operational Management ---- - -A backup is a copy of persisted data from a disk store. A backup is used to restore the disk store to the state it was in when the backup was made. The appropriate back up and restore procedures differ based upon whether the distributed system is online or offline. An online system has currently running members. An offline system does not have any running members. - -- [Making a Backup While the System Is Online](backup_restore_disk_store.html#backup_restore_disk_store__section_63AB5917BF24432898A79DBE8E4071FF) -- [What a Full Online Backup Saves](backup_restore_disk_store.html#backup_restore_disk_store__section_C08E52E65DAD4CD5AE076BBDCF1DB340) -- [What an Incremental Online Backup Saves](backup_restore_disk_store.html#backup_restore_disk_store__section_59E23EEA4AB24374A45B99A8B44FD49B) -- [Disk Store Backup Directory Structure and Contents](backup_restore_disk_store.html#backup_restore_disk_store__section_22809A237A344015B40C962B704D8F34) -- [Offline Members—Manual Catch-Up to an Online Backup](backup_restore_disk_store.html#backup_restore_disk_store__section_6F998080AF7640D1A9E951D155A75E3A) -- [Restore Using a Backup Made While the System Was Online](backup_restore_disk_store.html#backup_restore_disk_store__section_D08DC489B9D947DE97B8F96261E4A977) - -## Making a Backup While the System Is Online - -The gfsh command `backup disk-store` creates a backup of the disk stores for all members running in the distributed system. The backup works by passing commands to the running system members; therefore, the members need to be online for this operation to succeed. Each member with persistent data creates a backup of its own configuration and disk stores. The backup does not block any activities within the distributed system, but it does use resources. - -**Note:** -Do not try to create backup files from a running system by using your operating system's file copy commands. This would create incomplete and unusable copies. - -**Preparing to Make a Backup** - -- Consider compacting your disk store before making a backup. If auto-compaction is turned off, you may want to do a manual compaction to save on the quantity of data copied over the network by the backup. For more information on configuring a manual compaction, see [Manual Compaction](compacting_disk_stores.html#compacting_disk_stores__li_63CF8C35153D4173AADF7DC35FEC61F9). -- Run the backup during a period of low activity in your system. The backup does not block system activities, but it uses file system resources on all hosts in your distributed system, and it can affect performance. -- Configure each member with any additional files or directories to be backed up by modifying the member's `cache.xml` file. Additional items that ought to be included in the backup: - - - application jar files - - other files that the application needs when starting, such as a file that sets the classpath - - For example, to include file `myExtraBackupStuff` in the backup, the `cache.xml` file specification of the data store would include: - - ``` pre - ./myExtraBackupStuff - ``` - - Directories are recursively copied, with any disk stores that are found excluded from this user-specified backup. - -- Back up to a SAN (recommended) or to a directory that all members can access. Make sure the directory exists and has the proper permissions for all members to write to the directory and create subdirectories. - - The directory specified for the backup can be used multiple times. Each time a backup is made, a new subdirectory is created within the specified directory, and that new subdirectory's name represents the date and time. - - You can use one of two locations for the backup: - - - a single physical location, such as a network file server, for example: - - ``` pre - /export/fileServerDirectory/gemfireBackupLocation - ``` - - - a directory that is local to all host machines in the system, for example: - - ``` pre - ./gemfireBackupLocation - ``` - -- Make sure all members with persistent data are running in the system, because offline members cannot back up their disk stores. Output from the backup command will not identify members hosting replicated regions that are offline. - -**How to Do a Full Online Backup** - -1. If auto-compaction is disabled, and manual compaction is needed: - - ``` pre - gfsh>compact disk-store --name=Disk1 - ``` - -2. Run the `gfsh backup disk-store` command, specifying the backup directory location. For example: - - ``` pre - gfsh>backup disk-store --dir=/export/fileServerDirectory/gemfireBackupLocation - ``` - - The output will list information for each member that has successfully backed up disk stores. The tabular information will contain the member's name, its UUID, the directory backed up, and the host name of the member. - - Any online member that fails to complete its backup will leave a file named `INCOMPLETE_BACKUP` in its highest level backup directory. The existence of this file identifies that the backup file contains only a partial backup, and it cannot be used in a restore operation. - -3. Validate the backup for later recovery use. On the command line, each backup can be checked with commands such as - - ``` pre - cd 2010-04-10-11-35/straw_14871_53406_34322/diskstores/ds1 - gfsh validate offline-disk-store --name=ds1 --disk-dirs=/home/dsmith/dir1 - ``` - -**How to Do an Incremental Backup** - -An incremental backup contains items that have changed since a previous backup was made. - -To do an incremental backup, specify the backup directory that the incremental backup will be based upon with the `--baseline-dir` argument. For example: - -``` pre -gfsh>backup disk-store --dir=/export/fileServerDirectory/gemfireBackupLocation ---baseline-dir=/export/fileServerDirectory/gemfireBackupLocation/2012-10-01-12-30 -``` - -The output will appear the same as the output for a full online backup. - -Any online member that fails to complete its incremental backup will leave a file named `INCOMPLETE_BACKUP` in its highest level backup directory. The existence of this file identifies that the backup file contains only a partial backup, and it cannot be used in a restore operation. The next time a backup is made, a full backup will be made. - -## What a Full Online Backup Saves - -For each member with persistent data, a full backup includes the following: - -- Disk store files for all members containing persistent region data. -- Files and directories specified in the `cache.xml` configuration file as `` elements. For example: - - ``` pre - ./systemConfig/gf.jar - /users/user/gfSystemInfo/myCustomerConfig.doc - ``` - -- Deployed JAR files that were deployed using the gfsh [deploy](../../tools_modules/gfsh/command-pages/deploy.html) command. -- Configuration files from the member startup. - - `gemfire.properties`, including the properties with which the member was started. - - `cache.xml`, if used. - - These configuration files are not automatically restored, to avoid interfering with more recent configurations. In particular, if these are extracted from a master `jar` file, copying the separate files into your working area can override the files in the `jar`. If you want to back up and restore these files, add them as custom `` elements. -- A restore script, called `restore.bat` on Windows, and called `restore.sh` on Linux. This script may later be used to do a restore. The script copies files back to their original locations. - -## What an Incremental Online Backup Saves - -An incremental backup saves the difference between the last backup and the current data. An incremental backup copies only operations logs that are not already present in the baseline directories for each member. For incremental backups, the restore script contains explicit references to operation logs in one or more previously chained incremental backups. When the restore script is run from an incremental backup, it also restores the operation logs from previous incremental backups that are part of the backup chain. - -If members are missing from the baseline directory because they were offline or did not exist at the time of the baseline backup, those members place full backups of all their files into the incremental backup directory. - -## Disk Store Backup Directory Structure and Contents - -``` pre -$ cd thebackupdir -$ ls -R -./2012-10-18-13-44-53: -dasmith_e6410_server1_8623_v1_33892 dasmith_e6410_server2_8940_v2_45565 - -./2012-10-18-13-44-53/dasmith_e6410_server1_8623_v1_33892: -config diskstores README.txt restore.sh user - -./2012-10-18-13-44-53/dasmith_e6410_server1_8623_v1_33892/config: -cache.xml - -./2012-10-18-13-44-53/dasmith_e6410_server1_8623_v1_33892/diskstores: -DEFAULT - -./2012-10-18-13-44-53/dasmith_e6410_server1_8623_v1_33892/diskstores/DEFAULT: -dir0 - -./2012-10-18-13-44-53/dasmith_e6410_server1_8623_v1_33892/diskstores/DEFAULT/dir0: -BACKUPDEFAULT_1.crf BACKUPDEFAULT_1.drf BACKUPDEFAULT.if - -./2012-10-18-13-44-53/dasmith_e6410_server1_8623_v1_33892/user: -``` - -## Offline Members—Manual Catch-Up to an Online Backup - -If you must have a member offline during an online backup, you can manually back up its disk stores. Bring this member’s files into the online backup framework manually, and create a restore script by hand starting with a copy of another member’s script: - -1. Duplicate the directory structure of a backed up member for this member. -2. Rename directories as needed to reflect this member’s particular backup, including disk store names. -3. Clear out all files other than the restore script. -4. Copy in this member’s files. -5. Modify the restore script to work for this member. - -## Restore Using a Backup Made While the System Was Online - -The `restore.sh` or `restore.bat` script copies files back to their original locations. - -1. Restore your disk stores while cache members are offline and the system is down. -2. Look at each of the restore scripts to see where they will place the files and make sure the destination locations are ready. A restore script will refuse to copy over files with the same names. -3. Run each restore script on the host where the backup originated. - -The restore copies these files back to their original location: - -- Disk store files for all stores containing persistent region data. -- Any files or directories you have configured to be backed up in the `cache.xml` `` elements. - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/disk_storage/chapter_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/disk_storage/chapter_overview.html.md.erb b/geode-docs/managing/disk_storage/chapter_overview.html.md.erb deleted file mode 100644 index 70cba24..0000000 --- a/geode-docs/managing/disk_storage/chapter_overview.html.md.erb +++ /dev/null @@ -1,39 +0,0 @@ ---- -title: Disk Storage ---- - -With Apache Geode disk stores, you can persist data to disk as a backup to your in-memory copy and overflow data to disk when memory use gets too high. - -- **[How Disk Stores Work](../../managing/disk_storage/how_disk_stores_work.html)** - - Overflow and persistence use disk stores individually or together to store data. - -- **[Disk Store File Names and Extensions](../../managing/disk_storage/file_names_and_extensions.html)** - - Disk store files include store management files, access control files, and the operation log, or oplog, files, consisting of one file for deletions and another for all other operations. - -- **[Disk Store Operation Logs](../../managing/disk_storage/operation_logs.html)** - - At creation, each operation log is initialized at the disk store's `max-oplog-size`, with the size divided between the `crf` and `drf` files. When the oplog is closed, Apache Geode shrinks the files to the space used in each file. - -- **[Configuring Disk Stores](../../managing/disk_storage/overview_using_disk_stores.html)** - - In addition to the disk stores you specify, Apache Geode has a default disk store that it uses when disk use is configured with no disk store name specified. You can modify default disk store behavior. - -- **[Optimizing a System with Disk Stores](../../managing/disk_storage/optimize_availability_and_performance.html)** - - Optimize availability and performance by following the guidelines in this section. - -- **[Start Up and Shut Down with Disk Stores](../../managing/disk_storage/starting_system_with_disk_stores.html)** - - This section describes what happens during startup and shutdown and provides procedures for those operations. - -- **[Disk Store Management](../../managing/disk_storage/managing_disk_stores.html)** - - The `gfsh` command-line tool has a number of options for examining and managing your disk stores. The `gfsh` tool, the `cache.xml` file and the DiskStore APIs are your management tools for online and offline disk stores. - -- **[Creating Backups for System Recovery and Operational Management](../../managing/disk_storage/backup_restore_disk_store.html)** - - A backup is a copy of persisted data from a disk store. A backup is used to restore the disk store to the state it was in when the backup was made. The appropriate back up and restore procedures differ based upon whether the distributed system is online or offline. An online system has currently running members. An offline system does not have any running members. - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/disk_storage/compacting_disk_stores.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/disk_storage/compacting_disk_stores.html.md.erb b/geode-docs/managing/disk_storage/compacting_disk_stores.html.md.erb deleted file mode 100644 index 8357948..0000000 --- a/geode-docs/managing/disk_storage/compacting_disk_stores.html.md.erb +++ /dev/null @@ -1,116 +0,0 @@ ---- -title: Running Compaction on Disk Store Log Files ---- - - -When a cache operation is added to a disk store, any preexisting operation record for the same entry becomes obsolete, and Apache Geode marks it as garbage. For example, when you create an entry, the create operation is added to the store. If you update the entry later, the update operation is added and the create operation becomes garbage. Geode does not remove garbage records as it goes, but it tracks the percentage of garbage in each operation log, and provides mechanisms for removing garbage to compact your log files. - -Geode compacts an old operation log by copying all non-garbage records into the current log and discarding the old files. As with logging, oplogs are rolled as needed during compaction to stay within the max oplog setting. - -You can configure the system to automatically compact any closed operation log when its garbage content reaches a certain percentage. You can also manually request compaction for online and offline disk stores. For the online disk store, the current operation log is not available for compaction, no matter how much garbage it contains. - -## Log File Compaction for the Online Disk Store - -

- -Offline compaction runs essentially in the same way, but without the incoming cache operations. Also, because there is no current open log, the compaction creates a new one to get started. - -## Run Online Compaction - -Old log files become eligible for online compaction when their garbage content surpasses a configured percentage of the total file. A record is garbage when its operation is superseded by a more recent operation for the same object. During compaction, the non-garbage records are added to the current log along with new cache operations. Online compaction does not block current system operations. - -- **Automatic compaction**. When `auto-compact` is true, Geode automatically compacts each oplog when its garbage content surpasses the `compaction-threshold`. This takes cycles from your other operations, so you may want to disable this and only do manual compaction, to control the timing. -- **Manual compaction**. To run manual compaction: - - Set the disk store attribute `allow-force-compaction` to true. This causes Geode to maintain extra data about the files so it can compact on demand. This is disabled by default to save space. You can run manual online compaction at any time while the system is running. Oplogs eligible for compaction based on the `compaction-threshold` are compacted into the current oplog. - - Run manual compaction as needed. Geode has two types of manual compaction: - - Compact the logs for a single online disk store through the API, with the `forceCompaction` method. This method first rolls the oplogs and then compacts them. Example: - - ``` pre - myCache.getDiskStore("myDiskStore").forceCompaction(); - ``` - - - Using `gfsh`, compact a disk store in a distributed system with the [compact disk-store](../../tools_modules/gfsh/command-pages/compact.html#topic_F113C95C076F424E9AA8AC4F1F6324CC) command. Examples: - - ``` pre - gfsh>compact disk-store --name=Disk1 - - gfsh>compact disk-store --name=Disk1 --group=MemberGroup1,MemberGroup2 - ``` - - **Note:** - You need to be connected to a JMX Manager in `gfsh` to run this command. - -## Run Offline Compaction - -Offline compaction is a manual process. All log files are compacted as much as possible, regardless of how much garbage they hold. Offline compaction creates new log files for the compacted log records. - -Using `gfsh`, compact individual offline disk stores with the [compact offline-disk-store](../../tools_modules/gfsh/command-pages/compact.html#topic_9CCFCB2FA2154E16BD775439C8ABC8FB) command: - -``` pre -gfsh>compact offline-disk-store --name=Disk2 --disk-dirs=/Disks/Disk2 - -gfsh>compact offline-disk-store --name=Disk2 --disk-dirs=/Disks/Disk2 ---max-oplog-size=512 -J=-Xmx1024m -``` - -**Note:** -Do not perform offline compaction on the baseline directory of an incremental backup. - -You must provide all of the directories in the disk store. If no oplog max size is specified, Geode uses the system default. - -Offline compaction can take a lot of memory. If you get a `java.lang.OutOfMemory` error while running this, you may need to increase your heap size with the `-J=-Xmx` parameter. - -## Performance Benefits of Manual Compaction - -You can improve performance during busy times if you disable automatic compaction and run your own manual compaction during lighter system load or during downtimes. You could run the API call after your application performs a large set of data operations. You could run `compact disk-store` command every night when system use is very low. - -To follow a strategy like this, you need to set aside enough disk space to accommodate all non-compacted disk data. You might need to increase system monitoring to make sure you do not overrun your disk space. You may be able to run only offline compaction. If so, you can set `allow-force-compaction` to false and avoid storing the information required for manual online compaction. - -## Directory Size Limits - -Reaching directory size limits during has different results depending on whether you are running an automatic or manual compaction: - -- For automatic compaction, the system logs a warning, but does not stop. -- For manual compaction, the operation stops and returns a `DiskAccessException` to the calling process, reporting that the system has run out of disk space. - -## Example Compaction Run - -In this example offline compaction run listing, the disk store compaction had nothing to do in the `*_3.*` files, so they were left alone. The `*_4.*` files had garbage records, so the oplog from them was compacted into the new `*_5.*` files. - -``` pre -bash-2.05$ ls -ltra backupDirectory -total 28 --rw-rw-r-- 1 user users 3 Apr 7 14:56 BACKUPds1_3.drf --rw-rw-r-- 1 user users 25 Apr 7 14:56 BACKUPds1_3.crf -drwxrwxr-x 3 user users 1024 Apr 7 15:02 .. --rw-rw-r-- 1 user users 7085 Apr 7 15:06 BACKUPds1.if --rw-rw-r-- 1 user users 18 Apr 7 15:07 BACKUPds1_4.drf --rw-rw-r-- 1 user users 1070 Apr 7 15:07 BACKUPds1_4.crf -drwxrwxr-x 2 user users 512 Apr 7 15:07 . - -bash-2.05$ gfsh - -gfsh>validate offline-disk-store --name=ds1 --disk-dirs=backupDirectory - -/root: entryCount=6 -/partitioned_region entryCount=1 bucketCount=10 -Disk store contains 12 compactable records. -Total number of region entries in this disk store is: 7 - -gfsh>compact offline-disk-store --name=ds1 --disk-dirs=backupDirectory -Offline compaction removed 12 records. -Total number of region entries in this disk store is: 7 - -gfsh>exit - -bash-2.05$ ls -ltra backupDirectory -total 16 --rw-rw-r-- 1 user users 3 Apr 7 14:56 BACKUPds1_3.drf --rw-rw-r-- 1 user users 25 Apr 7 14:56 BACKUPds1_3.crf -drwxrwxr-x 3 user users 1024 Apr 7 15:02 .. --rw-rw-r-- 1 user users 0 Apr 7 15:08 BACKUPds1_5.drf --rw-rw-r-- 1 user users 638 Apr 7 15:08 BACKUPds1_5.crf --rw-rw-r-- 1 user users 2788 Apr 7 15:08 BACKUPds1.if -drwxrwxr-x 2 user users 512 Apr 7 15:09 . -bash-2.05$ -``` http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/disk_storage/disk_free_space_monitoring.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/disk_storage/disk_free_space_monitoring.html.md.erb b/geode-docs/managing/disk_storage/disk_free_space_monitoring.html.md.erb deleted file mode 100644 index b8df31b..0000000 --- a/geode-docs/managing/disk_storage/disk_free_space_monitoring.html.md.erb +++ /dev/null @@ -1,40 +0,0 @@ ---- -title: Configuring Disk Free Space Monitoring ---- - -To modify `disk-usage-warning-percentage` and `disk-usage-critical-percentage` thresholds, specify the parameters when executing the `gfsh create disk-store` command. - -``` pre -gfsh>create disk-store --name=serverOverflow --dir=c:\overflow_data#20480 \ ---compaction-threshold=40 --auto-compact=false --allow-force-compaction=true \ ---max-oplog-size=512 --queue-size=10000 --time-interval=15 --write-buffer-size=65536 \ ---disk-usage-warning-percentage=80 --disk-usage-critical-percentage=98 -``` - -By default, disk usage above 80% triggers a warning message. Disk usage above 99% generates an error and shuts down the member cache that accesses that disk store. To disable disk store monitoring, set the parameters to 0. - -To view the current threshold values set for an existing disk store, use the `gfsh describe` disk-store command: - -``` pre -gfsh>describe disk-store --member=server1 --name=DiskStore1 -``` - -You can also use the following `DiskStoreMXBean` method APIs to configure and obtain these thresholds programmatically. - -- `getDiskUsageCriticalPercentage` -- `getDiskUsageWarningPercentage` -- `setDiskUsageCriticalPercentage` -- `setDiskUsageWarningPercentage` - -You can obtain statistics on disk space usage and the performance of disk space monitoring by accessing the following statistics: - -- `diskSpace` -- `maximumSpace` -- `volumeSize` -- `volumeFreeSpace` -- `volumeFreeSpaceChecks` -- `volumeFreeSpaceTime` - -See [Disk Space Usage (DiskDirStatistics)](../../reference/statistics/statistics_list.html#section_6C2BECC63A83456190B029DEDB8F4BE3). - - http://git-wip-us.apache.org/repos/asf/geode-native/blob/ff80a931/geode-docs/managing/disk_storage/disk_store_configuration_params.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/disk_storage/disk_store_configuration_params.html.md.erb b/geode-docs/managing/disk_storage/disk_store_configuration_params.html.md.erb deleted file mode 100644 index 53ad195..0000000 --- a/geode-docs/managing/disk_storage/disk_store_configuration_params.html.md.erb +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: Disk Store Configuration Parameters ---- - -You define your disk stores by using the `gfsh create disk-store` command or in `` subelements of your cache declaration in `cache.xml`. All disk stores are available for use by all of your regions and queues. - -These `` attributes and subelements have corresponding `gfsh create disk-store` command-line parameters as well as getter and setter methods in the `org.apache.geode.cache.DiskStoreFactory` and `org.apache.geode.cache.DiskStore` APIs. - -## Disk Store Configuration Attributes and Elements - - ----- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

disk-store attribute	Description	Default
`name`	String used to identify this disk store. All regions and queues select their disk store by specifying this name.	DEFAULT
`allow-force-compaction`	Boolean indicating whether to allow manual compaction through the API or command-line tools.	false
`auto-compact`	Boolean indicating whether to automatically compact a file when it reaches the `compaction-threshold`.	true
`compaction-threshold`	Percentage of garbage allowed in the file before it is eligible for compaction. Garbage is created by entry destroys, entry updates, and region destroys and creates. Surpassing this percentage does not make compaction occur—it makes the file eligible to be compacted when a compaction is done.	50
`disk-usage-critical-percentage`	Disk usage above this threshold generates an error message and shuts down the member's cache. For example, if the threshold is set to 99%, then falling under 10 GB of free disk space on a 1 TB drive generates the error and shuts down the cache. - Set to "0" (zero) to disable.	99
`disk-usage-warning-percentage`	Disk usage above this threshold generates a warning message. For example, if the threshold is set to 90%, then on a 1 TB drive falling under 100 GB of free disk space generates the warning. - Set to "0" (zero) to disable.	90
`max-oplog-size`	The largest size, in megabytes, to allow an operation log to become before automatically rolling to a new file. This size is the combined sizes of the oplog files.	1024
`queue-size`	For asynchronous queueing. The maximum number of operations to allow into the write queue before automatically flushing the queue. Operations that would add entries to the queue block until the queue is flushed. A value of zero implies no size limit. Reaching this limit or the time-interval limit will cause the queue to flush.	0
`time-interval`	For asynchronous queueing. The number of milliseconds that can elapse before data is flushed to disk. Reaching this limit or the queue-size limit causes the queue to flush.	1000
`write-buffer-size`	Size of the buffer, in bytes, used to write to disk.	32768

- -| `disk-store` subelement | Description | Default | -|-------------------------|-----------------------------------------------------------------------------------------|------------------------| -| `` | Defines the system directories where the disk store is written and their maximum sizes. | `.` with no size limit | - -## disk-dirs Element - -The `` element defines the host system directories to use for the disk store. It contains one or more single `` elements with the following contents: - -- The directory specification, provided as the text of the `disk-dir` element. -- An optional `dir-size` attribute specifying the maximum amount of space, in megabytes, to use for the disk store in the directory. By default, there is no limit. The space used is calculated as the combined sizes of all oplog files. - -You can specify any number of `disk-dir` subelements to the `disk-dirs` element. The data is spread evenly among the active disk files in the directories, keeping within any limits you set. - -Example: - -``` pre - - /host1/users/gf/memberA_DStore - /host2/users/gf/memberA_DStore - /host3/users/gf/memberA_DStore - -``` - -**Note:** -The directories must exist when the disk store is created or the system throws an exception. Geode does not create directories. - -Use different disk-dir specifications for different disk stores. You cannot use the same directory for the same named disk store in two different members.