jackrabbit-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Jackrabbit Wiki] Update of "DataStore" by ThomasMueller
Date Tue, 08 Feb 2011 07:20:25 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Jackrabbit Wiki" for change notification.

The "DataStore" page has been changed by ThomasMueller.
http://wiki.apache.org/jackrabbit/DataStore?action=diff&rev1=63&rev2=64

--------------------------------------------------

  == Data Store ==
- 
  <<TableOfContents>>
  
  == Overview ==
- 
  The data store is optionally used to store large binary values. Normally all node and property
data is stored in a persistence manager, but for large binaries such as files special treatment
can improve performance and reduce disk usage.
  
- The main features of the data store are: 
+ The main features of the data store are:
  
   * Space saving: only one copy per unique object it kept
   * Fast copy: only the identifier is copied
@@ -20, +18 @@

   * Clustering: all cluster nodes use the same data store
  
  == Requirements ==
- 
  Jackrabbit 1.4 is required, it is not available in the previous releases.
  
- A Bundle persistence manager is required, or any other persistence manager that supports
data stores. The Simple``Db``Persistence``Manager and subclasses do not support the data store,
meaning large objects are still saved multiple times if it is used.
+ A Bundle persistence manager is required, or any other persistence manager that supports
data stores. The SimpleDbPersistenceManager and subclasses do not support the data store,
meaning large objects are still saved multiple times if it is used.
  
  The file system must supports files as large as the largest object you want to store. Please
note that the file size limit of FAT32 is 2 GB.
  
- 
  == How to Configure the File Data Store ==
- 
  To use the file based data store, add this to your repository.xml before the </Repository>:
  
  {{{
-     <DataStore class="org.apache.jackrabbit.core.data.FileDataStore"/> 
+     <DataStore class="org.apache.jackrabbit.core.data.FileDataStore"/>
  }}}
- 
  == File Data Store ==
- 
  The file data store stores each binary in a file. The file name is the hash code of the
content. When reading, the data is streamed directly from the file (no local or temporary
copy of the file is created). The file data store does not use any local cache, that means
content is directly read from the files as needed. New content is first stored in a temporary
file, and later renamed / moved to the right place.
  
- Because the data store is append-only, the File``Data``Store is guaranteed to be consistent
after a crash (unlike the Bundle``Fs``Persistence``Manager). It is usually faster than the
Db``Data``Store, and the preferred choice unless you have strict operational reasons to put
everything into a database.
+ Because the data store is append-only, the FileDataStore is guaranteed to be consistent
after a crash (unlike the BundleFsPersistenceManager). It is usually faster than the DbDataStore,
and the preferred choice unless you have strict operational reasons to put everything into
a database.
  
  === Configuration ===
- 
  This is a full configuration using the default values:
  
  {{{
@@ -52, +44 @@

          <param name="minRecordLength" value="100"/>
      </DataStore>
  }}}
- 
  All configuration options are optional:
+ 
   * path: The name of the directory where this data store keeps the files. The default is
/repository/datastore.
-  * minRecordLength: The minimum object length. The default is 100 bytes; smaller objects
are stored inline (not in the data store). There is a limitation on the minRecordLength: the
maximum value is around 32000. The reason for this is that Java doesn't support strings longer
than 64 KB in writeUTF.
+  * minRecordLength: The minimum object length. The default is 100 bytes; smaller objects
are stored inline (not in the data store). Using a low value means more objects are kept in
the data store (which may result in a smaller repository, if the same object is used in many
places). Using a high value means less objects are stored in the datastore (which may result
in better performance, because less datastore access is required). There is a limitation on
the minRecordLength: the maximum value is around 32000. The reason for this is that Java doesn't
support strings longer than 64 KB in writeUTF.
  
  == Database Data Store ==
- 
- The database data store stores data in a relational database. All content is stored in one
table, the unique key of the table is the hash code of the content. 
+ The database data store stores data in a relational database. All content is stored in one
table, the unique key of the table is the hash code of the content.
  
  When reading, the data may be first copied to a temporary file on the server, or streamed
directly from the database (depending on the copyWhenReading setting). New content is first
stored in the table under a unique temporary identifier, and later the key is updated to the
hash of the content.
  
  When adding a record, by default the stream is first copied to a temporary file. If you
get the exception "Can not insert new record java.io.IOException: No space left on device"
that means your temporary directory is too small. The reason for the temp file is: most databases
need to know the stream size when adding a record, and the JCR API doesn't provide a way to
do that. The mechanism used to add a record depends on the property "storeStream" in resource
org/apache/jackrabbit/core/data/db/<databaseType>.properties. Implemented mechanisms
are: "tempFile" (default; create a temp file before adding a record), "-1" (use the length
-1 when adding the record; currently only supported by the H2 database), and "max" (use the
length Integer.MAX_VALUE).
  
  === Configuration ===
- 
  Here is a possible configuration using the database data store:
  
  {{{
@@ -83, +73 @@

          <param name="schemaObjectPrefix" value=""/>
      </DataStore>
  }}}
- 
  The configuration options are:
+ 
   * url: The database URL (required).
   * user: The database user name (required).
   * password: The database password (required).
@@ -96, +86 @@

   * schemaObjectPrefix: The schema object prefix. The default is empty. Available from version
1.6+. Before Jackrabbit version 2.0, this setting is case sensitive (needs to be lowercase
for PostgreSQL and MySQL, and uppercase for other databases).
  
  === Limitations ===
- 
  MySQL does not support sending very large binaries from the JDBC driver to the database.
See also: http://bugs.mysql.com/bug.php?id=10859 and http://forums.mysql.com/read.php?10,258333,260405#msg-260405
.
  
  === Temporary Files ===
- 
  By default, the database data store creates temporary files that start with "dbRecord" in
the temp directory (the directory where the system property "java.io.tmpdir" points to). To
disable this behavior, set the configuration property "copyWhenReading" to "false" (see above).
  
  == FAQ ==
- 
  Q: Can I disable the data store? A: Only if there are no elements in the data store. If
there it is not empty, you need to copy the data to a new repository.
  
  Q: When I use the database data store I get the message: 'Table or view does not exists'.
A: Maybe the data store table already exists in another schema. When starting the repository,
the database data store checks if the table already exists (using a database meta data call),
and will create the table if not. If the table exists, but is in another schema, the table
is not created, but accessing it may fail (if the other schema is not in the schema search
path for this user).
  
- Q: What would happen when multiple users download large files from the store? A: Db``Data``Store:
It depends on whether the copyWhenReading option is enabled or not (see there). File``Data``Store:
The same file would be opened multiple times. The file would be in the operating system /
file system block cache, so that wouldn't be a problem.
+ Q: What would happen when multiple users download large files from the store? A: DbDataStore:
It depends on whether the copyWhenReading option is enabled or not (see there). FileDataStore:
The same file would be opened multiple times. The file would be in the operating system /
file system block cache, so that wouldn't be a problem.
  
  Q: Can I change the file content in the data store? A: No. The whole point of the data store
is that the content is immutable.
  
  Clustering is supported if you use a shared file system, such as SAN or NFS (Windows file
sharing works as well). You need to set data store path of all cluster nodes to the same location.
  
- Blob Store: When the data store is enabled, the blob store is not used. The data store solves
the same (and more) problems than the blob store. Therefore, the blob store is now deprecated,
however it will be supported for quite some time. 
+ Blob Store: When the data store is enabled, the blob store is not used. The data store solves
the same (and more) problems than the blob store. Therefore, the blob store is now deprecated,
however it will be supported for quite some time.
  
  Transaction: transactional semantics are guaranteed.
  
@@ -128, +115 @@

  Migration: currently there is no special mechanism to migrate data from a blob store to
a data store. You will have to convert the whole repository, see also BackupAndMigration.
  
  == How Does It Work ==
- 
  When adding a binary object, Jackrabbit checks the size of it. When it is larger than minRecordLength,
it is added to the data store, otherwise it is kept in-memory. This is done very early (possible
when calling Property.setValue(stream)). Only the unique data identifier is stored in the
persistence manager (except for in-memory objects, where the data is stored). When updating
a value, the old value is kept there (potentially becoming garbage) an the new value is added.
There is no update operation.
  
  The current implementation still stores temporary files in some situations, for example
in the RMI client. Those cases will be changed to use the data store directly where it makes
sense.
  
  Very small objects (where it does not make sense to create a file) are stored in the persistence
manager (in-place).
  
- Objects in the data store are only removed when they are not reachable (that means, objects
referenced in the cache or in memory are not collected). There is no 'update' operation, only
'add new entry'. Data is added before the transaction is committed. Additions are globally
atomic, cluster nodes can share the same data store. Even different repositories can share
the same store, as long as garbage collection is done correctly. 
+ Objects in the data store are only removed when they are not reachable (that means, objects
referenced in the cache or in memory are not collected). There is no 'update' operation, only
'add new entry'. Data is added before the transaction is committed. Additions are globally
atomic, cluster nodes can share the same data store. Even different repositories can share
the same store, as long as garbage collection is done correctly.
  
- Overview: 
+ Overview:
  
  {{attachment:DataStoreOverview.png}}
  
  Objects are usually stored early in the data store, even before the transaction is committed.
Only the the identifier is stored in the persistence manager. The blob store is not used any
longer (except for backward compatibility). When using the RMI client, large objects are not
stored directly in the data store, instead they are first transferred to the server.
  
  == API ==
- 
  The regular JCR API is used read and write entries in the data store.
  
  === Retrieve the Identifier ===
+ The identifier can help you locate the file in the data store backend;  however, creating
a value instance from a identifier is currently not supported. To get the identifier of a
binary value, use [[http://jackrabbit.apache.org/api/2.1/org/apache/jackrabbit/api/JackrabbitValue.html#getContentIdentity()|JackrabbitValue.getContentIdentity()]].
 Example:
- 
- The identifier can help you locate the file in the data store backend; 
- however, creating a value instance from a identifier is currently not supported.
- To get the identifier of a binary value, use [[http://jackrabbit.apache.org/api/2.1/org/apache/jackrabbit/api/JackrabbitValue.html#getContentIdentity()|JackrabbitValue.getContentIdentity()]].

- Example:
  
  {{{
  InputStream in;
@@ -163, +144 @@

      String id = jv.getContentIdentity();
  }
  }}}
- 
  == Data Store Garbage Collection ==
- 
  The data store never deletes entries except when running data store garbage collection.
Similar to Java heap garbage collection, data store garbage collection will first mark all
used entries, and later remove unused items.
  
  Data store garbage collection does not delete entries if the identifier is still in the
Java heap memory. To delete as many unreferenced entries as possible, call System.gc() a few
times before running the data store garbage collection. Please note System.gc() does not guarantee
all objects are garbage collected.
  
  === Running Data Store Garbage Collection (Jackrabbit 1.x) ===
- 
  Running the garbage collection is currently a manual process. You can run this as a separate
thread concurrently to your application:
  
  {{{
@@ -189, +167 @@

  // delete old data
  gc.deleteUnused();
  }}}
- 
  The process above applies to a standalone repository. When clustered, the garbage collection
can be run from any cluster node.
  
  If multiple distinct repositories use the same data store, the process is a bit different:
First, call gc.scan() on the first repository, then on the second and so on. At the end, call
gc.deleteUnused() on the first repository:
@@ -203, +180 @@

  gc2.stopScan();
  gc3.stopScan();
  }}}
- 
  An alternative is:
  
   1. Write down the current time = X
-  1. Run gc.scan() on each repository 
+  1. Run gc.scan() on each repository
   1. Manually delete files with last modified date older than X
  
- 
  === Running Data Store Garbage Collection (Jackrabbit 2.x) ===
- 
  Running the garbage collection is currently a manual process. You can run this as a separate
thread concurrently to your application:
  
  {{{
@@ -237, +211 @@

  session.logout();
  rm.stop();
  }}}
- 
  The process above applies to a standalone repository. When clustered, the garbage collection
can be run from any cluster node.
  
  If multiple distinct repositories use the same data store, the process is a bit different:
First, call gc.mark() on the first repository, then on the second and so on. At the end, call
gc.sweep() on the first repository:
@@ -251, +224 @@

  gc2.close();
  gc3.close();
  }}}
- 
  An alternative is:
  
   1. Write down the current time = X
-  1. Run gc.mark() on each repository 
+  1. Run gc.mark() on each repository
   1. Manually delete files with last modified date older than X
  
- 
  == How to Write a New Data Store Implementation ==
- 
  New implementations are welcome! Cool would be a S3 data store (http://en.wikipedia.org/wiki/Amazon_S3).
A caching data store would be great as well (items that are used a lot are stored in fast
file system, others in a slower one).
  
  == Future Ideas ==
- 
  Theoretically the data store could be split to different directories / hard drives. Currently
this can be done manually moving directories to different disks and by creating softlinks.
Content that is accessed more often could be moved to a faster disk, and less used data could
eventually be moved to slower / cheaper disk. That would be an extension of the 'memory hierarchy'
(see also http://en.wikipedia.org/wiki/Memory_hierarchy). Of course this wouldn't limit the
space used per workspace, but would improve system performance if done right. Maybe we need
to do that anyway in the near future to better support solid state disk.
  
  Other feature requests:
+ 
   * A replicating data store
-  * Currently the File``Data``Store creates a lot of directories (and files). If possible
the number of directories (and maybe files) should be reduced to improve performance. 
+  * Currently the FileDataStore creates a lot of directories (and files). If possible the
number of directories (and maybe files) should be reduced to improve performance.
-  * Fulltext search and meta data extraction could be done when storing the object (only
once per object) and stored next to the object. 
+  * Fulltext search and meta data extraction could be done when storing the object (only
once per object) and stored next to the object.
   * Client should first send the checksum and size of large objects when they store something
(import, adding or updating data), in many cases the actual data does not need to be sent.
   * Speed up garbage collection. One idea is to use 'back references' for larger objects:
each larger object would know the set of nodes that reference it. This would be an 'append
only' set, that means at runtime links only added, not removed. Only the garbage collection
process removes links. The garbage collection would first update links for large objects (this
process could stop at the first link that still exists). That way large objects can be removed
quickly if they are not used any more. Afterwards, objects with a low use count should be
scanned. This algorithm wouldn't necessarily speed up the total garbage collection time, but
it would free up space more quickly.
   * Auto-Compressing Datastore (file, db) - if a specific file's content type and size make
it likely to have large disk space savings if compressed, set the datastore to auto-compress
(whether zip, gzip, bz, etc.).  File datastore is more likely to have this feature than DB,
and should not impact retrieval or other normal usage.  (user added 7/17/2009)

Mime
View raw message