Intervall and Time Budget

Housekeeping interval and time budget is configured by setting up a StorageHousekeepingController.

Available properties are:

• House keeping interval interval the housekeeping is triggered in milliseconds, default once per every second

• House keeping budget time budget for housekeeping in nanoseconds, default is 0.01 seconds

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setHousekeepingController(Storage.HousekeepingController(1000, 10_000_000))
		.createConfiguration())
	.start();

File Sizes and Payload

The desired file min and max sizes and payload ratio is configured by the StorageDataFileEvaluator:

available properties are:

• Data file minimum size Files smaller then minimum file size will be merged with other files if possible, default is 1 MB.

• Data file maximum size Files larger then maximum file size will be split in smaller ones, default is 8 MB.

• Data file minimum use ratio Ratio of non-gap data contained in a storage file to prevent the file from being dissolved, default is 0.75 (75%).

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setDataFileEvaluator(Storage.DataFileEvaluator(1024*1024, 1024*1024*8, 0.75))
		.createConfiguration())
	.start();

Cache

The lifetime of objects in the internal entity cache can be configured by the StorageEntityCacheEvaluator:

Available properties are:

• Entity cache threshold

  Abstract threshold value, roughly comparable to size in bytes with a time component, at which a cache must be cleared of some entities. Default is 1000000000.

• Entity cache timeout Time in milliseconds after that an entity is considered to be old if not read meanwhile. must be greater zero, default is 86400000ms (1 day).

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setEntityCacheEvaluator(Storage.EntityCacheEvaluator(
			86_400_000, 
			1_000_000_000))
		.createConfiguration())
	.start();

Using a Storage File Provider (one.microstream.storage.types.StorageFileProvider) allows to specify the location and naming rules for all storage related files.

available properties are:

• BaseDirectory The Microstream storages location base directory. Contains channel directories and type dictionary file.

• DeletionDirectory If configured, the storage will not delete files. Instead of deleting a file it will be moved to this directory.

• TruncationDirectory If configured, files that will get truncated are copied into this directory.

• ChannelDirectoryPrefix Channel directory prefix string

• StorageFilePrefix Storage file prefix string

• StorageFileSuffix storage file extension

• TransactionsFilePrefix transactions file prefix

• TransactionsFileSuffix transaction file extension

• TypeDictionaryFileName filename of the type dictionary

StorageFileProvider fileProvider = Storage.FileProviderBuilder()
	.setBaseDirectory         (WORKINGDIR.getPath())
	.setDeletionDirectory     (DELETIONDIR.getPath())
	.setTruncationDirectory   (TRUNKATIONDIR.getPath())
	.setChannelDirectoryPrefix("canal_")
	.setStorageFilePrefix     ("canal_")
	.setStorageFileSuffix     (".bin")
	.setTransactionsFilePrefix("events_")
	.setTransactionsFileSuffix(".bin")
	.setTypeDictionaryFileName("typeDictionary.txt")
	.createFileProvider();

Channel Usage

Channels are the IO threads used by the MicroStream storage engine. A single channel represents the unity of a thread, a storage directory and cached data. Increasing the number of channels means to run more IO threads.

The channel count is an important configuration value that impacts to IO performance.

Channel Configuration

For the channel configuration the following configuration properties are available:

• Channel count

  The number of channels that MicroStream will use. Must be $2^n$

• Channel directory prefix

  The channel directory will be prefix+channelNumber e.g. "ch_0" if prefix is "ch_"

• Data file prefix default is "channel_"

• Data file suffix deflaut id ".dat"

Channel file size configuration is done by the the Storage Data File Evaluator.

They can be set By storage.embedded.configuration API:

embeddedStorageManager storageManager = Configuration.Default()
	.setChannelCount(4)
	.setChannelDirectoryPrefix("channel_")
	.setDataFilePrefix("channel_")
	.setDataFileSuffix(".bin")
	.createEmbeddedStorageFoundation()
	.createEmbeddedStorageManager();

<properties>
	<property name="channelCount" value="4" />
	<property name="channelDirectoryPrefix" value="channel_" />
	<property name="dataFilePrefix value="channel_" />
	<property name="dataFileSuffix" value=".dat" />
</properties>

channelCount = 4
channelDirectoryPrefix = prefix
dataFilePrefix = channel_
dataFileSuffix = .dat

See also: Configuration

Or by setting a StorageFileProvider using theEmbeddedStorageFoundationfactory

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setChannelCountProvider(Storage.ChannelCountProvider(4))
		.setStorageFileProvider(StorageFileProvider.Builder()
			.setBaseDirectory("storage")
			.setChannelDirectoryPrefix("prefix")
			.setStorageFilePrefix("channel-")
        	.setStorageFileSuffix(".bin")
			.createFileProvider()).createConfiguration()
		).start();

The EmbeddedStorageManager is mostly created with factory methods of EmbeddedStorage, where the most common settings, like database directory or the root instance, can be configured.

EmbeddedStorageManager storageManager = EmbeddedStorage.start(
    myRoot,                 // root object of entity graph
    Paths.get("data-dir")    // storage data directory
);

Foundations

To achieve a more detailed customization, you can utilize the EmbeddedStorageFoundation factory type. It holds and creates on demand all the parts that form an EmbeddedStorageManager.

EmbeddedStorageManager storageManager = EmbeddedStorageFoundation.New()
    .setConfiguration(
        StorageConfiguration.Builder()
            .setChannelCountProvider(StorageChannelCountProvider.New(4))
            .setBackupSetup(StorageBackupSetup.New(Paths.get("backupdir")))
            .createConfiguration()
    )
    .createEmbeddedStorageManager();

External Configuration

The artifact storage.embedded.configuration provides a convenience layer for configuration purposes, as well as facilities to read external configuration.

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded.configuration</artifactId>
        <version>03.00.00-MS-GA</version>
    </dependency>
</dependencies>

The Configuration type consolidates the most widely used parameters from the storage foundations in one place. It's output is an EmbeddedStorageFoundation from which a EmbeddedStorageManager can be created.

EmbeddedStorageManager storageManager = Configuration.Default()
    .setBaseDirectoryInUserHome("data-dir")
    .setBackupDirectory("backup-dir")
    .setChannelCount(4)
    .createEmbeddedStorageFoundation()
    .createEmbeddedStorageManager();

To read an external configuration use ConfigurationLoader and ConfigurationParser or the Load*() methods of Configuration. Currently XML and INI files are supported.

Configuration configuration = Configuration.LoadXml(
    HelloWorld.class.getResource("/META-INF/microstream/storage.xml")
);

<?xml version="1.0" encoding="UTF-8"?>
<properties>
    <property name="baseDirectory" value ="data" />
    <property name="channelCount" value ="4" />
</properties>

Configuration configuration = Configuration.LoadIni(
    HelloWorld.class.getResource("/META-INF/microstream/storage.ini")
);

baseDirectory = data
channelCount = 4

If you just use Configuration.Load() the default configuration file is used, which is either a file in the classpath root named microstream-storage.properties, or the path configured via the system property microstream.storage.configuration.path.

Continuous Backup

By default, the continuous backup is disabled. If enabled the MicroStream instance will clone all changes to another directory. The backup is identical to the primary MicroStream storage.

To enable the continuous backup just set the backup directory:

With storage.embedded.configuration API:

EmbeddedStorageManager storageManager = Configuration.Default()
	.setBackupDirectory("A safe place")
	.createEmbeddedStorageFoundation()
	.createEmbeddedStorageManager();

<properties>
	<property name="backupDirectory" value ="/save/backup" />
	...
</properties>

backupDirectory = backupDir

With MicroStream foundation classes:

StorageBackupSetup backupSetup = StorageBackupSetup.New(
    Storage.FileProviderBuilder()
		.setBaseDirectory(BACKUPDIR.getPath())						
		.setTruncationDirectory(TRUNKATIONDIR.getPath())
		.setDeletionDirectory(DELETIONDIR.getPath())
		.createFileProvider()
	);	
				
StorageConfiguration configuration = StorageConfiguration.Builder()
	.setBackupSetup(backupSetup)
	.setStorageFileProvider(StorageFileProvider.New(WORKINGDIR))
	.createConfiguration();

These are the available properties of the Configuration type. The names are used accordingly in the external configuration files. They can be found as constants in ConfigurationPropertyNames.

Detailed Description

channelCount

Number of threads used by the storage engine. It depicts the numbers of subdirectories as well. Each thread manages one directory in which it writes to and reads from exclusively. The unity of thread, directory and the cached data therefor is called a "Channel".

__________________________________________________
                               [RAM ]{ Code }    |    (      Filesystem      )
               ,- "Channel 0": [Data]{Thread} <-I|O-> (Storage Subdirectory 0)
              /-- "Channel 1": [Data]{Thread} <-I|O-> (Storage Subdirectory 1)
StorageManager                                   |
              \-- "Channel 2": [Data]{Thread} <-I|O-> (Storage Subdirectory 2)
               '- "Channel 3": [Data]{Thread} <-I|O-> (Storage Subdirectory 3)
_________________________________________________|

houseKeepingIntervalMs

Number of milliseconds for the house keeping interval. House keeping tasks are, among others:

• Garbage Collection

• Cache Check

• File Cleanup Check

In combination with houseKeepingNanoTimeBudget, it can be specified how many CPU time should be used for house keeping. E.g. interval=1000ms and budget=10000000ns means every second there's 0.01 seconds time for house keeping, so max 1% CPU time used for house keeping. This CPU time window is only used if house keeping work is pending. If nothing has to be done, no time is wasted.

housekeepingTimeBudgetNs

Number of nanoseconds used for each housekeeping cycle. However, no matter how low the number is, one item of work will always be completed. But if there is nothing to clean up, no processor time will be wasted. Default is 10000000 (10 million nanoseconds = 10 milliseconds = 0.01 seconds). However, no matter how small the time is, one item is done at least. This is to avoid no-ops, if a too small time window is configured. This time budget is a "best effort" threshold, meaning when at 1ns left, a huge file has to be cleaned or the references of a huge collection have to be marked for GC, then this budget can be exceeded considerably.

dataFileMinimumSize

Minimum file size in bytes of a storage file to avoid merging with other files during housekeeping. Must be greater than 1, maximum is 2GB.

dataFileMaximumSize

Maximum file size in bytes of a storage file to avoid splitting in more files during housekeeping. Must be greater than 1, maximum is 2GB.

dataFileMinimumUseRatio

The ratio (value in ]0.0;1.0]) of non-gap data contained in a storage file to prevent the file from being dissolved. "Gap" data is anything that is not the latest version of an entity's data, inluding older versions of an entity and "comment" bytes (a sequence of bytes beginning with its length as a negative value length header). The closer this value is to 1.0 (100%), the less disk space is occupied by storage files, but the more file dissolving (data transfers to new files) is required and vice versa.

Involved Types

This list shows which property configures which type, used by the foundation types, to create the storage manager.

By default, MicroStream uses the operation system's standard file locking mechanism to prevent simultaneous access to the storage files. In the rare case that this is not sufficient to control the file access MicroStream provides a proprietary file lock implementation to ensure exclusive access to the storage files from different applications using MicroStream.

Using this file lock may only be necessary if, while a MicroStream application is running, a second MicroStream application may try to access the same storage and the default file locks are not reliable.

You don't need to activate this feature if:

• Only one MicroStream application will access the storage,

• MicroStream applications that may access the same storage run on the same system,

• other applications that may access the storage files don't use MicroStream to access them.

To activate the internal file lock you need to setup StorageLockFileSetup:

The default interval the locks are updated is 10 seconds, you can set a custom value in milliseconds with:

Storage.LockFileSetupProvider( final long updateInterval )

To specify the charset used by the lock files use:

Storage.LockFileSetupProvider( final Charset charset )

or, to customize both:

LockFileSetupProvider( final Charset charset , final long updateInterval )