Welcome to the MicroStream Reference Manual. This manual includes concepts, instructions and examples to guide you on how to use MicroStream Data-Store and Cache, version 4.0.

You should be familiar with the Java programming language and you should have installed your preferred Integrated Development Environment (IDE). But since you are here we guest you got that covered ;)

What's new in 4.0

API Docs

The API documentation is available at .

Support

For information on the commercial support for MicroStream see .

License

By default, MicroStream uses the operation system's standard file locking mechanism to prevent simultaneous access to the storage files. In the rare case that this is not sufficient to control the file access MicroStream provides a proprietary file lock implementation to ensure exclusive access to the storage files from different applications using MicroStream.

Using this file lock may only be necessary if, while a MicroStream application is running, a second MicroStream application may try to access the same storage and the default file locks are not reliable.

You don't need to activate this feature if:

• Only one MicroStream application will access the storage,

MicroStream Data-Store is a native Java object graph storage engine. From a technical point of view it serves one purpose only: To fully or partially persist and restore a Java object graph in the simplest way possible for the user.

MicroStream Data-Store is a storage engine, but no database management system (DBMS). Many features that typical DBMS provide have been left out on purpose. The reason is that those features exist to make a DBMS something of a server application platform of an "old kind" on top of its data store functionality: A standalone process with user management, connection management, session handling, often even with a programming language of its own, a querying interface (SQL), etc. Today, all of those server application features are already and much better handled by dedicated server applications (the "new kind"), implemented in a modern language like Java. They have their built-in user, connection and session management, the querying interface to the outside world are typically web services instead of SQL, etc. But those modern server applications still lack one important thing: an easy to use and technically efficient way to store and restore their application's data. So a "new kind" server often uses an "old kind" server just to do the data storing. This comes at the price of catching all the overhead and problems of redundant user, connection and session management AND the outdated concepts and limitations of the old querying interface (SQL). Isn't that very weird and frustratingly complicated? Why not simply include a modern data storing library in the modern server and be done with it? A storing library that perfectly fits the modern technology and brings in no redundant overhead or complication of a secondary outdated wannabe server process. This is exactly what MicroStream Data-Store is and the reason why it is intentionally not a DBMS but "only" a storage engine.

One might think the easiest way to store and load data in Java would be Java's built-in serialization. However, it turned out long ago to be very limited, making it hard, if not impossible, to be used as a replacement for a DBMS:

• Only complete object graphs can be stored and restored, which is unacceptable for all but very small databases.

• It is very inefficient in terms of storage size and performance.

• It does not handle changing class structures very well, basically forbidding classes of persisted entities to ever change or introducing massive manual effort to compensate.

In short: The Java Serialization is not an acceptable data store solution and hence no valid replacement for those outdated DBMS.

MicroStream Data-store is such a solution:

• It can persist, load or update object graphs partially and on-demand.

• It is very efficient both size- and performance-wise.

• It handles changing class structures by mapping data in the old structure to the current structure during loading; implicitly via internal heuristics or explicitly via a user-defined mapping strategy.

MicroStream is what the Java Serialization should have been and it is the first and only really fitting data storing solution for modern applications, completely removing the need to attach a wannabe secondary server DBMS just to store data.

Using a Storage Live File Provider (one.microstream.storage.types.StorageLiveFileProvider) allows to specify the location and naming rules for all storage related files.

available properties are:

• BaseDirectory The Microstream storages location base directory. Contains channel directories and type dictionary file.

• DeletionDirectory If configured, the storage will not delete files. Instead of deleting a file it will be moved to this directory.

• TruncationDirectory If configured, files that will get truncated are copied into this directory.

• ChannelDirectoryPrefix Channel directory prefix string

• StorageFilePrefix Storage file prefix string

• StorageFileSuffix storage file extension

• TransactionsFilePrefix transactions file prefix

• TransactionsFileSuffix transaction file extension

• TypeDictionaryFileName filename of the type dictionary

Continuous Backup

By default, the continuous backup is disabled. If enabled the MicroStream instance will clone all changes to another directory. The backup is identical to the primary MicroStream storage.

To enable the continuous backup just set the backup directory:

With storage.embedded.configuration API:

With MicroStream foundation classes:

The NIO connector can access the local or mounted file systems, as well as different in-memory file systems. This is probably the easiest way to start, especially for prototyping and testing purposes. For productive use, the other file systems are preferred.

NioFileSystem fileSystem = NioFileSystem.New();
EmbeddedStorage.start(fileSystem.ensureDirectoryPath("path", "to", "storage"));

MicroStream – Persistence – License Agreement Version 1, September 16, 2019, MicroStream Software GmbH

IMPORTANT: READ THE TERMS OF THIS AGREEMENT CAREFULLY BEFORE DOWNLOADING AND USING THE SOFTWARE. BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, OR USING THE SOFTWARE YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON BEHALF OF ANOTHER PERSON OR A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE FULL AUTHORITY TO BIND THAT PERSON, COMPANY, OR LEGAL ENTITY TO THESE TERMS. IF YOU DO NOT AGREE TO ALL THESE TERMS, YOU MUST NOT DOWNLOAD, INSTALL, COPY, ACCESS, OR USE THE SOFTWARE; AND YOU MUST PROMPTLY DESTROY ALL COPIES OF SOFTWARE AND ALL RELATED MATERIALS.

1. DEFINITIONS. "MicroStream Software GmbH" is a software developing firm based in Regensburg, Germany, hereafter referred to as "MicroStream" or "Licensor", www.microstream.one. "Licensor" refers either to an individual person or to a single legal entity. "Software" is the following, including the original and all whole or partial copies: (i) machine-readable instructions and data, (ii) components, (iii) audio-visual content (such as images, text, recordings, or pictures), (iv) related licensed materials, and (v) license use documents or keys, and documentation. "Agreement" refers to this MicroStream License Agreement.

Convenience Methods

Beside long store(Object instance) MicroStream provides some convenience methods to store several objects at once:

void storeAll(Iterable<?> instances)

Stores the passed instance in any case and all referenced instances of persistable references recursively, but stores referenced instances only if they are newly encountered (e.g. don't have an id associated with them in the object registry, yet and are therefore required to be handled). This is useful for the common case of just storing an updated instance and potentially newly created instances along with it while skipping all existing (and normally unchanged) referenced instances.

Intervall and Time Budget

Housekeeping interval and time budget is configured by setting up a StorageHousekeepingController.

Available properties are:

• interval the housekeeping is triggered in milliseconds, default once per every second

• time budget for housekeeping in nanoseconds, default is 0.01 seconds

File Sizes and Payload

The desired file min and max sizes and payload ratio is configured by the StorageDataFileEvaluator:

available properties are:

• Files smaller then minimum file size will be merged with other files if possible, default is 1 MB.

• Files larger then maximum file size will be split in smaller ones, default is 8 MB.

• Ratio of non-gap data contained in a storage file to prevent the file from being dissolved, default is 0.75 (75%).

Cache

The lifetime of objects in the internal entity cache can be configured by the StorageEntityCacheEvaluator:

Available properties are:

• Entity cache threshold

  Abstract threshold value, roughly comparable to size in bytes with a time component, at which a cache must be cleared of some entities. Default is 1000000000.

• Entity cache timeout Time in milliseconds after that an entity is considered to be old if not read meanwhile. must be greater zero, default is 86400000ms (1 day).

Channel Usage

Channels are the IO threads used by the MicroStream storage engine. A single channel represents the unity of a thread, a storage directory and cached data. Increasing the number of channels means to run more IO threads.

The channel count is an important configuration value that impacts to IO performance.

Channel Configuration

For the channel configuration the following configuration are available:

• Channel count

  The number of channels that MicroStream will use. Must be

• Channel directory prefix

  The channel directory will be prefix+channelNumber e.g. "ch_0" if prefix is "ch_"

Channel file size configuration is done by the the .

They can be set By storage.embedded.configuration API:

See also:

Or by setting a StorageFileProvider using theEmbeddedStorageFoundationfactory

MicroStream supports a variety of storage targets. Through an abstracted file system (AFS), it is possible to connect to a lot of different back ends. The AFS allows to use folders and files, like in all common file systems, but with different connectors it is possible to use different solutions as the actual storage.

To connect to the local file system use the Java Non-Blocking IO (NIO) connector, which is part of the base module, so no additional dependency is needed.

EmbeddedStorage.start(Paths.get("path", "to", "storage"));

Internally this creates and uses a NioFileSystem and is a shortcut for:

NioFileSystem fileSystem = NioFileSystem.New();
EmbeddedStorage.start(fileSystem.ensureDirectoryPath("path", "to", "storage"));

The file system API is the same for all connectors, like for MySQL. This is part of another module.

<!-- sql file system -->
<dependency>
       <groupId>one.microstream</groupId>
       <artifactId>filesystem.sql</artifactId>
       <version>04.00.00-MS-GA</version>
</dependency>
<!-- driver of your choice -->
<dependency>
       <groupId>mysql</groupId>
       <artifactId>mysql-connector-java</artifactId>
       <version>8.0.21</version>
</dependency>

// create JDBC data source
MysqlDataSource dataSource = new MysqlDataSource();
dataSource.setUrl("jdbc:mysql://host:3306/mydb");
dataSource.setUser("user");
dataSource.setPassword("secret");

// create sql file system
SqlFileSystem fileSystem = SqlFileSystem.New(
     // use caching connector
     SqlConnector.Caching(
          SqlProviderMySql.New(dataSource)
     )
);

EmbeddedStorage.start(fileSystem.ensureDirectoryPath("path", "to", "storage"));

MicroStream is designed to work with object graphs. Thus, storing data means to store an object graph. This includes the object's value fields and references to other objects. Storing an object will also store all instances referenced by this objects that have not been stored before. While storing your data most of the work MicroStream performs for you. You only need to call the store method on the correct object. The rule is: "The Object that has been modified has to be stored!".

Storing objects that are not part of an object graph is most likely pointless.

Storing Root Instances

See Getting Started how to create a database with a root instance.

To store the registered root instance just call the storeRoot() method of a EmbeddedStorageManager instance.

Storing New Objects

To store a newly created object, store the "owner" of the object. In the example below a new object is created and added to the myObjects list of the root object. Then the modified list gets stored. This will also store the new object.

Storing Modified Objects

Before storing a modified object keep in your mind that the modified object needs to be stored.

In case of a value types, like int, it is the object that has the int field as a member:

The data layer is always immutable. In order to update the values we have to replace the data layer completely. This is done with the updater. The property setter methods can be chained, so it is easy to update multiple properties, for example:

PersonUpdater.New(mike)
			.firstName("Jim")
			.lastName("Hope")
			.update();

If only one property needs to be updated, the updater class offers static convenience methods for that:

PersonUpdater.setFirstName(mike, "Jim");

Where is the data of my database located

MicroStream connects your application's entity graph residing in memory to a physical form of data (i.e. persistent data) to/from which entity data is stored/loaded as required.

Where does MicroStream store persistent data?

MicroStream stores persistent data in a physical form, typically in native file-system files.

Does MicroStream work with the Java Module System (Jigsaw)?

Yes, since version 2.2 we provide an all-in-one multi-release jar which is compatible with Jigsaw projects. This single jar solution was necessary because different MicroStream modules use the same packages, but the module system doesn't allow that multiple modules declare the same exports.

Examples Collection

A collection of examples with different topics:

BookStore Demo

The default way to configure a JCache provider is to use the classjavax.cache.configuration.MutableConfiguration. This is mostly used to avoid provider specific code.

If you want to use all of MicroStream's Cache features, you can use our configuration implementation: one.microstream.cache.CacheConfiguration

To read an external configuration use CacheConfigurationLoader and CacheConfigurationParser or the Load*() methods of CacheConfiguration.

Entities can be created with an arbitrary amount of layers, so feel free to combine them as you like:

Prerequisites

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded.configuration</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
</dependencies>

Hello World

// Initialize a storage manager ("the database") with purely defaults.
final EmbeddedStorageManager storageManager = EmbeddedStorage.start();

// print the last loaded root instance, 
// replace it with a current version and store it
System.out.println(storageManager.root());
storageManager.setRoot("Hello World! @ " + new Date());
storageManager.storeRoot();

// shutdown storage
storageManager.shutdown();

This simplest example will create a new storage if no existing storage is found, if a existing storage is found it will be loaded (this is all done at line 2 in the example above).

In line 6 the current storage's content is printed.

Line 7 assigns some data to the storage, replacing existing data if there is some.

In line 8 everything gets stored.

The Root Instance

When using MicroStream, your entire database is accessed starting at a root instance. This instance is the root object of an object graph that gets persisted by the MicroStream storage logic. While the root instance can be of any type (for example just a collection or an array), it is a good idea to define an explicit root type specific for the application. In this simple example, it is a class called DataRoot, which wraps a single String.

More about root instances:

Creating a Database

The following code is all that is required to setup a an application backed by a MicroStream database. The application's convenience root instance is defined and an EmbeddedStorageManager instance, linked to the root, is created (and its database managing threads are started). This is a fully operational Java database application.

Storing Data

This call is all that is necessary to store data in the simplest case.

Stopping a Live Database

Best practice is to safely shutdown the storage manager by simply calling:

The EmbeddedStorageManager is mostly created with factory methods of EmbeddedStorage, where the most common settings, like database directory or the root instance, can be configured.

EmbeddedStorageManager storageManager = EmbeddedStorage.start(
    myRoot,                 // root object of entity graph
    Paths.get("data-dir")    // storage data directory
);

Foundations

To achieve a more detailed customization, you can utilize the EmbeddedStorageFoundation factory type. It holds and creates on demand all the parts that form an EmbeddedStorageManager.

External Configuration

The artifact storage.embedded.configuration provides a convenience layer for configuration purposes, as well as facilities to read external configuration.

The Configuration type consolidates the most widely used parameters from the storage foundations in one place. It's output is an EmbeddedStorageFoundation from which a EmbeddedStorageManager can be created.

To read an external configuration use ConfigurationLoader and ConfigurationParser or the Load*() methods of Configuration. Currently XML and INI files are supported.

If you just use Configuration.Load() the default configuration file is used, which is either a file in the classpath root named microstream-storage.properties, or the path configured via the system property microstream.storage.configuration.path.

Since version 3 MicroStream provides a JCache (JSR-107) implementation, which is optionally backed by the MicroStream Storage.

JCache standardizes caching for the Java platform. It provides a common mechanism to cache values in a map-like structure. It expedites the mainstream adoption of in-memory computing by giving all Java developers an easy way to access memory from within Java. Businesses can change providers without rewriting their applications or maintaining a proprietary cache abstraction layer.

This caching standard is used in a wide variety of environments. The most common use cases are:

• Second-level cache in JPA (e.g. Hibernate)

• and many more

Motivation

Why another JCache implementation, you may wonder. There is already a myriad of providers out there. MicroStream's very own serialization and storage can be utilized to get the best out of caches. For example you are not limited to java.io.Serializable types, when storing by value. That means every key-value pair gets copied every time you put and get entries in and out of a cache. And if a cache is used with a backing store, MicroStream's storage can be used to get the best possible performance.

And now, MicroStream being a JCache provider, you can use it as a drop-in replacement in your existing application.

Basic Concepts of JCache

CachingProvider

The caching provider represents the implementation of JCache that you are using. You can use more than one JCache implementation in your project if you wish, and CachingProvider is how you access the different providers that are in use.

CacheManager

It is responsible for managing and providing access to many named caches.

Cache

The cache holds the different values being cached. You can have several caches, each of which may be holding data for a different purpose. Each one can have a different configuration; for example, different caches may evict old data using different techniques.

Entry

Each item of data in a cache is an entry, which is a key-value pair. The key is a unique value used to store and look up the data. The value is the actual data you wish to cache. Caches have some different properties than Maps, but the calls that you would use to store and lookup data is very similar.

Prerequisites

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>cache</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
</dependencies>

Hello World

CachingProvider provider     = Caching.getCachingProvider();  
CacheManager    cacheManager = provider.getCacheManager();   
MutableConfiguration<Integer, String> configuration = new MutableConfiguration<>()  
    .setTypes(Integer.class, String.class)   
    .setStoreByValue(false)   
    .setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.ONE_MINUTE));  
Cache<Integer, String> cache = cacheManager.createCache("jCache", configuration); 
cache.put(1, "Hello World"); 
String value = cache.get(1); 

1) Get the default CachingProvider implementation from the application’s classpath. This method will work if and only if there is only one JCache implementation available in the classpath. If there are multiple providers then use the fully qualified name Caching.getCachingProvider("one.microstream.cache.CachingProvider") instead.

2) Get the default CacheManager instance using the provider.

3) Create a cache configuration using MutableConfiguration 4) with key type and value type as Integer and String respectively 5) configured to store the cache entries by reference (not by value) 6) and with an expiry time of one minute defined for entries from the moment they are created.

7) Using the cache manager, create a cache named jCache with the configuration created in step 3.

8) Put some data into the cache

9) And retrieve it.

The same can be done using MicroSteam's CacheConfiguration API. This time we use a EmbeddedStorageManager as a backing store for the cache.

Another predefined logic layer is for logging purposes. Since there is a myriad of loggers out there, MicroStream doesn't provide any special adapter, but a generic type which can be used to adapt to the logging framework of your choice.

Just create a class and implement EntityLogger, and you are good to go.

public class JulLogger implements EntityLogger
{
	@Override
	public void afterUpdate(
		final Entity identity, 
		final Entity data, 
		final boolean successful)
	{
		Logger.getLogger(identity.getClass().getName())
			.info("Entity updated");
	}
}

Additional to afterUpdate there are further hooks:

• entityCreated

• afterRead

• beforeUpdate

Now just add the logger when creating entities:

When you call

the logger's output is

Are transactions possible in MicroStream?

Yes. In fact, every storing of data is executed as a transaction, an atomic all-or-nothing action. When one or more entities are stored, their data is collected into a continuous block of bytes and that block is written to the physical form (the "files") in one fell swoop. Any problem during the IO-operation causes the whole block to be deleted (rolled back).

Is MicroStream multi-threaded?

Yes. The storing and loading process can be parallelized by using multiple threads and thus be strongly accelerated. There is no limitation on how many threads can be used, apart from the mathematical constraint that the thread count must be a power of 2 (1, 2, 4, 8, 16, etc.).

Does MicroStream support a backup strategy?

Yes. There are currently two options available to create backups: An on-the-fly backup that copies and validates stored entity data after it has been written and the possibility to export all database files to a target location (which is in fact just a low-level file copying, but executed in a concurrency-safe way).

Does MicroStream provide data export/import functionality?

Yes. MicroStream provides a per-type export of binary data and a utility to convert its binary data into the CSV format. The other way around (convert CSV to binary an import binary files) is also possible.

Does MicroStream provide a user management, authentication and authorization etc., like conventional DBMS ?

No, because it doesn't need to. Such concerns are long covered by the application itself, with the DBMS usually being degraded to only being the application's exclusive database. Thus, all that is needed for a modern business application is just an application-exclusive data storage solution, which is exactely what MicroStream is.

Is it possible to migrate an existing database to MicroStream?

Yes, if the data is structured in a format conforming to the entity classes and with references being represented in globally unique and bijective numbers. How hard that is for a given database depends on its specifics, but it can be as easy as executing one generically created SELECT per table.

Continuous Backup

Continuous backup mode allows the MicroStream engine to clone all storage actions to a second storage. This storage should be placed on another file system.

To configure and enable the continuous backup see Backup.

Data Export

MicroStream provides the possibility to import and export data, see .

Deleting data does not require performing explicit deleting actions like DELETE FROM table WHERE... . Instead you just need to clear any references to the object in your object-graph and store those changes. If a stored object is not reachable anymore its data will be deleted from the storage later. This behavior is comparable to Java's garbage collector.

root.MyArrayList.remove(0);
storage.store(root.MyArrayList);

The erasing from the storage files is done by the housekeeping process.

Object instances can be stored as simple records. One value after another as a trivial byte stream. References between objects are mapped with unique numbers, called ObjectId, or short OID. With both combined, byte streams and OIDs, an object graph can be stored in a simple and quick way, as well as loaded, as a whole or partially.

But there is a small catch. Where does it start? What is the first instance or reference at startup? Strictly speaking "nothing". That's why at least one instance or a reference to an instance must be registered in a special way, so that the application has a starting point from where the object graph can be loaded. This is a "Root Instance".

Same difference, another problem are instances which are references by constant fields in Java classes. These aren't created when the records are loaded from the database, but by the JVM while loading the classes. Without special treatment, this would be a problem:

Supported JDKs

Tested and officially supported JDKs:

These are the available properties of the CacheConfiguration type. The names are used accordingly in the external configuration files. They can be found as constants in CacheConfigurationPropertyNames.

MicroStream offers a Hibernate cache region factory, which can be found in the cache.hibernate module.

The region factory's class name is one.microstream.cache.hibernate.CacheRegionFactory. It is configured via the property hibernate.cache.region.factory_class.

Depending on your environment it can be configured in different ways.

If you use a good old persistence.xml, set the property there:

Or for Spring applications:

The wrapper code generator is an annotation processor, provided by the base module.

The maven configuration looks like this:

There are following ways to get the base wrapper types generated. If you want it for your own types, the best way is to use the GenerateWrapper annotation.

Or, if you want it for interfaces in libraries, like PersistenceStoring, you cannot add an annotation. That's what the microstream.wrapper.types parameter is for. This is just a comma separated list of types. Alternatively you can use the GenerateWrapperFor

Given is the following entity:

So how is it done? Since the code generator provides a creator, we can use it to create a new Person.

Let's see what the debugger displays if we run this code:

There's always an entity chain, with

First of all add the MicroStream Cache dependency:

The core caching abstraction provided by Spring comes in the module.

If you use Spring Boot, then add the package to add the caching dependencies:

To enable caching, Spring makes good use of annotations, much like enabling any other configuration level feature in the framework. The caching feature can be enabled by simply providing a cache setup component.

The layered entities code generator is an annotation processor, provided by the base module.

The maven configuration looks like this:

If you don't want the HashEqualator to be generated, just set the microstream.entity.hashequalator argument to false. You can leave it out otherwise, the default value is true.

The same applies to the Appendable

When a reference to the loading storage is needed in entities, e.g. usage of different tenants or to store its internal state in a tailored fashion, this can be done by this little trick.

If an entity type contains one or more transient fields with field type compatible to Persister, the updating processing during loading will set the Persister instance (e.g. an EmbeddedStorageManager instance) used to load the entity instance to those fields.

The fields must be transient to exclude them from the persistent form of the entity. Checking for transient fields is only the default implementation. The checking logic can be customized via PersistenceFoundation#setFieldEvaluatorPersistable.

Implementing the PersistenceEagerStoringFieldEvaluator interface allows you to handle the eager/lazy storing behavior of any known member. The default implementation of the MicroStream engine threads all fields as lazy storing. See for details on lazy and eager storing.

The PersistenceEagerStoringFieldEvaluator has only one method to be implemented: public boolean isEagerStoring(Class<?> t, Field u) return true if the field has to be eager, otherwise return false.

To register the customized PersistenceEagerStoringFieldEvaluator add it using the one.microstream.persistence.types.PersistenceFoundation .setReferenceFieldEagerEvaluator(PersistenceEagerStoringFieldEvaluator) method during the storage initialization.

Manually

The Lazy class has a .clear() method. When called, the reference held in the Lazy Reference is removed and only the ID is kept so that the instance can be reloaded when needed.

In addition to the methods for legacy type mapping described in chapter there is also the possibility to implement custom legacy type handlers. Those handlers are the most flexible way to do the mapping from old to new types.

The basic interface that has to be implemented is one.microstream.persistence.types.PersistenceLegacyTypeHandler.

Fortunately the standard persistence implementation provides the abstract class one.microstream.persistence.binary.types.BinaryLegacyTypeHandler.AbstractCustom that should be sufficient to start with a custom implementation in most cases.

The MicroStream engine supports two general storing strategies: lazy and eager storing. By default, MicroStream uses the lazy storing strategy.

These storing strategies differ in the way how objects, referenced by the object to be stored are handled if those referenced objects had already been stored.

Lazy Storing

Lazy storing is the default storing mode of the MicroStream engine.

Referenced instances are stored only if they have not been stored yet. If a referenced instance has been stored previously it is not stored again even if it has been modified.

In certain environments or setups it is necessary to provide specific ClassLoader instances. This can be done by customizing the connection foundation.

If a single ClassLoader is sufficient, just create a new provider by handing over the instance:

Or return a ClassLoader depending on the requested type:

Loading data can be done in two ways, eager and Lazy. The basic, default way of loading is eager loading. This means that all objects of a stored object graph are loaded immediately. This is done during startup of the MicroStream database instance automatically if an already existing database is found.

Contrary to , eager loading has no requirements to your entity model.

To load your data you just need to create an EmbeddedStorageManager instance:

After that just get the root instance of your object graph from the StorageManager by calling EmbeddedStorageManager.root() and check for null as this indicates a non-existing database

The default MicroStream implementation fully supports the Java transient field modifier. Class members marked transient will not be persisted.

It is possible to override the default behavior by implementing a custom PersistenceFieldEvaluator.

Null-safe Lazy Reference Access

For convenience MicroStream provides Null-safe static access methods for lazy references.

These are

The MicroStream engine takes care of persisting your object graph. When you do queries, they are not run on the data stored by MicroStream, queries run on your data in the local system memory. There is no need to use special query languages like SQL. All operations can be done with plain Java. MicroStream does not restrict you in the way you query your object graph. You are totally free to choose the best method fitting to your application.

One possibility may be Streams if you use the standard Java collections.

Do I have to adapt my object-model to MicroStream?

No. MicroStream allows you to store any Java object. Instances of any and all types can be handled, there are no special restrictions like having to implement an interface, using annotations or having a default constructor (see POJO). Only types bound to JVM-internals like Thread, IO-streams and the like are deliberately excluded from being persistable since they could not be properly recreated upon loading, but such instances should not be part of entity data models, anyway.

MicroStream's storage can be used as a backing store for the cache. It functions as a CacheWriter as well as a CacheReader, depending on the writeThrough and readThrough configuration. Per default it is used for both.

EmbeddedStorageManager storageManager = EmbeddedStorage.start();
CachingProvider        provider       = Caching.getCachingProvider();  
CacheManager           cacheManager   = provider.getCacheManager();   
CacheConfiguration<Integer, String> configuration = CacheConfiguration
			.Builder(Integer.class, String.class, "my-cache", storageManager)
			.build(); 
Cache<Integer, String> cache = cacheManager.createCache("jCache", configuration);

If you prefer an external configuration, you can link the storage configuration:

keyType = java.lang.Integer
valueType = java.lang.String

readThrough = true
writeThrough = true

storageConfigurationResourceName = microstream-storage.properties

baseDirectory = ~/cache-data
channelCount = 4

Or you can embed the storage configuration using the storage. prefix:

keyType = java.lang.Integer
valueType = java.lang.String

readThrough = true
writeThrough = true

storage.baseDirectory = ~/cache-data
storage.channelCount = 4

Spring example:

spring.jpa.properties.hibernate.cache.microstream.missing_cache_strategy = create
spring.jpa.properties.hibernate.cache.microstream.readThrough = true
spring.jpa.properties.hibernate.cache.microstreamwriteThrough = true
spring.jpa.properties.hibernate.cache.microstream.storage.baseDirectory = ~/cache-data
spring.jpa.properties.hibernate.cache.microstream.storage.channelCount = 4
spring.jpa.properties.hibernate.cache.region.factory_class = one.microstream.cache.hibernate.CacheRegionFactory
spring.jpa.properties.hibernate.cache.use_query_cache = true
spring.jpa.properties.hibernate.cache.use_second_level_cache = true

REST Service

First of all we have to connect a storage to a REST service.

Just add the REST service implementation to your dependencies, the logger is optional.

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.restservice.sparkjava</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-simple</artifactId>
			<version>1.7.30</version>
		</dependency>
</dependencies>

Now use the resolver to connect the service to a storage, start it, and you're good to go.

EmbeddedStorageManager storage = EmbeddedStorage.start();
if(storage.root() == null)
{
		storage.setRoot(new Object[] {
				LocalDate.now(),
				X.List("a", "b", "c"),
				1337
		});
		storage.storeRoot();
}

StorageRestService service = StorageRestServiceResolver.resolve(storage);
service.start();

1) An example storage manager 12) Create the REST service, 13) and start it.

That's all you have to do to open the REST endpoints to access the storage data.

The base URL of the opened endpoints is per default: http://localhost:4567/microstream/

Configuration

If you want to change the default port (4567) or instance name (microstream) it can be done by using the rest service implementation directly. The spark service can then be customized by your liking.

This will change the base URL to http://localhost/my-name/

Based on the REST API we provide a client, which serves a convenient web user interface to browse through the storage data.

It is a runnable jar which starts a simple web server which then can be accessed by a browser of your choice.

To download it use maven

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.restclient.app</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
</dependencies>

or this direct link:

Start the client. The port parameter is optional, default port is 8080.

java -jar storage.restclient.app-04.00.00-MS-GA.jar --port=80

Then just open http://localhost in your browser, select the base URL of the REST service and click connect.

Now you can browse through the data of the storage:

Or view the statistics:

The MicroStream Storage isn't a typical database server with administrative tooling and stuff like that. It is just a Java library which runs embedded in your application. The storage data layer, per default the file system, contains the serialized and persisted data. But it is not really accessible, or more precise, human-readable. Nor do we provide a query language to access the storage data.

The Java objects, which reside in memory, are easy to inspect and traverse, e.g. with a debugger.

But for various purposes, like monitoring, the requirement to read the actual stored data has come up.

Since version 3.0 a REST interface for the storage data is included. It enables you to access the data via REST calls or a convenient user interface.

It is made up of the following modules:

Stopping or Crashing a Live Database

Actually, a database is a passive collection of persisted data that can never be live on its own. But the managing thread accessing it can.

When an EmbeddedStorageManager is "started" it is actually just setup with all kinds of default and user-defined settings and definitions. What is actually "started" are the database managing threads that process storing and loading requests.

// Setup the database manager and start the managing threads
EmbeddedStorageManager storageManager = EmbeddedStorage.start();

Of course, for every start() method, there needs to be something like a shutdown() method. So there is in MicroStream:

// Stop accessing the database
storageManager.shutdown();

But is it really necessary to call shutdown? Should it be? What if there's an error and the process stops without calling shutdown()? Will that cause the database to become inconsistent, corrupted, maybe even destroyed?

The answer is: It wouldn't be much of a database solution if a crash could cause any problem in the persisted data. MicroStream data-store is carefully designed in such a fashion that the process it runs in can simply vanish at any point in time and the persisted data will never be corrupted.

This is surprisingly simple and reliable to solve: Whenever a .store() call returns, it is guaranteed that the data stored by it has been physically written to the underlying storage layer, usually a file system. Before that, there is no guarantee regarding written data at all. In fact, should the process die before the last byte has been written and secured, the next StorageManager initialization will recognize that and truncate the last partially written store. Either way, all the data that was guaranteed to be written will be consistently available after the next .start().

As a consequence, this safety mechanism makes an explicit .shutdown() call pretty much unnecessary. It doesn't hurt, but it is effectively more-less the same as just calling System.exit(0);.

The only time when an explicit shutdown is really needed is, if the database managing threads shall be stopped but the application itself keeps running. For example, it is perfectly valid to start the StorageManager, work with the database, then stop it, maybe change some configuration or copy files or something like that and then start it up again to continue working.

In any other case, the shutdown method can be ignored and the live database can happily just be "killed" while running. It is specifically designed to withstand such a treatment.

Multiple Databases

Any live MicroStream database basically consists of three major parts:

• A place where the persisted data is located. Usually a file system directory.

• The managing threads accessing (read and write) the persisted data.

• The EmbeddedStorageManager instance to use and control the database in the application.

Apart from a lot of internal components (configuration, processing logic, housekeeping state, etc.), that's all there is. There is nothing special or "magic" about it, no static state, no global registration in the JVM process or something like that.

The consequence of this is: If two EmbeddedStorageManager instances are started, each one with a different location for its persistend data, then the application has two live databases! If three or ten or 100 are started, then that's the number of live databases the application has. There is no limit and no conflict between different databases inside the same application process. The only important thing is that no two running StorageManagers can access the same data location.

Storing Hidden Encapsulated Objects

In some cases, it can be necessary to store modified encapsulated objects that cannot be a accessed from your code.

public class ForeignObject
{
    ...
    private HiddenObject hidden;
    ...
}

In the upper code snippet the "hidden" object cannot be accessed by store(myForeignObject.hidden) if no getter is available. To allow such hidden objects to be stored after they have been modified you have to options:

1. Set the global storing strategy of the MicroStream instance to or

2. Implement and set a custom for this field.

Use Immutable data models

To increase performance use immutable sub-graphs as often as possible. Storing those with the provided or using a thread local to insert those sub-graphs concurrently can give a great performance boost.

MicroStream uses a strictly interface-based architecture. All types in the public API are, whenever possible, interfaces. This offers the best possibilities to extend or exchange parts of the engine. A good ways to enrich a type with features, is the wrapper (decorator) pattern.

For example, let's say we want to add logging to the PersistenceStoring's store(object) method.

Conventionally it would be done that way: A new type, implementing the original interface, would be handed over the wrapped instance, all interface methods have to be implemented and delegated. And in the single method, we wanted to add functionality; the actual implementation of the logging is done.

This produces a lot of overhead. In this case, three methods are just boilerplate code to delegate the calls to the wrapped instance. A common solution for that is to create an abstract base wrapper type for the designated interface, and to reuse it whenever needed.

MicroStream's wrapper code generator generates following wrapper type for PersistenceStoring:

It is not an abstract class, but an interface, which extends the Wrapper interface of the base module, and the wrapped type itself. This offers you the most flexible way to use it in your application.

The Wrapper type is just a typed interface and an abstract implementation of itself.

You can either implement the Wrapper interface and provide the wrapped instance via the wrapped()

An arbitrary amount of logic layers can be added to entities. Let's use the predefined versioning layer. It will keep track of all changes. Technically every new data layer which is added by the updater, will create a new version entry.

Let's have a look at the debugger:

Now the versioning layer is chained between the identity layer and the data layer.

If we update the entity a few times, we will see how the versioning layer works. In this case we use an auto-incrementing Long as key.

If you want to access older versions use the context:

Housekeeping is an internal background logic to optimize the database's usage of memory and persistent storage space (typically disc space). It is comprised of mechanisms for cleaning up storage files, clearing unneeded cached data and recognizing deleted entities via garbage collection. Housekeeping is performed with a configurable time budget in configurable intervals to make sure it never interferes with the application's work load too much (see ).

File cleanup:

If new versions of an entity are stored or if entities become no longer reachable (meaning the become effectively deleted or "garbage" data), their older data is no longer needed. However, the byte sequences representing that old data still exist in the storage files. But since they will never be needed again, they become logical "gaps" in the storage files. Space that is occupied, but will never be read again. It might as well be all zeroes or not exist at all. Sadly, unwanted areas cannot simple by "cut" from files. Above all because that would ruin all file offsets coming after them. So with every newly stored version of an entity and every entity that is recognized as unreachable "garbage", a storage file consists more and more of useless "gaps" and less and less of actually used data. This makes the storage space less and less efficient. To prevent eventually ending up with a drive that is filled with useless bytes despite an actually not that big database, the files need to be "cleaned up" from time to time. To do this, the Housekeeping occasionaly scans the storage files. If their "payload" ratio goes below the configured limit, the affected files will be retired: all data that belongs to still live entities is copied to a new file. Then the old file consists of 100% unneeded gap data and can safely be deleted.

Custom type handlers allow taking control over the storing and loading procedure of specific java types. This is useful to optimize the performance for storing complex objects or in the rare case that it is not possible to store a type with the default type handlers.

Implementation

There are two strategies for a simplified type handler implementation.

A Custom Binary Handler

Implementing a class that extends CustomBinaryHandler and defines a sequence of BinaryFields via the #Field~ static pseudo-constructor methods. Everything else like setting the name, calculating the binary offsets, etc. is then done implicitly via reflection.

The custom type handler must be registered in the CustomTypeHandlerRegistry to enable it:

A Static Provider Method

Implementing a class can be skipped altogether by using the method Binary#TypeHandler and passing the BinaryFields directly. Registering such a generically created TypeHandler is not required, either, since Version 3 of MicroStream brought a solution to just define a static method in the entity class that will be recognized and used by MicroStream.

The following is a simple technical example on how a custom binary handler can be easily defined and technically leveraged to optimize storage behavior. E.g. imagine having millions of such objects that now only create 1 database record with a fraction of the required storage space instead of 4 records but hold the same information.

In this chapter it is explained how Lazy Loading is done with MicroStream.

Of course, it's not really about the technical implementation of procrastination, but about efficiency: why bloat the limited RAM with stuff before you even need it?

Classic example: The application has self-contained data areas that contain a large amount of data. The data for an area is not loaded if the area is not worked at all. Instead, you only load a tiny amount of "head data" for each area (name or other for displaying purposes) and the actual data only when the application really needs it. E.g. fiscal years with hundreds of thousands or millions of sales. One million revenue records for 2010, one million for 2011, for 2012, etc. In 2019, most of the time only 2019 and 2018 will be needed. The previous few, and the year 2000 sales are not of great interest anymore. Therefore: load data only when needed. Super efficient.

For example let's say the app "MyBusinessApp" has a root instance class, looking like this:

public class MyBusinessApp
{
    // ...

    private HashMap<Integer, BusinessYear> businessYears = new HashMap<>(); 

    // ...
}

The business year hold the turnovers:

public class BusinessYear
{
    // ...

    private ArrayList<Turnover> turnovers = new ArrayList<>();

    // ...
}

This approach would be a problem: During initialization, the root instance would be loaded, from there its HashMap with all BusinessYear instances, each with its ArrayList and thus all conversions. For all years. 20 years of approximately 1 million sales makes 20 million entities, which are pumped directly into the RAM at the start. It does not matter if someone needs it or not. We don't want it that way.

It would be nice if you could simply add a "lazy" to the turnover list. And that's exactly how it works:

And bingo, the turnovers are now loaded lazily.

What do you have to do now to get the actual ArrayList<Turnover> instance?

Just as with WeakReference, or simply as one would expect from a reference intermediate type in general: a simple get method.

The .get() call reloads the data as needed. But you do not want to mess around with that yourself. No "SELECT bla FROM turnovers WHERE ID =" + this.turnovers.getId(). Since you want to program your application you don' t have to mess around with low-level database ID-loading stuff. That's what the MicroStream Code does internally. You do not even need to access the ID, you just have to say "get!".

That's it.

What about the "..." ?

There are different strategies, what you write here. Analogous to the code example before it would be simply:

So always a new ArrayList instance, wrapped in a Lazy instance. If the actual ArrayList reference should be null at first, it works the same way:

The this.turnovers.get() also just always returns null. Completely transparent.

But you could also do this:

If there is no list, then you do not make any intermediate reference instance for any list. A separate instance for null is indeed a bit ... meh.

But that has a nasty problem elsewhere: this.turnovers.get() does not work then. Because NullPointerException.

Anytime you need to write this here, the readability of code is not exactly conducive:

But there is a simple solution: Just move this check into a static util method. Just like that:

This is the same .get(), just with a static null-check around it. This always puts you on the safe side.

In Short

For Lazy Loading, simply wrap Lazy<> around the actual field and then call .get() or maybe better Lazy.get(...).

It's as simple as that.

Side Note

Why do you have to replace your actual instance with a lazy loading intermediate instance and fiddle around with generics? Why is not something like this:

Put simply: If it were just that it would be bare Java bytecode for accessing an ArrayList. There would be no way for a middleware library to get access and look it up and perhaps reload it. What's written there is an ArrayList reference. There is no lazy anymore. Either, the instance is null, or it is not null. If you wanted to reach in there, you would have to start with bytecode manipulation. Technically possible, but something you really don't want in your application.

So there must always be some form of intermediary.

Hibernate solves this through its own collection implementations that do lazy loading internally. Although the lazy loading is nicely hidden in some way (or not, if you need an annotation for that), it also comes with all sorts of limitations. You can only use interfaces instead of concrete classes for collections. At first, the instance is not the one you dictate, the code becomes intransparent and difficult to debug, you have to use a collection, even if it's just a single instance, and so on. You want to be able to write anything you want and you want full insight and control (debugability, etc.) over the code.

All this can be done with the tiny Lazy Interim Reference class. No restrictions, no incomprehensible "magic" under the hood (proxy instances and stuff) and also usable for individual instances.

4.0

Features

MicroStream provides an API to import and export persisted data of the storage. It is pretty much the same as writing and reading a backup.

The records in the storage are distributed in lots of files and folders, depending on channel count and other . To get order in the chaos the export produces one file per type. This files are used again by the import to read the data into the storage.

Data Conversion

It is also possible to convert the exported binary files to a human readable format, namely CSV.

Root

GET [instance-name]/root

Returns the name and object id of the current storage root element.