Welcome to the MicroStream Reference Manual. This manual includes concepts, instructions and examples to guide you on how to use MicroStream Data-Store and Cache, version 4.0.

You should be familiar with the Java programming language and you should have installed your preferred Integrated Development Environment (IDE). But since you are here we guest you got that covered ;)

What's new in 4.0

API Docs

The API documentation is available at https://api.docs.microstream.one/4.0/.

Support

For information on the commercial support for MicroStream see microstream.one.

License

The NIO connector can access the local or mounted file systems, as well as different in-memory file systems. This is probably the easiest way to start, especially for prototyping and testing purposes. For productive use, the other file systems are preferred.

NioFileSystem fileSystem = NioFileSystem.New();
EmbeddedStorage.start(fileSystem.ensureDirectoryPath("path", "to", "storage"));

Continuous Backup

Continuous backup mode allows the MicroStream engine to clone all storage actions to a second storage. This storage should be placed on another file system.

To configure and enable the continuous backup see Backup.

Data Export

MicroStream provides the possibility to import and export data, see Import / Export.

Tested and officially supported JDKs:

By default, MicroStream uses the operation system's standard file locking mechanism to prevent simultaneous access to the storage files. In the rare case that this is not sufficient to control the file access MicroStream provides a proprietary file lock implementation to ensure exclusive access to the storage files from different applications using MicroStream.

Using this file lock may only be necessary if, while a MicroStream application is running, a second MicroStream application may try to access the same storage and the default file locks are not reliable.

You don't need to activate this feature if:

• Only one MicroStream application will access the storage,

Beside long store(Object instance) MicroStream provides some convenience methods to store several objects at once:

void storeAll(Iterable<?> instances)

Stores the passed instance in any case and all referenced instances of persistable references recursively, but stores referenced instances only if they are newly encountered (e.g. don't have an id associated with them in the object registry, yet and are therefore required to be handled). This is useful for the common case of just storing an updated instance and potentially newly created instances along with it while skipping all existing (and normally unchanged) referenced instances.

long[] storeAll(Object... instances)

The MicroStream engine supports two general storing strategies: lazy and eager storing. By default, MicroStream uses the lazy storing strategy.

These storing strategies differ in the way how objects, referenced by the object to be stored are handled if those referenced objects had already been stored.

Lazy storing is the default storing mode of the MicroStream engine.

Referenced instances are stored only if they have not been stored yet. If a referenced instance has been stored previously it is not stored again even if it has been modified.

In eager storing mode referenced instances are stored even if they had been stored before. Contrary to Lazy storing this will also store modified child objects at the cost of performance.

To use lazy or eager storing explicitly, get an instance of the required Storer

The default MicroStream implementation fully supports the Java transient field modifier. Class members marked transient will not be persisted.

It is possible to override the default behavior by implementing a custom PersistenceFieldEvaluator.

Loading data can be done in two ways, eager and Lazy. The basic, default way of loading is eager loading. This means that all objects of a stored object graph are loaded immediately. This is done during startup of the MicroStream database instance automatically if an already existing database is found.

Contrary to , eager loading has no requirements to your entity model.

To load your data you just need to create an EmbeddedStorageManager instance:

After that just get the root instance of your object graph from the StorageManager by calling EmbeddedStorageManager.root() and check for null as this indicates a non-existing database

For convenience MicroStream provides Null-safe static access methods for lazy references.

These are

• Lazy.get(Lazy) Gets the lazy referenced object, loads it if required. return value: null if the lazy reference itself is null otherwise the referenced object

The Lazy class has a .clear() method. When called, the reference held in the Lazy Reference is removed and only the ID is kept so that the instance can be reloaded when needed.

So that the Lazy References do not have to be managed manually, but the whole goes automatically, there is the following mechanism: Each Lazy instance has a lastTouched timestamp. Each .get() call sets it to the current time. This will tell you how long a Lazy Reference has not been used, i.e. if it is needed at all.

The LazyReferenceManager

Deleting data does not require performing explicit deleting actions like DELETE FROM table WHERE... . Instead you just need to clear any references to the object in your object-graph and store those changes. If a stored object is not reachable anymore its data will be deleted from the storage later. This behavior is comparable to Java's garbage collector.

The erasing from the storage files is done by the process.

The MicroStream engine takes care of persisting your object graph. When you do queries, they are not run on the data stored by MicroStream, queries run on your data in the local system memory. There is no need to use special query languages like SQL. All operations can be done with plain Java. MicroStream does not restrict you in the way you query your object graph. You are totally free to choose the best method fitting to your application.

One possibility may be Streams if you use the standard Java collections.

In addition to the methods for legacy type mapping described in chapter there is also the possibility to implement custom legacy type handlers. Those handlers are the most flexible way to do the mapping from old to new types.

The basic interface that has to be implemented is one.microstream.persistence.types.PersistenceLegacyTypeHandler.

Fortunately the standard persistence implementation provides the abstract class one.microstream.persistence.binary.types.BinaryLegacyTypeHandler.AbstractCustom that should be sufficient to start with a custom implementation in most cases.

In certain environments or setups it is necessary to provide specific ClassLoader instances. This can be done by customizing the connection foundation.

If a single ClassLoader is sufficient, just create a new provider by handing over the instance:

Or return a ClassLoader depending on the requested type:

Most application servers load the session's classes with the context class loader. Just use the one of the current thread:

Implementing the PersistenceEagerStoringFieldEvaluator interface allows you to handle the eager/lazy storing behavior of any known member. The default implementation of the MicroStream engine threads all fields as lazy storing. See for details on lazy and eager storing.

The PersistenceEagerStoringFieldEvaluator has only one method to be implemented: public boolean isEagerStoring(Class<?> t, Field u) return true if the field has to be eager, otherwise return false.

To register the customized PersistenceEagerStoringFieldEvaluator add it using the one.microstream.persistence.types.PersistenceFoundation .setReferenceFieldEagerEvaluator(PersistenceEagerStoringFieldEvaluator) method during the storage initialization.

When a reference to the loading storage is needed in entities, e.g. usage of different tenants or to store its internal state in a tailored fashion, this can be done by this little trick.

If an entity type contains one or more transient fields with field type compatible to Persister, the updating processing during loading will set the Persister instance (e.g. an EmbeddedStorageManager instance) used to load the entity instance to those fields.

The fields must be transient to exclude them from the persistent form of the entity. Checking for transient fields is only the default implementation. The checking logic can be customized via PersistenceFoundation#setFieldEvaluatorPersistable.

No. MicroStream allows you to store any Java object. Instances of any and all types can be handled, there are no special restrictions like having to implement an interface, using annotations or having a default constructor (see POJO). Only types bound to JVM-internals like Thread, IO-streams and the like are deliberately excluded from being persistable since they could not be properly recreated upon loading, but such instances should not be part of entity data models, anyway.

During initialization, MicroStream automatically checks if your runtime entity classes are still matching the persistent data. Mismatches are automatically mapped when loading data based on predefined rules that you can extend and overwrite on a per-case basis if needed.

MicroStream connects your application's entity graph residing in memory to a physical form of data (i.e. persistent data) to/from which entity data is stored/loaded as required.

MicroStream uses the common concept of "", allowing you to define which parts of your data (entity sub-graphs) are loaded only when required instead of eagerly at startup. A few well-placed lazy references in your entity model make your application load only a tiny bit of "head" entities at startup time and load everything else later on demand. This allows the handling of arbitrarily big databases with relatively small memory requirements.

MicroStream stores persistent data in a physical form, typically in native file-system files.

Yes, as many as you like. Each MicroStream instance represents one coherent entity graph of persistent data.

Yes. This is already done automatically. The minimum and maximum size of every partial file can be configured, although this is a very technical detail that should not be relevant in most cases.

At any given time, only one JVM process may directly access the files representing a unique set of data. Such a restriction is crucial for the correct execution of any application: changes to an application's persistent data have to be guarded by the rules of the application's business logic, i.e. the process that currently runs the application. Allowing another process to bypass these rules would eventually result in catastrophic consistency errors. The requirement to distribute an application over multiple processes must be solved by a clustering approach (e.g. by distributing logic AND persistent data over multiple processes or by having one process to serve as the data master for multiple worker processes).

Yes, since version 2.2 we provide an all-in-one multi-release jar which is compatible with Jigsaw projects. This single jar solution was necessary because different MicroStream modules use the same packages, but the module system doesn't allow that multiple modules declare the same exports.

Due to changes of the reflective behavior of records in Java 15, they are currently not supported. Java 16 will contain the final release of records, and MicroStream will support them with the release 5.0, which will be available in April 2021.

Yes. In fact, every storing of data is executed as a transaction, an atomic all-or-nothing action. When one or more entities are stored, their data is collected into a continuous block of bytes and that block is written to the physical form (the "files") in one fell swoop. Any problem during the IO-operation causes the whole block to be deleted (rolled back).

Yes. The storing and loading process can be parallelized by using multiple threads and thus be strongly accelerated. There is no limitation on how many threads can be used, apart from the mathematical constraint that the thread count must be a power of 2 (1, 2, 4, 8, 16, etc.).

Yes. There are currently two options available to create backups: An on-the-fly backup that copies and validates stored entity data after it has been written and the possibility to export all database files to a target location (which is in fact just a low-level file copying, but executed in a concurrency-safe way).

Yes. MicroStream provides a per-type export of binary data and a utility to convert its binary data into the CSV format. The other way around (convert CSV to binary an import binary files) is also possible.

No, because it doesn't need to. Such concerns are long covered by the application itself, with the DBMS usually being degraded to only being the application's exclusive database. Thus, all that is needed for a modern business application is just an application-exclusive data storage solution, which is exactely what MicroStream is.

MicroStream Data-Store is a native Java object graph storage engine. From a technical point of view it serves one purpose only: To fully or partially persist and restore a Java object graph in the simplest way possible for the user.

MicroStream Data-Store is a storage engine, but no database management system (DBMS). Many features that typical DBMS provide have been left out on purpose. The reason is that those features exist to make a DBMS something of a server application platform of an "old kind" on top of its data store functionality: A standalone process with user management, connection management, session handling, often even with a programming language of its own, a querying interface (SQL), etc. Today, all of those server application features are already and much better handled by dedicated server applications (the "new kind"), implemented in a modern language like Java. They have their built-in user, connection and session management, the querying interface to the outside world are typically web services instead of SQL, etc. But those modern server applications still lack one important thing: an easy to use and technically efficient way to store and restore their application's data. So a "new kind" server often uses an "old kind" server just to do the data storing. This comes at the price of catching all the overhead and problems of redundant user, connection and session management AND the outdated concepts and limitations of the old querying interface (SQL). Isn't that very weird and frustratingly complicated? Why not simply include a modern data storing library in the modern server and be done with it? A storing library that perfectly fits the modern technology and brings in no redundant overhead or complication of a secondary outdated wannabe server process. This is exactly what MicroStream Data-Store is and the reason why it is intentionally not a DBMS but "only" a storage engine.

One might think the easiest way to store and load data in Java would be Java's built-in serialization. However, it turned out long ago to be very limited, making it hard, if not impossible, to be used as a replacement for a DBMS:

• Only complete object graphs can be stored and restored, which is unacceptable for all but very small databases.

• It is very inefficient in terms of storage size and performance.

• It does not handle changing class structures very well, basically forbidding classes of persisted entities to ever change or introducing massive manual effort to compensate.

In short: The Java Serialization is not an acceptable data store solution and hence no valid replacement for those outdated DBMS.

MicroStream Data-store is such a solution:

• It can persist, load or update object graphs partially and on-demand.

• It is very efficient both size- and performance-wise.

• It handles changing class structures by mapping data in the old structure to the current structure during loading; implicitly via internal heuristics or explicitly via a user-defined mapping strategy.

MicroStream is what the Java Serialization should have been and it is the first and only really fitting data storing solution for modern applications, completely removing the need to attach a wannabe secondary server DBMS just to store data.

Using a Storage Live File Provider (one.microstream.storage.types.StorageLiveFileProvider) allows to specify the location and naming rules for all storage related files.

available properties are:

• BaseDirectory The Microstream storages location base directory. Contains channel directories and type dictionary file.

• DeletionDirectory If configured, the storage will not delete files. Instead of deleting a file it will be moved to this directory.

• TruncationDirectory If configured, files that will get truncated are copied into this directory.

• ChannelDirectoryPrefix Channel directory prefix string

• StorageFilePrefix Storage file prefix string

• StorageFileSuffix storage file extension

• TransactionsFilePrefix transactions file prefix

• TransactionsFileSuffix transaction file extension

• TypeDictionaryFileName filename of the type dictionary

Continuous Backup

By default, the continuous backup is disabled. If enabled the MicroStream instance will clone all changes to another directory. The backup is identical to the primary MicroStream storage.

To enable the continuous backup just set the backup directory:

With storage.embedded.configuration API:

EmbeddedStorageManager storageManager = Configuration.Default()

<properties>
	<property name="backupDirectory" value ="/save/backup" />
	...
</properties>

backupDirectory = backupDir

With MicroStream foundation classes:

NioFileSystem      fileSystem  = NioFileSystem.New();
		
StorageBackupSetup backupSetup = StorageBackupSetup.New(
  Storage.BackupFileProviderBuilder(fileSystem)
		.setDirectory(fileSystem.ensureDirectoryPath(BACKUPDIR))						
		.setTruncationDirectory(fileSystem.ensureDirectoryPath(TRUNCATIONDIR))
		.setDeletionDirectory(fileSystem.ensureDirectoryPath(DELETIONDIR))
		.createFileProvider()
);	
				
StorageConfiguration configuration = StorageConfiguration.Builder()
	.setBackupSetup(backupSetup)
	.setStorageFileProvider(StorageLiveFileProvider.New(
	  fileSystem.ensureDirectoryPath(WORKINGDIR)
	))
	.createConfiguration()
;

MicroStream supports a variety of storage targets. Through an abstracted file system (AFS), it is possible to connect to a lot of different back ends. The AFS allows to use folders and files, like in all common file systems, but with different connectors it is possible to use different solutions as the actual storage.

To connect to the local file system use the Java Non-Blocking IO (NIO) connector, which is part of the base module, so no additional dependency is needed.

EmbeddedStorage.start(Paths.get("path", "to", "storage"));

Internally this creates and uses a NioFileSystem and is a shortcut for:

NioFileSystem fileSystem = NioFileSystem.New();
EmbeddedStorage.start(fileSystem.ensureDirectoryPath("path", "to", "storage"));

The file system API is the same for all connectors, like for MySQL. This is part of another module.

<!-- sql file system -->
<dependency>
       <groupId>one.microstream</groupId>
       <artifactId>filesystem.sql</artifactId>
       <version>04.00.00-MS-GA</version>
</dependency>
<!-- driver of your choice -->
<dependency>
       <groupId>mysql</groupId>
       <artifactId>mysql-connector-java</artifactId>
       <version>8.0.21</version>
</dependency>

// create JDBC data source
MysqlDataSource dataSource = new MysqlDataSource();
dataSource.setUrl("jdbc:mysql://host:3306/mydb");
dataSource.setUser("user");
dataSource.setPassword("secret");

// create sql file system
SqlFileSystem fileSystem = SqlFileSystem.New(
     // use caching connector
     SqlConnector.Caching(
          SqlProviderMySql.New(dataSource)
     )
);

EmbeddedStorage.start(fileSystem.ensureDirectoryPath("path", "to", "storage"));

MicroStream is designed to work with object graphs. Thus, storing data means to store an object graph. This includes the object's value fields and references to other objects. Storing an object will also store all instances referenced by this objects that have not been stored before. While storing your data most of the work MicroStream performs for you. You only need to call the store method on the correct object. The rule is: "The Object that has been modified has to be stored!".

Storing objects that are not part of an object graph is most likely pointless.

Storing Root Instances

See Getting Started how to create a database with a root instance.

To store the registered root instance just call the storeRoot() method of a EmbeddedStorageManager instance.

// Init storage manager
final EmbeddedStorageManager storageManager = EmbeddedStorage.start(root);
    
// Store the root object
storageManager.storeRoot();

Storing New Objects

To store a newly created object, store the "owner" of the object. In the example below a new object is created and added to the myObjects list of the root object. Then the modified list gets stored. This will also store the new object.

Before storing a modified object keep in your mind that the modified object needs to be stored.

In case of a value types, like int, it is the object that has the int field as a member:

Storing Hidden Encapsulated Objects

In some cases, it can be necessary to store modified encapsulated objects that cannot be a accessed from your code.

public class ForeignObject
{
    ...
    private HiddenObject hidden;
    ...
}

In the upper code snippet the "hidden" object cannot be accessed by store(myForeignObject.hidden) if no getter is available. To allow such hidden objects to be stored after they have been modified you have to options:

1. Set the global storing strategy of the MicroStream instance to eager storing or

2. Implement and set a custom for this field.

To increase performance use immutable sub-graphs as often as possible. Storing those with the provided or using a thread local to insert those sub-graphs concurrently can give a great performance boost.

The MicroStream Storage isn't a typical database server with administrative tooling and stuff like that. It is just a Java library which runs embedded in your application. The storage data layer, per default the file system, contains the serialized and persisted data. But it is not really accessible, or more precise, human-readable. Nor do we provide a query language to access the storage data.

The Java objects, which reside in memory, are easy to inspect and traverse, e.g. with a debugger.

But for various purposes, like monitoring, the requirement to read the actual stored data has come up.

Since version 3.0 a REST interface for the storage data is included. It enables you to access the data via REST calls or a convenient user interface.

It is made up of the following modules:

REST Service

First of all we have to connect a storage to a REST service.

Just add the REST service implementation to your dependencies, the logger is optional.

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.restservice.sparkjava</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-simple</artifactId>
			<version>1.7.30</version>
		</dependency>
</dependencies>

Now use the resolver to connect the service to a storage, start it, and you're good to go.

EmbeddedStorageManager storage = EmbeddedStorage.start();
if(storage.root() == null)
{
		storage.setRoot(new Object[] {
				LocalDate.now(),
				X.List("a", "b", "c"),
				1337
		});
		storage.storeRoot();
}

StorageRestService service = StorageRestServiceResolver.resolve(storage);
service.start();

1) An example storage manager 12) Create the REST service, 13) and start it.

That's all you have to do to open the REST endpoints to access the storage data.

The base URL of the opened endpoints is per default: http://localhost:4567/microstream/

Configuration

If you want to change the default port (4567) or instance name (microstream) it can be done by using the rest service implementation directly. The spark service can then be customized by your liking.

This will change the base URL to http://localhost/my-name/

Based on the REST API we provide a client, which serves a convenient web user interface to browse through the storage data.

It is a runnable jar which starts a simple web server which then can be accessed by a browser of your choice.

To download it use maven

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.restclient.app</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
</dependencies>

or this direct link:

Start the client. The port parameter is optional, default port is 8080.

java -jar storage.restclient.app-04.00.00-MS-GA.jar --port=80

Then just open http://localhost in your browser, select the base URL of the REST service and click connect.

Now you can browse through the data of the storage:

Or view the statistics:

Examples Collection

A collection of examples with different topics:

BookStore Demo

The BookStore Demo is a fully fledged sample application. It shows how to design an application with MicroStream from the ground up.

If you want to compare MicroStream with a database of your choice, it can be done with following project, which is based on the BookStore Demo.

Spring Crud

Combination of two great frameworks.

Further reading:

Android RecyclerView

This example shows how to effectively use MicroStream storage with Android RecyclerView.

Further reading:

The default way to configure a JCache provider is to use the classjavax.cache.configuration.MutableConfiguration. This is mostly used to avoid provider specific code.

If you want to use all of MicroStream's Cache features, you can use our configuration implementation: one.microstream.cache.CacheConfiguration

CachingProvider provider     = Caching.getCachingProvider();  
CacheManager    cacheManager = provider.getCacheManager();   
CacheConfiguration<Integer, String> configuration = CacheConfiguration
			.Builder(Integer.class, String.class)
			.storeByValue()
			.expiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.ONE_MINUTE))
			.build(); 
Cache<Integer, String> cache = cacheManager.createCache("jCache", configuration); 
cache.put(1, "Hello World"); 
String value = cache.get(1); 

To read an external configuration use CacheConfigurationLoader and CacheConfigurationParser or the Load*() methods of CacheConfiguration.

CacheConfiguration<Integer, String> configuration = CacheConfiguration
			.Load("cache-config.properties", Integer.class, String.class);

If you just use CacheConfiguration.Load() the default configuration file is used, which is either a file in the classpath root named microstream-cache.properties, or the path configured via the system property microstream.cache.configuration.path.

The data layer is always immutable. In order to update the values we have to replace the data layer completely. This is done with the updater. The property setter methods can be chained, so it is easy to update multiple properties, for example:

PersonUpdater.New(mike)
			.firstName("Jim")
			.lastName("Hope")
			.update();

If only one property needs to be updated, the updater class offers static convenience methods for that:

PersonUpdater.setFirstName(mike, "Jim");

Housekeeping is an internal background logic to optimize the database's usage of memory and persistent storage space (typically disc space). It is comprised of mechanisms for cleaning up storage files, clearing unneeded cached data and recognizing deleted entities via garbage collection. Housekeeping is performed with a configurable time budget in configurable intervals to make sure it never interferes with the application's work load too much (see ).

If new versions of an entity are stored or if entities become no longer reachable (meaning the become effectively deleted or "garbage" data), their older data is no longer needed. However, the byte sequences representing that old data still exist in the storage files. But since they will never be needed again, they become logical "gaps" in the storage files. Space that is occupied, but will never be read again. It might as well be all zeroes or not exist at all. Sadly, unwanted areas cannot simple by "cut" from files. Above all because that would ruin all file offsets coming after them. So with every newly stored version of an entity and every entity that is recognized as unreachable "garbage", a storage file consists more and more of useless "gaps" and less and less of actually used data. This makes the storage space less and less efficient. To prevent eventually ending up with a drive that is filled with useless bytes despite an actually not that big database, the files need to be "cleaned up" from time to time. To do this, the Housekeeping occasionaly scans the storage files. If their "payload" ratio goes below the configured limit, the affected files will be retired: all data that belongs to still live entities is copied to a new file. Then the old file consists of 100% unneeded gap data and can safely be deleted.

Which ratio value to set in the configuration is a matter of taste or, more precisely, depends on each indivudual application's demands. A value of 1.0 (100%) means: only files with 100% payload, so no gaps at all, are acceptable. This means that for every store that contains at least one new version of an already existing entity, the corresponding storage file will contain the slightest gap, thus dropping below the demanded ratio of 100% and as a consequence, will be retired on the next occasion. This very aggressive cleanup strategy will keep the disc space usage at a perfect minimum, but at the cost of enormous amounts of copied data, since virtually every store will cause one or more storage files to be retired and their content be shifted into a new file. Respectively, a value of 0.0 (0%) means something like: "Never care about gaps, just fill up the disc until it bursts." This keeps the disc write loads for the file cleanup at 0, but at the cost of rapidly eating up disc space.

Since version 3 MicroStream provides a JCache () implementation, which is optionally backed by the MicroStream Storage.

JCache standardizes caching for the Java platform. It provides a common mechanism to cache values in a map-like structure. It expedites the mainstream adoption of in-memory computing by giving all Java developers an easy way to access memory from within Java. Businesses can change providers without rewriting their applications or maintaining a proprietary cache abstraction layer.

This caching standard is used in a wide variety of environments. The most common use cases are:

• Second-level cache in JPA (e.g. )

1) Get the default CachingProvider implementation from the application’s classpath. This method will work if and only if there is only one JCache implementation available in the classpath. If there are multiple providers then use the fully qualified name Caching.getCachingProvider("one.microstream.cache.CachingProvider") instead.

2) Get the default CacheManager instance using the provider.

3) Create a cache configuration using MutableConfiguration 4) with key type and value type as Integer and String respectively 5) configured to store the cache entries by reference (not by value) 6) and with an expiry time of one minute defined for entries from the moment they are created.

First of all add the MicroStream Cache dependency:

The core caching abstraction provided by Spring comes in the module.

If you use Spring Boot, then add the package to add the caching dependencies:

To enable caching, Spring makes good use of annotations, much like enabling any other configuration level feature in the framework. The caching feature can be enabled by simply providing a cache setup component.

Entities can be created with an arbitrary amount of layers, so feel free to combine them as you like:

MicroStream – Persistence – License Agreement Version 1, September 16, 2019, MicroStream Software GmbH

IMPORTANT: READ THE TERMS OF THIS AGREEMENT CAREFULLY BEFORE DOWNLOADING AND USING THE SOFTWARE. BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, OR USING THE SOFTWARE YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON BEHALF OF ANOTHER PERSON OR A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE FULL AUTHORITY TO BIND THAT PERSON, COMPANY, OR LEGAL ENTITY TO THESE TERMS. IF YOU DO NOT AGREE TO ALL THESE TERMS, YOU MUST NOT DOWNLOAD, INSTALL, COPY, ACCESS, OR USE THE SOFTWARE; AND YOU MUST PROMPTLY DESTROY ALL COPIES OF SOFTWARE AND ALL RELATED MATERIALS.

1. DEFINITIONS. "MicroStream Software GmbH" is a software developing firm based in Regensburg, Germany, hereafter referred to as "MicroStream" or "Licensor", www.microstream.one. "Licensor" refers either to an individual person or to a single legal entity. "Software" is the following, including the original and all whole or partial copies: (i) machine-readable instructions and data, (ii) components, (iii) audio-visual content (such as images, text, recordings, or pictures), (iv) related licensed materials, and (v) license use documents or keys, and documentation. "Agreement" refers to this MicroStream License Agreement.

2. LICENSE TO USE. As between the parties, Licensor reserves all rights in and to the MicroStream software.

3. License to internal use and development. Subject to the terms and conditions of this agreement, MicroStream grants you a non-exclusive, non-transferable, limited license to

   reproduce and use internally Software complete and unmodified for the sole purpose of running programs, unless additional rights to use are explicitly granted in a written document. As appropriate additional licenses for developers are granted in the Supplemental License Terms. The Licensee is entitled to use the MicroStream software for an unlimited number of open-source projects. The prerequisite for this is (i) that the MicroStream software remains closed source, and (ii) that you have registered online.

4. License to distribute software. Subject to the terms and conditions of this agreement, MicroStream grants you a non-exclusive, nontransferable, limited license to reproduce and distribute the Software, provided that (i) you distribute the Software complete and unmodified and only bundled as part of, and for the sole purpose of running, your Programs, (ii) the Programs add significant and primary functionality to the Software, (iii) you do not distribute additional software intended to replace any component(s) of the Software, (iv) you do not remove or alter any proprietary legends or notices contained in the Software, (v) you only distribute the Software subject to a license agreement that protects MicroStream's interests consistent with the terms contained in this Agreement, (vi) you agree to defend and indemnify MicroStream and its licensors from and against any damages, costs, liabilities, settlement amounts and/or expenses (including attorneys' fees) incurred in connection with any claim, lawsuit or action by any third party that arises or results from the use or distribution of any and all Programs and/or Software, and (vii) that you have registered online.

5. RESTRICTIONS.

6. Proprietary Notices. The software is owned by the licensor, confidential, copyrighted, and licensed, not sold. Title to Software and all associated intellectual property rights is retained by MicroStream and/or its licensors.

7. Reverse Engineering. THE LICENSEE MAY NOT REVERSE ENGINEER, DECOMPILE OR DISASSEMBLE THE SOFTWARE. THE LICENSEE MAY NOT MODIFY, ADAPT, TRANSLATE, RENT, LEASE, LOAN OR CREATE DERIVATIVE WORKS BASED UPON THE MICROSTREAM SOFTWARE OR ANY PART THEREOF.

8. Ethnic Restriction. The Licensee acknowledges that the software is not intended for use in the design, construction, operation or maintenance of any nuclear facilities, aircraft navigation or communication systems, air traffic control systems, life support, machines or machines or other equipment in which failure of the software could lead to death, personal injury, or severe physical or environmental damage. MicroStream disclaims any express or implied warranty of fitness for such uses.

9. Using Trademarks. No right, title or interest in or to any trademark, service mark, logo or trade name of MicroStream or its licensors is granted under this agreement.

10. TRANSFER. The Licensee may not transfer or assign its rights under this license to another party without MicroStream's prior written consent.

11. CHANGES TO THIS AGREEMENT.

12. The Licensor reserves the right at its discretion to change, modify, add or remove terms of use of this Agreement at any time.

13. Any change, modification, addition or removal of the terms of use of this Agreement must be notified to licensee as soon as possible. Such notification will be done by announcement on the MicroStream website.

14. The Licensee will have to agree on such change, modification, addition or removal of the terms of use of this Agreement before the use of the latest version of the MicroStream software will be allowed again. In case of a missing renewed consent by licensee, any further use of the MicroStream software will be automatically denied without any right of compensation or reimbursement of payment being due.

15. In case of modifications and changes of any national or international legal framework having compulsory effect on this Agreement as well as on the provision of any contractual duties, rights and services formerly negotiated between licensor and licensee, licensor shall be allowed to change this Agreement without the explicit consent of the licensee.

16. TERMINATION. This Agreement is effective until terminated. The Licensee may terminate this Agreement at any time by destroying all copies of Software. This Agreement will terminate immediately without notice from MicroStream if the Licensee fails to comply with any provision of this Agreement. Either party may terminate this Agreement immediately should any Software become, or in either party's opinion be likely to become, the subject of a claim of infringement of any intellectual property right. Upon termination, the Licensee must destroy all copies of Software. MicroStream may terminate your license if you fail to comply with the terms of this Agreement. If MicroStream does so, you must destroy all copies of the program and its proof of entitlement.

17. EXPORT REGULATIONS. The Licensee may not use or otherwise export or re-export the Software except as authorized by United States law and the laws of the jurisdiction in which the Software was obtained. In particular, but without limitation, the Software may not be exported or re-exported, (i) into any U.S. embargoed countries or, (ii) to anyone on the U.S. Treasury Department's list of Specially Designated Nationals or the U.S. Department of Commerce's Denied Person's List or Entity List or any other restricted party lists. By using the Software, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use the Software for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture or production of missiles, nuclear, chemical or biological weapons.

18. LIABILITY. Licensor shall only be liable for damages occurring on wilful intent or gross negligence. Licensor shall not be liable for any material defects/damages, including consequential damages, loss of income, business or profit, special, indirect or incidental damages due to the use of the MicroStream software. Licensor's liability for material defects is restricted to those taking place during the transfer of the MicroStream software from the original source to Licensee. Licensee indemnifies Licensor against any claim of third parties due to the use of the MicroStream software. Licensee must assume the entire risk of using the MicroStream software.

19. LIMITED WARRANTIES AND DISCLAIMERS. Unless otherwise set forth in this Agreement, MicroStream warrants for a period of ninety (90) days from your date of download that the Software as provided by MicroStream will perform substantially in accordance with the accompanying documentation. MicroStream's entire liability and your sole and exclusive remedy for any breach of the foregoing limited warranty will be, at MicroStream's option, replacement or repair of the Software.

THIS LIMITED WARRANTY IS THE ONLY WARRANTY PROVIDED BY MICROSTREAM AND MICROSTREAM AND ITS LICENSORS EXPRESSLY DISCLAIM ALL OTHER WARRANTIES, CONDITIONS OR OTHER TERMS, EITHER EXPRESS OR IMPLIED (WHETHER COLLATERALLY, BY STATUTE OR OTHERWISE), INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES, CONDITIONS OR OTHER TERMS OF MERCHANTABILITY, SATISFACTORY QUALITY AND/OR FITNESS FOR A PARTICULAR PURPOSE WITH REGARD TO THE SOFTWARE AND ACCOMPANYING WRITTEN MATERIALS. FURTHERMORE, THERE IS NO WARRANTY AGAINST INTERFERENCE WITH YOUR ENJOYMENT OF THE SOFTWARE OR AGAINST INFRINGEMENT OF THIRD PARTY PROPRIETARY RIGHTS BY THE SOFTWARE. MICROSTREAM DOES NOT WARRANT THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT DEFECTS IN THE SOFTWARE WILL BE CORRECTED. NO ORAL OR WRITTEN INFORMATION OR ADVICE GIVEN BY MICROSTREAM OR AN MICROSTREAM AUTHORIZED REPRESENTATIVE SHALL CREATE A WARRANTY. BECAUSE SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF IMPLIED WARRANTIES, CONDITIONS OR OTHER TERMS THE ABOVE LIMITATION MAY NOT APPLY TO YOU. THE TERMS OF THIS DISCLAIMER AND THE LIMITED WARRANTY UNDER THIS SECTION 9 DO NOT AFFECT OR PREJUDICE THE STATUTORY RIGHTS OF A CONSUMER ACQUIRING THE SOFTWARE OTHERWISE THAN IN THE COURSE OF A BUSINESS, NEITHER DO THEY LIMIT OR EXCLUDE ANY LIABILITY FOR DEATH OR PERSONAL INJURY CAUSED BY MICROSTREAM'S NEGLIGENCE.

1. EXCLUSION AND LIMITATIONS OF REMEDIES AND DAMAGES.

2. Exclusion. IN NO EVENT WILL MICROSTREAM, ITS PARENT, SUBSIDIARIES, OR ANY OF ITS LICENSORS, DIRECTORS, OFFICERS, EMPLOYEES OR AFFILIATES OF ANY OF THE FOREGOING BE LIABLE TO YOU FOR ANY CONSEQUENTIAL, INCIDENTAL, INDIRECT OR SPECIAL DAMAGES WHATSOEVER (INCLUDING WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF BUSINESS INFORMATION AND THE LIKE) OR DIRECT LOSS OF BUSINESS, BUSINESS PROFITS OR REVENUE, WHETHER FORESEEABLE OR UNFORESEEABLE, ARISING OUT OF THE USE OF OR INABILITY TO USE THE SOFTWARE OR ACCOMPANYING WRITTEN MATERIALS, REGARDLESS OF THE BASIS OF THE CLAIM (WHETHER UNDER CONTRACT, NEGLIGENCE OR OTHER TORT OR UNDER STATUTE OR OTHERWISE HOWSOEVER ARISING) AND EVEN IF MICROSTREAM OR A MICROSTREAM REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

END OF TERMS AND CONDITIONS

Prerequisites

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded.configuration</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
</dependencies>

Hello World

// Initialize a storage manager ("the database") with purely defaults.
final EmbeddedStorageManager storageManager = EmbeddedStorage.start();

// print the last loaded root instance, 
// replace it with a current version and store it
System.out.println(storageManager.root());
storageManager.setRoot("Hello World! @ " + new Date());
storageManager.storeRoot();

// shutdown storage
storageManager.shutdown();

This simplest example will create a new storage if no existing storage is found, if a existing storage is found it will be loaded (this is all done at line 2 in the example above).

In line 6 the current storage's content is printed.

Line 7 assigns some data to the storage, replacing existing data if there is some.

In line 8 everything gets stored.

The Root Instance

When using MicroStream, your entire database is accessed starting at a root instance. This instance is the root object of an object graph that gets persisted by the MicroStream storage logic. While the root instance can be of any type (for example just a collection or an array), it is a good idea to define an explicit root type specific for the application. In this simple example, it is a class called DataRoot, which wraps a single String.

public class DataRoot
{
    private String content;

    public DataRoot()
    {
        super();
    }

    public String getContent()
    {
        return this.content;
    }

    public void setContent(final String content)
    {
        this.content = content;
    }

    @Override
    public String toString()
    {
        return "Root: " + this.content;
    }
}

More about root instances:

The following code is all that is required to setup a an application backed by a MicroStream database. The application's convenience root instance is defined and an EmbeddedStorageManager instance, linked to the root, is created (and its database managing threads are started). This is a fully operational Java database application.

This call is all that is necessary to store data in the simplest case.

Best practice is to safely shutdown the storage manager by simply calling:

The EmbeddedStorageManager is mostly created with factory methods of EmbeddedStorage, where the most common settings, like database directory or the root instance, can be configured.

EmbeddedStorageManager storageManager = EmbeddedStorage.start(
    myRoot,                 // root object of entity graph
    Paths.get("data-dir")    // storage data directory
);

Foundations

To achieve a more detailed customization, you can utilize the EmbeddedStorageFoundation factory type. It holds and creates on demand all the parts that form an EmbeddedStorageManager.

NioFileSystem          fileSystem     = NioFileSystem.New();
EmbeddedStorageManager storageManager = EmbeddedStorageFoundation.New()
    .setConfiguration(
        StorageConfiguration.Builder()
          .setStorageFileProvider(
        		    Storage.FileProviderBuilder(fileSystem)
        			    .setDirectory(fileSystem.ensureDirectoryPath("storageDir"))
        			    .createFileProvider()
        	)
          .setChannelCountProvider(StorageChannelCountProvider.New(4))
          .setBackupSetup(StorageBackupSetup.New(
              fileSystem.ensureDirectoryPath("backupDir")
          ))
          .createConfiguration()
    )
    .createEmbeddedStorageManager();

External Configuration

The artifact storage.embedded.configuration provides a convenience layer for configuration purposes, as well as facilities to read external configuration.

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded.configuration</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
</dependencies>

The Configuration type consolidates the most widely used parameters from the storage foundations in one place. It's output is an EmbeddedStorageFoundation from which a EmbeddedStorageManager can be created.

To read an external configuration use ConfigurationLoader and ConfigurationParser or the Load*() methods of Configuration. Currently XML and INI files are supported.

If you just use Configuration.Load() the default configuration file is used, which is either a file in the classpath root named microstream-storage.properties, or the path configured via the system property microstream.storage.configuration.path.

Channel Usage

Channels are the IO threads used by the MicroStream storage engine. A single channel represents the unity of a thread, a storage directory and cached data. Increasing the number of channels means to run more IO threads.

The channel count is an important configuration value that impacts to IO performance.

Channel Configuration

For the channel configuration the following configuration properties are available:

• Channel count

  The number of channels that MicroStream will use. Must be $2^n$

• Channel directory prefix

  The channel directory will be prefix+channelNumber e.g. "ch_0" if prefix is "ch_"

• Data file prefix default is "channel_"

• Data file suffix deflaut id ".dat"

Channel file size configuration is done by the the .

They can be set By storage.embedded.configuration API:

See also:

Or by setting a StorageFileProvider using theEmbeddedStorageFoundationfactory

Intervall and Time Budget

Housekeeping interval and time budget is configured by setting up a StorageHousekeepingController.

Available properties are:

• House keeping interval interval the housekeeping is triggered in milliseconds, default once per every second

• time budget for housekeeping in nanoseconds, default is 0.01 seconds

The desired file min and max sizes and payload ratio is configured by the StorageDataFileEvaluator:

available properties are:

• Files smaller then minimum file size will be merged with other files if possible, default is 1 MB.

• Files larger then maximum file size will be split in smaller ones, default is 8 MB.

• Ratio of non-gap data contained in a storage file to prevent the file from being dissolved, default is 0.75 (75%).

The lifetime of objects in the internal entity cache can be configured by the StorageEntityCacheEvaluator:

Available properties are:

• Entity cache threshold

  Abstract threshold value, roughly comparable to size in bytes with a time component, at which a cache must be cleared of some entities. Default is 1000000000.

• Entity cache timeout Time in milliseconds after that an entity is considered to be old if not read meanwhile. must be greater zero, default is 86400000ms (1 day).

In this chapter it is explained how Lazy Loading is done with MicroStream.

Of course, it's not really about the technical implementation of procrastination, but about efficiency: why bloat the limited RAM with stuff before you even need it?

Classic example: The application has self-contained data areas that contain a large amount of data. The data for an area is not loaded if the area is not worked at all. Instead, you only load a tiny amount of "head data" for each area (name or other for displaying purposes) and the actual data only when the application really needs it. E.g. fiscal years with hundreds of thousands or millions of sales. One million revenue records for 2010, one million for 2011, for 2012, etc. In 2019, most of the time only 2019 and 2018 will be needed. The previous few, and the year 2000 sales are not of great interest anymore. Therefore: load data only when needed. Super efficient.

For example let's say the app "MyBusinessApp" has a root instance class, looking like this:

public class MyBusinessApp
{
    // ...

    private HashMap<Integer, BusinessYear> businessYears = new HashMap<>(); 

    // ...
}

The business year hold the turnovers:

public class BusinessYear
{
    // ...

    private ArrayList<Turnover> turnovers = new ArrayList<>();

    // ...
}

This approach would be a problem: During initialization, the root instance would be loaded, from there its HashMap with all BusinessYear instances, each with its ArrayList and thus all conversions. For all years. 20 years of approximately 1 million sales makes 20 million entities, which are pumped directly into the RAM at the start. It does not matter if someone needs it or not. We don't want it that way.

It would be nice if you could simply add a "lazy" to the turnover list. And that's exactly how it works:

And bingo, the turnovers are now loaded lazily.

What do you have to do now to get the actual ArrayList<Turnover> instance?

Just as with WeakReference, or simply as one would expect from a reference intermediate type in general: a simple get method.

The .get() call reloads the data as needed. But you do not want to mess around with that yourself. No "SELECT bla FROM turnovers WHERE ID =" + this.turnovers.getId(). Since you want to program your application you don' t have to mess around with low-level database ID-loading stuff. That's what the MicroStream Code does internally. You do not even need to access the ID, you just have to say "get!".

That's it.

There are different strategies, what you write here. Analogous to the code example before it would be simply:

So always a new ArrayList instance, wrapped in a Lazy instance. If the actual ArrayList reference should be null at first, it works the same way:

The this.turnovers.get() also just always returns null. Completely transparent.

But you could also do this:

If there is no list, then you do not make any intermediate reference instance for any list. A separate instance for null is indeed a bit ... meh.

But that has a nasty problem elsewhere: this.turnovers.get() does not work then. Because NullPointerException.

Anytime you need to write this here, the readability of code is not exactly conducive:

But there is a simple solution: Just move this check into a static util method. Just like that:

This is the same .get(), just with a static null-check around it. This always puts you on the safe side.

For Lazy Loading, simply wrap Lazy<> around the actual field and then call .get() or maybe better Lazy.get(...).

It's as simple as that.

Why do you have to replace your actual instance with a lazy loading intermediate instance and fiddle around with generics? Why is not something like this:

Put simply: If it were just that it would be bare Java bytecode for accessing an ArrayList. There would be no way for a middleware library to get access and look it up and perhaps reload it. What's written there is an ArrayList reference. There is no lazy anymore. Either, the instance is null, or it is not null. If you wanted to reach in there, you would have to start with bytecode manipulation. Technically possible, but something you really don't want in your application.

So there must always be some form of intermediary.

Hibernate solves this through its own collection implementations that do lazy loading internally. Although the lazy loading is nicely hidden in some way (or not, if you need an annotation for that), it also comes with all sorts of limitations. You can only use interfaces instead of concrete classes for collections. At first, the instance is not the one you dictate, the code becomes intransparent and difficult to debug, you have to use a collection, even if it's just a single instance, and so on. You want to be able to write anything you want and you want full insight and control (debugability, etc.) over the code.

All this can be done with the tiny Lazy Interim Reference class. No restrictions, no incomprehensible "magic" under the hood (proxy instances and stuff) and also usable for individual instances.

Custom type handlers allow taking control over the storing and loading procedure of specific java types. This is useful to optimize the performance for storing complex objects or in the rare case that it is not possible to store a type with the default type handlers.

Implementation

There are two strategies for a simplified type handler implementation.

A Custom Binary Handler

Implementing a class that extends CustomBinaryHandler and defines a sequence of BinaryFields via the #Field~ static pseudo-constructor methods. Everything else like setting the name, calculating the binary offsets, etc. is then done implicitly via reflection.

The custom type handler must be registered in the CustomTypeHandlerRegistry to enable it:

EmbeddedStorageManager storage = EmbeddedStorage
     .Foundation(WORKINGDIR)
     .onConnectionFoundation(f ->
          f.registerCustomTypeHandlers(new CustomBufferedImageHandler())
     )
     .start(ROOT);

A Static Provider Method

Implementing a class can be skipped altogether by using the method Binary#TypeHandler and passing the BinaryFields directly. Registering such a generically created TypeHandler is not required, either, since Version 3 of MicroStream brought a solution to just define a static method in the entity class that will be recognized and used by MicroStream.

The following is a simple technical example on how a custom binary handler can be easily defined and technically leveraged to optimize storage behavior. E.g. imagine having millions of such objects that now only create 1 database record with a fraction of the required storage space instead of 4 records but hold the same information.

MicroStream's storage can be used as a backing store for the cache. It functions as a CacheWriter as well as a CacheReader, depending on the writeThrough and readThrough configuration. Per default it is used for both.

EmbeddedStorageManager storageManager = EmbeddedStorage.start();
CachingProvider        provider       = Caching.getCachingProvider();  
CacheManager           cacheManager   = provider.getCacheManager();   
CacheConfiguration<Integer, String> configuration = CacheConfiguration
			.Builder(Integer.class, String.class, "my-cache", storageManager)
			.build(); 
Cache<Integer, String> cache = cacheManager.createCache("jCache", configuration);

If you prefer an external configuration, you can link the storage configuration:

keyType = java.lang.Integer
valueType = java.lang.String

readThrough = true
writeThrough = true

storageConfigurationResourceName = microstream-storage.properties

baseDirectory = ~/cache-data
channelCount = 4

Or you can embed the storage configuration using the storage. prefix:

keyType = java.lang.Integer
valueType = java.lang.String

readThrough = true
writeThrough = true

storage.baseDirectory = ~/cache-data
storage.channelCount = 4

Spring example:

These are the available properties of the CacheConfiguration type. The names are used accordingly in the external configuration files. They can be found as constants in CacheConfigurationPropertyNames.

Given is the following entity:

public interface Person extends Entity
{
	public String firstName();
	 
	public String lastName();
}

So how is it done? Since the code generator provides a creator, we can use it to create a new Person.

Person john = PersonCreator.New()
    .firstName("John")
    .lastName("Doe")
    .create();

Let's see what the debugger displays if we run this code:

There's always an entity chain, with

• The identity (PersonEntity) as outer layer

• Then the logic layers, none here in our example

• And the inner most layer is always the data (PersonData), which holds the properties.

The properties can be accessed like defined in the entity's interface:

The creator can also be used to create copies. Just hand over the existing one as template:

This will create a "Mike Doe".

Another predefined logic layer is for logging purposes. Since there is a myriad of loggers out there, MicroStream doesn't provide any special adapter, but a generic type which can be used to adapt to the logging framework of your choice.

Just create a class and implement EntityLogger, and you are good to go.

public class JulLogger implements EntityLogger
{
	@Override
	public void afterUpdate(
		final Entity identity, 
		final Entity data, 
		final boolean successful)
	{
		Logger.getLogger(identity.getClass().getName())
			.info("Entity updated");
	}
}

Additional to afterUpdate there are further hooks:

• entityCreated

• afterRead

• beforeUpdate

Now just add the logger when creating entities:

When you call

the logger's output is

The wrapper code generator is an annotation processor, provided by the base module.

The maven configuration looks like this:

<dependencies>
	<dependency>
		<groupId>one.microstream</groupId>
		<artifactId>base</artifactId>
		<version>04.00.00-MS-GA</version>
	</dependency>
</dependencies>

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.8.1</version>
			<configuration>
				<source>1.8</source>
				<target>1.8</target>
				<encoding>UTF-8</encoding>
				<annotationProcessors>
					<annotationProcessor>one.microstream.wrapping.codegen.WrapperProcessor</annotationProcessor>
				</annotationProcessors>
				<compilerArgs>
					<arg>-Amicrostream.wrapper.types=one.microstream.persistence.types.PersistenceStoring</arg>
				</compilerArgs>
			</configuration>
		</plugin>
	</plugins>
</build>

There are following ways to get the base wrapper types generated. If you want it for your own types, the best way is to use the GenerateWrapper annotation.

@GenerateWrapper
public interface MyInterface
{
	public void doStuff();
	
	public String getStuff();
}

Or, if you want it for interfaces in libraries, like PersistenceStoring, you cannot add an annotation. That's what the microstream.wrapper.types parameter is for. This is just a comma separated list of types. Alternatively you can use the GenerateWrapperFor annotation:

It accepts a list of type names. Plain strings have to be used instead of class literals, because it is read inside the compilation cycle which prohibits access to class elements.

The layered entities code generator is an annotation processor, provided by the base module.

The maven configuration looks like this:

<dependencies>
	<dependency>
		<groupId>one.microstream</groupId>
		<artifactId>base</artifactId>
		<version>04.00.00-MS-GA</version>
	</dependency>
</dependencies>

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.8.1</version>
			<configuration>
				<source>1.8</source>
				<target>1.8</target>
				<encoding>UTF-8</encoding>
				<annotationProcessors>
					<annotationProcessor>one.microstream.entity.codegen.EntityProcessor</annotationProcessor>
				</annotationProcessors>
				<compilerArgs>
					<arg>-Amicrostream.entity.hashequalator=true</arg>
					<arg>-Amicrostream.entity.appendable=true</arg>
				</compilerArgs>
			</configuration>
		</plugin>
	</plugins>
</build>

If you don't want the HashEqualator to be generated, just set the microstream.entity.hashequalator argument to false. You can leave it out otherwise, the default value is true.

The same applies to the Appendable.

Object instances can be stored as simple records. One value after another as a trivial byte stream. References between objects are mapped with unique numbers, called ObjectId, or short OID. With both combined, byte streams and OIDs, an object graph can be stored in a simple and quick way, as well as loaded, as a whole or partially.

But there is a small catch. Where does it start? What is the first instance or reference at startup? Strictly speaking "nothing". That's why at least one instance or a reference to an instance must be registered in a special way, so that the application has a starting point from where the object graph can be loaded. This is a "Root Instance".

Same difference, another problem are instances which are references by constant fields in Java classes. These aren't created when the records are loaded from the database, but by the JVM while loading the classes. Without special treatment, this would be a problem:

MicroStream provides an API to import and export persisted data of the storage. It is pretty much the same as writing and reading a backup.

The records in the storage are distributed in lots of files and folders, depending on channel count and other . To get order in the chaos the export produces one file per type. This files are used again by the import to read the data into the storage.

It is also possible to convert the exported binary files to a human readable format, namely CSV.

GET [instance-name]/root

Returns the name and object id of the current storage root element.

GET [instance-name]/object/:objectid

Returns description and values of a distinct object.

An arbitrary amount of logic layers can be added to entities. Let's use the predefined versioning layer. It will keep track of all changes. Technically every new data layer which is added by the updater, will create a new version entry.

Let's have a look at the debugger:

Now the versioning layer is chained between the identity layer and the data layer.

If we update the entity a few times, we will see how the versioning layer works. In this case we use an auto-incrementing Long as key.

If you want to access older versions use the context:

MicroStream offers a Hibernate cache region factory, which can be found in the cache.hibernate module.

The region factory's class name is one.microstream.cache.hibernate.CacheRegionFactory. It is configured via the property hibernate.cache.region.factory_class.

Depending on your environment it can be configured in different ways.

If you use a good old persistence.xml, set the property there:

Or for Spring applications:

Here is an overview of how to enable and configure different levels of user interaction for the Legacy Type Mapping.

Somewhere you have a foundation instance, a foundation in where everything is configured, from which the StorageManager is created.

EmbeddedStorageFoundation<?> foundation = 
    EmbeddedStorage.Foundation(); // or from somewhere else

It itself contains a foundation for connections. To access the inner thing needs a little detour.

Incidentally, that's not a JDBC connection, but this is just one thing that creates helper instances like Storer and Loader. Because Legacy Type Mapping affects loading, it has to go in there.

Either you access it directly, like this:

Or like this, that's better for method chaining.

If you have that, the configuration for the Legacy Type Mapping callback logic is just a one liner:

That's just the necessary logic, without anything further. If you do not change anything, this is done by default.

That wraps a printer around the necessary logic. All these storage and persistence classes are nothing sacred or super duper intertwined or anything. These are just interfaces and if you plug in another implementation then it will be used.

Resultor which asks the user to apply. More customization is possible, see below.

With the implementation of just one single interface method, you can build anything else you can imagine. For example, logging to a file instead of the console. Or in the personally preferred logging framework. Or write confirmed mappings into the refactorings file. Everything is possible.

For the inquiring implementation (InquiringLegacyTypeMappingResultor) there are a few settings: When should he ask? Always or only if something is unclear. Never does not make any sense of course, then you shouldn't use it, or alternatively the printing resultor.

When is a mapping unclear? If at least one field mapping is not completely clear. A field mapping is clear if:

1. If two fields are exactly the same (similarity 1.0 or 100%)

2. Or if two fields are specified by the explicit mapping.

So if all fields are clear according to the above rule, then there is no need to ask.

And there is another special case: If a field is discarded that is not explicitly marked as discardable, then as a precaution an inquiry is always done. Although no data is lost, but the data would not be available in the application, so better ask.

There are options to control this a bit finer. You can optionally specify a double as a threshold (from 0.0 to 1.0, otherwise Exception): The value determines how similar two matching fields automatically have to be so that they are not inquired. Example: The value is 0.9 (90%), but a match found is only 0.8 (80%) similar. This is according to the specification too little, there must be an inquiry as a precaution. If you specify 1.0, that means: always ask, everything is really perfectly clear. If you enter 0.0, this means: never ask or only for implicitly dropping fields.

Looks like this:

Here a small example with a Person class.

It should be changed to:

Without explicitly predefined mappings, the inquiry would look like this:

customerid and pin are too different to be automatically assigned to each other. Therefore, it is wrongly assumed that customerid is omitted and pin is new. comment and commerceId are surprisingly similar (75%) and are therefore assigned. But that's not what we want.

Incidentally, it would not matter here what is defined as a threshold: customerid would be eliminated by the implicit decision. This is too delicate not to inquire, so it is always necessary to ask.

To get the mapping right, you have to specify two entries:

• customerid is now called pin

• and comment should be omitted

Then the inquiry looks like this:

Due to the explicit mapping from customerid to pin, the similarity does not matter, it is the mapping that matters. To indicate this, it says "[mapped]" instead of the similarity. The rest is as usual. Only comment is now "[discarded]", according to the mapping. The difference to the above is namely: This is an explicitly predetermined absence. That does not force inquiry.

This clears the way for the threshold:

• If you enter 0.7 or more then you will be asked. As far as everything would be clear, but the mapping of surname to lastName is below the required "minimum similarity", so rather ask.

• If you enter 0.6 or less, you will no longer be asked. Because all assignments are either explicitly specified or they are according to "minimum similarity" similar enough to rely on it.

A recommendation for a good value for the "minimum similarity" is difficult. As soon as one softens rules, there is always the danger of a mistake. See comment example above: is 75% similar to commerceId. Still wrong. Then prefer 80%? Or 90%? Of course it is better, but the danger is still there.

If you want to be sure, just make 1.0 or omit the parameter, then by default 1.0 is taken.

The most important is the explicit mapping anyway : if "enough" is given by the user, there is no need to ask.

MicroStream uses a strictly interface-based architecture. All types in the public API are, whenever possible, interfaces. This offers the best possibilities to extend or exchange parts of the engine. A good ways to enrich a type with features, is the wrapper (decorator) pattern.

For example, let's say we want to add logging to the PersistenceStoring's store(object) method.

Conventionally it would be done that way: A new type, implementing the original interface, would be handed over the wrapped instance, all interface methods have to be implemented and delegated. And in the single method, we wanted to add functionality; the actual implementation of the logging is done.

This produces a lot of overhead. In this case, three methods are just boilerplate code to delegate the calls to the wrapped instance. A common solution for that is to create an abstract base wrapper type for the designated interface, and to reuse it whenever needed.

MicroStream's wrapper code generator generates following wrapper type for PersistenceStoring:

It is not an abstract class, but an interface, which extends the Wrapper interface of the base module, and the wrapped type itself. This offers you the most flexible way to use it in your application.

The Wrapper type is just a typed interface and an abstract implementation of itself.

You can either implement the Wrapper interface and provide the wrapped instance via the wrapped()

Actually, a database is a passive collection of persisted data that can never be live on its own. But the managing thread accessing it can.

When an EmbeddedStorageManager is "started" it is actually just setup with all kinds of default and user-defined settings and definitions. What is actually "started" are the database managing threads that process storing and loading requests.

Of course, for every start() method, there needs to be something like a shutdown() method. So there is in MicroStream:

But is it really necessary to call shutdown? Should it be? What if there's an error and the process stops without calling shutdown()? Will that cause the database to become inconsistent, corrupted, maybe even destroyed?

The answer is: It wouldn't be much of a database solution if a crash could cause any problem in the persisted data. MicroStream data-store is carefully designed in such a fashion that the process it runs in can simply vanish at any point in time and the persisted data will never be corrupted.