The wrapper code generator is an annotation processor, provided by the base module.

The maven configuration looks like this:

<dependencies>
	<dependency>
		<groupId>one.microstream</groupId>
		<artifactId>base</artifactId>
		<version>04.00.00-MS-GA</version>
	</dependency>
</dependencies>

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.8.1</version>
			<configuration>
				<source>1.8</source>
				<target>1.8</target>
				<encoding>UTF-8</encoding>
				<annotationProcessors>
					<annotationProcessor>one.microstream.wrapping.codegen.WrapperProcessor</annotationProcessor>
				</annotationProcessors>
				<compilerArgs>
					<arg>-Amicrostream.wrapper.types=one.microstream.persistence.types.PersistenceStoring</arg>
				</compilerArgs>
			</configuration>
		</plugin>
	</plugins>
</build>

There are following ways to get the base wrapper types generated. If you want it for your own types, the best way is to use the GenerateWrapper annotation.

@GenerateWrapper
public interface MyInterface
{
	public void doStuff();
	
	public String getStuff();
}

Or, if you want it for interfaces in libraries, like PersistenceStoring, you cannot add an annotation. That's what the microstream.wrapper.types parameter is for. This is just a comma separated list of types. Alternatively you can use the GenerateWrapperFor annotation:

@GenerateWrapperFor("one.microstream.persistence.types.PersistenceStoring")
public class WrapperGenerationDummy
{
}

It accepts a list of type names. Plain strings have to be used instead of class literals, because it is read inside the compilation cycle which prohibits access to class elements.

4.0

Features

• File System Abstraction to support a variety of storage targets.

• Exporter API for storage configuration.

Bugfixes

• Better error handling in housekeeping, to avoid storage channel thread interruptions.

• Fixed XML loading bug under Android.

Migration guide

All configuration code, which includes file paths, may be rewritten to use the new Abstract File System. For example see Storage Files or Backup.

3.0.1

Bugfix release for projects using an old non public version.

Bugfixes

• Fixed bug in type dictionary parser for old legacy type handlers

3.0

Features

• Added a convenience layer for defining custom type handlers

• Fully automatic lazy reference managing for clearing older Lazy references as free memory gets lower.

• Completed optimized persistence handling for remaining types in JDK that are reasonable to appear in entity graphs. E.g. Currency, Locale, URI, URL, Path

• Compatibility with projects using java modules ("jigsaw")

• Added JavaDoc for common types like StorageConnection, StorageManager, StorageController, etc.

• Auto-recognition for CSV separator character in configuration files (e.g. legacy type mapping)

• JSR-107 compatible JCache provider with additional Hibernate cache region factory

• Storage REST Service and Client are available to access the storage data via REST, a Java-Wrapper thereof and a Web-UI

Bugfixes

• Fixed a potential race condition during loading

• Fixed a potential race condition and robustness issue during storing

• StorageConnectionFoundation instance is now properly created on demand.

• Removed unnecessary memory consumption exponential to channel count.

• Improved exception messages on invalid configuration values (file sizes etc.)

• Workarounded a JDK bug regarding file locking when importing files (JDK exception says another process is locking the file which is outright wrong)

• Fixed type resolving problems when different ClassLoaders are involved.

• Fixed a bug that caused loading of zero-length arrays to throw an exception despite everything being fine.

• Various smaller bugfixes for unnecessary exceptions in special cases and state robustness after exceptions.

Migration guide

From 2.2 no actions necessary, for older versions see below.

2.2

Features

• Removed SelfStoring without replacement since it could not be used recursively and has no advantages over just creating a static storing utility method for a certain entity.

• Added state validation of value type objects (e.g. String, Integer, BigDecimal, etc.) upon loading. This is hardly relevant in practice, but not having it can lead to confusing hello-world-like test applications.

• EmbeddedStorageManager now implements java.lang.AutoClosable.

• Replaced all provisional RuntimeExceptions with either PersistenceException or StorageException, depending on the architectural level the corresponding source code it located.

• The two technically different root handling concepts ("default" and "custom") have been consolidated in a way that they are the same thing on the API level and interchangeable, meaning no more confusion with those root exception messages.

• All entity fields of type transient EmbeddedStorageManager now get a reference to the used EmbeddedStorageManager instance set upon loading/updating.

• The interfaces around storage managing have been enhanced so that it is now equally valid to just write StorageManager instead of EmbeddedStorageManager. (An EmbeddedStorageManager "is a" StorageManager)

• Slight technical consolidation of Lazy reference handling caused the type Lazy to be moved from the package one.microstream.persistence.lazy to one.microstream.reference. The reason is that the lazy handling has actually no inherent connection to persistence or storage. It's actually just a generic concept that can be used by those layers. See Migration Guide below on how to adjust existing projects.

Bugfixes

• Fixed an off-heap memory leak when restarting the storage multiple times in the same process.

• Fixed a bug where changing the fields of an entity type caused an exception. This was a regression bug from fixing a similar problem for another case in version 2.1. Now, both cases work correctly.

Migration Guide

All occurrences in user code of one.microstream.persistence.lazy.Lazy have to be refactored to one.microstream.reference.Lazy. Modern IDEs provide a functionality to "auto-import" missing types or automatically "organize imports", so this should be resolved with a proverbial push of a button.

2.1

Features

• Android support MicroStream is now Java-wise fully compatible with Android.

• Replaced all usages of java.util.File with java.nio.file.Path to allow using custom file implementations.

• Improved skipping functionality of Storers (see EmbeddedStorageManager#createStorer and Storer#skip).

• The class Lazy is now an interface to allow custom implementations. See Migration guide below.

Bugfixes

• Fixed a few minor bugs in the skipping functionality of Storers.

• Fixed a bug where files remained locked after the storage was shut down.

• Fixed a bug where files remained locked after an exception in storage initialization.

• Enums defining an abstract method are now handled correctly.

• By default, all threads created by MicroStream now start with the prefix "MicroStream-". This can be customized by the new interface StorageThreadNameProvider.

• Fixed a NullPointerException in import.

• Fixed a bug that caused enums with a certain field layout to be loaded inconsistently.

• java.util.Locale is now persisted and created using Locale's #toLanguageTag and #forLanguageTag.

Migration Guide

In the directory of an existing storage, in the TypeDictionary file (default name "PersistenceTypeDictionary.ptd"), all occurances of "one.microstream.persistence.lazy.Lazy" must be replaced with "one.microstream.persistence.lazy.Lazy$Default".

Welcome to the MicroStream Reference Manual. This manual includes concepts, instructions and examples to guide you on how to use MicroStream Data-Store and Cache, version 4.0.

You should be familiar with the Java programming language and you should have installed your preferred Integrated Development Environment (IDE). But since you are here we guest you got that covered ;)

What's new in 4.0

API Docs

The API documentation is available at https://api.docs.microstream.one/4.0/.

Support

For information on the commercial support for MicroStream see microstream.one.

License

MicroStream Data-Store is a native Java object graph storage engine. From a technical point of view it serves one purpose only: To fully or partially persist and restore a Java object graph in the simplest way possible for the user.

MicroStream Data-Store is a storage engine, but no database management system (DBMS). Many features that typical DBMS provide have been left out on purpose. The reason is that those features exist to make a DBMS something of a server application platform of an "old kind" on top of its data store functionality: A standalone process with user management, connection management, session handling, often even with a programming language of its own, a querying interface (SQL), etc. Today, all of those server application features are already and much better handled by dedicated server applications (the "new kind"), implemented in a modern language like Java. They have their built-in user, connection and session management, the querying interface to the outside world are typically web services instead of SQL, etc. But those modern server applications still lack one important thing: an easy to use and technically efficient way to store and restore their application's data. So a "new kind" server often uses an "old kind" server just to do the data storing. This comes at the price of catching all the overhead and problems of redundant user, connection and session management AND the outdated concepts and limitations of the old querying interface (SQL). Isn't that very weird and frustratingly complicated? Why not simply include a modern data storing library in the modern server and be done with it? A storing library that perfectly fits the modern technology and brings in no redundant overhead or complication of a secondary outdated wannabe server process. This is exactly what MicroStream Data-Store is and the reason why it is intentionally not a DBMS but "only" a storage engine.

One might think the easiest way to store and load data in Java would be Java's built-in serialization. However, it turned out long ago to be very limited, making it hard, if not impossible, to be used as a replacement for a DBMS:

• Only complete object graphs can be stored and restored, which is unacceptable for all but very small databases.

• It is very inefficient in terms of storage size and performance.

• It does not handle changing class structures very well, basically forbidding classes of persisted entities to ever change or introducing massive manual effort to compensate.

• It cannot handle third-party classes that do not implement Serializable but cannot be changed.

In short: The Java Serialization is not an acceptable data store solution and hence no valid replacement for those outdated DBMS.

MicroStream Data-store is such a solution:

• It can persist, load or update object graphs partially and on-demand.

• It is very efficient both size- and performance-wise.

• It handles changing class structures by mapping data in the old structure to the current structure during loading; implicitly via internal heuristics or explicitly via a user-defined mapping strategy.

• It can automatically handle any Java constructs, only excluding those that are technically or reasonably not persistable (e.g. lambdas, proxies or instances with ties to JVM-internals like threads, etc.).

MicroStream is what the Java Serialization should have been and it is the first and only really fitting data storing solution for modern applications, completely removing the need to attach a wannabe secondary server DBMS just to store data.

Supported JDKs

Tested and officially supported JDKs:

In theory MicroStream is compatible with all JDK distributions from Version 8 on.

Supported Operating Systems

• Every desktop or server operating system which the supported JVMs are available for

• Android 8+

Third Party Libraries

MicroStream's core itself doesn't have any dependencies to other libraries whatsoever. So you don't have to worry about potential conflicts in your environment. This was a matter of choice of ours to keep the life of the developers using MicroStream as simple as possible. On the other hand feel free to include any dependencies you need, MicroStream will play along well, e.g. a logging framework of your choice.

MicroStream – Persistence – License Agreement Version 1, September 16, 2019, MicroStream Software GmbH

IMPORTANT: READ THE TERMS OF THIS AGREEMENT CAREFULLY BEFORE DOWNLOADING AND USING THE SOFTWARE. BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, OR USING THE SOFTWARE YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON BEHALF OF ANOTHER PERSON OR A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE FULL AUTHORITY TO BIND THAT PERSON, COMPANY, OR LEGAL ENTITY TO THESE TERMS. IF YOU DO NOT AGREE TO ALL THESE TERMS, YOU MUST NOT DOWNLOAD, INSTALL, COPY, ACCESS, OR USE THE SOFTWARE; AND YOU MUST PROMPTLY DESTROY ALL COPIES OF SOFTWARE AND ALL RELATED MATERIALS.

1. DEFINITIONS. "MicroStream Software GmbH" is a software developing firm based in Regensburg, Germany, hereafter referred to as "MicroStream" or "Licensor", www.microstream.one. "Licensor" refers either to an individual person or to a single legal entity. "Software" is the following, including the original and all whole or partial copies: (i) machine-readable instructions and data, (ii) components, (iii) audio-visual content (such as images, text, recordings, or pictures), (iv) related licensed materials, and (v) license use documents or keys, and documentation. "Agreement" refers to this MicroStream License Agreement.

2. LICENSE TO USE. As between the parties, Licensor reserves all rights in and to the MicroStream software.

3. License to internal use and development. Subject to the terms and conditions of this agreement, MicroStream grants you a non-exclusive, non-transferable, limited license to

   reproduce and use internally Software complete and unmodified for the sole purpose of running programs, unless additional rights to use are explicitly granted in a written document. As appropriate additional licenses for developers are granted in the Supplemental License Terms. The Licensee is entitled to use the MicroStream software for an unlimited number of open-source projects. The prerequisite for this is (i) that the MicroStream software remains closed source, and (ii) that you have registered online.

4. License to distribute software. Subject to the terms and conditions of this agreement, MicroStream grants you a non-exclusive, nontransferable, limited license to reproduce and distribute the Software, provided that (i) you distribute the Software complete and unmodified and only bundled as part of, and for the sole purpose of running, your Programs, (ii) the Programs add significant and primary functionality to the Software, (iii) you do not distribute additional software intended to replace any component(s) of the Software, (iv) you do not remove or alter any proprietary legends or notices contained in the Software, (v) you only distribute the Software subject to a license agreement that protects MicroStream's interests consistent with the terms contained in this Agreement, (vi) you agree to defend and indemnify MicroStream and its licensors from and against any damages, costs, liabilities, settlement amounts and/or expenses (including attorneys' fees) incurred in connection with any claim, lawsuit or action by any third party that arises or results from the use or distribution of any and all Programs and/or Software, and (vii) that you have registered online.

5. RESTRICTIONS.

6. Proprietary Notices. The software is owned by the licensor, confidential, copyrighted, and licensed, not sold. Title to Software and all associated intellectual property rights is retained by MicroStream and/or its licensors.

7. Reverse Engineering. THE LICENSEE MAY NOT REVERSE ENGINEER, DECOMPILE OR DISASSEMBLE THE SOFTWARE. THE LICENSEE MAY NOT MODIFY, ADAPT, TRANSLATE, RENT, LEASE, LOAN OR CREATE DERIVATIVE WORKS BASED UPON THE MICROSTREAM SOFTWARE OR ANY PART THEREOF.

8. Ethnic Restriction. The Licensee acknowledges that the software is not intended for use in the design, construction, operation or maintenance of any nuclear facilities, aircraft navigation or communication systems, air traffic control systems, life support, machines or machines or other equipment in which failure of the software could lead to death, personal injury, or severe physical or environmental damage. MicroStream disclaims any express or implied warranty of fitness for such uses.

9. Using Trademarks. No right, title or interest in or to any trademark, service mark, logo or trade name of MicroStream or its licensors is granted under this agreement.

10. TRANSFER. The Licensee may not transfer or assign its rights under this license to another party without MicroStream's prior written consent.

11. CHANGES TO THIS AGREEMENT.

12. The Licensor reserves the right at its discretion to change, modify, add or remove terms of use of this Agreement at any time.

13. Any change, modification, addition or removal of the terms of use of this Agreement must be notified to licensee as soon as possible. Such notification will be done by announcement on the MicroStream website.

14. The Licensee will have to agree on such change, modification, addition or removal of the terms of use of this Agreement before the use of the latest version of the MicroStream software will be allowed again. In case of a missing renewed consent by licensee, any further use of the MicroStream software will be automatically denied without any right of compensation or reimbursement of payment being due.

15. In case of modifications and changes of any national or international legal framework having compulsory effect on this Agreement as well as on the provision of any contractual duties, rights and services formerly negotiated between licensor and licensee, licensor shall be allowed to change this Agreement without the explicit consent of the licensee.

16. TERMINATION. This Agreement is effective until terminated. The Licensee may terminate this Agreement at any time by destroying all copies of Software. This Agreement will terminate immediately without notice from MicroStream if the Licensee fails to comply with any provision of this Agreement. Either party may terminate this Agreement immediately should any Software become, or in either party's opinion be likely to become, the subject of a claim of infringement of any intellectual property right. Upon termination, the Licensee must destroy all copies of Software. MicroStream may terminate your license if you fail to comply with the terms of this Agreement. If MicroStream does so, you must destroy all copies of the program and its proof of entitlement.

17. EXPORT REGULATIONS. The Licensee may not use or otherwise export or re-export the Software except as authorized by United States law and the laws of the jurisdiction in which the Software was obtained. In particular, but without limitation, the Software may not be exported or re-exported, (i) into any U.S. embargoed countries or, (ii) to anyone on the U.S. Treasury Department's list of Specially Designated Nationals or the U.S. Department of Commerce's Denied Person's List or Entity List or any other restricted party lists. By using the Software, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use the Software for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture or production of missiles, nuclear, chemical or biological weapons.

18. LIABILITY. Licensor shall only be liable for damages occurring on wilful intent or gross negligence. Licensor shall not be liable for any material defects/damages, including consequential damages, loss of income, business or profit, special, indirect or incidental damages due to the use of the MicroStream software. Licensor's liability for material defects is restricted to those taking place during the transfer of the MicroStream software from the original source to Licensee. Licensee indemnifies Licensor against any claim of third parties due to the use of the MicroStream software. Licensee must assume the entire risk of using the MicroStream software.

19. LIMITED WARRANTIES AND DISCLAIMERS. Unless otherwise set forth in this Agreement, MicroStream warrants for a period of ninety (90) days from your date of download that the Software as provided by MicroStream will perform substantially in accordance with the accompanying documentation. MicroStream's entire liability and your sole and exclusive remedy for any breach of the foregoing limited warranty will be, at MicroStream's option, replacement or repair of the Software.

THIS LIMITED WARRANTY IS THE ONLY WARRANTY PROVIDED BY MICROSTREAM AND MICROSTREAM AND ITS LICENSORS EXPRESSLY DISCLAIM ALL OTHER WARRANTIES, CONDITIONS OR OTHER TERMS, EITHER EXPRESS OR IMPLIED (WHETHER COLLATERALLY, BY STATUTE OR OTHERWISE), INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES, CONDITIONS OR OTHER TERMS OF MERCHANTABILITY, SATISFACTORY QUALITY AND/OR FITNESS FOR A PARTICULAR PURPOSE WITH REGARD TO THE SOFTWARE AND ACCOMPANYING WRITTEN MATERIALS. FURTHERMORE, THERE IS NO WARRANTY AGAINST INTERFERENCE WITH YOUR ENJOYMENT OF THE SOFTWARE OR AGAINST INFRINGEMENT OF THIRD PARTY PROPRIETARY RIGHTS BY THE SOFTWARE. MICROSTREAM DOES NOT WARRANT THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT DEFECTS IN THE SOFTWARE WILL BE CORRECTED. NO ORAL OR WRITTEN INFORMATION OR ADVICE GIVEN BY MICROSTREAM OR AN MICROSTREAM AUTHORIZED REPRESENTATIVE SHALL CREATE A WARRANTY. BECAUSE SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF IMPLIED WARRANTIES, CONDITIONS OR OTHER TERMS THE ABOVE LIMITATION MAY NOT APPLY TO YOU. THE TERMS OF THIS DISCLAIMER AND THE LIMITED WARRANTY UNDER THIS SECTION 9 DO NOT AFFECT OR PREJUDICE THE STATUTORY RIGHTS OF A CONSUMER ACQUIRING THE SOFTWARE OTHERWISE THAN IN THE COURSE OF A BUSINESS, NEITHER DO THEY LIMIT OR EXCLUDE ANY LIABILITY FOR DEATH OR PERSONAL INJURY CAUSED BY MICROSTREAM'S NEGLIGENCE.

1. EXCLUSION AND LIMITATIONS OF REMEDIES AND DAMAGES.

2. Exclusion. IN NO EVENT WILL MICROSTREAM, ITS PARENT, SUBSIDIARIES, OR ANY OF ITS LICENSORS, DIRECTORS, OFFICERS, EMPLOYEES OR AFFILIATES OF ANY OF THE FOREGOING BE LIABLE TO YOU FOR ANY CONSEQUENTIAL, INCIDENTAL, INDIRECT OR SPECIAL DAMAGES WHATSOEVER (INCLUDING WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF BUSINESS INFORMATION AND THE LIKE) OR DIRECT LOSS OF BUSINESS, BUSINESS PROFITS OR REVENUE, WHETHER FORESEEABLE OR UNFORESEEABLE, ARISING OUT OF THE USE OF OR INABILITY TO USE THE SOFTWARE OR ACCOMPANYING WRITTEN MATERIALS, REGARDLESS OF THE BASIS OF THE CLAIM (WHETHER UNDER CONTRACT, NEGLIGENCE OR OTHER TORT OR UNDER STATUTE OR OTHERWISE HOWSOEVER ARISING) AND EVEN IF MICROSTREAM OR A MICROSTREAM REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

3. Limitation. MICROSTREAM'S TOTAL LIABILITY TO THE LICENSEE FOR DAMAGES FOR ANY CAUSE WHATSOEVER NOT EXCLUDED BY SECTION 10.1. ABOVE HOWSOEVER CAUSED (WHETHER IN CONTRACT, NEGLIGENCE OR OTHER TORT, UNDER STATUTE OR OTHERWISE HOWSOEVER ARISING) WILL BE LIMITED TO THE GREATER OF U.S.$5.00 OR THE MONEY PAID FOR THE SOFTWARE THAT CAUSED THE DAMAGES. THE PARTIES AGREE THAT THIS LIMITATION OF REMEDIES AND DAMAGES PROVISION SHALL BE ENFORCED INDEPENDENTLY OF AND SURVIVE THE FAILURE OF ESSENTIAL PURPOSE OF ANY WARRANTY REMEDY. THIS LIMITATION WILL NOT APPLY IN THE CASE OF DEATH OR PERSONAL INJURY CAUSED BY FMI'S NEGLIGENCE ONLY WHERE AND TO THE EXTENT THAT APPLICABLE LAW REQUIRES SUCH LIABILITY. BECAUSE SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR CONSEQUENTIAL OR INCIDENTAL DAMAGES, THE LIMITATION OF LIABILITY IN THIS SECTION 6 MAY NOT APPLY TO YOU. NOTHING IN THIS LICENSE AFFECTS OR PREJUDICES THE STATUTORY RIGHTS OF A CONSUMER ACQUIRING THE SOFTWARE OTHERWISE THAN IN THE COURSE OF A BUSINESS.

4. SUBLICENSING. Licensee agrees that all distribution of the runtime and extras will be subject to a written agreement, the terms and conditions of which will, at a minimum: (i) grant a nonexclusive right to use only one copy of the Runtime application and/or Extras for each copy of your own Runtime Solutions which you license to your customer, (ii) provide that any subsequent transfer is subject to the restrictions set forth in this Section 11, (iii) state that the Runtime and Extras (or as renamed) are licensed, not sold, to the end-user and that title to all copies of the Runtime and Extras remain with MicroStream and its licensors, (iv) include restrictions substantially similar to those set forth in Section 3 (RESTRICTIONS) and Section 7 (EXPORT REGULATIONS) of this License, and (v) include Warranty Disclaimer and Disclaimer of Liability provisions which are consistent with and substantially similar to the terms set forth in Sections 5 and 6 of this License.

5. TECHNICAL SUPPORT. You are solely responsible for providing all technical support to your sublicensees of your own runtime solution, and you will not direct any sublicensee to contact MicroStream for technical support regarding your own runtime solution. You further agree to include your name and contact information in your own License Agreement as part of your own runtime solution.

6. INDEMNIFICATION. You will indemnify and hold MicroStream harmless from any and all claims, damages, losses, liabilities, costs and expenses (including reasonable fees of attorneys and other professionals) arising out of or in connection with any runtime solutions distributed by you and which is based on your contributions to such runtime solution.

7. GENERAL. The parties agree that the United Nations Convention on Contracts for the International Sale of Goods (1980), as amended, is specifically excluded from application to this License. This License constitutes the entire agreement between the parties with respect to the Software licensed under these terms, and it supersedes all prior or contemporaneous agreement, arrangement and understanding regarding such subject matter. You acknowledge and agree that you have not relied on any representations made by MicroStream, however, nothing in this license shall limit or exclude liability for any representation made fraudulently. No amendment to or modification of this License will be binding unless in writing and signed by MicroStream.

8. APPLICABLE LAW AND COURT OF JURISDICTION. This agreement shall be governed, subjected to, and construed in accordance with the laws of Germany. All disputes arising from and/or in connection with the present agreement, and/or from any further agreements resulting therefrom, and which the parties are unable to resolve between themselves, shall exclusively be brought before the competent court of jurisdiction in Regensburg, Germany. No choice of law rules of any jurisdiction will apply.

9. SEVERABILITY CLAUSE. If any provision of this agreement shall be held by a court of competent jurisdiction to be contrary to law, that provision will be enforced to the maximum extent permissible, The concerned provision is superseded in accordance with the legal laws, and the remaining provisions of this agreement will remain in full force and effect.

END OF TERMS AND CONDITIONS

Build Configuration

You can find the MicroStream libraries in our :

Modules

These are the different modules that make up MicroStream.

Dependency Graph

Using a Storage Live File Provider (one.microstream.storage.types.StorageLiveFileProvider) allows to specify the location and naming rules for all storage related files.

available properties are:

• BaseDirectory The Microstream storages location base directory. Contains channel directories and type dictionary file.

• DeletionDirectory If configured, the storage will not delete files. Instead of deleting a file it will be moved to this directory.

• TruncationDirectory If configured, files that will get truncated are copied into this directory.

• ChannelDirectoryPrefix Channel directory prefix string

• StorageFilePrefix Storage file prefix string

• StorageFileSuffix storage file extension

• TransactionsFilePrefix transactions file prefix

• TransactionsFileSuffix transaction file extension

• TypeDictionaryFileName filename of the type dictionary

StorageFileNameProvider fileNameProvider = StorageFileNameProvider.Builder()
	.setChannelDirectoryPrefix("canal_")
	.setDataFilePrefix        ("canal_")
	.setDataFileSuffix        (".bin")
	.setTransactionsFilePrefix("events_")
	.setTransactionsFileSuffix(".bin")
	.setTypeDictionaryFileName("typeDictionary.txt")
	.createFileNameProvider   ()
;

NioFileSystem           fileSystem   = NioFileSystem.New();
StorageLiveFileProvider fileProvider = Storage
	.FileProviderBuilder   (fileSystem)
	.setDirectory          (fileSystem.ensureDirectoryPath(WORKINGDIR))
	.setDeletionDirectory  (fileSystem.ensureDirectoryPath(DELETIONDIR))
	.setTruncationDirectory(fileSystem.ensureDirectoryPath(TRUNCATIONDIR))
	.setFileNameProvider   (fileNameProvider)
	.createFileProvider    ()
;

The EmbeddedStorageManager is mostly created with factory methods of EmbeddedStorage, where the most common settings, like database directory or the root instance, can be configured.

EmbeddedStorageManager storageManager = EmbeddedStorage.start(
    myRoot,                 // root object of entity graph
    Paths.get("data-dir")    // storage data directory
);

Foundations

To achieve a more detailed customization, you can utilize the EmbeddedStorageFoundation factory type. It holds and creates on demand all the parts that form an EmbeddedStorageManager.

NioFileSystem          fileSystem     = NioFileSystem.New();
EmbeddedStorageManager storageManager = EmbeddedStorageFoundation.New()
    .setConfiguration(
        StorageConfiguration.Builder()
          .setStorageFileProvider(
        		    Storage.FileProviderBuilder(fileSystem)
        			    .setDirectory(fileSystem.ensureDirectoryPath("storageDir"))
        			    .createFileProvider()
        	)
          .setChannelCountProvider(StorageChannelCountProvider.New(4))
          .setBackupSetup(StorageBackupSetup.New(
              fileSystem.ensureDirectoryPath("backupDir")
          ))
          .createConfiguration()
    )
    .createEmbeddedStorageManager();

External Configuration

The artifact storage.embedded.configuration provides a convenience layer for configuration purposes, as well as facilities to read external configuration.

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded.configuration</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
</dependencies>

The Configuration type consolidates the most widely used parameters from the storage foundations in one place. It's output is an EmbeddedStorageFoundation from which a EmbeddedStorageManager can be created.

EmbeddedStorageManager storageManager = Configuration.Default()
    .setBaseDirectoryInUserHome("data-dir")
    .setBackupDirectory("backup-dir")
    .setChannelCount(4)
    .createEmbeddedStorageFoundation()
    .createEmbeddedStorageManager();

To read an external configuration use ConfigurationLoader and ConfigurationParser or the Load*() methods of Configuration. Currently XML and INI files are supported.

Configuration configuration = Configuration.LoadXml(
    HelloWorld.class.getResource("/META-INF/microstream/storage.xml")
);

<?xml version="1.0" encoding="UTF-8"?>
<properties>
    <property name="baseDirectory" value ="data" />
    <property name="channelCount" value ="4" />
</properties>

Configuration configuration = Configuration.LoadIni(
    HelloWorld.class.getResource("/META-INF/microstream/storage.ini")
);

baseDirectory = data
channelCount = 4

If you just use Configuration.Load() the default configuration file is used, which is either a file in the classpath root named microstream-storage.properties, or the path configured via the system property microstream.storage.configuration.path.

Channel Usage

Channels are the IO threads used by the MicroStream storage engine. A single channel represents the unity of a thread, a storage directory and cached data. Increasing the number of channels means to run more IO threads.

The channel count is an important configuration value that impacts to IO performance.

Channel Configuration

For the channel configuration the following configuration properties are available:

• Channel count

  The number of channels that MicroStream will use. Must be $2^n$

• Channel directory prefix

  The channel directory will be prefix+channelNumber e.g. "ch_0" if prefix is "ch_"

• Data file prefix default is "channel_"

• Data file suffix deflaut id ".dat"

Channel file size configuration is done by the the Storage Data File Evaluator.

They can be set By storage.embedded.configuration API:

EmbeddedStorageManager storageManager = Configuration.Default()
	.setChannelCount(4)
	.setChannelDirectoryPrefix("channel_")
	.setDataFilePrefix("channel_")
	.setDataFileSuffix(".bin")
	.createEmbeddedStorageFoundation()
	.createEmbeddedStorageManager();

<properties>
	<property name="channelCount" value="4" />
	<property name="channelDirectoryPrefix" value="channel_" />
	<property name="dataFilePrefix value="channel_" />
	<property name="dataFileSuffix" value=".dat" />
</properties>

channelCount = 4
channelDirectoryPrefix = prefix
dataFilePrefix = channel_
dataFileSuffix = .dat

See also: Configuration

Or by setting a StorageFileProvider using theEmbeddedStorageFoundationfactory

NioFileSystem          fileSystem = NioFileSystem.New();
EmbeddedStorageManager storage    = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setChannelCountProvider(Storage.ChannelCountProvider(4))
		.setStorageFileProvider(
			StorageLiveFileProvider.Builder()
				.setDirectory(fileSystem.ensureDirectoryPath("storage"))
				.createFileProvider()
		)
		.createConfiguration()
)
.start();

Prerequisites

<repositories>
    <repository>
        <id>microstream-releases</id>
        <url>https://repo.microstream.one/repository/maven-public/</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
    <dependency>
        <groupId>one.microstream</groupId>
        <artifactId>storage.embedded.configuration</artifactId>
        <version>04.00.00-MS-GA</version>
    </dependency>
</dependencies>

Hello World

// Initialize a storage manager ("the database") with purely defaults.
final EmbeddedStorageManager storageManager = EmbeddedStorage.start();

// print the last loaded root instance, 
// replace it with a current version and store it
System.out.println(storageManager.root());
storageManager.setRoot("Hello World! @ " + new Date());
storageManager.storeRoot();

// shutdown storage
storageManager.shutdown();

This simplest example will create a new storage if no existing storage is found, if a existing storage is found it will be loaded (this is all done at line 2 in the example above).

In line 6 the current storage's content is printed.

Line 7 assigns some data to the storage, replacing existing data if there is some.

In line 8 everything gets stored.

The Root Instance

When using MicroStream, your entire database is accessed starting at a root instance. This instance is the root object of an object graph that gets persisted by the MicroStream storage logic. While the root instance can be of any type (for example just a collection or an array), it is a good idea to define an explicit root type specific for the application. In this simple example, it is a class called DataRoot, which wraps a single String.

public class DataRoot
{
    private String content;

    public DataRoot()
    {
        super();
    }

    public String getContent()
    {
        return this.content;
    }

    public void setContent(final String content)
    {
        this.content = content;
    }

    @Override
    public String toString()
    {
        return "Root: " + this.content;
    }
}

More about root instances:

Creating a Database

The following code is all that is required to setup a an application backed by a MicroStream database. The application's convenience root instance is defined and an EmbeddedStorageManager instance, linked to the root, is created (and its database managing threads are started). This is a fully operational Java database application.

// Application-specific root instance
final DataRoot root = new DataRoot();

// Initialize a storage manager ("the database") with the given directory.
final EmbeddedStorageManager storageManager = EmbeddedStorage.start(
    root,             // root object
    Paths.get("data") // storage directory
);

Storing Data

// Set content data to the root element, including the time to visualize 
// changes on the next execution.
root.setContent("Hello World! @ " + new Date());

// Store the modified root and its content.
storageManager.storeRoot();

This call is all that is necessary to store data in the simplest case.

Stopping a Live Database

Best practice is to safely shutdown the storage manager by simply calling:

storageManager.shutdown();

Object instances can be stored as simple records. One value after another as a trivial byte stream. References between objects are mapped with unique numbers, called ObjectId, or short OID. With both combined, byte streams and OIDs, an object graph can be stored in a simple and quick way, as well as loaded, as a whole or partially.

But there is a small catch. Where does it start? What is the first instance or reference at startup? Strictly speaking "nothing". That's why at least one instance or a reference to an instance must be registered in a special way, so that the application has a starting point from where the object graph can be loaded. This is a "Root Instance".

Same difference, another problem are instances which are references by constant fields in Java classes. These aren't created when the records are loaded from the database, but by the JVM while loading the classes. Without special treatment, this would be a problem:

1. The application, meaning the JVM or the JVM process, starts, the constant instances are created by the JVM, one or more of them are stored, then the application shuts down.

2. The stored data of the constants are now stored with a certain OID in the database.

3. The application starts again.

4. The Constant instances are created again by the JVM. The data records are read by MicroStream.

The problem is: How should the application know what values, which are stored with a certain OID, belong to which constant? The JVM created everything from scratch at startup and doesn't know anything about OIDs. To resolve this, the constant instances must be registered, just like the entity graph's root instance. Then MicroStream can associate the constant instances with the stored data via the OIDs. Constant instances can be thought of as JVM-created implicit root instances for the object graph.

In both cases, root and constant instances, it is about registering special starting points for the object graph in order to load it correctly. For MicroStream, from a plain technical view, both cases don't make a difference.

What Must Be Done in the Application?

In the most common cases, nothing at all. The default behavior is enough to get things going.

By default, a single instance can be registered as the entity graph's root, accessible via EmbeddedStorage.root(). Therefore, this is already a fully fledged (although tiny) database application:

// Start the database manager
EmbeddedStorageManager storageManager = EmbeddedStorage.start();
 
// Set the entity (graph) as root
storageManager.setRoot("Hello World");
 
// Store root
storageManager.storeRoot();

Custom Root Instances

The simple default approach has its limits when the application defines an explicit root instance that must be updated/filled from the database directly during database initialization.

Something like this:

// Empty application-specific root, to be filled during start()
MyApplicationRoot root = new MyApplicationRoot();

// Start the database manager
EmbeddedStorageManager storage = EmbeddedStorage.start();

// root must be filled at this point... but how?
root.printAllMyEntities();

To solve this, a custom root instance can be directly registered at the database setup. In the simplest case, is just has to be passed to .start();:

// Empty application-specific root, to be filled during start()
MyApplicationRoot root = new MyApplicationRoot();

// Start the database manager with a reference to the application's root.
EmbeddedStorageManager storageManager = EmbeddedStorage.start(root);

// root is "magically" filled at this point. (Yay!)
root.printAllMyEntities();

Internally, the two concepts (default root and custom root) and handled by different mechanisms. This can be seen from the two different methods

storageManager.defaultRoot();
storageManager.customRoot();

The simplified method storageManager.root(); automatically chooses the variant that is used. Since neither of those three methods can know the concrete type of the root instance (and adding a type parameter just for that would have been a complication overkill), they all can only be typed to return Object. So, to avoid annoying and dangerous casts, it is best to keep a direct reference to a custom root instance as shown in the code snippet above.

Likewise, storageManager.storeRoot(); works for both variants, so there is no need to worry about how to store which one.

These are the available properties of the Configuration type. The names are used accordingly in the external configuration files. They can be found as constants in ConfigurationPropertyNames.

Detailed Description

channelCount

Number of threads used by the storage engine. It depicts the numbers of subdirectories as well. Each thread manages one directory in which it writes to and reads from exclusively. The unity of thread, directory and the cached data therefor is called a "Channel".

__________________________________________________
                               [RAM ]{ Code }    |    (      Filesystem      )
               ,- "Channel 0": [Data]{Thread} <-I|O-> (Storage Subdirectory 0)
              /-- "Channel 1": [Data]{Thread} <-I|O-> (Storage Subdirectory 1)
StorageManager                                   |
              \-- "Channel 2": [Data]{Thread} <-I|O-> (Storage Subdirectory 2)
               '- "Channel 3": [Data]{Thread} <-I|O-> (Storage Subdirectory 3)
_________________________________________________|

houseKeepingIntervalMs

Number of milliseconds for the house keeping interval. House keeping tasks are, among others:

• Garbage Collection

• Cache Check

• File Cleanup Check

In combination with houseKeepingNanoTimeBudget, it can be specified how many CPU time should be used for house keeping. E.g. interval=1000ms and budget=10000000ns means every second there's 0.01 seconds time for house keeping, so max 1% CPU time used for house keeping. This CPU time window is only used if house keeping work is pending. If nothing has to be done, no time is wasted.

housekeepingTimeBudgetNs

Number of nanoseconds used for each housekeeping cycle. However, no matter how low the number is, one item of work will always be completed. But if there is nothing to clean up, no processor time will be wasted. Default is 10000000 (10 million nanoseconds = 10 milliseconds = 0.01 seconds). However, no matter how small the time is, one item is done at least. This is to avoid no-ops, if a too small time window is configured. This time budget is a "best effort" threshold, meaning when at 1ns left, a huge file has to be cleaned or the references of a huge collection have to be marked for GC, then this budget can be exceeded considerably.

dataFileMinimumSize

Minimum file size in bytes of a storage file to avoid merging with other files during housekeeping. Must be greater than 1, maximum is 2GB.

dataFileMaximumSize

Maximum file size in bytes of a storage file to avoid splitting in more files during housekeeping. Must be greater than 1, maximum is 2GB.

dataFileMinimumUseRatio

The ratio (value in ]0.0;1.0]) of non-gap data contained in a storage file to prevent the file from being dissolved. "Gap" data is anything that is not the latest version of an entity's data, inluding older versions of an entity and "comment" bytes (a sequence of bytes beginning with its length as a negative value length header). The closer this value is to 1.0 (100%), the less disk space is occupied by storage files, but the more file dissolving (data transfers to new files) is required and vice versa.

Involved Types

This list shows which property configures which type, used by the foundation types, to create the storage manager.

Intervall and Time Budget

interval and time budget is configured by setting up a StorageHousekeepingController.

Available properties are:

• interval the housekeeping is triggered in milliseconds, default once per every second

• time budget for housekeeping in nanoseconds, default is 0.01 seconds

File Sizes and Payload

The desired file min and max sizes and payload ratio is configured by the StorageDataFileEvaluator:

available properties are:

Cache

The lifetime of objects in the internal entity cache can be configured by the StorageEntityCacheEvaluator:

Available properties are:

• Entity cache threshold

  Abstract threshold value, roughly comparable to size in bytes with a time component, at which a cache must be cleared of some entities. Default is 1000000000.

• Entity cache timeout Time in milliseconds after that an entity is considered to be old if not read meanwhile. must be greater zero, default is 86400000ms (1 day).

The NIO connector can access the local or mounted file systems, as well as different in-memory file systems. This is probably the easiest way to start, especially for prototyping and testing purposes. For productive use, the other file systems are preferred.

By default, MicroStream uses the operation system's standard file locking mechanism to prevent simultaneous access to the storage files. In the rare case that this is not sufficient to control the file access MicroStream provides a proprietary file lock implementation to ensure exclusive access to the storage files from different applications using MicroStream.

Using this file lock may only be necessary if, while a MicroStream application is running, a second MicroStream application may try to access the same storage and the default file locks are not reliable.

You don't need to activate this feature if:

• Only one MicroStream application will access the storage,

• MicroStream applications that may access the same storage run on the same system,

• other applications that may access the storage files don't use MicroStream to access them.

To activate the internal file lock you need to setup StorageLockFileSetup:

The default interval the locks are updated is 10 seconds, you can set a custom value in milliseconds with:

Storage.LockFileSetupProvider( final long updateInterval )

To specify the charset used by the lock files use:

Storage.LockFileSetupProvider( final Charset charset )

or, to customize both:

LockFileSetupProvider( final Charset charset , final long updateInterval )

Continuous Backup

By default, the continuous backup is disabled. If enabled the MicroStream instance will clone all changes to another directory. The backup is identical to the primary MicroStream storage.

To enable the continuous backup just set the backup directory:

With storage.embedded.configuration API:

With MicroStream foundation classes:

MicroStream supports a variety of storage targets. Through an abstracted file system (AFS), it is possible to connect to a lot of different back ends. The AFS allows to use folders and files, like in all common file systems, but with different connectors it is possible to use different solutions as the actual storage.

To connect to the local file system use the Java Non-Blocking IO (NIO) connector, which is part of the base module, so no additional dependency is needed.

Internally this creates and uses a NioFileSystem and is a shortcut for:

The file system API is the same for all connectors, like for MySQL. This is part of another module.

Convenience Methods

Beside long store(Object instance) MicroStream provides some convenience methods to store several objects at once:

void storeAll(Iterable<?> instances)

Stores the passed instance in any case and all referenced instances of persistable references recursively, but stores referenced instances only if they are newly encountered (e.g. don't have an id associated with them in the object registry, yet and are therefore required to be handled). This is useful for the common case of just storing an updated instance and potentially newly created instances along with it while skipping all existing (and normally unchanged) referenced instances.

List subset = myCollection.getSubset();
subset.foreach(item -> item.modify());
storage.storeAll(subset);

long[] storeAll(Object... instances)

Convenience method to store multiple instances. The passed array (maybe implicitly created by the compiler) itself is NOT stored.

storage.storeAll(itemA, iteamB, iteamC);

Transactions

MicroStream does not provide explicit transactions, every call to a store method is automatically a transaction. A store operation is an atomic all or nothing operation. if the store call is successful all data is written to the storage. Otherwise no data is persisted. Partially persisted data will be reverted.

The MicroStream engine supports two general storing strategies: lazy and eager storing. By default, MicroStream uses the lazy storing strategy.

These storing strategies differ in the way how objects, referenced by the object to be stored are handled if those referenced objects had already been stored.

Lazy Storing

Lazy storing is the default storing mode of the MicroStream engine.

Referenced instances are stored only if they have not been stored yet. If a referenced instance has been stored previously it is not stored again even if it has been modified.

Eager Storing

In eager storing mode referenced instances are stored even if they had been stored before. Contrary to Lazy storing this will also store modified child objects at the cost of performance.

Usage

To use lazy or eager storing explicitly, get an instance of the required Storer and use it's store methods:

Storer storer = storage.createEagerStorer();
storer.store(myData);
storer.commit();

Available Storers are:

storage.createLazyStorer()

storage.createEagerStorer()

Standard storing:

storage.createStorer()

will provide corresponding Storer instances.

Custom Handling

Beside the 'global' lazy or eager storing strategies MicroStream allows to implement an individual handling for the storing behavior. See PersistenceEagerStoringFieldEvaluator for details.

The default MicroStream implementation fully supports the Java transient field modifier. Class members marked transient will not be persisted.

public class User 
{
	private String username;
	
	//do not persist the users password
	private transient String password; 
}

It is possible to override the default behavior by implementing a custom PersistenceFieldEvaluator.

PersistenceFieldEvaluator fieldEvaluator = 
	(clazz, field) -> !field.getName().startsWith("_");
			
EmbeddedStorageManager storage = EmbeddedStorage.Foundation()
	.onConnectionFoundation(
		c -> c.setFieldEvaluatorPersistable(fieldEvaluator)
	)
	.createEmbeddedStorageManager()
	.start();

MicroStream is designed to work with object graphs. Thus, storing data means to store an object graph. This includes the object's value fields and references to other objects. Storing an object will also store all instances referenced by this objects that have not been stored before. While storing your data most of the work MicroStream performs for you. You only need to call the store method on the correct object. The rule is: "The Object that has been modified has to be stored!".

Storing objects that are not part of an object graph is most likely pointless.

Storing Root Instances

See Getting Started how to create a database with a root instance.

To store the registered root instance just call the storeRoot() method of a EmbeddedStorageManager instance.

// Init storage manager
final EmbeddedStorageManager storageManager = EmbeddedStorage.start(root);
    
// Store the root object
storageManager.storeRoot();

Storing New Objects

To store a newly created object, store the "owner" of the object. In the example below a new object is created and added to the myObjects list of the root object. Then the modified list gets stored. This will also store the new object.

// Add a new data object to the list in root
MyData dataItem = new MyData("Alice");
root.myObjects.add(dataItem);
    	
// Store the modified list
storageManager.store(root.myObjects);

Storing Modified Objects

Before storing a modified object keep in your mind that the modified object needs to be stored.

In case of a value types, like int, it is the object that has the int field as a member:

// Modify a value type memeber and store it
dataItem.setIntValue(100);
storageManager.store(dataItem);

// Change a string object and store it
dataItem.setName("Bob");
storageManager.store(dataItem);

Storing Hidden Encapsulated Objects

In some cases, it can be necessary to store modified encapsulated objects that cannot be a accessed from your code.

In the upper code snippet the "hidden" object cannot be accessed by store(myForeignObject.hidden) if no getter is available. To allow such hidden objects to be stored after they have been modified you have to options:

1. Set the global storing strategy of the MicroStream instance to or

2. Implement and set a custom for this field.

Use Immutable data models

Manually

The Lazy class has a .clear() method. When called, the reference held in the Lazy Reference is removed and only the ID is kept so that the instance can be reloaded when needed.

Automatically

So that the Lazy References do not have to be managed manually, but the whole goes automatically, there is the following mechanism: Each Lazy instance has a lastTouched timestamp. Each .get() call sets it to the current time. This will tell you how long a Lazy Reference has not been used, i.e. if it is needed at all.

The LazyReferenceManager audits this. It is enabled by default, with a timeout of 1,000,000 milliseconds, which is about 15 minutes.

A custom manager can be set easily, which should happen before a storage is started.

LazyReferenceManager.set(LazyReferenceManager.New(
    Lazy.Checker(
        Duration.ofMinutes(30).toMillis(), // timeout of lazy access
        0.75                               // memory quota
    )
);

The timeout of lazy references is set to 30 minutes, meaning references which haven't been touched for this time are cleared. In combination with a memory quota of 0.75.

The MicroStream engine takes care of persisting your object graph. When you do queries, they are not run on the data stored by MicroStream, queries run on your data in the local system memory. There is no need to use special query languages like SQL. All operations can be done with plain Java. MicroStream does not restrict you in the way you query your object graph. You are totally free to choose the best method fitting to your application.

One possibility may be Streams if you use the standard Java collections.

public List<Article> getUnAvailableArticles()
{
    return shop.getArticles().stream()
        .filter(a -> !a.available())
        .collect(Collectors.toList())
    ;
}

Null-safe Lazy Reference Access

For convenience MicroStream provides Null-safe static access methods for lazy references.

These are

• Lazy.get(Lazy) Gets the lazy referenced object, loads it if required. return value: null if the lazy reference itself is null otherwise the referenced object

• Lazy.peek(Lazy) Get the lazy referenced object if it is loaded, no lazy loading is done. If the object has been unloaded before peek will return null. return value: null if the lazy reference itself is null otherwise the current reference without on-demand loading

• Lazy.clear(Lazy) Clears the lazy reference if it is not null.

Touched Timestamp

All lazy references track the time of their last access (creation or querying) as a timestamp in milliseconds. If an instance is deemed timed out by a LazyReferenzManager its subject gets cleared.

The timestamp is currently not public accessible.

Loading data can be done in two ways, eager and Lazy. The basic, default way of loading is eager loading. This means that all objects of a stored object graph are loaded immediately. This is done during startup of the MicroStream database instance automatically if an already existing database is found.

Contrary to lazy loading, eager loading has no requirements to your entity model.

To load your data you just need to create an EmbeddedStorageManager instance:

EmbeddedStorageManager storage = EmbeddedStorage.start();

After that just get the root instance of your object graph from the StorageManager by calling EmbeddedStorageManager.root() and check for null as this indicates a non-existing database

final EmbeddedStorageManager storage = EmbeddedStorage.start();

if(storage.root() == null)
{
    //No existing Database found
}
else
{    
    MyRoot root = (MyRoot) storage.root();          
}

In this chapter it is explained how Lazy Loading is done with MicroStream.

Of course, it's not really about the technical implementation of procrastination, but about efficiency: why bloat the limited RAM with stuff before you even need it?

Classic example: The application has self-contained data areas that contain a large amount of data. The data for an area is not loaded if the area is not worked at all. Instead, you only load a tiny amount of "head data" for each area (name or other for displaying purposes) and the actual data only when the application really needs it. E.g. fiscal years with hundreds of thousands or millions of sales. One million revenue records for 2010, one million for 2011, for 2012, etc. In 2019, most of the time only 2019 and 2018 will be needed. The previous few, and the year 2000 sales are not of great interest anymore. Therefore: load data only when needed. Super efficient.

For example let's say the app "MyBusinessApp" has a root instance class, looking like this:

public class MyBusinessApp
{
    // ...

    private HashMap<Integer, BusinessYear> businessYears = new HashMap<>(); 

    // ...
}

The business year hold the turnovers:

public class BusinessYear
{
    // ...

    private ArrayList<Turnover> turnovers = new ArrayList<>();

    // ...
}

This approach would be a problem: During initialization, the root instance would be loaded, from there its HashMap with all BusinessYear instances, each with its ArrayList and thus all conversions. For all years. 20 years of approximately 1 million sales makes 20 million entities, which are pumped directly into the RAM at the start. It does not matter if someone needs it or not. We don't want it that way.

It would be nice if you could simply add a "lazy" to the turnover list. And that's exactly how it works:

public class BusinessYear
{
    // ...

    private Lazy<ArrayList<Turnover>> turnovers  = ...; // we will get to that

    // ...
}

And bingo, the turnovers are now loaded lazily.

What do you have to do now to get the actual ArrayList<Turnover> instance?

ArrayList<Turnover> turnovers = this.turnovers.get();

Just as with WeakReference, or simply as one would expect from a reference intermediate type in general: a simple get method.

The .get() call reloads the data as needed. But you do not want to mess around with that yourself. No "SELECT bla FROM turnovers WHERE ID =" + this.turnovers.getId(). Since you want to program your application you don' t have to mess around with low-level database ID-loading stuff. That's what the MicroStream Code does internally. You do not even need to access the ID, you just have to say "get!".

That's it.

What about the "..." ?

There are different strategies, what you write here. Analogous to the code example before it would be simply:

private Lazy<ArrayList<Turnover>> turnovers = Lazy.Reference(new ArrayList<>());

So always a new ArrayList instance, wrapped in a Lazy instance. If the actual ArrayList reference should be null at first, it works the same way:

private Lazy<ArrayList<Turnover>> turnovers = Lazy.Reference(null);

The this.turnovers.get() also just always returns null. Completely transparent.

But you could also do this:

private Lazy<ArrayList<Turnover>> turnovers = null;

If there is no list, then you do not make any intermediate reference instance for any list. A separate instance for null is indeed a bit ... meh.

But that has a nasty problem elsewhere: this.turnovers.get() does not work then. Because NullPointerException.

Anytime you need to write this here, the readability of code is not exactly conducive:

return this.turnovers == null ? null : this.turnovers.get();

But there is a simple solution: Just move this check into a static util method. Just like that:

return Lazy.get(this.turnovers);

This is the same .get(), just with a static null-check around it. This always puts you on the safe side.

In Short

For Lazy Loading, simply wrap Lazy<> around the actual field and then call .get() or maybe better Lazy.get(...).

It's as simple as that.

Side Note

Why do you have to replace your actual instance with a lazy loading intermediate instance and fiddle around with generics? Why is not something like this:

@Lazy
private ArrayList<Turnover> turnovers = new ArrayList<>();

Put simply: If it were just that it would be bare Java bytecode for accessing an ArrayList. There would be no way for a middleware library to get access and look it up and perhaps reload it. What's written there is an ArrayList reference. There is no lazy anymore. Either, the instance is null, or it is not null. If you wanted to reach in there, you would have to start with bytecode manipulation. Technically possible, but something you really don't want in your application.

So there must always be some form of intermediary.

Hibernate solves this through its own collection implementations that do lazy loading internally. Although the lazy loading is nicely hidden in some way (or not, if you need an annotation for that), it also comes with all sorts of limitations. You can only use interfaces instead of concrete classes for collections. At first, the instance is not the one you dictate, the code becomes intransparent and difficult to debug, you have to use a collection, even if it's just a single instance, and so on. You want to be able to write anything you want and you want full insight and control (debugability, etc.) over the code.

All this can be done with the tiny Lazy Interim Reference class. No restrictions, no incomprehensible "magic" under the hood (proxy instances and stuff) and also usable for individual instances.

Stopping or Crashing a Live Database

Actually, a database is a passive collection of persisted data that can never be live on its own. But the managing thread accessing it can.

When an EmbeddedStorageManager is "started" it is actually just setup with all kinds of default and user-defined settings and definitions. What is actually "started" are the database managing threads that process storing and loading requests.

// Setup the database manager and start the managing threads
EmbeddedStorageManager storageManager = EmbeddedStorage.start();

Of course, for every start() method, there needs to be something like a shutdown() method. So there is in MicroStream:

// Stop accessing the database
storageManager.shutdown();

But is it really necessary to call shutdown? Should it be? What if there's an error and the process stops without calling shutdown()? Will that cause the database to become inconsistent, corrupted, maybe even destroyed?

The answer is: It wouldn't be much of a database solution if a crash could cause any problem in the persisted data. MicroStream data-store is carefully designed in such a fashion that the process it runs in can simply vanish at any point in time and the persisted data will never be corrupted.

This is surprisingly simple and reliable to solve: Whenever a .store() call returns, it is guaranteed that the data stored by it has been physically written to the underlying storage layer, usually a file system. Before that, there is no guarantee regarding written data at all. In fact, should the process die before the last byte has been written and secured, the next StorageManager initialization will recognize that and truncate the last partially written store. Either way, all the data that was guaranteed to be written will be consistently available after the next .start().

As a consequence, this safety mechanism makes an explicit .shutdown() call pretty much unnecessary. It doesn't hurt, but it is effectively more-less the same as just calling System.exit(0);.

The only time when an explicit shutdown is really needed is, if the database managing threads shall be stopped but the application itself keeps running. For example, it is perfectly valid to start the StorageManager, work with the database, then stop it, maybe change some configuration or copy files or something like that and then start it up again to continue working.

In any other case, the shutdown method can be ignored and the live database can happily just be "killed" while running. It is specifically designed to withstand such a treatment.

Multiple Databases

Any live MicroStream database basically consists of three major parts:

• A place where the persisted data is located. Usually a file system directory.

• The managing threads accessing (read and write) the persisted data.

• The EmbeddedStorageManager instance to use and control the database in the application.

Apart from a lot of internal components (configuration, processing logic, housekeeping state, etc.), that's all there is. There is nothing special or "magic" about it, no static state, no global registration in the JVM process or something like that.

The consequence of this is: If two EmbeddedStorageManager instances are started, each one with a different location for its persistend data, then the application has two live databases! If three or ten or 100 are started, then that's the number of live databases the application has. There is no limit and no conflict between different databases inside the same application process. The only important thing is that no two running StorageManagers can access the same data location.

Here is an overview of how to enable and configure different levels of user interaction for the Legacy Type Mapping.

Somewhere you have a foundation instance, a foundation in where everything is configured, from which the StorageManager is created.

It itself contains a foundation for connections. To access the inner thing needs a little detour.

Incidentally, that's not a JDBC connection, but this is just one thing that creates helper instances like Storer and Loader. Because Legacy Type Mapping affects loading, it has to go in there.

Either you access it directly, like this:

Or like this, that's better for method chaining.

If you have that, the configuration for the Legacy Type Mapping callback logic is just a one liner:

Default

That's just the necessary logic, without anything further. If you do not change anything, this is done by default.

With Console Output

That wraps a printer around the necessary logic. All these storage and persistence classes are nothing sacred or super duper intertwined or anything. These are just interfaces and if you plug in another implementation then it will be used.

With Inquiry

Resultor which asks the user to apply. More customization is possible, see below.

And so on

With the implementation of just one single interface method, you can build anything else you can imagine. For example, logging to a file instead of the console. Or in the personally preferred logging framework. Or write confirmed mappings into the refactorings file. Everything is possible.

For the inquiring implementation (InquiringLegacyTypeMappingResultor) there are a few settings: When should he ask? Always or only if something is unclear. Never does not make any sense of course, then you shouldn't use it, or alternatively the printing resultor.

When is a mapping unclear? If at least one field mapping is not completely clear. A field mapping is clear if:

1. If two fields are exactly the same (similarity 1.0 or 100%)

2. Or if two fields are specified by the explicit mapping.

So if all fields are clear according to the above rule, then there is no need to ask.

And there is another special case: If a field is discarded that is not explicitly marked as discardable, then as a precaution an inquiry is always done. Although no data is lost, but the data would not be available in the application, so better ask.

There are options to control this a bit finer. You can optionally specify a double as a threshold (from 0.0 to 1.0, otherwise Exception): The value determines how similar two matching fields automatically have to be so that they are not inquired. Example: The value is 0.9 (90%), but a match found is only 0.8 (80%) similar. This is according to the specification too little, there must be an inquiry as a precaution. If you specify 1.0, that means: always ask, everything is really perfectly clear. If you enter 0.0, this means: never ask or only for implicitly dropping fields.

Looks like this:

Here a small example with a Person class.

It should be changed to:

Without explicitly predefined mappings, the inquiry would look like this:

customerid and pin are too different to be automatically assigned to each other. Therefore, it is wrongly assumed that customerid is omitted and pin is new. comment and commerceId are surprisingly similar (75%) and are therefore assigned. But that's not what we want.

Incidentally, it would not matter here what is defined as a threshold: customerid would be eliminated by the implicit decision. This is too delicate not to inquire, so it is always necessary to ask.

To get the mapping right, you have to specify two entries:

• customerid is now called pin

• and comment should be omitted

Then the inquiry looks like this:

Due to the explicit mapping from customerid to pin, the similarity does not matter, it is the mapping that matters. To indicate this, it says "[mapped]" instead of the similarity. The rest is as usual. Only comment is now "[discarded]", according to the mapping. The difference to the above is namely: This is an explicitly predetermined absence. That does not force inquiry.

This clears the way for the threshold:

• If you enter 0.7 or more then you will be asked. As far as everything would be clear, but the mapping of surname to lastName is below the required "minimum similarity", so rather ask.

• If you enter 0.6 or less, you will no longer be asked. Because all assignments are either explicitly specified or they are according to "minimum similarity" similar enough to rely on it.

A recommendation for a good value for the "minimum similarity" is difficult. As soon as one softens rules, there is always the danger of a mistake. See comment example above: is 75% similar to commerceId. Still wrong. Then prefer 80%? Or 90%? Of course it is better, but the danger is still there.

If you want to be sure, just make 1.0 or omit the parameter, then by default 1.0 is taken.

The most important is the explicit mapping anyway : if "enough" is given by the user, there is no need to ask.

Continuous Backup

Continuous backup mode allows the MicroStream engine to clone all storage actions to a second storage. This storage should be placed on another file system.

To configure and enable the continuous backup see Backup.

Data Export

MicroStream provides the possibility to import and export data, see Import / Export.

MicroStream provides an API to import and export persisted data of the storage. It is pretty much the same as writing and reading a backup.

The records in the storage are distributed in lots of files and folders, depending on channel count and other settings. To get order in the chaos the export produces one file per type. This files are used again by the import to read the data into the storage.

NioFileSystem          fileSystem = NioFileSystem.New();
	
EmbeddedStorageManager storage    = EmbeddedStorage.start(
  fileSystem.ensureDirectoryPath("storage")
);

String                            fileSuffix      = "bin";
StorageConnection                 connection      = storage.createConnection();
StorageEntityTypeExportStatistics exportResult    = connection.exportTypes(
	new StorageEntityTypeExportFileProvider.Default(
	  fileSystem.ensureDirectoryPath("export-dir"), 
	  fileSuffix
	),
	typeHandler -> true // export all, customize if necessary
);
XSequence<Path> exportFiles = CQL
	.from(exportResult.typeStatistics().values())
	.project(s -> Paths.get(s.file().identifier()))
	.execute()
;

NioFileSystem          fileSystem = NioFileSystem.New();
	
EmbeddedStorageManager storage    = EmbeddedStorage.start(
  fileSystem.ensureDirectoryPath("storage")
);

StorageConnection      connection = storage.createConnection();
connection.importFiles(X.Enum(
	fileSystem.ensureFilePath("type1.bin"),
	fileSystem.ensureFilePath("type2.bin")
));