Object instances can be stored as simple records. One value after another as a trivial byte stream. References between objects are mapped with unique numbers, called ObjectId, or short OID. With both combined, byte streams and OIDs, an object graph can be stored in a simple and quick way, as well as loaded, as a whole or partially.

But there is a small catch. Where does it start? What is the first instance or reference at startup? Strictly speaking "nothing". That's why at least one instance or a reference to an instance must be registered in a special way, so that the application has a starting point from where the object graph can be loaded. This is a "Root Instance".

Same difference, another problem are instances which are references by constant fields in Java classes. These aren't created when the records are loaded from the database, but by the JVM while loading the classes. Without special treatment, this would be a problem:

1. The application, meaning the JVM or the JVM process, starts, the constant instances are created by the JVM, one or more of them are stored, then the application shuts down.

2. The stored data of the constants are now stored with a certain OID in the database.

3. The application starts again.

4. The Constant instances are created again by the JVM. The data records are read by MicroStream.

The problem is: How should the application know what values, which are stored with a certain OID, belong to which constant? The JVM created everything from scratch at startup and doesn't know anything about OIDs. To resolve this, the constant instances must be registered, just like the entity graph's root instance. Then MicroStream can associate the constant instances with the stored data via the OIDs. Constant instances can be thought of as JVM-created implicit root instances for the object graph.

In both cases, root and constant instances, it is about registering special starting points for the object graph in order to load it correctly. For MicroStream, from a plain technical view, both cases don't make a difference.

What Must Be Done in the Application?

In the most common cases, nothing at all. The default behavior is enough to get things going.

By default, a single instance can be registered as the entity graph's root, accessible via EmbeddedStorage.root(). Therefore, this is already a fully fledged (although tiny) database application:

// Start the database manager
EmbeddedStorageManager storageManager = EmbeddedStorage.start();
 
// Set the entity (graph) as root
storageManager.setRoot("Hello World");
 
// Store root
storageManager.storeRoot();

Custom Root Instances

The simple default approach has its limits when the application defines an explicit root instance that must be updated/filled from the database directly during database initialization.

Something like this:

// Empty application-specific root, to be filled during start()
MyApplicationRoot root = new MyApplicationRoot();

// Start the database manager
EmbeddedStorageManager storage = EmbeddedStorage.start();

// root must be filled at this point... but how?
root.printAllMyEntities();

To solve this, a custom root instance can be directly registered at the database setup. In the simplest case, is just has to be passed to .start();:

// Empty application-specific root, to be filled during start()
MyApplicationRoot root = new MyApplicationRoot();

// Start the database manager with a reference to the application's root.
EmbeddedStorageManager storageManager = EmbeddedStorage.start(root);

// root is "magically" filled at this point. (Yay!)
root.printAllMyEntities();

Internally, the two concepts (default root and custom root) and handled by different mechanisms. This can be seen from the two different methods

storageManager.defaultRoot();
storageManager.customRoot();

The simplified method storageManager.root(); automatically chooses the variant that is used. Since neither of those three methods can know the concrete type of the root instance (and adding a type parameter just for that would have been a complication overkill), they all can only be typed to return Object. So, to avoid annoying and dangerous casts, it is best to keep a direct reference to a custom root instance as shown in the code snippet above.

Likewise, storageManager.storeRoot(); works for both variants, so there is no need to worry about how to store which one.

Build Configuration

You can find the MicroStream libraries in our :

Modules

These are the different modules that make up MicroStream.

Dependency Graph

MicroStream Data-Store is a native Java object graph storage engine. From a technical point of view it serves one purpose only: To fully or partially persist and restore a Java object graph in the simplest way possible for the user.

MicroStream Data-Store is a storage engine, but no database management system (DBMS). Many features that typical DBMS provide have been left out on purpose. The reason is that those features exist to make a DBMS something of a server application platform of an "old kind" on top of its data store functionality: A standalone process with user management, connection management, session handling, often even with a programming language of its own, a querying interface (SQL), etc. Today, all of those server application features are already and much better handled by dedicated server applications (the "new kind"), implemented in a modern language like Java. They have their built-in user, connection and session management, the querying interface to the outside world are typically web services instead of SQL, etc. But those modern server applications still lack one important thing: an easy to use and technically efficient way to store and restore their application's data. So a "new kind" server often uses an "old kind" server just to do the data storing. This comes at the price of catching all the overhead and problems of redundant user, connection and session management AND the outdated concepts and limitations of the old querying interface (SQL). Isn't that very weird and frustratingly complicated? Why not simply include a modern data storing library in the modern server and be done with it? A storing library that perfectly fits the modern technology and brings in no redundant overhead or complication of a secondary outdated wannabe server process. This is exactly what MicroStream Data-Store is and the reason why it is intentionally not a DBMS but "only" a storage engine.

One might think the easiest way to store and load data in Java would be Java's built-in serialization. However, it turned out long ago to be very limited, making it hard, if not impossible, to be used as a replacement for a DBMS:

• Only complete object graphs can be stored and restored, which is unacceptable for all but very small databases.

• It is very inefficient in terms of storage size and performance.

• It does not handle changing class structures very well, basically forbidding classes of persisted entities to ever change or introducing massive manual effort to compensate.

• It cannot handle third-party classes that do not implement Serializable but cannot be changed.

In short: The Java Serialization is not an acceptable data store solution and hence no valid replacement for those outdated DBMS.

MicroStream Data-store is such a solution:

• It can persist, load or update object graphs partially and on-demand.

• It is very efficient both size- and performance-wise.

• It handles changing class structures by mapping data in the old structure to the current structure during loading; implicitly via internal heuristics or explicitly via a user-defined mapping strategy.

• It can automatically handle any Java constructs, only excluding those that are technically or reasonably not persistable (e.g. lambdas, proxies or instances with ties to JVM-internals like threads, etc.).

MicroStream is what the Java Serialization should have been and it is the first and only really fitting data storing solution for modern applications, completely removing the need to attach a wannabe secondary server DBMS just to store data.

MicroStream – Persistence – License Agreement Version 1, September 16, 2019, MicroStream Software GmbH

IMPORTANT: READ THE TERMS OF THIS AGREEMENT CAREFULLY BEFORE DOWNLOADING AND USING THE SOFTWARE. BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, OR USING THE SOFTWARE YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON BEHALF OF ANOTHER PERSON OR A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE FULL AUTHORITY TO BIND THAT PERSON, COMPANY, OR LEGAL ENTITY TO THESE TERMS. IF YOU DO NOT AGREE TO ALL THESE TERMS, YOU MUST NOT DOWNLOAD, INSTALL, COPY, ACCESS, OR USE THE SOFTWARE; AND YOU MUST PROMPTLY DESTROY ALL COPIES OF SOFTWARE AND ALL RELATED MATERIALS.

1. DEFINITIONS. "MicroStream Software GmbH" is a software developing firm based in Regensburg, Germany, hereafter referred to as "MicroStream" or "Licensor", www.microstream.one. "Licensor" refers either to an individual person or to a single legal entity. "Software" is the following, including the original and all whole or partial copies: (i) machine-readable instructions and data, (ii) components, (iii) audio-visual content (such as images, text, recordings, or pictures), (iv) related licensed materials, and (v) license use documents or keys, and documentation. "Agreement" refers to this MicroStream License Agreement.

2. LICENSE TO USE. As between the parties, Licensor reserves all rights in and to the MicroStream software.

3. License to internal use and development. Subject to the terms and conditions of this agreement, MicroStream grants you a non-exclusive, non-transferable, limited license to

   reproduce and use internally Software complete and unmodified for the sole purpose of running programs, unless additional rights to use are explicitly granted in a written document. As appropriate additional licenses for developers are granted in the Supplemental License Terms. The Licensee is entitled to use the MicroStream software for an unlimited number of open-source projects. The prerequisite for this is (i) that the MicroStream software remains closed source, and (ii) that you have registered online.

4. License to distribute software. Subject to the terms and conditions of this agreement, MicroStream grants you a non-exclusive, nontransferable, limited license to reproduce and distribute the Software, provided that (i) you distribute the Software complete and unmodified and only bundled as part of, and for the sole purpose of running, your Programs, (ii) the Programs add significant and primary functionality to the Software, (iii) you do not distribute additional software intended to replace any component(s) of the Software, (iv) you do not remove or alter any proprietary legends or notices contained in the Software, (v) you only distribute the Software subject to a license agreement that protects MicroStream's interests consistent with the terms contained in this Agreement, (vi) you agree to defend and indemnify MicroStream and its licensors from and against any damages, costs, liabilities, settlement amounts and/or expenses (including attorneys' fees) incurred in connection with any claim, lawsuit or action by any third party that arises or results from the use or distribution of any and all Programs and/or Software, and (vii) that you have registered online.

5. RESTRICTIONS.

6. Proprietary Notices. The software is owned by the licensor, confidential, copyrighted, and licensed, not sold. Title to Software and all associated intellectual property rights is retained by MicroStream and/or its licensors.

7. Reverse Engineering. THE LICENSEE MAY NOT REVERSE ENGINEER, DECOMPILE OR DISASSEMBLE THE SOFTWARE. THE LICENSEE MAY NOT MODIFY, ADAPT, TRANSLATE, RENT, LEASE, LOAN OR CREATE DERIVATIVE WORKS BASED UPON THE MICROSTREAM SOFTWARE OR ANY PART THEREOF.

8. Ethnic Restriction. The Licensee acknowledges that the software is not intended for use in the design, construction, operation or maintenance of any nuclear facilities, aircraft navigation or communication systems, air traffic control systems, life support, machines or machines or other equipment in which failure of the software could lead to death, personal injury, or severe physical or environmental damage. MicroStream disclaims any express or implied warranty of fitness for such uses.

9. Using Trademarks. No right, title or interest in or to any trademark, service mark, logo or trade name of MicroStream or its licensors is granted under this agreement.

10. TRANSFER. The Licensee may not transfer or assign its rights under this license to another party without MicroStream's prior written consent.

11. CHANGES TO THIS AGREEMENT.

12. The Licensor reserves the right at its discretion to change, modify, add or remove terms of use of this Agreement at any time.

13. Any change, modification, addition or removal of the terms of use of this Agreement must be notified to licensee as soon as possible. Such notification will be done by announcement on the MicroStream website.

14. The Licensee will have to agree on such change, modification, addition or removal of the terms of use of this Agreement before the use of the latest version of the MicroStream software will be allowed again. In case of a missing renewed consent by licensee, any further use of the MicroStream software will be automatically denied without any right of compensation or reimbursement of payment being due.

15. In case of modifications and changes of any national or international legal framework having compulsory effect on this Agreement as well as on the provision of any contractual duties, rights and services formerly negotiated between licensor and licensee, licensor shall be allowed to change this Agreement without the explicit consent of the licensee.

16. TERMINATION. This Agreement is effective until terminated. The Licensee may terminate this Agreement at any time by destroying all copies of Software. This Agreement will terminate immediately without notice from MicroStream if the Licensee fails to comply with any provision of this Agreement. Either party may terminate this Agreement immediately should any Software become, or in either party's opinion be likely to become, the subject of a claim of infringement of any intellectual property right. Upon termination, the Licensee must destroy all copies of Software. MicroStream may terminate your license if you fail to comply with the terms of this Agreement. If MicroStream does so, you must destroy all copies of the program and its proof of entitlement.

17. EXPORT REGULATIONS. The Licensee may not use or otherwise export or re-export the Software except as authorized by United States law and the laws of the jurisdiction in which the Software was obtained. In particular, but without limitation, the Software may not be exported or re-exported, (i) into any U.S. embargoed countries or, (ii) to anyone on the U.S. Treasury Department's list of Specially Designated Nationals or the U.S. Department of Commerce's Denied Person's List or Entity List or any other restricted party lists. By using the Software, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use the Software for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture or production of missiles, nuclear, chemical or biological weapons.

18. LIABILITY. Licensor shall only be liable for damages occurring on wilful intent or gross negligence. Licensor shall not be liable for any material defects/damages, including consequential damages, loss of income, business or profit, special, indirect or incidental damages due to the use of the MicroStream software. Licensor's liability for material defects is restricted to those taking place during the transfer of the MicroStream software from the original source to Licensee. Licensee indemnifies Licensor against any claim of third parties due to the use of the MicroStream software. Licensee must assume the entire risk of using the MicroStream software.

19. LIMITED WARRANTIES AND DISCLAIMERS. Unless otherwise set forth in this Agreement, MicroStream warrants for a period of ninety (90) days from your date of download that the Software as provided by MicroStream will perform substantially in accordance with the accompanying documentation. MicroStream's entire liability and your sole and exclusive remedy for any breach of the foregoing limited warranty will be, at MicroStream's option, replacement or repair of the Software.

THIS LIMITED WARRANTY IS THE ONLY WARRANTY PROVIDED BY MICROSTREAM AND MICROSTREAM AND ITS LICENSORS EXPRESSLY DISCLAIM ALL OTHER WARRANTIES, CONDITIONS OR OTHER TERMS, EITHER EXPRESS OR IMPLIED (WHETHER COLLATERALLY, BY STATUTE OR OTHERWISE), INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES, CONDITIONS OR OTHER TERMS OF MERCHANTABILITY, SATISFACTORY QUALITY AND/OR FITNESS FOR A PARTICULAR PURPOSE WITH REGARD TO THE SOFTWARE AND ACCOMPANYING WRITTEN MATERIALS. FURTHERMORE, THERE IS NO WARRANTY AGAINST INTERFERENCE WITH YOUR ENJOYMENT OF THE SOFTWARE OR AGAINST INFRINGEMENT OF THIRD PARTY PROPRIETARY RIGHTS BY THE SOFTWARE. MICROSTREAM DOES NOT WARRANT THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT DEFECTS IN THE SOFTWARE WILL BE CORRECTED. NO ORAL OR WRITTEN INFORMATION OR ADVICE GIVEN BY MICROSTREAM OR AN MICROSTREAM AUTHORIZED REPRESENTATIVE SHALL CREATE A WARRANTY. BECAUSE SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF IMPLIED WARRANTIES, CONDITIONS OR OTHER TERMS THE ABOVE LIMITATION MAY NOT APPLY TO YOU. THE TERMS OF THIS DISCLAIMER AND THE LIMITED WARRANTY UNDER THIS SECTION 9 DO NOT AFFECT OR PREJUDICE THE STATUTORY RIGHTS OF A CONSUMER ACQUIRING THE SOFTWARE OTHERWISE THAN IN THE COURSE OF A BUSINESS, NEITHER DO THEY LIMIT OR EXCLUDE ANY LIABILITY FOR DEATH OR PERSONAL INJURY CAUSED BY MICROSTREAM'S NEGLIGENCE.

1. EXCLUSION AND LIMITATIONS OF REMEDIES AND DAMAGES.

2. Exclusion. IN NO EVENT WILL MICROSTREAM, ITS PARENT, SUBSIDIARIES, OR ANY OF ITS LICENSORS, DIRECTORS, OFFICERS, EMPLOYEES OR AFFILIATES OF ANY OF THE FOREGOING BE LIABLE TO YOU FOR ANY CONSEQUENTIAL, INCIDENTAL, INDIRECT OR SPECIAL DAMAGES WHATSOEVER (INCLUDING WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF BUSINESS INFORMATION AND THE LIKE) OR DIRECT LOSS OF BUSINESS, BUSINESS PROFITS OR REVENUE, WHETHER FORESEEABLE OR UNFORESEEABLE, ARISING OUT OF THE USE OF OR INABILITY TO USE THE SOFTWARE OR ACCOMPANYING WRITTEN MATERIALS, REGARDLESS OF THE BASIS OF THE CLAIM (WHETHER UNDER CONTRACT, NEGLIGENCE OR OTHER TORT OR UNDER STATUTE OR OTHERWISE HOWSOEVER ARISING) AND EVEN IF MICROSTREAM OR A MICROSTREAM REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

3. Limitation. MICROSTREAM'S TOTAL LIABILITY TO THE LICENSEE FOR DAMAGES FOR ANY CAUSE WHATSOEVER NOT EXCLUDED BY SECTION 10.1. ABOVE HOWSOEVER CAUSED (WHETHER IN CONTRACT, NEGLIGENCE OR OTHER TORT, UNDER STATUTE OR OTHERWISE HOWSOEVER ARISING) WILL BE LIMITED TO THE GREATER OF U.S.$5.00 OR THE MONEY PAID FOR THE SOFTWARE THAT CAUSED THE DAMAGES. THE PARTIES AGREE THAT THIS LIMITATION OF REMEDIES AND DAMAGES PROVISION SHALL BE ENFORCED INDEPENDENTLY OF AND SURVIVE THE FAILURE OF ESSENTIAL PURPOSE OF ANY WARRANTY REMEDY. THIS LIMITATION WILL NOT APPLY IN THE CASE OF DEATH OR PERSONAL INJURY CAUSED BY FMI'S NEGLIGENCE ONLY WHERE AND TO THE EXTENT THAT APPLICABLE LAW REQUIRES SUCH LIABILITY. BECAUSE SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR CONSEQUENTIAL OR INCIDENTAL DAMAGES, THE LIMITATION OF LIABILITY IN THIS SECTION 6 MAY NOT APPLY TO YOU. NOTHING IN THIS LICENSE AFFECTS OR PREJUDICES THE STATUTORY RIGHTS OF A CONSUMER ACQUIRING THE SOFTWARE OTHERWISE THAN IN THE COURSE OF A BUSINESS.

4. SUBLICENSING. Licensee agrees that all distribution of the runtime and extras will be subject to a written agreement, the terms and conditions of which will, at a minimum: (i) grant a nonexclusive right to use only one copy of the Runtime application and/or Extras for each copy of your own Runtime Solutions which you license to your customer, (ii) provide that any subsequent transfer is subject to the restrictions set forth in this Section 11, (iii) state that the Runtime and Extras (or as renamed) are licensed, not sold, to the end-user and that title to all copies of the Runtime and Extras remain with MicroStream and its licensors, (iv) include restrictions substantially similar to those set forth in Section 3 (RESTRICTIONS) and Section 7 (EXPORT REGULATIONS) of this License, and (v) include Warranty Disclaimer and Disclaimer of Liability provisions which are consistent with and substantially similar to the terms set forth in Sections 5 and 6 of this License.

5. TECHNICAL SUPPORT. You are solely responsible for providing all technical support to your sublicensees of your own runtime solution, and you will not direct any sublicensee to contact MicroStream for technical support regarding your own runtime solution. You further agree to include your name and contact information in your own License Agreement as part of your own runtime solution.

6. INDEMNIFICATION. You will indemnify and hold MicroStream harmless from any and all claims, damages, losses, liabilities, costs and expenses (including reasonable fees of attorneys and other professionals) arising out of or in connection with any runtime solutions distributed by you and which is based on your contributions to such runtime solution.

7. GENERAL. The parties agree that the United Nations Convention on Contracts for the International Sale of Goods (1980), as amended, is specifically excluded from application to this License. This License constitutes the entire agreement between the parties with respect to the Software licensed under these terms, and it supersedes all prior or contemporaneous agreement, arrangement and understanding regarding such subject matter. You acknowledge and agree that you have not relied on any representations made by MicroStream, however, nothing in this license shall limit or exclude liability for any representation made fraudulently. No amendment to or modification of this License will be binding unless in writing and signed by MicroStream.

8. APPLICABLE LAW AND COURT OF JURISDICTION. This agreement shall be governed, subjected to, and construed in accordance with the laws of Germany. All disputes arising from and/or in connection with the present agreement, and/or from any further agreements resulting therefrom, and which the parties are unable to resolve between themselves, shall exclusively be brought before the competent court of jurisdiction in Regensburg, Germany. No choice of law rules of any jurisdiction will apply.

9. SEVERABILITY CLAUSE. If any provision of this agreement shall be held by a court of competent jurisdiction to be contrary to law, that provision will be enforced to the maximum extent permissible, The concerned provision is superseded in accordance with the legal laws, and the remaining provisions of this agreement will remain in full force and effect.

END OF TERMS AND CONDITIONS

The EmbeddedStorageManager is mostly created with factory methods of EmbeddedStorage, where the most common settings, like database directory or the root instance, can be configured.

Foundations

To achieve a more detailed customization, you can utilize the EmbeddedStorageFoundation factory type. It holds and creates on demand all the parts that form an EmbeddedStorageManager.

External Configuration

The artifact storage.embedded.configuration provides a convenience layer for configuration purposes, as well as facilities to read external configuration.

The Configuration type consolidates the most widely used parameters from the storage foundations in one place. It's output is an EmbeddedStorageFoundation from which a EmbeddedStorageManager can be created.

To read an external configuration use ConfigurationLoader and ConfigurationParser or the Load*() methods of Configuration. Currently XML and INI files are supported.

If you just use Configuration.Load() the default configuration file is used, which is either a file in the classpath root named microstream-storage.properties, or the path configured via the system property microstream.storage.configuration.path.

Prerequisites

Hello World

This simplest example will create a new storage if no existing storage is found, if a existing storage is found it will be loaded (this is all done at line 2 in the example above).

In line 6 the current storage's content is printed.

Line 7 assigns some data to the storage, replacing existing data if there is some.

In line 8 everything gets stored.

The Root Instance

When using MicroStream, your entire database is accessed starting at a root instance. This instance is the root object of an object graph that gets persisted by the MicroStream storage logic. While the root instance can be of any type (for example just a collection or an array), it is a good idea to define an explicit root type specific for the application. In this simple example, it is a class called DataRoot, which wraps a single String.

More about root instances:

Creating a Database

The following code is all that is required to setup a an application backed by a MicroStream database. The application's convenience root instance is defined and an EmbeddedStorageManager instance, linked to the root, is created (and its database managing threads are started). This is a fully operational Java database application.

Storing Data

This call is all that is necessary to store data in the simplest case.

Stopping a Live Database

Best practice is to safely shutdown the storage manager by simply calling:

Welcome to the MicroStream Reference Manual. This manual includes concepts, instructions and examples to guide you on how to use MicroStream and , version 3.0.

You should be familiar with the Java programming language and you should have installed your preferred Integrated Development Environment (IDE). But since you are here we guest you got that covered ;)

What's new in 3.0

API Docs

The API documentation is available at .

Support

For information on the commercial support for MicroStream see .

License

Supported JDKs

Tested and officially supported JDKs:

In theory MicroStream is compatible with all JDK distributions from Version 8 on.

Supported Operating Systems

• Every desktop or server operating system which the supported JVMs are available for

• Android 8+

Third Party Libraries

MicroStream's core itself doesn't have any dependencies to other libraries whatsoever. So you don't have to worry about potential conflicts in your environment. This was a matter of choice of ours to keep the life of the developers using MicroStream as simple as possible. On the other hand feel free to include any dependencies you need, MicroStream will play along well, e.g. a logging framework of your choice.

Using a Storage File Provider (one.microstream.storage.types.StorageFileProvider) allows to specify the location and naming rules for all storage related files.

available properties are:

• BaseDirectory The Microstream storages location base directory. Contains channel directories and type dictionary file.

• DeletionDirectory If configured, the storage will not delete files. Instead of deleting a file it will be moved to this directory.

• TruncationDirectory If configured, files that will get truncated are copied into this directory.

• ChannelDirectoryPrefix Channel directory prefix string

• StorageFilePrefix Storage file prefix string

• StorageFileSuffix storage file extension

• TransactionsFilePrefix transactions file prefix

• TransactionsFileSuffix transaction file extension

• TypeDictionaryFileName filename of the type dictionary

StorageFileProvider fileProvider = Storage.FileProviderBuilder()
	.setBaseDirectory         (WORKINGDIR.getPath())
	.setDeletionDirectory     (DELETIONDIR.getPath())
	.setTruncationDirectory   (TRUNKATIONDIR.getPath())
	.setChannelDirectoryPrefix("canal_")
	.setStorageFilePrefix     ("canal_")
	.setStorageFileSuffix     (".bin")
	.setTransactionsFilePrefix("events_")
	.setTransactionsFileSuffix(".bin")
	.setTypeDictionaryFileName("typeDictionary.txt")
	.createFileProvider();

Intervall and Time Budget

Housekeeping interval and time budget is configured by setting up a StorageHousekeepingController.

Available properties are:

• House keeping interval interval the housekeeping is triggered in milliseconds, default once per every second

• House keeping budget time budget for housekeeping in nanoseconds, default is 0.01 seconds

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setHousekeepingController(Storage.HousekeepingController(1000, 10_000_000))
		.createConfiguration())
	.start();

File Sizes and Payload

The desired file min and max sizes and payload ratio is configured by the StorageDataFileEvaluator:

available properties are:

• Data file minimum size Files smaller then minimum file size will be merged with other files if possible, default is 1 MB.

• Data file maximum size Files larger then maximum file size will be split in smaller ones, default is 8 MB.

• Data file minimum use ratio Ratio of non-gap data contained in a storage file to prevent the file from being dissolved, default is 0.75 (75%).

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setDataFileEvaluator(Storage.DataFileEvaluator(1024*1024, 1024*1024*8, 0.75))
		.createConfiguration())
	.start();

Cache

The lifetime of objects in the internal entity cache can be configured by the StorageEntityCacheEvaluator:

Available properties are:

• Entity cache threshold

  Abstract threshold value, roughly comparable to size in bytes with a time component, at which a cache must be cleared of some entities. Default is 1000000000.

• Entity cache timeout Time in milliseconds after that an entity is considered to be old if not read meanwhile. must be greater zero, default is 86400000ms (1 day).

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setEntityCacheEvaluator(Storage.EntityCacheEvaluator(
			86_400_000, 
			1_000_000_000))
		.createConfiguration())
	.start();

3.0.1

Bugfix release for projects using an old non public version.

Bugfixes

• Fixed bug in type dictionary parser for old legacy type handlers

3.0

Features

• Added a convenience layer for defining custom type handlers

• Fully automatic lazy reference managing for clearing older Lazy references as free memory gets lower.

• Completed optimized persistence handling for remaining types in JDK that are reasonable to appear in entity graphs. E.g. Currency, Locale, URI, URL, Path

• Compatibility with projects using java modules ("jigsaw")

• Added JavaDoc for common types like StorageConnection, StorageManager, StorageController, etc.

• Auto-recognition for CSV separator character in configuration files (e.g. legacy type mapping)

• JSR-107 compatible JCache provider with additional Hibernate cache region factory

• Storage REST Service and Client are available to access the storage data via REST, a Java-Wrapper thereof and a Web-UI

Bugfixes

• Fixed a potential race condition during loading

• Fixed a potential race condition and robustness issue during storing

• StorageConnectionFoundation instance is now properly created on demand.

• Removed unnecessary memory consumption exponential to channel count.

• Improved exception messages on invalid configuration values (file sizes etc.)

• Workarounded a JDK bug regarding file locking when importing files (JDK exception says another process is locking the file which is outright wrong)

• Fixed type resolving problems when different ClassLoaders are involved.

• Fixed a bug that caused loading of zero-length arrays to throw an exception despite everything being fine.

• Various smaller bugfixes for unnecessary exceptions in special cases and state robustness after exceptions.

Migration guide

From 2.2 no actions necessary, for older versions see below.

2.2

Features

• Removed SelfStoring without replacement since it could not be used recursively and has no advantages over just creating a static storing utility method for a certain entity.

• Added state validation of value type objects (e.g. String, Integer, BigDecimal, etc.) upon loading. This is hardly relevant in practice, but not having it can lead to confusing hello-world-like test applications.

• EmbeddedStorageManager now implements java.lang.AutoClosable.

• Replaced all provisional RuntimeExceptions with either PersistenceException or StorageException, depending on the architectural level the corresponding source code it located.

• The two technically different root handling concepts ("default" and "custom") have been consolidated in a way that they are the same thing on the API level and interchangeable, meaning no more confusion with those root exception messages.

• All entity fields of type transient EmbeddedStorageManager now get a reference to the used EmbeddedStorageManager instance set upon loading/updating.

• The interfaces around storage managing have been enhanced so that it is now equally valid to just write StorageManager instead of EmbeddedStorageManager. (An EmbeddedStorageManager "is a" StorageManager)

• Slight technical consolidation of Lazy reference handling caused the type Lazy to be moved from the package one.microstream.persistence.lazy to one.microstream.reference. The reason is that the lazy handling has actually no inherent connection to persistence or storage. It's actually just a generic concept that can be used by those layers. See Migration Guide below on how to adjust existing projects.

Bugfixes

• Fixed an off-heap memory leak when restarting the storage multiple times in the same process.

• Fixed a bug where changing the fields of an entity type caused an exception. This was a regression bug from fixing a similar problem for another case in version 2.1. Now, both cases work correctly.

Migration Guide

All occurrences in user code of one.microstream.persistence.lazy.Lazy have to be refactored to one.microstream.reference.Lazy. Modern IDEs provide a functionality to "auto-import" missing types or automatically "organize imports", so this should be resolved with a proverbial push of a button.

2.1

Features

• Android support MicroStream is now Java-wise fully compatible with Android.

• Replaced all usages of java.util.File with java.nio.file.Path to allow using custom file implementations.

• Improved skipping functionality of Storers (see EmbeddedStorageManager#createStorer and Storer#skip).

• The class Lazy is now an interface to allow custom implementations. See Migration guide below.

Bugfixes

• Fixed a few minor bugs in the skipping functionality of Storers.

• Fixed a bug where files remained locked after the storage was shut down.

• Fixed a bug where files remained locked after an exception in storage initialization.

• Enums defining an abstract method are now handled correctly.

• By default, all threads created by MicroStream now start with the prefix "MicroStream-". This can be customized by the new interface StorageThreadNameProvider.

• Fixed a NullPointerException in import.

• Fixed a bug that caused enums with a certain field layout to be loaded inconsistently.

• java.util.Locale is now persisted and created using Locale's #toLanguageTag and #forLanguageTag.

Migration Guide

In the directory of an existing storage, in the TypeDictionary file (default name "PersistenceTypeDictionary.ptd"), all occurances of "one.microstream.persistence.lazy.Lazy" must be replaced with "one.microstream.persistence.lazy.Lazy$Default".

Continuous Backup

By default, the continuous backup is disabled. If enabled the MicroStream instance will clone all changes to another directory. The backup is identical to the primary MicroStream storage.

To enable the continuous backup just set the backup directory:

With storage.embedded.configuration API:

EmbeddedStorageManager storageManager = Configuration.Default()
	.setBackupDirectory("A safe place")
	.createEmbeddedStorageFoundation()
	.createEmbeddedStorageManager();

<properties>
	<property name="backupDirectory" value ="/save/backup" />
	...
</properties>

backupDirectory = backupDir

With MicroStream foundation classes:

StorageBackupSetup backupSetup = StorageBackupSetup.New(
    Storage.FileProviderBuilder()
		.setBaseDirectory(BACKUPDIR.getPath())						
		.setTruncationDirectory(TRUNKATIONDIR.getPath())
		.setDeletionDirectory(DELETIONDIR.getPath())
		.createFileProvider()
	);	
				
StorageConfiguration configuration = StorageConfiguration.Builder()
	.setBackupSetup(backupSetup)
	.setStorageFileProvider(StorageFileProvider.New(WORKINGDIR))
	.createConfiguration();

Channel Usage

Channels are the IO threads used by the MicroStream storage engine. A single channel represents the unity of a thread, a storage directory and cached data. Increasing the number of channels means to run more IO threads.

The channel count is an important configuration value that impacts to IO performance.

Channel Configuration

For the channel configuration the following configuration properties are available:

• Channel count

  The number of channels that MicroStream will use. Must be $2^n$

• Channel directory prefix

  The channel directory will be prefix+channelNumber e.g. "ch_0" if prefix is "ch_"

• Data file prefix default is "channel_"

• Data file suffix deflaut id ".dat"

Channel file size configuration is done by the the Storage Data File Evaluator.

They can be set By storage.embedded.configuration API:

embeddedStorageManager storageManager = Configuration.Default()
	.setChannelCount(4)
	.setChannelDirectoryPrefix("channel_")
	.setDataFilePrefix("channel_")
	.setDataFileSuffix(".bin")
	.createEmbeddedStorageFoundation()
	.createEmbeddedStorageManager();

<properties>
	<property name="channelCount" value="4" />
	<property name="channelDirectoryPrefix" value="channel_" />
	<property name="dataFilePrefix value="channel_" />
	<property name="dataFileSuffix" value=".dat" />
</properties>

channelCount = 4
channelDirectoryPrefix = prefix
dataFilePrefix = channel_
dataFileSuffix = .dat

See also: Configuration

Or by setting a StorageFileProvider using theEmbeddedStorageFoundationfactory

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setChannelCountProvider(Storage.ChannelCountProvider(4))
		.setStorageFileProvider(StorageFileProvider.Builder()
			.setBaseDirectory("storage")
			.setChannelDirectoryPrefix("prefix")
			.setStorageFilePrefix("channel-")
        	.setStorageFileSuffix(".bin")
			.createFileProvider()).createConfiguration()
		).start();

These are the available properties of the Configuration type. The names are used accordingly in the external configuration files. They can be found as constants in ConfigurationPropertyNames.

Detailed Description

channelCount

Number of threads used by the storage engine. It depicts the numbers of subdirectories as well. Each thread manages one directory in which it writes to and reads from exclusively. The unity of thread, directory and the cached data therefor is called a "Channel".

__________________________________________________
                               [RAM ]{ Code }    |    (      Filesystem      )
               ,- "Channel 0": [Data]{Thread} <-I|O-> (Storage Subdirectory 0)
              /-- "Channel 1": [Data]{Thread} <-I|O-> (Storage Subdirectory 1)
StorageManager                                   |
              \-- "Channel 2": [Data]{Thread} <-I|O-> (Storage Subdirectory 2)
               '- "Channel 3": [Data]{Thread} <-I|O-> (Storage Subdirectory 3)
_________________________________________________|

houseKeepingIntervalMs

Number of milliseconds for the house keeping interval. House keeping tasks are, among others:

• Garbage Collection

• Cache Check

• File Cleanup Check

In combination with houseKeepingNanoTimeBudget, it can be specified how many CPU time should be used for house keeping. E.g. interval=1000ms and budget=10000000ns means every second there's 0.01 seconds time for house keeping, so max 1% CPU time used for house keeping. This CPU time window is only used if house keeping work is pending. If nothing has to be done, no time is wasted.

housekeepingTimeBudgetNs

Number of nanoseconds used for each housekeeping cycle. However, no matter how low the number is, one item of work will always be completed. But if there is nothing to clean up, no processor time will be wasted. Default is 10000000 (10 million nanoseconds = 10 milliseconds = 0.01 seconds). However, no matter how small the time is, one item is done at least. This is to avoid no-ops, if a too small time window is configured. This time budget is a "best effort" threshold, meaning when at 1ns left, a huge file has to be cleaned or the references of a huge collection have to be marked for GC, then this budget can be exceeded considerably.

dataFileMinimumSize

Minimum file size in bytes of a storage file to avoid merging with other files during housekeeping. Must be greater than 1, maximum is 2GB.

dataFileMaximumSize

Maximum file size in bytes of a storage file to avoid splitting in more files during housekeeping. Must be greater than 1, maximum is 2GB.

dataFileMinimumUseRatio

The ratio (value in ]0.0;1.0]) of non-gap data contained in a storage file to prevent the file from being dissolved. "Gap" data is anything that is not the latest version of an entity's data, inluding older versions of an entity and "comment" bytes (a sequence of bytes beginning with its length as a negative value length header). The closer this value is to 1.0 (100%), the less disk space is occupied by storage files, but the more file dissolving (data transfers to new files) is required and vice versa.

Involved Types

This list shows which property configures which type, used by the foundation types, to create the storage manager.

The default MicroStream implementation fully supports the Java transient field modifier. Class members marked transient will not be persisted.

It is possible to override the default behavior by implementing a custom PersistenceFieldEvaluator.

MicroStream is designed to work with object graphs. Thus, storing data means to store an object graph. This includes the object's value fields and references to other objects. Storing an object will also store all instances referenced by this objects that have not been stored before. While storing your data most of the work MicroStream performs for you. You only need to call the store method on the correct object. The rule is: "The Object that has been modified has to be stored!".

Storing objects that are not part of an object graph is most likely pointless.

Storing Root Instances

See how to create a database with a root instance.

To store the registered root instance just call the storeRoot() method of a EmbeddedStorageManager instance.

Storing New Objects

To store a newly created object, store the "owner" of the object. In the example below a new object is created and added to the myObjects list of the root object. Then the modified list gets stored. This will also store the new object.

Storing Modified Objects

Before storing a modified object keep in your mind that the modified object needs to be stored.

In case of a value types, like int, it is the object that has the int field as a member:

The MicroStream engine supports two general storing strategies: Lazy and eager storing. By default, MicroStream uses the lazy storing strategy.

These storing strategies differ in the way how objects, referenced by the object to be stored are handled if those referenced objects had already been stored.

Lazy Storing

Lazy storing is the default storing mode of the MicroStream engine.

Referenced instances are stored only if they have not been stored yet. If a referenced instance has been stored previously it is not stored again even if it has been modified.

Eager Storing

In eager storing mode referenced instances are stored even if they had been stored before. Contrary to Lazy storing this will also store modified child objects at the cost of performance.

Usage

To use lazy or eager storing explicitly, get an instance of the required Storer and use it's store methods:

Available Storers are:

storage.createLazyStorer()

storage.createEagerStorer()

Standard storing:

storage.createStorer()

will provide corresponding Storer instances.

Custom Handling

By default, MicroStream uses the operation system's standard file locking mechanism to prevent simultaneous access to the storage files. In the rare case that this is not sufficient to control the file access MicroStream provides a proprietary file lock implementation to ensure exclusive access to the storage files from different applications using MicroStream.

Using this file lock may only be necessary if, while a MicroStream application is running, a second MicroStream application may try to access the same storage and the default file locks are not reliable.

You don't need to activate this feature if:

• Only one MicroStream application will access the storage,

• MicroStream applications that may access the same storage run on the same system,

• other applications that may access the storage files don't use MicroStream to access them.

To activate the internal file lock you need to setup StorageLockFileSetup:

The default interval the locks are updated is 10 seconds, you can set a custom value in milliseconds with:

Storage.LockFileSetupProvider( final long updateInterval )

To specify the charset used by the lock files use:

Storage.LockFileSetupProvider( final Charset charset )

or, to customize both:

LockFileSetupProvider( final Charset charset , final long updateInterval )

Convenience Methods

Beside long store(Object instance) MicroStream provides some convenience methods to store several objects at once:

void storeAll(Iterable<?> instances)

Stores the passed instance in any case and all referenced instances of persistable references recursively, but stores referenced instances only if they are newly encountered (e.g. don't have an id associated with them in the object registry, yet and are therefore required to be handled). This is useful for the common case of just storing an updated instance and potentially newly created instances along with it while skipping all existing (and normally unchanged) referenced instances.

long[] storeAll(Object... instances)

Convenience method to store multiple instances. The passed array (maybe implicitly created by the compiler) itself is NOT stored.

Transactions

MicroStream does not provide explicit transactions, every call to a store method is automatically a transaction. A store operation is an atomic all or nothing operation. if the store call is successful all data is written to the storage. Otherwise no data is persisted. Partially persisted data will be reverted.

Storing Hidden Encapsulated Objects

In some cases, it can be necessary to store modified encapsulated objects that cannot be a accessed from your code.

In the upper code snippet the "hidden" object cannot be accessed by store(myForeignObject.hidden) if no getter is available. To allow such hidden objects to be stored after they have been modified you have to options:

1. Set the global storing strategy of the MicroStream instance to or

2. Implement and set a custom for this field.

Use Immutable data models

The MicroStream engine takes care of persisting your object graph. When you do queries, they are not run on the data stored by MicroStream, queries run on your data in the local system memory. There is no need to use special query languages like SQL. All operations can be done with plain Java. MicroStream does not restrict you in the way you query your object graph. You are totally free to choose the best method fitting to your application.

One possibility may be Streams if you use the standard Java collections.

public List<Article> getUnAvailableArticles()
{
    return shop.getArticles().stream()
        .filter(a -> !(a.available())).collect(Collectors.toList());
}

In this chapter it is explained how Lazy Loading is done with MicroStream.

Of course, it's not really about the technical implementation of procrastination, but about efficiency: why bloat the limited RAM with stuff before you even need it?

Classic example: The application has self-contained data areas that contain a large amount of data. The data for an area is not loaded if the area is not worked at all. Instead, you only load a tiny amount of "head data" for each area (name or other for displaying purposes) and the actual data only when the application really needs it. E.g. fiscal years with hundreds of thousands or millions of sales. One million revenue records for 2010, one million for 2011, for 2012, etc. In 2019, most of the time only 2019 and 2018 will be needed. The previous few, and the year 2000 sales are not of great interest anymore. Therefore: load data only when needed. Super efficient.

For example let's say the app "MyBusinessApp" has a root instance class, looking like this:

public class MyBusinessApp
{
    // ...

    private HashMap<Integer, BusinessYear> businessYears = new HashMap<>(); 

    // ...
}

The business year hold the turnovers:

public class BusinessYear
{
    // ...

    private ArrayList<Turnover> turnovers = new ArrayList<>();

    // ...
}

This approach would be a problem: During initialization, the root instance would be loaded, from there its HashMap with all BusinessYear instances, each with its ArrayList and thus all conversions. For all years. 20 years of approximately 1 million sales makes 20 million entities, which are pumped directly into the RAM at the start. It does not matter if someone needs it or not. We don't want it that way.

It would be nice if you could simply add a "lazy" to the turnover list. And that's exactly how it works:

public class BusinessYear
{
    // ...

    private Lazy<ArrayList<Turnover>> turnovers  = ...; // we will get to that

    // ...
}

And bingo, the turnovers are now loaded lazily.

What do you have to do now to get the actual ArrayList<Turnover> instance?

ArrayList<Turnover> turnovers = this.turnovers.get();

Just as with WeakReference, or simply as one would expect from a reference intermediate type in general: a simple get method.

The .get() call reloads the data as needed. But you do not want to mess around with that yourself. No "SELECT bla FROM turnovers WHERE ID =" + this.turnovers.getId(). Since you want to program your application you don' t have to mess around with low-level database ID-loading stuff. That's what the MicroStream Code does internally. You do not even need to access the ID, you just have to say "get!".

That's it.

What about the "..." ?

There are different strategies, what you write here. Analogous to the code example before it would be simply:

private Lazy<ArrayList<Turnover>> turnovers = Lazy.Reference(new ArrayList<>());

So always a new ArrayList instance, wrapped in a Lazy instance. If the actual ArrayList reference should be null at first, it works the same way:

private Lazy<ArrayList<Turnover>> turnovers = Lazy.Reference(null);

The this.turnovers.get() also just always returns null. Completely transparent.

But you could also do this:

private Lazy<ArrayList<Turnover>> turnovers = null;

If there is no list, then you do not make any intermediate reference instance for any list. A separate instance for null is indeed a bit ... meh.

But that has a nasty problem elsewhere: this.turnovers.get() does not work then. Because NullPointerException.

Anytime you need to write this here, the readability of code is not exactly conducive:

return this.turnovers == null ? null : this.turnovers.get();

But there is a simple solution: Just move this check into a static util method. Just like that:

return Lazy.get(this.turnovers);

This is the same .get(), just with a static null-check around it. This always puts you on the safe side.

In Short

For Lazy Loading, simply wrap Lazy<> around the actual field and then call .get() or maybe better Lazy.get(...).

It's as simple as that.

Side Note

Why do you have to replace your actual instance with a lazy loading intermediate instance and fiddle around with generics? Why is not something like this:

@Lazy
private ArrayList<Turnover> turnovers = new ArrayList<>();

Put simply: If it were just that it would be bare Java bytecode for accessing an ArrayList. There would be no way for a middleware library to get access and look it up and perhaps reload it. What's written there is an ArrayList reference. There is no lazy anymore. Either, the instance is null, or it is not null. If you wanted to reach in there, you would have to start with bytecode manipulation. Technically possible, but something you really don't want in your application.

So there must always be some form of intermediary.

Hibernate solves this through its own collection implementations that do lazy loading internally. Although the lazy loading is nicely hidden in some way (or not, if you need an annotation for that), it also comes with all sorts of limitations. You can only use interfaces instead of concrete classes for collections. At first, the instance is not the one you dictate, the code becomes intransparent and difficult to debug, you have to use a collection, even if it's just a single instance, and so on. You want to be able to write anything you want and you want full insight and control (debugability, etc.) over the code.

All this can be done with the tiny Lazy Interim Reference class. No restrictions, no incomprehensible "magic" under the hood (proxy instances and stuff) and also usable for individual instances.

Manually

The Lazy class has a .clear() method. When called, the reference held in the Lazy Reference is removed and only the ID is kept so that the instance can be reloaded when needed.

Automatically

So that the Lazy References do not have to be managed manually, but the whole goes automatically, there is the following mechanism: Each Lazy instance has a lastTouched timestamp. Each .get() call sets it to the current time. This will tell you how long a Lazy Reference has not been used, i.e. if it is needed at all.

The LazyReferenceManager audits this. It is enabled by default, with a timeout of 1,000,000 milliseconds, which is about 15 minutes.

A custom manager can be set easily, which should happen before a storage is started.

LazyReferenceManager.set(LazyReferenceManager.New(
    Lazy.Checker(
        Duration.ofMinutes(30).toMillis(), // timeout of lazy access
        0.75                               // memory quota
    )
));

The timeout of lazy references is set to 30 minutes, meaning references which haven't been touched for this time are cleared. In combination with a memory quota of 0.75.

Loading data can be done in two ways, eager and Lazy. The basic, default way of loading is eager loading. This means that all objects of a stored object graph are loaded immediately. This is done during startup of the MicroStream database instance automatically if an already existing database is found.

Contrary to lazy loading, eager loading has no requirements to your entity model.

To load your data you just need to create an EmbeddedStorageManager instance:

EmbeddedStorageManager storage = EmbeddedStorage.start();

After that just get the root instance of your object graph from the StorageManager by calling EmbeddedStorageManager.root() and check for null as this indicates a non-existing database

final EmbeddedStorageManager storage = EmbeddedStorage.start();

if(storage.root() == null)
{
    //No existing Database found
}
else
{    
    MyRoot root = (MyRoot) storage.root();          
}

Continuous Backup

Continuous backup mode allows the MicroStream engine to clone all storage actions to a second storage. This storage should be placed on another file system.

to configure and enable the continuous backup see .

Data Export

MicroStream provides the possibility to import and export data, see .

Deleting data does not require performing explicit deleting actions like DELETE FROM table WHERE... . Instead you just need to clear any references to the object in your object-graph and store those changes. If a stored object is not reachable anymore its data will be deleted from the storage later. This behavior is comparable to Java's garbage collector.

The erasing from the storage files is done by the process.

Stopping or Crashing a Live Database

Actually, a database is a passive collection of persisted data that can never be live on its own. But the managing thread accessing it can.

When an EmbeddedStorageManager is "started" it is actually just setup with all kinds of default and user-defined settings and definitions. What is actually "started" are the database managing threads that process storing and loading requests.

Of course, for every start() method, there needs to be something like a shutdown() method. So there is in MicroStream:

But is it really necessary to call shutdown? Should it be? What if there's an error and the process stops without calling shutdown()? Will that cause the database to become inconsistent, corrupted, maybe even destroyed?

The answer is: It wouldn't be much of a database solution if a crash could cause any problem in the persisted data. MicroStream data-store is carefully designed in such a fashion that the process it runs in can simply vanish at any point in time and the persisted data will never be corrupted.

This is surprisingly simple and reliable to solve: Whenever a .store() call returns, it is guaranteed that the data stored by it has been physically written to the underlying storage layer, usually a file system. Before that, there is no guarantee regarding written data at all. In fact, should the process die before the last byte has been written and secured, the next StorageManager initialization will recognize that and truncate the last partially written store. Either way, all the data that was guaranteed to be written will be consistently available after the next .start().

As a consequence, this safety mechanism makes an explicit .shutdown() call pretty much unnecessary. It doesn't hurt, but it is effectively more-less the same as just calling System.exit(0);.

The only time when an explicit shutdown is really needed is, if the database managing threads shall be stopped but the application itself keeps running. For example, it is perfectly valid to start the StorageManager, work with the database, then stop it, maybe change some configuration or copy files or something like that and then start it up again to continue working.

In any other case, the shutdown method can be ignored and the live database can happily just be "killed" while running. It is specifically designed to withstand such a treatment.

Multiple Databases

Any live MicroStream database basically consists of three major parts:

• A place where the persisted data is located. Usually a file system directory.

• The managing threads accessing (read and write) the persisted data.

• The EmbeddedStorageManager instance to use and control the database in the application.

Apart from a lot of internal components (configuration, processing logic, housekeeping state, etc.), that's all there is. There is nothing special or "magic" about it, no static state, no global registration in the JVM process or something like that.

The consequence of this is: If two EmbeddedStorageManager instances are started, each one with a different location for its persistend data, then the application has two live databases! If three or ten or 100 are started, then that's the number of live databases the application has. There is no limit and no conflict between different databases inside the same application process. The only important thing is that no two running StorageManagers can access the same data location.

Here is an overview of how to enable and configure different levels of user interaction for the Legacy Type Mapping.

Somewhere you have a foundation instance, a foundation in where everything is configured, from which the StorageManager is created.

EmbeddedStorageFoundation<?> foundation = 
    EmbeddedStorage.Foundation(); // or from somewhere else

It itself contains a foundation for connections. To access the inner thing needs a little detour.

Incidentally, that's not a JDBC connection, but this is just one thing that creates helper instances like Storer and Loader. Because Legacy Type Mapping affects loading, it has to go in there.

Either you access it directly, like this:

EmbeddedStorageConnectionFoundation f = foundation.getConnectionFoundation();

Or like this, that's better for method chaining.

foundation.onConnectionFoundation(f ->
{
    ...
});

If you have that, the configuration for the Legacy Type Mapping callback logic is just a one liner:

f.setLegacyTypeMappingResultor(...);

Default

PersistenceLegacyTypeMappingResultor.New()

That's just the necessary logic, without anything further. If you do not change anything, this is done by default.

With Console Output

PrintingLegacyTypeMappingResultor.New(PersistenceLegacyTypeMappingResultor.New())

That wraps a printer around the necessary logic. All these storage and persistence classes are nothing sacred or super duper intertwined or anything. These are just interfaces and if you plug in another implementation then it will be used.

With Inquiry

InquiringLegacyTypeMappingResultor.New(PersistenceLegacyTypeMappingResultor.New())

Resultor which asks the user to apply. More customization is possible, see below.

And so on

With the implementation of just one single interface method, you can build anything else you can imagine. For example, logging to a file instead of the console. Or in the personally preferred logging framework. Or write confirmed mappings into the refactorings file. Everything is possible.

For the inquiring implementation (InquiringLegacyTypeMappingResultor) there are a few settings: When should he ask? Always or only if something is unclear. Never does not make any sense of course, then you shouldn't use it, or alternatively the printing resultor.

When is a mapping unclear? If at least one field mapping is not completely clear. A field mapping is clear if:

1. If two fields are exactly the same (similarity 1.0 or 100%)

2. Or if two fields are specified by the explicit mapping.

So if all fields are clear according to the above rule, then there is no need to ask.

And there is another special case: If a field is discarded that is not explicitly marked as discardable, then as a precaution an inquiry is always done. Although no data is lost, but the data would not be available in the application, so better ask.

There are options to control this a bit finer. You can optionally specify a double as a threshold (from 0.0 to 1.0, otherwise Exception): The value determines how similar two matching fields automatically have to be so that they are not inquired. Example: The value is 0.9 (90%), but a match found is only 0.8 (80%) similar. This is according to the specification too little, there must be an inquiry as a precaution. If you specify 1.0, that means: always ask, everything is really perfectly clear. If you enter 0.0, this means: never ask or only for implicitly dropping fields.

Looks like this:

InquiringLegacyTypeMappingResultor.New(
    PersistenceLegacyTypeMappingResultor.New()) // implicitely 1.0
InquiringLegacyTypeMappingResultor.New(
    PersistenceLegacyTypeMappingResultor.New(), 0.7) // 0.7 threshold

Here a small example with a Person class.

int    customerid  ; // -> pin
String firstname   ; // -> firstName
String surname     ; // -> lastName
String comment     ; // discarded, NOT new commerceId

It should be changed to:

Integer pin       ; // <- customerid
String  firstName ; // <- firstname
String  lastName  ; // <- surname
String  commerceId; // new, NOT old comment
Address address   ; // new

Without explicitly predefined mappings, the inquiry would look like this:

             [***new***] pin
firstname    -0,944----> firstName
surname      -0,688----> lastName
comment      -0,750----> commerceId
             [***new***] address
customerid   [discarded]

customerid and pin are too different to be automatically assigned to each other. Therefore, it is wrongly assumed that customerid is omitted and pin is new. comment and commerceId are surprisingly similar (75%) and are therefore assigned. But that's not what we want.

Incidentally, it would not matter here what is defined as a threshold: customerid would be eliminated by the implicit decision. This is too delicate not to inquire, so it is always necessary to ask.

To get the mapping right, you have to specify two entries:

• customerid is now called pin

• and comment should be omitted

Then the inquiry looks like this:

customerid   -[mapped]-> pin
firstname    -0,944----> firstName
surname      -0,688----> lastName
             [***new***] commerceId
             [***new***] address
comment      [discarded]

Due to the explicit mapping from customerid to pin, the similarity does not matter, it is the mapping that matters. To indicate this, it says "[mapped]" instead of the similarity. The rest is as usual. Only comment is now "[discarded]", according to the mapping. The difference to the above is namely: This is an explicitly predetermined absence. That does not force inquiry.

This clears the way for the threshold:

• If you enter 0.7 or more then you will be asked. As far as everything would be clear, but the mapping of surname to lastName is below the required "minimum similarity", so rather ask.

• If you enter 0.6 or less, you will no longer be asked. Because all assignments are either explicitly specified or they are according to "minimum similarity" similar enough to rely on it.

A recommendation for a good value for the "minimum similarity" is difficult. As soon as one softens rules, there is always the danger of a mistake. See comment example above: is 75% similar to commerceId. Still wrong. Then prefer 80%? Or 90%? Of course it is better, but the danger is still there.

If you want to be sure, just make 1.0 or omit the parameter, then by default 1.0 is taken.

The most important is the explicit mapping anyway : if "enough" is given by the user, there is no need to ask.

MicroStream provides an API to import and export persisted data of the storage. It is pretty much the same as writing and reading a backup.

The records in the storage are distributed in lots of files and folders, depending on channel count and other . To get order in the chaos the export produces one file per type. This files are used again by the import to read the data into the storage.

Data Conversion

It is also possible to convert the exported binary files to a human readable format, namely CSV.

Implementing the PersistenceEagerStoringFieldEvaluator interface allows you to handle the eager/lazy storing behavior of any known member. The default implementation of the MicroStream engine threads all fields as lazy storing. See for details on lazy and eager storing.

The PersistenceEagerStoringFieldEvaluator has only one method to be implemented: public boolean isEagerStoring(Class<?> t, Field u) return true if the field has to be eager, otherwise return false.

To register the customized PersistenceEagerStoringFieldEvaluator add it using the one.microstream.persistence.types.PersistenceFoundation .setReferenceFieldEagerEvaluator(PersistenceEagerStoringFieldEvaluator) method during the storage initialization.

In certain environments or setups it is necessary to provide specific ClassLoader instances. This can be done by customizing the connection foundation.

If a single ClassLoader is sufficient, just create a new provider by handing over the instance:

Or return a ClassLoader depending on the requested type:

Class Loader in Application Server

Most application servers load the session's classes with the context class loader. Just use the one of the current thread:

Custom type handlers allow taking control over the storing and loading procedure of specific java types. This is useful to optimize the performance for storing complex objects or in the rare case that it is not possible to store a type with the default type handlers.

Implementation

There are two strategies for a simplified type handler implementation.

A Custom Binary Handler

Implementing a class that extends CustomBinaryHandler and defines a sequence of BinaryFields via the #Field~ static pseudo-constructor methods. Everything else like setting the name, calculating the binary offsets, etc. is then done implicitly via reflection.

The custom type handler must be registered in the CustomTypeHandlerRegistry to enable it:

EmbeddedStorageManager storage = EmbeddedStorage
     .Foundation(WORKINGDIR)
     .onConnectionFoundation(f ->
          f.registerCustomTypeHandlers(new CustomBufferedImageHandler())
     )
     .start(ROOT);

A Static Provider Method

Implementing a class can be skipped altogether by using the method Binary#TypeHandler and passing the BinaryFields directly. Registering such a generically created TypeHandler is not required, either, since Version 3 of MicroStream brought a solution to just define a static method in the entity class that will be recognized and used by MicroStream.

The following is a simple technical example on how a custom binary handler can be easily defined and technically leveraged to optimize storage behavior. E.g. imagine having millions of such objects that now only create 1 database record with a fraction of the required storage space instead of 4 records but hold the same information.

public class Employee
{
	/*
	 * Fields with primitive data are (for whatever reason, e.g. project 
	 * design rules) all object types, but records should be stored as 
	 * efficient as possible, i.e. without overhead of references and value objects.
	 * 
	 * MicroStream's generic type analysis does not know of this and hence cannot 
	 * do it. But defining a custom type handler can
	 */

	String id         ;
	Double salary     ;
	Date   dateOfBirth;
	
	// constructor, getters, setters, etc
	
	/*
	 * The entity class must just contain "any" method returning a suitable type 
	 * handler and MicroStream will recognize it and use the returned handler 
	 * automatically.
	 * 
	 * Type type handler just needs to specify the entity class and define a list 
	 * of fields comprised of (name, getter, setter) in arbitrary order.
	 */
	static BinaryTypeHandler<Employee> provideTypeHandler()
	{
		return Binary.TypeHandler(
		  Employee.class,
			Binary.Field_long("id",
				e -> Long.parseLong(e.id),
				(e, value) -> e.id = String.valueOf(value)
			),
			Binary.Field_long("dateOfBirth",
				e -> e.dateOfBirth.getTime(),
				(e, value) -> e.dateOfBirth = new Date(value)
			),
			Binary.Field_double("salary",
				e -> e.salary.longValue(),
				(e, value) -> e.salary = Double.valueOf(value)
			)
		);
	}
	
}

When a reference to the loading storage is needed in entities, e.g. usage of different tenants or to store its internal state in a tailored fashion, this can be done by this little trick.

If an entity type contains one or more transient fields with field type compatible to Persister, the updating processing during loading will set the Persister instance (e.g. an EmbeddedStorageManager instance) used to load the entity instance to those fields.

The fields must be transient to exclude them from the persistent form of the entity. Checking for transient fields is only the default implementation. The checking logic can be customized via PersistenceFoundation#setFieldEvaluatorPersistable.

A more precise check for Persister fields can be customized via PersistenceFoundation#setFieldEvaluatorPersister. Note, however, that the check for compatibility with the Persister type is done in any case to avoid inconsistencies/crashes.

If no applicable field is found, nothing happens and no additional meta data is kept in memory. This feature is completely optional.

class MyEntity
{
	String name ;
	int    value;
	
	transient EmbeddedStorageManager storage;
}

Upon loading an instance of class MyEntity, a reference to the EmbeddedStorageManager used to load it is set to its field storage.

Housekeeping is an internal background logic to optimize the database's usage of memory and persistent storage space (typically disc space). It is comprised of mechanisms for cleaning up storage files, clearing unneeded cached data and recognizing deleted entities via garbage collection. Housekeeping is performed with a configurable time budget in configurable intervals to make sure it never interferes with the application's work load too much (see Housekeeping Configuration).

File cleanup:

If new versions of an entity are stored or if entities become no longer reachable (meaning the become effectively deleted or "garbage" data), their older data is no longer needed. However, the byte sequences representing that old data still exist in the storage files. But since they will never be needed again, they become logical "gaps" in the storage files. Space that is occupied, but will never be read again. It might as well be all zeroes or not exist at all. Sadly, unwanted areas cannot simple by "cut" from files. Above all because that would ruin all file offsets coming after them. So with every newly stored version of an entity and every entity that is recognized as unreachable "garbage", a storage file consists more and more of useless "gaps" and less and less of actually used data. This makes the storage space less and less efficient. To prevent eventually ending up with a drive that is filled with useless bytes despite an actually not that big database, the files need to be "cleaned up" from time to time. To do this, the Housekeeping occasionaly scans the storage files. If their "payload" ratio goes below the configured limit, the affected files will be retired: all data that belongs to still live entities is copied to a new file. Then the old file consists of 100% unneeded gap data and can safely be deleted.

Which ratio value to set in the configuration is a matter of taste or, more precisely, depends on each indivudual application's demands. A value of 1.0 (100%) means: only files with 100% payload, so no gaps at all, are acceptable. This means that for every store that contains at least one new version of an already existing entity, the corresponding storage file will contain the slightest gap, thus dropping below the demanded ratio of 100% and as a consequence, will be retired on the next occasion. This very aggressive cleanup strategy will keep the disc space usage at a perfect minimum, but at the cost of enormous amounts of copied data, since virtually every store will cause one or more storage files to be retired and their content be shifted into a new file. Respectively, a value of 0.0 (0%) means something like: "Never care about gaps, just fill up the disc until it bursts." This keeps the disc write loads for the file cleanup at 0, but at the cost of rapidly eating up disc space.

The best strategy most probably lies somewhere in between. Somewhere betwen 0.1 and 0.9 (10% and 90%). The default vaue is 0.75 (75%). So a storage file containing up to 25% of unused gap data is okay. Containing more gaps that 25% will cause a storage file to be retired.

In addition to the payload ratio check, the file cleanup also retired files tha are too small or too big. For example: The application logic might commit a single store that is 100 MB in size. But the storage files are configured to be no larger than 10 MB (for example to keep a single file cleanup nice and fast). A single store is always written as a whole in the currently last storage file. The reason for this is to process the store as fast as possible and quickly return control to the application logic. When the housekeeping file cleanup scan encounters such an oversized file, it will retire it immediately by spliting it up accross 10 smaller files and then deleting the oversized file. A similar logic applies to files that are too small. Upper and lower size bounds can be freely configured to arbitrary values. The defaults are 1 MB and 8 MB.

Cache cleanup

To avoid repeated reads to storage files (which are incredibly expensive compared to just reading memory), data of once loaded entities is cached in memory. If an entity's cached data is not requested again for a certain amount of time in relation to how much data is already cached, it is cleared from the cache to avoid unnecessarily consuming memory. The mechanism to constantly evaluate and clear cached data where applicable, is part of the housekeeping. The aggressiveness of this mechanism can be configured via the Housekeeping Configuration.

Garbage collection

In a reference-based (or graph-like) data paradigm, instances never have to be deleted explicitely. For example, there is no "delete" in the java language. There are only references. If those references are utilized correctly, deleting can be done fully automatically without any need for the developer to care about it. This is called "garbage collection". The concept is basically very simple: when the last reference to an instance is cut, that instance can never be accessed again. It becomes "garbage" that occupies memory with it data that is not needed any longer. To identify those garbage instances, all an algorithm (the "garbage collector") has to do is to follow every reference, starting at some defined root instance (or several) of a graph and mark every instance it encounters as "reachable". When it has no more unvisited instances in its queue, the marking is completed. Every instance that is not marked as reachable by then must be unreachable garbage and will be deleted from memory.

Similar to the JVM's garbage collection to optimize its memory consumption, MicroStream has a garbage collection of its own, but for the level of persistent storage space instead of memory space.

However, MicroStreams multi-threaded garbage collector is currently still in development and not activated, yet.

Explicit Housekeeping

Housekeeping can also be triggered manually from the StorageConnection . Related methods are:

• issueCacheCheck(nanoTimeBudgetBound)

• issueCacheCheck(nanoTimeBudgetBound, entityEvaluator)

• issueFileCheck(nanoTimeBudgetBound)

• issueFileCheck(nanoTimeBudgetBound, fileDissolvingEvaluator)

• issueFullCacheCheck()

• issueFullCacheCheck(entityEvaluator)

• issueFullFileCheck()

• issueFullFileCheck(fileDissolvingEvaluator)

• issueFullGarbageCollection()

• issueGarbageCollection(nanoTimeBudget)

All Housekeeping methods can be given a defined time budget or can be run until full completion.

If one or more fields in a class have changed, the data structure of this class doesn't match anymore with the records in the database. This renders the application and the database incompatible.

It's like in an IDE. You change the structure of a class and the tooling takes care of the rest. Problem is, in a database, the "rest" can be, in some circumstances, several gigabytes or even more, that have to be refactored and written again. It's one way to do it, but there are better alternatives.

At best, the data is transformed when it's accessed only. The old (legacy) type data is being mapped to the new type when it's being loaded, hence: Legacy Type Mapping.

Nothing needs to be rewritten. All records are, as they were saved, compatible with all other versions of their type. Simply by mapping while loading.

Automatic Mapping

What has to be done to achieve this? In the most common cases, nothing!

The heuristic attempts to automatically detect which fields are new, have been removed, reordered or altered.

The fields in the Contact entity class have been renamed, reordered, one was removed, one is new.

What the heuristic is doing now is something like this: String firstname is equal in both classes, so it has to be the same, pretty much as int age. name and lastname is pretty similar, type is the same too. If there is nothing better for the two, they probably belong together. Same with the other two fields. In the end, the ominous link and postalAddress remain. The heuristic can not make sense of that, so it assumes that one thing falls away and the other one is added. In this particular example, that worked perfectly. Well done, heuristic.

But: Just as people can make mistakes in estimating similarities ("I would have thought ..."), even programs can make mistakes as soon as they logically go on thin ice. There is nothing more with absolute correctness that you actually know from (bug-free) software. Such a similarity matching will be correct in the most cases, but sometimes it will also fall by the wayside. Example: perhaps only PostalAddress instances were referenced in the application under link and the two fields would actually be the same, only now properly typed and named. How should heuristics know that? Nobody could know that either, if he is not privy to the details of the concrete application.

That's why Legacy Type Mapping has two mechanisms that prevent things from going wrong:

1. A callback interface is used to create the desired mapping result: PersistenceLegacyTypeMappingResultor

If you do not want that, you can simply set another resultor. Like in this example each suspected mapping is submitted once to the user for control in the console. This is done with the InquiringLegacyTypeMappingResultor. Maybe even one, where the user can "rewire" the mapping itself, write out the mapping, and then return an appropriate result.

Explicit Mapping

All you need is two columns of strings: from old to new. By default MicroStream uses a CSV file, but you can also write something else. In the end, a lot of string pairs for "old -> new" mappings have to come into the program somewhere.

The concept is simple:

• If there are two strings, this is interpreted as a mapping from an old thing to a new thing.

• If the second value is missing, it is interpreted as an old thing to be deleted.

• Missing the first value, then it's as a new thing.

Why call it "thing"? Because this applies to several structural elements:

• Constant identifier

• Class names

• Field names

Example: count; articleCount means: the field named earlier count is called articleCount in the current version of the class. count;means: the early field count should be ignored during the mapping. More specifically, the values ​​of this field per record. ;articleCount means, this is a newly added field, DO NOT try to match it with anything else heuristically.

You can also mix explicit mapping and heuristics. Only explicitly specify so many changes until the analysis gets the rest right by itself. That means you never have to specify the annoying trivial cases explicitly. Only the tricky ones. Usually, nothing should be necessary at all, or maybe a view indications at most to avoid mishaps.

However, those who strictly prefer to make any change explicitly, instead of trusting a "guessing" software, can also do that. No problem.

Explicit Mapping of Classes