Import / Export

Search…
Import / Export
This is the manual for older MicroStream versions (Version < 5.0).
The new documentation (Version >= 5.0) is located at:
​https://docs.microstream.one/​
MicroStream provides an API to import and export persisted data of the storage. It is pretty much the same as writing and reading a backup.
The records in the storage are distributed in lots of files and folders, depending on channel count and other settings. To get order in the chaos the export produces one file per type. This files are used again by the import to read the data into the storage.
The created binary type data files contain only records of the according type, nevertheless they have the same format as the channel storage files.
Export
1
NioFileSystem          fileSystem = NioFileSystem.New();
2
	
3
EmbeddedStorageManager storage    = EmbeddedStorage.start(
4
  fileSystem.ensureDirectoryPath("storage")
5
);
6
​
7
String                            fileSuffix      = "bin";
8
StorageConnection                 connection      = storage.createConnection();
9
StorageEntityTypeExportStatistics exportResult    = connection.exportTypes(
10
	new StorageEntityTypeExportFileProvider.Default(
11
	  fileSystem.ensureDirectoryPath("export-dir"), 
12
	  fileSuffix
13
	),
14
	typeHandler -> true // export all, customize if necessary
15
);
16
XSequence<Path> exportFiles = CQL
17
	.from(exportResult.typeStatistics().values())
18
	.project(s -> Paths.get(s.file().identifier()))
19
	.execute()
20
;
Copied!
Import
1
NioFileSystem          fileSystem = NioFileSystem.New();
2
	
3
EmbeddedStorageManager storage    = EmbeddedStorage.start(
4
  fileSystem.ensureDirectoryPath("storage")
5
);
6
​
7
StorageConnection      connection = storage.createConnection();
8
connection.importFiles(X.Enum(
9
	fileSystem.ensureFilePath("type1.bin"),
10
	fileSystem.ensureFilePath("type2.bin")
11
));
Copied!
Data Conversion
It is also possible to convert the exported binary files to a human readable format, namely CSV.
Why CSV?
Contrary to XML or JSON, CSV is perfectly suited to represent records with the least possible overhead. There are a lot of tools, like spreadsheet editors, which can read and modify CSV files. The file's size is at the possible minimum and the performance of the converter is significantly better than with the other formats.
Binary to CSV
1
NioFileSystem          fileSystem = NioFileSystem.New();
2
	
3
EmbeddedStorageManager storage    = EmbeddedStorage.start(
4
	fileSystem.ensureDirectoryPath("storage")
5
);
6
​
7
StorageDataConverterTypeBinaryToCsv converter = 
8
    new StorageDataConverterTypeBinaryToCsv.UTF8(
9
		StorageDataConverterCsvConfiguration.defaultConfiguration(),
10
		new StorageEntityTypeConversionFileProvider.Default(
11
			fileSystem.ensureDirectoryPath("csv-dir"), 
12
			"csv"
13
		),
14
		storage.typeDictionary(),
15
		null, // no type name mapping
16
		4096, // read buffer size
17
		4096  // write buffer size
18
);
19
AReadableFile dataFile = fileSystem.ensureFilePath("type1.bin").useReading();
20
try
21
{
22
	converter.convertDataFile(dataFile);
23
}
24
finally
25
{
26
	dataFile.close();
27
}
Copied!
CSV to Binary
1
NioFileSystem          fileSystem = NioFileSystem.New();
2
	
3
EmbeddedStorageManager storage    = EmbeddedStorage.start(
4
	fileSystem.ensureDirectoryPath("storage")
5
);
6
​
7
StorageDataConverterTypeCsvToBinary<AFile> converter = 
8
	StorageDataConverterTypeCsvToBinary.New(
9
		StorageDataConverterCsvConfiguration.defaultConfiguration(),
10
		storage.typeDictionary(),
11
		new StorageEntityTypeConversionFileProvider.Default(
12
			fileSystem.ensureDirectoryPath("bin-dir"), 
13
			"dat"
14
		)
15
);
16
converter.convertCsv(fileSystem.ensureFilePath("type1.csv"));
Copied!
User Interaction
Next - Data-Store
Housekeeping
Last modified 3mo ago
Export as PDF
Copy link
Contents
Data Conversion