Memory

Classes
class	BorgIv::File
	File values are stored locally but managed centrally. More...
class	BorgIv::Map
	maps associate keys with values and are centrally stored More...
class	BorgIv::Value
	wraps value identifiers into a read-only C++ class. More...

---

Detailed Description

As described many times before. The data format of the flow mechanism is crucial. A first attempt tried to wrap tokens in files that could be read anywhere. That however is a not so optimal attempt (see Data) because each external program needs the ability to read these. Therefore we changed the format to something that is stored at two locations: files on disk from which the content itself is unknown to the flow and metadata which can be read by the flow mechanism. We have data which can be string (binary blobs actually), arrays or files. The files are hardlinked on disk when possible, and copied when that is not possible.

The BorgIV runtime depends on a distributed memory. This memory is a readonly/createonly memory, meaning that all values once they are made can only be read. This allows a functional programming practice and avoids a lot of concurrency problems. It also allows caching of many values at the different clients. It also allows a content based comparison at the server side that makes storage optimalization and object comparisons faster.

The memory model further only provides support for the following datatypes

• primtive values, which are binary blobs (similar to strings)
• associations, which can be used to store arrays, multidimensional arrays and maps
• files, which are stored at the client side.

Currently, this memory system is stored in a central database and is the basis for the tokens, but is not the same as the token system. The advantage offereed by this system is that if two values refer to the same content then we can simply take the original value instread of recreating one. This is especially so beause we rely on a functional approach. We can never change data: only recreate copies of it. Probably in the future it will become usefull to create a more efficient centralized model. However for now we store all this data in 4 SQL tables

The main wrapper class is Value

Each cell must be set up with the following requirements:

• rsync connections must be able to get data from any other host, without entering passwords
• a virtual X frambeuffer must be running
• each machine must provide its ssh key to all other machines. These keys are listed in the keytable and must be used to update the authorized keys file.

Todo:

we need an update-keys program to exchange the keys automatically

we should reorganize this entire structure into a local subserver that connects to a parent server. Valuse that come down are cached and values that go up are updated this should make it straightforward to optimize many of the refernce count stuff. By avoiding double notifications we can effectively increase the throughput of the program

if we work with our own memory server then we can also delete files whenever we need to and have a more optimal file storage

---