Lifecycle
Last updated
Was this helpful?
1.2
Last updated
Was this helpful?
An object is represented by a sequence of Events that mutate it. The object is associated with a state machine.
In gm-data, an append-only design is used. Records are never mutated. An oid will have a lifetime determined by the (oid, tstamp) pairs on every event that gets written. Due to concurrency and parallelism (in which "parallel" means that the order in which items is accomplished does not matter), it is possible for records to get inserted in an order that is not exactly the same as tstamp order; particularly in cases involving replication.
Even in the simple cases, such as gm-catalog, where a service that uses gm-data wants to write, we have a problem to deal with. This issue is not any different from problems that arise from any shared filesystem, such as a shared NFS mount. Some definitions first:
Define mutation to mean that there is an update that creates new information while also destroying old information.
An "update" that only appends new records without modifying old records is not the same thing as an outright mutation.
Conflicting records can be stored in such a way that all databases at least agree on the final result when there is a semantic conflict. This does not imply that any database got the result that it wanted.
A classic data race that destroys the old value is a serious problem, because this can cause a pair of databases to converge on different states. When two databases end up with different states, with no way to converge them to one consistent state, we call that database corruption.
As an example, if on startup, gm-catalog has two instances that both decide to write the same file, they are racing to perform that file write. It may be that one file does the create (action C), and the other one does update on that (action U). It could also be that they both managed to do a create (C), creating two directories called /world with different oids. Even with a locking mechanism, this could still have happened during a network partition; because locks require consensus, which is generally impossible during a network partition (ie: use cases where datacenters are disconnected from the world, and need to keep collecting data).
This race happens when we need to create the home directory into which to start writing, such as /world. But, because we do not throw away the records that lose in a conflict, it is actually not a real problem.
When there are no conflicts, checkedtstamp will always point to the tstamp of the previous record in history. If multiple records have the same checkedtstamp, then we know that two concurrent edits were created. They are like git branches that need a merge. This can be rendered in an user interface, so that a human can attend to a problem that may require manual conflict resolution.
The checkedtstamp for a create refers to the parentoid at the given tstamp, because action C means "permission to create, given by the parent".
The checkedtstamp for an update refers to the oid at the value tstamp.
Before you update a file, it is presumed that you have a tstamp of the current version you are looking at for that oid. After the upload, if checkedtstamp is not the tstamp you had on the event just before the upload, then some other client must have modified the same file. The conflict may or may not need a new update to resolve it. But what is important is that the conflicting versions are all available for inspection, to create a resolved update.
There are other strategies that can be employed. A service, such as gm-catalog, can elect a leader (ie: take a census, and lowest identifier wins) to be the only one trying to perform the write to gm-data. A general lock service might be taken out by a microservice for the duration of a write attempt (ie: 10 sec lease on exclusive update access to a file). gm-data might have a mechanism for leasing out a lock on updates to a file, though such a mechanism would only work within a strongly consistent cluster.
Another form of concurrency that is very real with documents is that a user may download a document and spend hours looking at it and editing it. When it is time to re-upload the document, fetching latest properties can tell us that our edits are not based on the current version.
Here it is possible to have applications define locks that periodically renew, to tell user interfaces to refrain from updates until the lock is removed or expired. This kind of functionality probably belongs in a different service, because this is about performing transactions across microservices in general.
We may want to tell the user to compare the two versions and upload a merged version of the document. When the upload is finally performed, if the last known tstamp is not equal to checkedtstamp, then we know that there was a concurrent update. We can't stop this with locks, because we may replicate in a conflicting update that happened in our past. What is important is that given the /history of an oid, we can show conflicts to the user when they happen, and we always have all versions required to produce a merged version. Users who have spent many hours creating modified versions of docs do not want their updates to be rejected outright. Instead, they need to be able to use the conflicting versions to produce a de-conflicted version.
In general, automated merging is done by having data structures that merge automatically. But when uploading arbitrary files, such as Word/PowerPoint/PDF, it is not possible to completely automate file merges to keep the user out of it.
