Design

The goal is to be focus on these attributes of Grey Matter Data:

• trivial and transparent change notification

• trivial historical listing

• trivial userPolicy/objectPolicy filtering

The design fits very closely to these principles:

CQRS/EventSourcing

This is an approach that dispenses with some distributed data issues and metadata encryption issues for now. It is critical to get security restrictions on what users can do to the data in place (which goes beyond just encrypting the data). We want a layer that simple reads and writes to S3, without worrying about caches or how bucket URLs are discovered.

• We use immutable data structures because we do plan on being able to do these things:

  • Migrate systems by replaying events into an empty database of a new version (TBD: a vPrevious to vNext conversion of incoming events).

  • Replicate systems by replaying events into an empty database of the same version.

  • We can already very easily see scaling out by replicating a master (that accepts /write calls) to other machines that only handle reads.

  • Allowing writes to multiple masters should be possible, but could be problematic if each node doesn't pick a part of the tree that it "owns".

Policy

The objectPolicy itself is self-contained, but the userPolicy will need to be generated by a service that knows what a user's attributes actually are. For now, we just pass in a userpolicy, and presume that it was injected into the header by a trusted proxy server.

JWT (json web tokens) are currently being used for userPolicy. There is still work to be done to allow more complex json in the policies though.

This allows us to not have to talk to any backend microservices. It is on the caller to get use a userPolicy. The end users design userPolicy and objectPolicy to work together, so that they can handle scenarios in a way that suits them. Example of the problem it solves:

• If a user comes in and tells us who they are, then they need to prove it.

• If a user gives us a username:password pair, then we still need to go look up the attributes of the user (organizations, birthdate, email, certifications, etc). We can't consult a microservice for every item in a listing.

• If we simply mark documents with the requirements, then we still need to present the pair of (userPolicy, objectPolicy) to something that can evaluate the combination; usually a microservice or compiled-in library.

• In order for the comparator to not need to be static, we can have the objectPolicy (on a specific object) literally be a chunk of code that evaluates userPolicy with the objectPolicy (which is a function) and returns permissions.

• End users design their objectPolicy and userPolicy formats to work together. This way, we don't need to lay down more than a few rules that makes them unambiguous to evaluate.

• The userPolicy came to us from somewhere else. It is either injected into a header from a trusted proxy, or we will need userPolicy objects to be actually signed.

You can think of userPolicy objects as short-term application-specific certificates granted in exchange for a login. They may attest to the user is, or just attest to some attributes that the user wishes to reveal. You can think of objectPolicy as functions that take userPolicy as input and return permissions (against a single object) as output. Enforcement of updates effectively ensures that updates are well formed. An object that is supposed to only be updated by Alice cannot be inserted due to a policy that says that only Bob can update in the latest version of the record being appended.