Structure

Grey Matter Data tracks information through a collection of Events that represent a Kafka messaging queue. Each Event is associated to a folder or file in the system and is used to capture all historical changes to it. New Events form a timeline of actions describing state modifications to a folder or file (e.g. create, update, delete).

The primary key for each Event in the system is identified by two timestamps:

1. An Object ID (oid) which is a nanosecond timestamp assigned at creation of the folder or file.

2. A tstamp which is recorded for each new Event belonging to the folder or file.

Events may also be used to define relationships between folders and files in two ways:

1. An Event's parentoid field defines folder-to-folder and file-to-folder relationships. For example, if the system receives an update action (U) that changes the parentoid for a given file's oid, the file will be "moved" from one folder to another.

2. An Event's derived field specifies a folder or file came from another. For example, a thumbnail of an image or a PDF copy of a Word document could point back to its original by assigning the original's oid to its derived field.

Event

Field	Data Type	Description
action	string	Describes a modification to an object's state
blobalgorithm	string	The blob algorithm. If not specified, we store in S3 with SSECustomerKey (?)
checkedtstamp	string	The tstamp of the previous version of the oid compared against (?)
custom	JSON (all string values)	Custom metadata fields
defaultfile	string	If the object is a folder, a Stream request will return this file; defaults to index.html
derived	Derived	See above
description	string	Describes the contents of the object
encrypted	(?)	Allow for fields to be encrypted on a case-by-case basis
expiration	string	The tstamp as of which this object and its Events are no longer accessible
isfile	string	Specifies that this object is a file
mimetype	string	Same as Content-Type
name	string	The object's folder name or filename without pathing
objectpolicy	string	Lisp-syntax rules that permit various actions to a given owner
oid	string	Numeric identifier for the object
parentoid	string	Numeric identifier for the parent folder of the object
purgetstamp	string	If set, purges an Event with the matching oid and tstamp
references	[]Reference (?)	An array of updates when uploading files into folders that don't yet exist. Indices are negative relative values (?)
rname	string	Used to define where this object is stored on disk (via ${FILE_BUCKET}/${FILE_PARTITION}/${rname})
schema	string	Version of the schema from which this object came
security	Security	A security label to assign to the object
sha256plain	string	sha256 of the plaintext; can be used in the client to calculate a minimum number of files to send for update
size	string	Same as Content-Length
tstamp	string	Timestamp unique to this Event
userpolicy	UserPolicy	The JWT claims used when creating this Event

Derived

Field	Data Type	Description
oid	string	Numeric identifier for the object
tstamp	string	Numeric identifier for the Event
type	string	Description of the derivation (e.g. doc-to-pdf)

Security

Field	Data Type	Description
label	string	Title of the security label (e.g. DECIPHER//GMDATA)
foreground	string	The text color (e.g. #FFFFFF)
background	string	The background color (e.g. #00FF00)

UserPolicy

Field	Data Type	Required	Description
label	string	yes	Title of the UserPolicy
exp	number	yes	Expiry in seconds
sub	string		Subject
iss	string		Issuer
aud	string		Audience
values	JSON (array of strings)		Hashtable for evaluating JWT tokens

Sample Events

For API usage, refer to the Data Modification section. These sample Events are only meant to be instructive.

Create

The parameters below are the bare minimum that must be specified for the create action to be successful.

When the action specified is C (create / upload), the system will backfill the oid.

{
  "action": "C",
  "name": "New Folder",
  "isfile": false,
  "parentoid": 1,
  "userpolicy": {
    "label": "asRoot"
  },
  "objectpolicy": "(if (contains email [e-mail])(yield-all)(yield R X))",
  "userpolicy": {
    "label": "asRob",
    "values": {
      "email": ["[e-mail]"]
    }
  }
}

Delete

When the action specified is D (delete), the system will backfill most of the parameters. It is only necessary to specify oid, action, parentoid, and objectpolicy.

{
  "action": "D",
  "oid": "42",
  "parentoid": 1,
  "objectpolicy": "(if (contains email [e-mail])(yield-all)(yield R X))"
}

Update

Important: Anytime the action specified is U (update), all parameters except tstamp must be specified, mimicking the previous Event. In the following examples, assume all other parameters are specified.

Renaming a file

{
  ...
  "action": "U",
  "oid": "42",
  "name": "new-filename.txt"
}

Moving a file

{
  ...
  "action": "U",
  "oid": "42",
  "parentoid": 2
}