Endpoints

This section covers all endpoints exposed by the Grey Matter Data API and their functionality.

/config

Get basic config information about the server, needed by apps to make decisions about setting up the UI

Request parameter	Parameter Value

There are no parameter values. But we get back config. This is an example of a system configured so that "/world" is the home directory where we are allowed self-service home directory creation. And in here, this directory must be named after an email that shows up in your JWT token. This /config

{
  "GMDATA_NAMESPACE": "world",
  "GMDATA_NAMESPACE_USERFIELD": "email"
}

Says that creating a home dir /world with parentoid: "42" is allowed if the directory name is equal to an email value from jwt. So, we propose to write an object with at least these fields,

{
  "parentoid": "42",
  "action": "C",
  "isfile": false,
  "name": "[e-mail]"
 }

Normally, a parentoid is just a numeric identifier for a directory (like pwd). It is now possible when doing updates to specify subdirectories like 42/books for a file to go into. If that directory does not yet exist, it will be created. But note that because it is perfectly permissible in gm-data to have multiple directories with the same name but a different oid, this may sometimes create an unwanted result where the user has to merge the directories together. This feature exists mostly for the sake of bulk upload scenarios, where it can be too painful to have to ensure that every directory exists beforehand. This can be the case when the directory is cut up by dates to keep directory size small in the face of a massive number of files being uploaded. ie: 42d335/collects/2013/02/03 as a parentoid

And the JWT claims assert at least

{
  "values": {
    "email": [ "[e-mail]", "[e-mail]" ]
  }
}

This write would be allowed, because the JWT does have an email with the name we are trying to write. So, we can create the directory /world/[e-mail]. This is an example of a "self-service" home directory, so that administrators do not need to get involved in users having a home directory to write into, because their userid, email in this case, is asserted. So /world/${email} is safe for this user to write into.

/read

Read endpoint allows for bulk querying of Grey Matter Data.

Request parameter	Parameter Value
URL	{host}/read
body	stringified( [{URL: '/list/${id1}/'}, {URL: '/list/${id2}/'}, {URL: '/props/${id1}/'}] )
method	POST
credentialsInclude	true

Query Strings

While query strings may not be applied to the /read endpoint directly, they may be applied to the URL parameters passed in the body of the request. An example stringified body of the request with query strings is presented below in JavaScript:

const stringifiedBulkRequest = JSON.stringify([
  {URL: '/history/${id1}/?count=10'},
  {URL: '/list/${id2}/?tstamp=1578a72437e23c00&?count=5'},
  {URL: '/notifications/${id1}/?last=1578a72437e23c00&?count=100'}
]);

Common use cases

1. File structure exploration, through bulk requests of the /list endpoint,

2. Unifying requests to reduce network traffic or improve performance on slow connections. For example, through bulk requests of /props endpoint data or /stream of Object thumbnails for an entire or part of a folder, and

3. Pre-loading deep file structure to improve user experience and speed of navigation on fast connections, through bulk requests of /list endpoint with each subsequent call requesting a whole level deeper info file structure to pre-load user selection.

/read JavaScript example

const bulkRequest = [
  {URL: "/list/32/?count=10"},
  {URL: "/list/12/?count=10&childCount=true"},
  {URL: "/history/12/?count=10"},
  {URL: "/notifications/1/"}
];

axios.post(${gmDataEndpoint}/read, JSON.stringify(bulkRequest), {
  // necessary to pass on server set HttpOnly authentication cookies
  withCredentials: "true"
})
.then(response => console.log(response.data))
.catch(error => console.log(error));

// readResponse will contain data for the 4 GET requests specified in bulkRequest constant

/props

Props endpoint returns the latest Event Object on an Object ID. Specifying ?tstamp= query parameter will return the latest Event Object as of that time.

Request parameter	Parameter Value
URL	{host}/props/{oid}/{path}?queryString1=queryValue1
body	none
method	GET
credentialsInclude	true

/props Query Strings

Query String	URL Example	Description
?tstamp	{host}/props/1578cfdd0241cbee/?tstamp=1578a72437e23c00	?tstamp may be must be specified in 64-bit hex (Unix Nanoseconds)
/path/	{host}/props/1578cfdd039b919b/testFolder/TestFile.csv	/path/ may be added, composed of names, which is an Object property, and separated by a /, which will be internally converted to Object IDs to construct a full path to desired file. This is necessary to allow deep linking into the system.

/props Common use cases

1. Getting information about an Object

2. Exploring Object's versions by specifying different time stamps

/props JavaScript example

const propsUrl = `${gmDataEndpoint}/props/1/Folder1/File1`;
// if File1 Object ID is 8888, then equivalent URL is ${gmDataEndpoint}/props/8888/

axios.get(propsUrl, {
  // necessary to pass on server set HttpOnly authentication cookies
  withCredentials: "true"
})
.then(response => console.log(response.data))
.catch(error => console.log(error));

// response.data will contain data for File1, which is child of Folder1 and root folder /1.

/list

List endpoint gets latest Event Object of each child Object IDs. Use this method to explore folders and navigate through file structure. This method will return an error if used on a file (no Event Objects in the system have this Objects ID in the parentoid parameter).

Request parameter	Parameter Value
URL	{host}/list/{oid}/{path}/?queryString1=queryValue1&?queryString2=queryValue2
body	none
method	GET
credentialsInclude	true

/list Query Strings

Query String	URL Example	Description
?childCount	{host}/list/1578cfdd0241cbee/?childCount=true	?childCount to count the children in each subdirectory
?tstamp	{host}/list/1578cfdd0241cbee/?tstamp=1578a72437e23c00	?tstamp may be must be specified in 64-bit hex (Unix Nanoseconds)
?last	{host}/list/1578cfdd039b919b/?last=1578cfdd0241cbee&count=10	?last may be added, as an Object ID, (tstamp for /notifications and /history) used in conjunction with ?count to control paging. Specify Object ID last seen as a cutoff.
?count	{host}/list/1578cfdd039b919b/?last=1578cfdd0241cbee&count=10	?count may be added, used in conjunction with ?last to control paging.
/path/	{host}/list/1578cfdd039b919b/testFolder/TestFile.csv	/path/ may be added, composed of names, which is an Object property, and separated by a /, which will be internally converted to Object IDs to construct a full path to desired file. This is necessary to allow deep linking into the system.

/list Common use cases

1. Exploring file structure of the system.

/list JavaScript example

const listUrl = `${gmDataEndpoint}/list/1/Folder1/Folder2/Folder3`;
// if Folder3 Object ID is 5555, then equivalent URL is ${gmDataEndpoint}/list/5555/

axios.get(listUrl, {
  // necessary to pass on server set HttpOnly authentication cookies
  withCredentials: "true"
})
.then(response => console.log(response.data))
.catch(error => console.log(error))

// response.data will contain data array of Event Objects

/history

Gets all Events associated with an Object ID.

Request parameter	Parameter Value
URL	{host}/history/{oid}/{path}?queryString1=queryValue1
body	none
method	GET
credentialsInclude	true

/history Query Strings

Query String	URL Example	Description
?tstamp	{host}/history/1578cfdd0241cbee/?tstamp=1578a72437e23c00	?tstamp may be must be specified in 64-bit hex (Unix Nanoseconds)
?last	{host}/notifications/1/?last=1578a72437e23c00&count=10	?last may be added, as a timestamp (for /notification and /history, but oid for /list) used in conjunction with ?count to control paging. Specify Object ID last seen as a cutoff.
?count	{host}/notifications/1/?last=1578a72437e23c00&count=10	?count may be added, used in conjunction with ?last to control paging.
/path/	{host}/history/1578cfdd039b919b/testFolder/TestFile.csv	/path/ may be added, composed of names, which is an Object property, and separated by a /, which will be internally converted to Object IDs to construct a full path to desired file. This is necessary to allow deep linking into the system.

/history Common use cases

1. Getting version history of an Object,

2. Polling this endpoint with Object ID specified to have notifications, and

3. Collecting system-wide analytics by querying this endpoint without inputting an Object ID.

/history JavaScript example

const historyUrl = `${gmDataEndpoint}/history/1/Folder1/Folder2/Folder3`;
// if Folder3 Object ID is 5555, then equivalent URL is ${gmDataEndpoint}/history/5555/

axios.get(historyUrl, {
  // necessary to pass on server set HttpOnly authentication cookies
  withCredentials: "true"
})
.then(response => console.log(response.data))
.catch(error => console.log(error))

// response.data will contain data array of Event Objects

/stream

Gets raw bytes of an Object ID.

Request parameter	Parameter Value
URL	{host}/stream/{oid}/{path}/?queryString1=queryValue1&?queryString2=queryValue2
body	none
method	GET
credentialsInclude	true

/stream Query Strings

Query String	URL Example	Description
?tstamp	{host}/stream/1578cfdd0241cbee/?tstamp=1578a72437e23c00	?tstamp may be must be specified in 64-bit hex (Unix Nanoseconds)
/path/	{host}/stream/1578cfdd039b919b/testFolder/TestFile.csv	/path/ may be added, composed of names, which is an Object property, and separated by a /, which will be internally converted to Object IDs to construct a full path to desired file. This is necessary to allow deep linking into the system.
?disposition	{host}/stream/41312434423/file.txt?disposition=attachment	The Content-Disposition header defaults to inline, so that browser shows the file. Use disposition=attachment to force a browser download.

Standard HTTP range requesting is supported on streams. Browsers will range request large media files without any special usage of the API

This endpoint can take range requesting parameters as well. rangeLow and rangeHigh are the upper and lower bound on bytes to be retrieved out of the stream. rangeHigh can be omitted, to get an open-ended stream that runs until the file ends. rangeLength is an alternative to using rangeHigh, to specify the desired content length of the partial download directly. Units are allowed on these. k, m, g, t are recognized. k is a unit of 1024 - computer kilobyte, etc. Notice that as required in the http range request specification, the rangeHigh turned into an actual number (with units multiplied in) is the location of the last byte in the data that comes back; which is probably one less than you may expect. ex: bytes=0-1023 asks for 1024 bytes. bytes=1k-2k also asks for 1024 bytes. If there is a unit on the upper-bound, then the number has 1 subtracted from it to be rendered the way that http wants it.

/stream Common use cases

1. Downloading Objects locally.

/stream JavaScript example

  const streamUrlEndpoint = `${gmDataEndpoint}/stream/1/Folder1/Document.csv`;
  axios
    .get(streamUrlEndpoint, { responseType: "blob", withCredentials: true })
    .then(res => {
      const URL = window.URL.createObjectURL(res.data);
      const a = document.createElement("a");
      a.download = "Document.csv";
      a.href = URL;
      document.body.appendChild(a);
      a.click();
      document.body.removeChild(a);
  });
  // The response of /stream endpoint is a stream of bytes that can be processed as a blob and downloaded locally through browser's link functionality

/show

Gets raw bytes of an Object ID and wraps it in an iFrame that shows Objects security meta data.

Request parameter	Parameter Value
URL	{host}/show/{oid}/{path}/?queryString1=queryValue1
body	none
method	GET
credentialsInclude	true

/show Query Strings

Query String	URL Example	Description
?tstamp	{host}/show/1578cfdd0241cbee/?tstamp=1578a72437e23c00	?tstamp may be must be specified in 64-bit hex (Unix Nanoseconds)
/path/	{host}/show/1578cfdd039b919b/testFolder/TestFile.csv	/path/ may be added, composed of names, which is an Object property, and separated by a /, which will be internally converted to Object IDs to construct a full path to desired file. This is necessary to allow deep linking into the system.

/show Common use cases

1. Preview Object in browser, and

2. Serve stored assets to remote clients in a CDN-like manner but with fine grained security policies.

/show JavaScript example

const showUrlEndpoint = "${gmDataEndpoint}/stream/1/Folder1/image.jpeg";
win = window.open(showUrlEndpoint, "_blank");
wSin.focus();
// This will open a new tab in your browser and show you file content with security meta data banner

/macros

An internal method used for verification of Object’s security policy.

Request parameter	Parameter Value
URL	{host}/macros/{oid}/
body	none
method	GET
credentialsInclude	true

/lisp-to-json

POST object policy lisp here to convert it to json, so that you do not need a port of the converter to your language

Request parameter	Parameter Value
URL	/lisp-to-json
body	object policy in lisp format
method	POST

Limited to 5MB POST body

/json-to-lisp

POST object policy lisp here to convert it to json, so that you do not need a port of the converter to your language

Request parameter	Parameter Value
URL	/json-to-lisp
body	object policy in json format
method	POST

Limited to 5MB POST body

/json-to-lisp Query Strings

none

/json-to-lisp Common use cases

1. Verify Objects security policy complies with security schema.

/macros JavaScript example

const macrosUrl = `${gmDataEndpoint}/macros/0000000000000001/`;
axios
  .get(macrosUrl, {
    // necessary to pass on server set HttpOnly authentication cookies
    withCredentials: "true"
  })
  .then(resp => {
    console.log(resp.data);
  })
  .catch(error => {
    console.log(error);
  });

// This will return an object that will contain description of a security policy.

/notifications

Returns all events for specified Object ID

Request parameter	Parameter Value
URL	{host}/notifications/{oid}/
body	none
method	GET
credentialsInclude	true

/notifications Query Strings

Query String	URL Example	Description
?tstamp	{host}/c/1578cfdd0241cbee/?tstamp=1578a72437e23c00	?tstamp may be must be specified in 64-bit hex (Unix Nanoseconds)
?last	{host}/notificationsQueryStrings/1578cfdd039b919b/?last=1578a72437e23c00&count=10	?last may be added, as a timestamp (for /notification and /history, but oid for /list) used in conjunction with ?count to control paging. Specify Object ID last seen as a cutoff.
?count	{host}/notificationsQueryStrings/1578cfdd039b919b/?last=1578a72437e23c00&count=10	?count may be added, used in conjunction with ?last to control paging.
/path/	{host}/notificationsQueryStrings/1578cfdd039b919b/testFolder/TestFile.csv	/path/ may be added, composed of names, which is an Object property, and separated by a /, which will be internally converted to Object IDs to construct a full path to desired file. This is necessary to allow deep linking into the system.

/notifications Common use cases

1. Get all changes to the system for monitoring purposes when requesting notifications from root directory

2. Get notifications for a specific folder in the system to keep track of updates

/notifications JavaScript example

const notificationsUrl = `${gmDataEndpoint}/notifications/0000000000000001/`;
axios
  .get(notificationsUrl, {
    // necessary to pass on server set HttpOnly authentication cookies
    withCredentials: "true"
})
.then(resp => console.log(resp.data))
.catch(error => console.log(error));
// This will return an object that will contain an array of all events sorted in reverse chronological order.

/derived

Returns list of Object IDs of files that are related to specified Object ID

Request parameter	Parameter Value
URL	{host}/derived/{oid}/
body	none
method	GET
credentialsInclude	true

/derived Query Strings

none

/derived Common use cases

1. Find all Objects that are related to another Object, for example, thumbnails of an image.

/derived JavaScript example

const derivedUrl = `${gmDataEndpoint}/derived/0000000000000001/`;
axios
  .get(derivedUrl, {
    // necessary to pass on server set HttpOnly authentication cookies
    withCredentials: "true"
})
.then(resp => console.log(resp.data))
.catch(error => console.log(error));
// This will return an object that will contain an array of all related Object IDs or nulls if none.

/write

Write endpoint is the only way to modify data within Grey Matter Data. Requests sent to /write endpoint must use POST and have form data in the body of the request, container ‘meta’ property with an Event Object. In case of file upload, form data must also have ‘blob’ property with file in Blob format attached.

Request parameter	Parameter Value
URL	{host}/write
body	form data
method	POST
credentialsInclude	true

Note: The root directory of a freshly initialized system is read only. The system will be configured with a root directory with self-service permissions. Usually, this directory will be called /world, and will allow you only to create a directory named after your email (using a POST /write call). It is possible that this can be re-configured to be something else (/home, /northeast) for the directory name, and possibly a different field that is signed in the user's JWT token (ie: username, dn, cn, etc.). Within this directory that you are allowed to create, you can have whatever permissions you want. This ensures that the root directory has a regular structure that UIs can predict, and is not filled with testing garbage.

/write Query Strings

none

/write Example JavaScript FormData with an Event being appended

In this example, we have a simple update in which we are not worried about losing any other fields other than the ones mentioned. See the next example of how updates and deletes should normally be done.

const events = [
  {
    action: "U",
    oid: 42,
    parentoid: 2,
    name: "some directory",
    // if isfile is not set to true, then it's assumed to be a directory!
    originalobjectpolicy: "(if (contains email [e-mail]) (yield-all))",
    security: {
      label: "DECIPHER//GMDATA",
      foreground: "#FFFFFF",
      background: "#00FF00"
    }
  }
];

const formData = new FormData();
formData.append("meta", JSON.stringify(events));

/write Example JavaScript of updating a file name

Note: when action: 'U' (update) is specified, all parameters, except the few that are being updated and tstamp, must be specified, mimicking previous Event associated with the Object ID that is being updated.

An update or delete will need to preserve previous values. So, get a copy of the previous /props for this oid. Then submit a modification of it with these changes:

• action must be U

• you can modify fields other than oid

• by default, if you leave the field rname in, it will not update the file, but only its properties. rname is the random name assigned to the file blob that is (usually) stored in S3.

• if you blank out rname, then the formData will insist that you append the file. you are telling the system "do not use the existing blob, use one created from this file instead".

var proposedobj = existingobj;
proposedobj.action = "U";
proposedobj.name = "myFileUpdatedName.txt";

const events = [ proposedobj ];

const formData = new FormData();
formData.append("meta", JSON.stringify(events));

axios.post(${gmDataEndpoint}/write, formData, {
  // necessary to pass on server set HttpOnly authentication cookies
  withCredentials: "true"
});

For all updates and deletes, start with the existing obj from /props/{oid}/ as a template. Modify action, then the desired fields that need to change. Whenever rname is blanked out on a create or update, the formData must supply a blob for it.

/write Example JavaScript of upload 2 files

In case of data upload, a request’s form data must have an appended {'blob': File BLOB} object for each file that is being uploaded in the same order as specified in the accompanying meta Event object. More information on this can be found in the /write endpoint section.

Note: When action: “C” (create / upload) is specified, the system will backfill Object ID when it’s created internally thus you shouldn’t specify oid parameter. Note: When action: "C" (create / upload) is specified, the parameters below are the bare minimum parameters that must be specified for the create action to complete internally Note: Files are in File Object format Note: When action: "U" (update) is specified, if an oid is supplied, it will update that exact object. Otherwise, it will try to find an oid with the given file name, and fill in the oid for you.

const filesToUpload = [ [File1], [File2] ];
// these two forms are equivalent... as LISP - lock down to owner for updates
const originalobjectpolicy = "(if (contains email [e-mail])(yield-all)(yield R X))";
// or compiled server-side as the JSON it is actually searched against
// note: f is "function", a is "args", v is "value".
const objectpolicy = {
  f: "if",
  a: [
    {f: "contains", a:[{v: "email"},{v: "[e-mail]"}]}
    {f: "yield-all"},
    {f: "yield", a:[{v: "R"}, {v: "X"}]}
  ]
};

const events = [
  {
    action: "C",
    name: "New File 1",
    isfile: true,
    mimetype: "image/jpeg",
    parentoid: 1,
    // was compiled client-side
    objectpolicy: objectpolicy,
    security: {
      label: "DECIPHER//GMDATA",
      foreground: "#FFFFFF",
      background: "#00FF00"
    }
  },
  {
    action: "C",
    name: "New File 2",
    isfile: true,
    mimetype: "image/jpeg",
    parentoid: 1,
    // compiled server-side, slower, but convenient equiv to using objectpolicy
    originalobjectpolicy: originalobjectpolicy,
    security: {
      label: "DECIPHER//GMDATA",
      foreground: "#FFFFFF",
      background: "#00FF00"
    }
  }
];

const formData = new FormData();
formData.append("meta", JSON.stringify(events));

for (let i in events) {
  formData.append("blob", filesToUpload[i]);
}

axios.post(${gmDataEndpoint}/write, formData, {
  // necessary to pass on server set HttpOnly authentication cookies
  withCredentials: "true"
})

Updates In General

In order to update an object, the strategy is to:

• Get the most recent item as an example and submit a modification to it

  • The /props for this oid, or first entry of /history will be the most recent state.

  • Many fields from this version were written by the server, and will be ignored if we leave them set. So we can ignore some of these fields for example:

    • tstamp

  • tstampend

  • jwthash

  • checkedtstamp

  • Just take the most recent state and submit a modification

    • Any field other than oid can be updated (if it's not one of the ignored fields)

  • Set action to U if we plan on making an update

    • If we plan on altering the file stream, then we must remove the field rname. What this means is "I am not going to reference an existing random name to an already uploaded stream. I am going to supply you with a new stream (which will be assigned its own rname.).

    • If we just want to modify properties and not update the stream, then rname should be left where it is.

    • You can use this functionality to pick an older rname in history to revert back to an older stream. But note that you cannot use an rname from another oid, as the oid is part of what determines the encryption key for the object.

    • Set an action to D if we plan on doing a delete. Note that the fields on this deleted object determine what it will look like in the trashcan. If it is resurrected from the trashcan by re-submitting with a U, then all fields (including recent rname) must be present.

  • If an object is already in state D, then it is eligible for a purge by setting the action to P. A simple P will remove the entire object oid from the database, and leave a temporary purge marker (for replication reasons) that will expire shortly. If the metadata about the purged file is sensitive, then all such fields can be redacted when writing the purge object. For example, set the name and objectpolicy (possibly every property) to redacted values. This file can not be resurrected from the trash, and the only thing really needed of it is the oid.

  • fulldir is a fully qualified directory that this item is in, at the time of the update. It must include the root oid, and terminate the fulldir (of the parent only) with a slash. If you want to be able to filter like: "everything under /0000000000000001/home/rob/", then set fulldir on objects as they are updated. Since we locate objects like (oid, name), this does not give you an easy way to know that an oid happens to be under another oid. So, for efficiency reasons, we leave this task to the client; who knows the full path to every object being written. Example:

    • /aae8f90aef/home/rob/ has a directory with name work. The fulldir is the directory that this object is in: /aae8f90aef/home/rob/, even though work is itself a directory. The path for this object, that happens to be a directory (isfile=false), is: /aae8f90aef/home/rob/work. So to list the work dir, the full dir is: /list/aae8f90aef/home/rob/work

  • /aae8f90aef/home/rob/ also has a file named resume.pdf. So, fulldir is /aae8f90aef/home/rob/, but stream the data with: /stream/aae8f90aef/home/rob/resume.pdf

Security Caveats:

• If you made a permissions mistake, then fix it with action: P; as action: D is an update with hide. The default system assumes that permissions mistakes are not being made, so performing a D on a file hides it from current version, but allows historical versions to be retrieved using /history and the tstamp parameter; like a delete from a version control system. If there was a permission mistake, then action: P the file to purge the entire file (redacting fields as necessary, for the action: P entry will remain in the database for a while), or include the purgetstamp to just purge one mistaken state (ie: an update with a wrong classification applied to it). This will keep it out of the database, but such records may have already escaped on Kafka queues.

• With Update permission (from CRUDXP), you effectively have full access to the file. If you have permission U on a file to update it, you can use this to make updates to grant all the other privileges on that file. So as far as enforcement goes, (R, X, U) are the main privileges. D and P show up as extra privileges to advise what the policy is supposed to be; and to stop accidents. This is a similar issue with chmod and chown under Unix. You may not have execute bits on a Unix file, but if you have w access, then you can grant it to yourself.

• If a bot writes a file on behalf of another user, the bot will need read and update access on the file. If you attempt to perform a write, without at least (R,U) access, you will not be able to correct any mistaken writes. So, the system will warn you that you may be trying to use wrong permissions (perhaps a misspelled role, attribute name, email,or userDN). But a bot may well try to write a file that it does not have permissions to update or read later. It is inadvisable to use allowpartialpermissions to make the write go through; because if the write succeeds with the permissions in error, then there is now data in the system to which NO user has access. It's technically possible to generate (contains privilege root) tokens that ignore permissions, but it's likely that your admin will not allow it, or help you to issue such tokens. Thus, there would be no way to clean up messes created by users writing things that they cannot edit themselves. So, it's best to make a write succeed by making sure that the author of the write has (R,U) privileges.

In this section we reviewed every endpoint provided by the Grey Matter Data API and the purpose it serves. Through combinations of endpoints and query strings, you will have complete, fine-grained control of your data.

/self

Reflect back the JWT claims that had been signed, that the server believes to be true due to the signature. The result is json.