Client Tutorial
In order to perform any operations in the system, we need claims (from a JWT token) to assert our privilege. All edits to the system come from a multipart mime POST /write call. We will start from an empty system, and add data. The root of the filesystem is only editable with root privilege, so we need to make user directories. Given this JWT claim, we need to generate a JWT token from some service (not gm-data) that knows user attributes and is trusted to sign off on them; like short-term certificates for tasks. Currently, the scripts ./makeurlroot, ./makeurljane invokes libraries to generate the tokens. You may want to read these short bash scripts to understand it.
{
"label": "asRoot",
"values": {
"privilege": [
"root"
]
}
}The token is generated when we go through a service that makes tokens, like /services/gm-jwt-security/1.0 
We can host multiple jwt servers in the system. A user passes through a JWT server to have an appropriate cookie set.
It encodes like this valid OAuth Bearer token, that we will use to create root directories.
eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTAxODI4MDMsImxhYmVsIjoiYXNSb290IiwidmFsdWVzIjp7InByaXZpbGVnZSI6WyJyb290Il19fQ.AXckbmNH0HVwuixpLPsj5AJR_cztOkQn0Ltju1klz1MaMu14Keg28R__4sUSSLJYQEKTtE8HfSwcpjnOWFKnW03YAI_XY4l5ZEAo7hjuNA5oQQN1wPPShGrG8tUpFAOSZNuyLJ4SfgHgUW_-Fk1kYhbGDZAkaZL8295wRQkJjvvvvwsTrecognized privilege: root (unrestricted use), readonly (no /write), blobkey (show S3 key)
We will create a directory for jane.doe@gmail.com that is readable by everyone (including anonymous), multipart/form-data POST /write with this as first part.
{
"name": "jane",
"action": "C",
"parentoid": "0000000000000001",
"objectpolicy": {
"label": "forAnonReadjaneFull",
"requirements": {
"f": "owner-full-ro-all",
"a": [
{
"v": "email"
},
{
"v": "jane.doe@deciphernow.com"
}
]
}
},
"derived": {},
"security": {
"label": "anonymousread ownedby jane.doe@gmail.com",
"foreground": "#FFFFFF",
"background": "#000000"
}
}We get back this:
We will create a directory for lucas.moten@deciphernow.com that is readable by everyone (including anonymous), multipart/form-data POST /write with this as first part.
We get back this:
Now we get a jwt token for jane with these claims.
Now we get a jwt token for lucas with these claims.
Create a file in the directory for jane.doe@gmail.com. multipart/form-data POST /write, with this in first part, and file blob next part
We get back this.
Create a file in the directory for lucas.moten@deciphernow.com. multipart/form-data POST /write, with this in first part, and file blob next part
We get back this.
Write Entire Directories In One Call
In order to do bulk uploads, it is possible to make a single large transaction. This transaction is an array of writes that should succeed or fail together. It is necessary for some writes to refer to previous writes, such as creating a directory in the 5th entry, and the 6th entry declaring that its parent directory is the 5th entry (ie: one entry prior to this one). This means that we need a notion of references to copy the oid of a previous write (that we won't know until it's inserted into the database) into the parentoid of this entry. If we POST /write this as multipart/form-data, this will be the first part:
... And the second part will be the actual file:
The first element of the first part (json) specified a directory. The second element of the first part specified a file. Because in this file we did not specify an Rname for where it goes in S3, it is going to expect that the next part of this multipart is this file.
Notice the References field used to fill in unknown parentoid with oid of previous directory.
The new directory was assigned an oid of 158340da87cde534, and the two files copied that in as their parent oids. The first file saw it -1 entry back, and second file saw it -2 entries back.
Update Metadata On An Existing Object (158340da87e468e0) Without Editing File
We will write this update record, where we specify ALL of the state required.. We might change anything on it.
Delete An Existing Object (158340da87e468e0)
Put into a deleted state. The key field is the oid.
Reference: Full Bytes Of Calls
Input: Create a directory
Output: The directory is created
Input: Create a directory
Output: The directory is created
Input: Create a file in the directory
Output: The file was created
Input: Create a file in the directory
Output: The file was created
Input: Do a multi-upload of one new directory, with two files into it.
Output: The directory was created, then file placed into directory
Input: Update (action U) the metadata on a file that was already created. We can re-name it, move it by changing its parent, etc. Most fields are editable, except for the oid.
Output: The metadata was modified, but its stream was not re-uploaded or modified.
Input: Put this file into a Deleted (action D) state. You can see old versions in history, but this file is no longer in current listings.
Output: The Delete only needs a few key fields, but including the entire state will allow for the trashcan to show all relevant information about it.
Reference: Event fields
name
type
format
description
schema
integer
int32
version of the schema from which this object came
jwt-hash
string
sha256hex
sha256 of JWT token we accepted
userpolicy
string
jwt claims used when creating this event
tstamp
string
tstamp
nanosecond timestamp of this objects creation time - unique per event.
tstampend
string
tstamp
end data for this record validity, which may change
name
string
filename
file/dir name without pathing
action
string
CRUDXP
actions on a file to put it into its current state
oid
string
tstamp
a numeric identifier assigned to a file/dir. happens to be a nanoseconds timestamp. (oid, tstamp) are the primary key for these events.
parentoid
string
tstamp
oid of the parent directory.
expiration
string
tstamp
tstamp as of which this record will not come back from queries. may legally require purges at some point.
checkedtstamp
string
tstamp
the tstamp of the previous version of the oid we compared against.
objectpolicy
string
the rules that allow action (C R U D X P)
mimetype
string
same as content-type. set for this object, and also for S3 blobs.
rname
string
s3name
${AWS_S3_BUCKET}/${AWS_S3_PARTITION}/${rname} is where this object is in S3. Name is assigned even if only IPFS is used.
size
integer
int64
Same as Content-Length.
derived
string
Fields to denote that a file is derived from another - ie: doc to PDS. (oid,tstamp) are a primary key for the original. type allows us to track what kind of derivation it is, such as doc-to-txt.
isfile
boolean
Set this if it is a file. You will get a directory if you don't set this.
custom
string
Put custom fields in here that gm-data does not understand, but need to be tracked by the application.
defaultfile
string
filename
For a directory, if we try to stream the directory, ${defaultfile} will be the assumed name. index.html is the default value.
security
string
Security labels are written here, along with their foreground/background pen colors.
description
string
Description of what is in the file or directory
sha256plain
string
sha256 of the plaintext, can be used in the client to calculate a minimum number of files to send for update
blobalgorithm
string
the blob algorithm. if not specified, we store in S3 with SSECustomerKey
purgetstamp
string
tstamp
In order to just purge a single event, supply a purgetstamp along with the oid.
fulldir
string
path
full dir for this version of this file, beginning and terminated with a slash
references
string
When uploading files, an array of updates can be supplied; so that we can upload files into directories that don't yet exist. Indices are negative relative values.
encrypted
string
Allow for fields to be encrypted on a case-by-case basis
Values for action
possible-value
description
C
create a file, system will make up an oid. parentoid is required.
R
permissions to read metadata about a file.
U
update a file, with existing oid. if oid not given, lookup by name will be done for upsert behavior.
D
delete a file. oid and objectPolicy are required.
X
execute on a file/dir. this means streaming a file, or listing dir.
P
purge a file, which deletes the entire history of the oid.
Values for security
possible-value
description
label
the banner string shown at top and bottom, such as SECRET
background
the background color in html color form such as #FF0000
foreground
the foreground color of the text in html format such as #FFFFFF
Values for blobalgorithm
possible-value
description
SSECustomerKey, where we must track a crypto key, and we send the key into the S3 request to decrypt and encrypt. This allows for the use of analytics APIs that can supply a crypto key.
none
Disable encryption in the S3 bucket. This facilitates use with APIs that cannot supply a crypto key, and don't need encryption.
sse
Server Side S3 encryption. This means that the client does not need to supply the crypto key, as S3 finds and plugs it in.
Values for references
possible-value
description
address
a relative value in the array of uploaded values
from
the field to copy from: parentoid, oid, tstamp
to
the field to copy to: oid, tstamp
Values for encrypted
possible-value
description
name
overwrite given name with encrypted name when this metadata is retrieved.
security
overwrite given security label with encrypted security when this metadata is retrieved.
blobkey
overwrite given blobkey label with encrypted blobkey when this metadata is retrieved.
custom
overwrite given custom attributes with encrypted custom attributes when this metadata is retrieved - per field.
Last updated
Was this helpful?