GraphGrid Ingest
2.0
API Version 1.0
Introduction
GraphGrid Ingest gathers data from assorted sources and transports it to a graph database that can be accessed and analyzed. This means that any disconnected data is ingested into a connected state. Ingest allows users to define a policy that maps values in their source database to a property graph model and thereby represent nodes, relationships and properties in the target property graph database. Ingest also includes the capability to automatically convert these values to standard formats if no policy map is found. Ingest is a part of Graphgrid Publish.
API
This Ingest API version is 1.0
and as such all endpoints are rooted at /1.0/
. For example, http://localhost/1.0/ingest
(requires auth) would be
the base context for this Ingest API under the GraphGrid API deployed at http://localhost
. All Ingest endpoints are routed through Graphgrid Publish.
Environment
Ingest requires the following integrations:
- ONgDB 1.0+
Ingest supports the following integrations:
- RabbitMQ 3.5+
- AWS SQS
Ingest
Get Status
Check the status of GraphGrid Ingest. Will return a 200 OK
response if healthy.
- BaseURL:
/1.0/ingest/status
- Method: GET
Request
curl --location --request GET -i "${API_BASE}/1.0/ingest/status"
Response
200 OK
Ingest Policy
The Ingest Policy provides a course of action determine transformations from a source database to the target property graph database. If a database needs to be updated, changes to the policy can be made. Changes to the policy take effect immediately.
Sample Policy
{
"name": "Movie Policy 123",
"displayName": "Movie Policy",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": ["MAPPED_AUTO"],
"attributes": ["MAPPED_AUTO"],
"schema": ["SCHEMA_ONLY"]
},
"ous": {
"PERSON": {
"mapping": {
"name": "Person",
"primaryKey": "personId",
"primaryKeyPrefix": "person:"
},
"attributes": {
"NAME": {
"mapping": {
"name": "personName",
"type": "PROPERTY",
"defaultValue": "John Smith",
"fixed": false
}
},
"likes": {
"mapping": {
"name": "LIKES",
"type": "RELATIONSHIP",
"direction": "OUTGOING"
}
}
}
},
"MOVIE": {
"mapping": {
"name": "Movie",
"primaryKey": "movieId",
"primaryKeyPrefix": "movie:",
"altNames": ["Media", "Visual"]
},
"attributes": {
"in": {
"mapping": {
"name": "IN",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
}
},
"schema": {
"delta": {
"constraints": [
{
"cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
}
],
"indexes": [
{
"cypher": "CREATE INDEX ON :Person(name);"
}
]
},
"full": {
"constraints": [
{
"cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
}
],
"indexes": [
{
"cypher": "CREATE INDEX ON :Person(name);"
}
]
}
},
"attributes": {
"NAME": {
"mapping": {
"name": "name"
}
},
"PERSON_ID": {
"mapping": {
"name": "pid"
}
},
"MOVIE_ID": {
"mapping": {
"name": "pid"
}
}
}
}
Components
Component | Description |
---|---|
name | The unique name of the policy. |
displayName | The name to display for this policy. |
policy | The type of policy and its details (ex. types, OUs, attributes, and schema) to implement. |
MAPPED_AUTO | Map everything according to the rules in the policy, and use the default mapping for anything not defined. |
MAPPED_ONLY | Map everything according to the rules in the policy and ignore anything that is not defined. |
Organizational Units (OUs) | A list of organizational units, along with their label, key, and relationship mappings. |
primaryKey | The name of the primary key to merge on. |
primaryKeyPrefix | The prefix to use in the target database before the label and id.For example, if a node has the label Person, the primaryKeyPrefix is "per" and its given id from the source database is "12345678", the generated value for the new primary key will be per:person:12345678 . |
altNames | A list of additional labels to be applied to all nodes of the given type. |
attributes | A list of global attribute key mappings. |
defaultValue | If an OU has an attribute with a defaultValue, it will always be applied whenever an update is sent that contains a new object of that OU type. In this example, any Person that's added to the graph would automatically get the personName "John Smith", unless the name is specified in the update. If "fixed" (by default false) is set to true, the default value would be immutable. |
Schema
A list of constraints and indexes in the database. "delta" defines the constraints and indexes to be applied or dropped when the current policy gets uploaded. "full" defines the full set of constraints and indexes within the database. If the schema type is "SCHEMA_ONLY", there must be a schema object in the policy. The schema will be updated according to the provided Geequel only. If the schema type is "SCHEMA_AUTO", there cannot be a "schema" object provided, since the schema will be defined automatically according to the primary keys provided in the list of OU objects.
SCHEMA_ONLY
The schema within the uploaded policy will also get new information in it. The state of each constraint, as well as the duration of the query to create it (in ms), will be added to the object. If there is any reason that the constraint could not be enforced, an "errors" object will be added as well.
"schema": {
"delta": {
"constraints": [
{
"cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;",
"state": "ONLINE",
"stats": {
"duration": 8,
}
]
}
}
You can also drop constraints and indexes through the policy. Here's an example using the same policy as the previous one, but with a change in the "schema" object:
"schema":{
"delta":{
"constraints":[
{
"cypher": "DROP CONTSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
}
]
}
}
SCHEMA_AUTO
By default, the schema is automatically generated. In this case, the policy cannot contain a "schema" object. The constraints will be applied based on the primary keys provided in the OU list. Every label in the OU list will also get a uniqueness constraint on the "grn" property, regardless of whether another primary key is provided. Here is an example:
{
"name": "something",
"displayName": "Something",
"createdAt": "2018-01-01T00:00Z",
"state": "ACTIVE",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": [
"MAPPED_AUTO"
],
"attributes": [
"MAPPED_AUTO"
],
"schema": [
"SCHEMA_AUTO"
]
},
"ous": {
"PERSON": {
"mapping": {
"name": "Person",
"primaryKey": "personId",
"primaryKeyPrefix": "person"
},
"attributes": {
}
},
"DEPARTMENT": {
"mapping": {
"name": "Department"
},
"attributes": {
}
},
"attributes": {
}
}
Versioning
Each time a policy is uploaded with the same name and cluster as an existing one, the previous one will be added in a "versions" list at the end of the new policy for easy review. If the following policy is uploaded:
{
"name": "something",
"displayName": "Something",
"createdAt": "2018-01-01T00:00Z",
"state": "ACTIVE",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": ["MAPPED_AUTO"],
"attributes": ["MAPPED_AUTO"],
"schema": ["SCHEMA_ONLY"]
},
"ous": {
"PERSON": {
"mapping": {
"name": "Person"
},
"attributes": {
"NAME": {
"mapping": {
"name": "personName",
"type": "PROPERTY"
}
},
"r1": {
"mapping": {
"name": "HAS",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
},
"DEPARTMENT": {
"mapping": {
"name": "Department"
},
"attributes": {
"membersOf": {
"mapping": {
"name": "MEMBERS_OF",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
}
},
"attributes": {
"NAME": {
"mapping": {
"name": "name"
}
},
"PERSON_ID": {
"mapping": {
"name": "pid"
}
},
"ENTITY_ID": {
"mapping": {
"name": "pid"
}
}
},
"schema": {}
}
Followed by this one under the same name and in the same cluster:
{
"name": "something",
"displayName": "Something",
"createdAt": "2018-01-02T00:00Z",
"state": "ACTIVE",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": ["MAPPED_AUTO"],
"attributes": ["MAPPED_AUTO"],
"schema": ["SCHEMA_ONLY"]
},
"ous": {
"PERSON": {
"mapping": {
"name": "Person"
},
"attributes": {
"NAME": {
"mapping": {
"name": "personName",
"type": "PROPERTY"
}
},
"r1": {
"mapping": {
"name": "HAS",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
},
"DEPARTMENT": {
"mapping": {
"name": "Department"
},
"attributes": {
"membersOf": {
"mapping": {
"name": "MEMBERS_OF",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
}
},
"attributes": {
"NAME": {
"mapping": {
"name": "name"
}
},
"PERSON_ID": {
"mapping": {
"name": "pid"
}
},
"ENTITY_ID": {
"mapping": {
"name": "pid"
}
}
},
"schema": {}
}
The resulting policy would look like this:
{
"name": "something",
"displayName": "Something",
"createdAt": "2018-01-02T00:00Z",
"state": "ACTIVE",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": ["MAPPED_AUTO"],
"attributes": ["MAPPED_AUTO"],
"schema": ["SCHEMA_ONLY"]
},
"ous": {
"PERSON": {
"mapping": {
"name": "Person"
},
"attributes": {
"NAME": {
"mapping": {
"name": "personName",
"type": "PROPERTY"
}
},
"r1": {
"mapping": {
"name": "HAS",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
},
"DEPARTMENT": {
"mapping": {
"name": "Department"
},
"attributes": {
"membersOf": {
"mapping": {
"name": "MEMBERS_OF",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
}
},
"attributes": {
"NAME": {
"mapping": {
"name": "name"
}
},
"PERSON_ID": {
"mapping": {
"name": "pid"
}
},
"ENTITY_ID": {
"mapping": {
"name": "pid"
}
}
},
"schema": {},
"versions": [
{
"name": "something",
"displayName": "Something",
"createdAt": "2018-01-01T00:00Z",
"state": "ACTIVE",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": ["MAPPED_AUTO"],
"attributes": ["MAPPED_AUTO"],
"schema": ["SCHEMA_ONLY"]
},
"ous": {
"PERSON": {
"mapping": {
"name": "Person"
},
"attributes": {
"NAME": {
"mapping": {
"name": "personName",
"type": "PROPERTY"
}
},
"r1": {
"mapping": {
"name": "HAS",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
},
"DEPARTMENT": {
"mapping": {
"name": "Department"
},
"attributes": {
"membersOf": {
"mapping": {
"name": "MEMBERS_OF",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
}
},
"attributes": {
"NAME": {
"mapping": {
"name": "name"
}
},
"PERSON_ID": {
"mapping": {
"name": "pid"
}
},
"ENTITY_ID": {
"mapping": {
"name": "pid"
}
}
},
"schema": {}
}
]
}
Sample XML Policy
In addition to a policy for updates in a JSON format, there is also an endpoint that allows uploading a policy for updates in an XML format. Instead of "OUs" and "attributes", an XML policy uses "elements", "attributes", and "elementAttributes". Elements can have other elements nested within them, along with their own attributes and element attributes. Other parts of the policy (mapping type, schema, versioning, etc.) behave the same way as a JSON policy. Here is a sample XML policy:
{
"name": "something",
"displayName": "Something",
"createdAt": "2018-01-01T00:00Z",
"state": "ACTIVE",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
},
"MAPPED_ONLY": {
"definition": "Does only what is mapped ignoring everything else."
}
},
"elements": ["MAPPED_AUTO"],
"attributes": ["MAPPED_AUTO"],
"elementAttributes": ["MAPPED_AUTO"]
},
"elements": {
"movie": {
"mapping": {
"name": "Movie",
"primaryKey": "movie-id",
"primaryKeyPrefix": "movie"
},
"elements": {},
"attributes": {
"rating": {
"mapping": {
"name": "rating",
"type": "IGNORE"
}
}
},
"elementAttributes": {
"studio": {
"mapping": {
"name": "studio",
"type": "NODE_RELATIONSHIP",
"direction": "OUTGOING"
}
}
}
}
},
"attributes": {},
"elementAttributes": {}
}
Create Policy
Creates a named Ingest Policy for a GraphGrid Cluster.
- BaseURL:
/1.0/ingest/{{clusterName}}/policy/{{policyName}}
- Method: POST
Parameters
Parameter | Description |
---|---|
clusterName string | The GraphGrid Cluster associated with this Ingest Policy. |
policyName string | The name for this Ingest Policy. |
A valid cluster name must follow the format {org}-{env}-{cluster}
.
Request
curl --location --request POST "${API_BASE}/1.0/ingest/gg-test-ongdb/policy/samplePolicy" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer ${BEARER_TOKEN}" \
--data-raw '{
"name": "Movie Policy 123",
"displayName": "Movie Policy",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": [
"MAPPED_AUTO"
],
"attributes": [
"MAPPED_AUTO"
],
"schema": [
"SCHEMA_ONLY"
]
},
"ous": {
"PERSON": {
"mapping": {
"name": "Person",
"primaryKey": "personId",
"primaryKeyPrefix": "person:"
},
"attributes": {
"NAME": {
"mapping": {
"name": "personName",
"type": "PROPERTY",
"defaultValue": "John Smith",
"fixed": false
}
},
"likes": {
"mapping": {
"name": "LIKES",
"type": "RELATIONSHIP",
"direction": "OUTGOING"
}
}
}
},
"MOVIE": {
"mapping": {
"name": "Movie",
"primaryKey": "movieId",
"primaryKeyPrefix": "movie:",
"altNames": [
"Media",
"Visual"
]
},
"attributes": {
"in": {
"mapping": {
"name": "IN",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
}
},
"schema": {
"delta": {
"constraints": [{
"cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
}],
"indexes": [{
"cypher": "CREATE INDEX ON :Person(name);"
}]
},
"full": {
"constraints": [{
"cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
}],
"indexes": [{
"cypher": "CREATE INDEX ON :Person(name);"
}]
}
},
"attributes": {
"NAME": {
"mapping": {
"name": "name"
}
},
"PERSON_ID": {
"mapping": {
"name": "pid"
}
},
"MOVIE_ID": {
"mapping": {
"name": "pid"
}
}
}
}'
Response
200 OK
Ingest Update
A database can be updated and will take effect immediately.
Sample Update
{
"update": {
"abc": {
"type": "PERSON",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"value": "Person Name"
},
"age": {
"change": "SET",
"value": 50
},
"likes": {
"change": "SET",
"id": "123",
"type": "MOVIE"
}
}
},
"123": {
"type": "MOVIE",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"value": "Movie Name"
},
"in": {
"change": "SET",
"id": "abc",
"type": "PERSON",
"direction": "INCOMING"
}
}
}
}
}
Components
Component | Description |
---|---|
update | A list of updates to be made to the database. Each key represents the unique id of some type of node to be modified. |
type | The type of the new node. This value will be mapped to a node label, either in accordance with the policy, or according to the default convention if the policy doesn't exist or uses MAPPED_AUTO. If the policy is MAPPED_ONLY, and this type isn't mapped in it, then this block will be ignored. This holds for values mapped to property keys and relationship types as well. |
change | Can be either SET (if the change is new or an update) or DELETE (if the node is to be deleted). |
changes | If the type of change is SET, this is a list of changes to be made to the node. Each key is either a property key on the node, or a relationship type attached to the node. |
value | If the change type is SET and it's for a property, this is the value to change the property to. |
id | The unique identifier of the node on the other side of the relationship being modified or deleted. |
type | he type of node on the other side of the relationship being modified or deleted. |
direction | The direction of the relationship being created (default is OUTGOING). |
Update
Saves an update to a GraphGrid Cluster. Optionally accepts an Ingest Policy to use during mapping, otherwise perform an automatic mapping.
- BaseURL:
/1.0/ingest/{{clusterName}}/update?policyName={{policyName}}
- Method: POST
Parameters
Parameter | Description |
---|---|
clusterName string | The GraphGrid Cluster to route to for this update. |
policyName string | The Ingest Policy name to be used during mapping which must correspond to an existing Ingest Policy that was created for this Cluster in Create Policy. |
Request
curl --location --request POST "${API_BASE}/1.0/ingest/gg-test-ongdb/update?policyName=samplePolicy" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer ${BEARER_TOKEN}" \
--data-raw '
{
"update": {
"abc": {
"type": "PERSON",
"change":"SET",
"changes": {
"NAME": {
"change": "SET",
"value": "Person Name"
},
"age": {
"change": "SET",
"value": 50
},
"likes": {
"change": "SET",
"id": "123",
"type": "MOVIE"
}
}
},
"123": {
"type": "MOVIE",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"value": "Movie Name"
},
"in": {
"change": "SET",
"id": "abc",
"type": "PERSON",
"direction": "INCOMING"
}
}
}
}
}
Response
{
"transactionId": "BSIDAdpbK73RmcDy8wNx-1612799404527",
"trackingId": "ZXhndy0yMDIxLTAyLTA4LTE1LTUwL21hc3Rlci9CU0lEQWRwYks3M1JtY0R5OHdOeC0xNjEyNzk5NDA0NTI3",
"update": {
"abc": {
"type": "PERSON",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"id": null,
"type": null,
"value": "Person Name",
"direction": null,
"time": null,
"properties": null
},
"age": {
"change": "SET",
"id": null,
"type": null,
"value": 50,
"direction": null,
"time": null,
"properties": null
},
"likes": {
"change": "SET",
"id": "123",
"type": "MOVIE",
"value": null,
"direction": null,
"time": null,
"properties": null
}
},
"transactionTime": null
},
"123": {
"type": "MOVIE",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"id": null,
"type": null,
"value": "Movie Name",
"direction": null,
"time": null,
"properties": null
},
"in": {
"change": "SET",
"id": "abc",
"type": "PERSON",
"value": null,
"direction": "INCOMING",
"time": null,
"properties": null
}
},
"transactionTime": null
}
},
"mapping": {
"policyName": "samplePolicy",
"transactionRequest": null
},
"transactionResult": {
"transactionState": {
"status": "UPLOADING"
},
"data": {
"uploadTime": "0ms",
"pollTime": null,
"writeTime": null,
"ingestTime": null,
"updatedAt": "2021-02-08T15:50:04.528Z",
"createdAt": "2021-02-08T15:50:04.527Z"
}
},
"xml": null
}
Update XML
Saves an update to a GraphGrid Cluster. Optionally accepts an Ingest Policy to use during mapping, otherwise perform an automatic mapping.
- BaseURL:
/1.0/ingest/{{clusterName}}/xmlupdate?policyName={{policyName}}
- Method: POST
Parameters
Parameter | Description |
---|---|
clusterName string | The GraphGrid Cluster to route to for this update. |
policyName string | The Ingest Policy name to be used during mapping which must correspond to an existing Ingest Policy that was created for this Cluster in Create Policy. |
Request
curl --location --request POST "${API_BASE}/1.0/ingest/gg-test-ongdb/xmlupdate?policyName=XMLPolicy" \
--header 'Content-Type: application/xml' \
--header "Authorization: Bearer ${BEARER_TOKEN}" \
--data-raw '<movie movie-id="abc" rating="5.0">
<actor>
<person person-id="def">
<date>19800101</date>
</person>
</actor>
<format>
<name>DVD</name>
<year>2000</year>
</format>
<studio>Mountain Studios</studio>
</movie>'
Response
{
"transactionId": "UVzhh3ZVsJTt8LJXddaV-1612815345327",
"trackingId": "clFxZC0yMDIxLTAyLTA4LTIwLTE1L21hc3Rlci9VVnpoaDNaVnNKVHQ4TEpYZGRhVi0xNjEyODE1MzQ1MzI3",
"update": null,
"mapping": {
"policyName": "XMLPolicy",
"transactionRequest": null
},
"transactionResult": {
"transactionState": {
"status": "UPLOADING"
},
"data": {
"uploadTime": "0ms",
"pollTime": null,
"writeTime": null,
"ingestTime": null,
"updatedAt": "2021-02-08T20:15:45.328Z",
"createdAt": "2021-02-08T20:15:45.327Z"
}
},
"xml": "<movie movie-id=\"abc\" rating=\"5.0\">\n <actor>\n <person person-id=\"def\">\n <date>19800101</date>\n </person>\n </actor>\n <format>\n <name>DVD</name>\n <year>2000</year>\n </format>\n <studio>Mountain Studios</studio>\n</movie>"
}
Get Update
Gets an update by trackingId
. Can be used to check status and transaction statements made for the update.
- BaseURL:
/ingest/{{clusterName}}/update/{{trackingId}}
- Method: GET
Parameters
Parameter | Description |
---|---|
clusterName string | The GraphGrid Cluster associated with this transaction update. |
trackingId string | A unique ID for tracking an update which is generated in Update. |
Request
curl --location --request GET "${API_BASE}/1.0/ingest/gg-test-ongdb/update/ZXhndy0yMDIxLTAyLTA4LTE1LTUwL21hc3Rlci9CU0lEQWRwYks3M1JtY0R5OHdOeC0xNjEyNzk5NDA0NTI3" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer ${BEARER_TOKEN}"
Response
{
"transactionId": "BSIDAdpbK73RmcDy8wNx-1612799404527",
"trackingId": "ZXhndy0yMDIxLTAyLTA4LTE1LTUwL21hc3Rlci9CU0lEQWRwYks3M1JtY0R5OHdOeC0xNjEyNzk5NDA0NTI3",
"update": {
"abc": {
"type": "PERSON",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"id": null,
"type": null,
"value": "Person Name",
"direction": null,
"time": null,
"properties": null
},
"age": {
"change": "SET",
"id": null,
"type": null,
"value": 50,
"direction": null,
"time": null,
"properties": null
},
"likes": {
"change": "SET",
"id": "123",
"type": "MOVIE",
"value": null,
"direction": null,
"time": null,
"properties": null
}
},
"transactionTime": null
},
"123": {
"type": "MOVIE",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"id": null,
"type": null,
"value": "Movie Name",
"direction": null,
"time": null,
"properties": null
},
"in": {
"change": "SET",
"id": "abc",
"type": "PERSON",
"value": null,
"direction": "INCOMING",
"time": null,
"properties": null
}
},
"transactionTime": null
}
},
"mapping": {
"policyName": "samplePolicy",
"transactionRequest": {
"statements": [
{
"statement": "MERGE (n:`Person` {`grn`: {`grn`}}) ON CREATE SET n.createdAt = {now} ON CREATE SET n.`personName` = {`nonStaticDefaultn1`} SET n.updatedAt = {now} SET n.`personName` = {`personName`};",
"parameters": {
"personName": "Person Name",
"nonStaticDefaultn1": "John Smith",
"grn": "person::abc",
"now": "2021-02-08T15:50:04.533Z"
},
"metadata": null
},
{
"statement": "MERGE (n:`Person` {`grn`: {`grn`}}) ON CREATE SET n.createdAt = {now} ON CREATE SET n.`personName` = {`nonStaticDefaultn1`} SET n.updatedAt = {now} SET n.`age` = {`age`};",
"parameters": {
"nonStaticDefaultn1": "John Smith",
"grn": "person::abc",
"age": 50,
"now": "2021-02-08T15:50:04.533Z"
},
"metadata": null
},
{
"statement": "MERGE (n:`Person` {`grn`: {`grn`}}) ON CREATE SET n.createdAt = {now} ON CREATE SET n.`personName` = {`nonStaticDefaultn1`} SET n.updatedAt = {now} WITH n MERGE (m:`Movie` {`grn`: {grn2}}) ON CREATE SET n.createdAt = {now} ON CREATE SET n.`personName` = {`nonStaticDefaultn1`} SET n.updatedAt = {now} SET m:`Media`:`Visual` WITH n, m MERGE (n)-[r:`LIKES`]->(m) ON CREATE SET r.grn = {relationshipGrn};",
"parameters": {
"nonStaticDefaultn1": "John Smith",
"relationshipGrn": "grn:gg:likes:068UYqJayp7XOQb0e0IgBqS9LUtfAuDmTjdMaJZfVfR9",
"grn": "person::abc",
"grn2": "movie::123",
"now": "2021-02-08T15:50:04.534Z"
},
"metadata": null
},
{
"statement": "MERGE (n:`Movie` {`grn`: {`grn`}}) ON CREATE SET n.createdAt = {now} SET n.updatedAt = {now} SET n.`name` = {`name`} SET n:`Media`:`Visual`;",
"parameters": {
"grn": "movie::123",
"now": "2021-02-08T15:50:04.534Z",
"name": "Movie Name"
},
"metadata": null
},
{
"statement": "MERGE (n:`Movie` {`grn`: {`grn`}}) ON CREATE SET n.createdAt = {now} SET n.updatedAt = {now} SET n:`Media`:`Visual` WITH n MERGE (m:`Person` {`grn`: {grn2}}) ON CREATE SET n.createdAt = {now} SET n.updatedAt = {now} WITH n, m MERGE (n)<-[r:`IN`]-(m) ON CREATE SET r.grn = {relationshipGrn};",
"parameters": {
"relationshipGrn": "grn:gg:in:G58HQa9x6nx1S8aaUtEsF1efnMSkaO97lglvJz6CNoZP",
"nonStaticDefaultm1": "John Smith",
"grn": "movie::123",
"grn2": "person::abc",
"now": "2021-02-08T15:50:04.534Z"
},
"metadata": null
}
]
}
},
"transactionResult": {
"transactionState": {
"status": "COMPLETED"
},
"data": {
"uploadTime": "0ms",
"pollTime": null,
"writeTime": "161ms",
"ingestTime": "168ms",
"updatedAt": "2021-02-08T15:50:04.696Z",
"createdAt": "2021-02-08T15:50:04.527Z"
}
},
"xml": null
}
Graph Projection
When the sample update is sent in using the sample policy, the resulting graph projection will look like this:
The Person node would have three properties: age = 50
, personId = "person:person:abc"
, and personName = "Person Name"
. The Movie node would have two additional
labels corresponding to its altNames
in the policy: Media
and Visual
. It would also have the properties movieId = "movie:movie:123"
and name = "Movie Name"
.
More Examples
Generate one person node with a property
{
"update": {
"abc": {
"type": "PERSON",
"change":"SET",
"changes": {
"NAME": {
"change": "SET",
"value": "Person Name"
}
}
}
}
Generate one Movie node
{
"update": {
"123": {
"type": "MOVIE",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"value": "Movie Name"
}
}
}
}
}
If you create a relationship, but haven't created a node beforehand, you'll end up with a shell node that only has a primary key on it.
{
"update": {
"123": {
"type": "MOVIE",
"change": "SET",
"changes": {
"NAME": {
"change": "SET",
"value": "Movie Name"
},
"in": {
"change": "SET",
"id": "abc",
"type": "PERSON",
"direction": "INCOMING"
}
}
}
}
}
This would create a shell Person node with only a primary key value (personId = "person:person:abc"
). It is important to be aware of this possibility and make
sure every node gets its data from the source database.
To delete the IN
relationship that was created in the above sample image:
{
"update": {
"123": {
"type": "MOVIE",
"change": "SET",
"changes": {
"in": {
"change": "DELETE",
"id": "abc",
"type": "PERSON",
"direction": "INCOMING"
}
}
}
}
}
This would be the result:
And to delete a node with all of its relationships:
{
"update": {
"abc": {
"type": "PERSON",
"change":"DELETE",
}
}
This would leave only the Movie node.
If an object in the source database has a property there, one might decide to turn it into a relationship to a new node. In that case, the following policy and updated json would achieve that:
{
"name": "testingRelationshipOuPolicy",
"displayName": "Relationship Ou Policy",
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": ["MAPPED_AUTO"],
"attributes": ["MAPPED_AUTO"]
},
"ous": {
"PERSON": {
"mapping": {
"name": "Person",
"primaryKey": "personId",
"primaryKeyPrefix": "person"
},
"attributes": {
"NAME": {
"mapping": {
"name": "personName",
"type": "PROPERTY"
}
},
"likes": {
"mapping": {
"name": "LIKES",
"type": "RELATIONSHIP",
"direction": "OUTGOING",
"ou": "MOVIE"
}
}
}
},
"MOVIE": {
"mapping": {
"name": "Movie",
"primaryKey": "movieId",
"primaryKeyPrefix": "movie",
"altNames": ["Media", "Visual"]
},
"attributes": {
"in": {
"mapping": {
"name": "IN",
"type": "RELATIONSHIP",
"direction": "INCOMING"
}
}
}
}
},
"attributes": {
"NAME": {
"mapping": {
"name": "name"
}
},
"PERSON_ID": {
"mapping": {
"name": "pid"
}
},
"MOVIE_ID": {
"mapping": {
"name": "pid"
}
}
}
}
The "OU" value in the "likes" mapping tells it what kind of node to put on the other end of the relationship. The update would then look like the following:
{
"update": {
"abc": {
"type": "PERSON",
"change": "SET",
"changes": {
"likes": {
"change": "SET",
"value": "123"
}
}
}
}
}
XML Graph Projection
If the previously defined XML policy is applied to the XML update, the resulting graph would look like this:
It is possible to be much more explicit about the mapping of each XML tag. Here is a sample with a much more manual policy:
Broker Integration Strategies
Currently, AWS SQS and RabbitMQ are supported. The broker is specified in the policy json object and can be updated by using the create policy endpoint. The broker object inside the json request body defines the broker type. AWS SQS is the default if no broker object is present.
Example
{
"name": "Movie Policy 123",
"displayName": "Movie Policy",
"createdAt": null,
"clusterName": null,
"state": null,
"policy": {
"types": {
"MAPPED_AUTO": {
"definition": "Does what is mapped otherwise filling in everything else."
}
},
"ous": [
"MAPPED_AUTO"
],
"attributes": [
"MAPPED_AUTO"
],
"schema": null,
"broker": {
"type": "RABBITMQ"
}
},
"ous": {
"INGEST_PERSON": {
"mapping": {
"name": "Ingest_Person",
"altNames": null,
"primaryKey": null,
"primaryKeyPrefix": "person"
},
This broker will be used for all updates associated with this policy. By default the service tries to process update request directly (without the use of any
messaging system) and only uses the messaging system if the load exceeds a configurable threshold. Direct processing can be disabled by using the
'directProcessingEnabled=false'
flag when sending update requests.
Example: /ingest/:clusterName/update?policyName=samplePolicy&directProcessingEnabled=false