Version: 2.0

GraphGrid Ingest

Platform Version 2.0

API Version 1.0

Introduction

GraphGrid Ingest gathers data from assorted sources and transports it to a graph database that can be accessed and analyzed. This means that any disconnected data is ingested into a connected state. Ingest allows users to define a policy that maps values in their source database to a property graph model and thereby represent nodes, relationships and properties in the target property graph database. Ingest also includes the capability to automatically convert these values to standard formats if no policy map is found. Ingest is a part of Graphgrid Publish.

API

This Ingest API version is 1.0 and as such all endpoints are rooted at /1.0/. For example, http://localhost/1.0/ingest (requires auth) would be the base context for this Ingest API under the GraphGrid API deployed at http://localhost. All Ingest endpoints are routed through Graphgrid Publish.

Environment

Ingest requires the following integrations:

ONgDB 1.0+

Ingest supports the following integrations:

RabbitMQ 3.5+
AWS SQS

Ingest

Get Status

Check the status of GraphGrid Ingest. Will return a 200 OK response if healthy.

BaseURL: /1.0/ingest/status
Method: GET

Request

curl --location --request GET -i "${API_BASE}/1.0/ingest/status"

Response

200 OK

Ingest Policy

The Ingest Policy provides a course of action determine transformations from a source database to the target property graph database. If a database needs to be updated, changes to the policy can be made. Changes to the policy take effect immediately.

Sample Policy

{
    "name": "Movie Policy 123",
    "displayName": "Movie Policy",
    "policy": {
        "types": {
            "MAPPED_AUTO": {
                "definition": "Does what is mapped otherwise filling in everything else."
            }
        },
        "ous": ["MAPPED_AUTO"],
        "attributes": ["MAPPED_AUTO"],
        "schema": ["SCHEMA_ONLY"]
    },
    "ous": {
        "PERSON": {
            "mapping": {
                "name": "Person",
                "primaryKey": "personId",
                "primaryKeyPrefix": "person:"
            },
            "attributes": {
                "NAME": {
                    "mapping": {
                        "name": "personName",
                        "type": "PROPERTY",
                        "defaultValue": "John Smith",
                        "fixed": false
                    }
                },
                "likes": {
                    "mapping": {
                        "name": "LIKES",
                        "type": "RELATIONSHIP",
                        "direction": "OUTGOING"
                    }
                }
            }
        },
        "MOVIE": {
            "mapping": {
                "name": "Movie",
                "primaryKey": "movieId",
                "primaryKeyPrefix": "movie:",
                "altNames": ["Media", "Visual"]
            },
            "attributes": {
                "in": {
                    "mapping": {
                        "name": "IN",
                        "type": "RELATIONSHIP",
                        "direction": "INCOMING"
                    }
                }
            }
        }
    },
    "schema": {
        "delta": {
            "constraints": [
                {
                    "cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
                }
            ],
            "indexes": [
                {
                    "cypher": "CREATE INDEX ON :Person(name);"
                }
            ]
        },
        "full": {
            "constraints": [
                {
                    "cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
                }
            ],
            "indexes": [
                {
                    "cypher": "CREATE INDEX ON :Person(name);"
                }
            ]
        }
    },
    "attributes": {
        "NAME": {
            "mapping": {
                "name": "name"
            }
        },
        "PERSON_ID": {
            "mapping": {
                "name": "pid"
            }
        },
        "MOVIE_ID": {
            "mapping": {
                "name": "pid"
            }
        }
    }
}

Components

Component	Description
name	The unique name of the policy.
displayName	The name to display for this policy.
policy	The type of policy and its details (ex. types, OUs, attributes, and schema) to implement.
`MAPPED_AUTO`	Map everything according to the rules in the policy, and use the default mapping for anything not defined.
`MAPPED_ONLY`	Map everything according to the rules in the policy and ignore anything that is not defined.
Organizational Units (OUs)	A list of organizational units, along with their label, key, and relationship mappings.
primaryKey	The name of the primary key to merge on.
primaryKeyPrefix	The prefix to use in the target database before the label and id.For example, if a node has the label Person, the primaryKeyPrefix is "per" and its given id from the source database is "12345678", the generated value for the new primary key will be `per:person:12345678`.
altNames	A list of additional labels to be applied to all nodes of the given type.
attributes	A list of global attribute key mappings.
defaultValue	If an OU has an attribute with a defaultValue, it will always be applied whenever an update is sent that contains a new object of that OU type. In this example, any Person that's added to the graph would automatically get the personName "John Smith", unless the name is specified in the update. If "fixed" (by default false) is set to true, the default value would be immutable.

Schema

A list of constraints and indexes in the database. "delta" defines the constraints and indexes to be applied or dropped when the current policy gets uploaded. "full" defines the full set of constraints and indexes within the database. If the schema type is "SCHEMA_ONLY", there must be a schema object in the policy. The schema will be updated according to the provided Geequel only. If the schema type is "SCHEMA_AUTO", there cannot be a "schema" object provided, since the schema will be defined automatically according to the primary keys provided in the list of OU objects.

SCHEMA_ONLY

The schema within the uploaded policy will also get new information in it. The state of each constraint, as well as the duration of the query to create it (in ms), will be added to the object. If there is any reason that the constraint could not be enforced, an "errors" object will be added as well.

"schema": {
  "delta": {
     "constraints": [
       {
         "cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;",
    "state": "ONLINE",
"stats": {
"duration": 8,
       }
     ]
  }
}

You can also drop constraints and indexes through the policy. Here's an example using the same policy as the previous one, but with a change in the "schema" object:

"schema":{
  "delta":{
    "constraints":[
      {
        "cypher": "DROP CONTSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
      }
    ]
  }
}

SCHEMA_AUTO

By default, the schema is automatically generated. In this case, the policy cannot contain a "schema" object. The constraints will be applied based on the primary keys provided in the OU list. Every label in the OU list will also get a uniqueness constraint on the "grn" property, regardless of whether another primary key is provided. Here is an example:

{
  "name": "something",
  "displayName": "Something",
  "createdAt": "2018-01-01T00:00Z",
  "state": "ACTIVE",
  "policy": {
  "types": {
    "MAPPED_AUTO": {
      "definition": "Does what is mapped otherwise filling in everything else."
    }
  },
  "ous": [
    "MAPPED_AUTO"
  ],
  "attributes": [
    "MAPPED_AUTO"
  ],
  "schema": [
    "SCHEMA_AUTO"
  ]
  },
  "ous": {
  "PERSON": {
      "mapping": {
          "name": "Person",
    "primaryKey": "personId",
    "primaryKeyPrefix": "person"
      },
      "attributes": {
    }
       },
  "DEPARTMENT": {
    "mapping": {
        "name": "Department"
    },
    "attributes": {
    }
  },
  "attributes": {
  }
}

Versioning

Each time a policy is uploaded with the same name and cluster as an existing one, the previous one will be added in a "versions" list at the end of the new policy for easy review. If the following policy is uploaded:

{
    "name": "something",
    "displayName": "Something",
    "createdAt": "2018-01-01T00:00Z",
    "state": "ACTIVE",
    "policy": {
        "types": {
            "MAPPED_AUTO": {
                "definition": "Does what is mapped otherwise filling in everything else."
            }
        },
        "ous": ["MAPPED_AUTO"],
        "attributes": ["MAPPED_AUTO"],
        "schema": ["SCHEMA_ONLY"]
    },
    "ous": {
        "PERSON": {
            "mapping": {
                "name": "Person"
            },
            "attributes": {
                "NAME": {
                    "mapping": {
                        "name": "personName",
                        "type": "PROPERTY"
                    }
                },
                "r1": {
                    "mapping": {
                        "name": "HAS",
                        "type": "RELATIONSHIP",
                        "direction": "INCOMING"
                    }
                }
            }
        },
        "DEPARTMENT": {
            "mapping": {
                "name": "Department"
            },
            "attributes": {
                "membersOf": {
                    "mapping": {
                        "name": "MEMBERS_OF",
                        "type": "RELATIONSHIP",
                        "direction": "INCOMING"
                    }
                }
            }
        }
    },
    "attributes": {
        "NAME": {
            "mapping": {
                "name": "name"
            }
        },
        "PERSON_ID": {
            "mapping": {
                "name": "pid"
            }
        },
        "ENTITY_ID": {
            "mapping": {
                "name": "pid"
            }
        }
    },
    "schema": {}
}

Followed by this one under the same name and in the same cluster:

{
    "name": "something",
    "displayName": "Something",
    "createdAt": "2018-01-02T00:00Z",
    "state": "ACTIVE",
    "policy": {
        "types": {
            "MAPPED_AUTO": {
                "definition": "Does what is mapped otherwise filling in everything else."
            }
        },
        "ous": ["MAPPED_AUTO"],
        "attributes": ["MAPPED_AUTO"],
        "schema": ["SCHEMA_ONLY"]
    },
    "ous": {
        "PERSON": {
            "mapping": {
                "name": "Person"
            },
            "attributes": {
                "NAME": {
                    "mapping": {
                        "name": "personName",
                        "type": "PROPERTY"
                    }
                },
                "r1": {
                    "mapping": {
                        "name": "HAS",
                        "type": "RELATIONSHIP",
                        "direction": "INCOMING"
                    }
                }
            }
        },
        "DEPARTMENT": {
            "mapping": {
                "name": "Department"
            },
            "attributes": {
                "membersOf": {
                    "mapping": {
                        "name": "MEMBERS_OF",
                        "type": "RELATIONSHIP",
                        "direction": "INCOMING"
                    }
                }
            }
        }
    },
    "attributes": {
        "NAME": {
            "mapping": {
                "name": "name"
            }
        },
        "PERSON_ID": {
            "mapping": {
                "name": "pid"
            }
        },
        "ENTITY_ID": {
            "mapping": {
                "name": "pid"
            }
        }
    },
    "schema": {}
}

The resulting policy would look like this:

{
    "name": "something",
    "displayName": "Something",
    "createdAt": "2018-01-02T00:00Z",
    "state": "ACTIVE",
    "policy": {
        "types": {
            "MAPPED_AUTO": {
                "definition": "Does what is mapped otherwise filling in everything else."
            }
        },
        "ous": ["MAPPED_AUTO"],
        "attributes": ["MAPPED_AUTO"],
        "schema": ["SCHEMA_ONLY"]
    },
    "ous": {
        "PERSON": {
            "mapping": {
                "name": "Person"
            },
            "attributes": {
                "NAME": {
                    "mapping": {
                        "name": "personName",
                        "type": "PROPERTY"
                    }
                },
                "r1": {
                    "mapping": {
                        "name": "HAS",
                        "type": "RELATIONSHIP",
                        "direction": "INCOMING"
                    }
                }
            }
        },
        "DEPARTMENT": {
            "mapping": {
                "name": "Department"
            },
            "attributes": {
                "membersOf": {
                    "mapping": {
                        "name": "MEMBERS_OF",
                        "type": "RELATIONSHIP",
                        "direction": "INCOMING"
                    }
                }
            }
        }
    },
    "attributes": {
        "NAME": {
            "mapping": {
                "name": "name"
            }
        },
        "PERSON_ID": {
            "mapping": {
                "name": "pid"
            }
        },
        "ENTITY_ID": {
            "mapping": {
                "name": "pid"
            }
        }
    },
    "schema": {},
    "versions": [
        {
            "name": "something",
            "displayName": "Something",
            "createdAt": "2018-01-01T00:00Z",
            "state": "ACTIVE",
            "policy": {
                "types": {
                    "MAPPED_AUTO": {
                        "definition": "Does what is mapped otherwise filling in everything else."
                    }
                },
                "ous": ["MAPPED_AUTO"],
                "attributes": ["MAPPED_AUTO"],
                "schema": ["SCHEMA_ONLY"]
            },
            "ous": {
                "PERSON": {
                    "mapping": {
                        "name": "Person"
                    },
                    "attributes": {
                        "NAME": {
                            "mapping": {
                                "name": "personName",
                                "type": "PROPERTY"
                            }
                        },
                        "r1": {
                            "mapping": {
                                "name": "HAS",
                                "type": "RELATIONSHIP",
                                "direction": "INCOMING"
                            }
                        }
                    }
                },
                "DEPARTMENT": {
                    "mapping": {
                        "name": "Department"
                    },
                    "attributes": {
                        "membersOf": {
                            "mapping": {
                                "name": "MEMBERS_OF",
                                "type": "RELATIONSHIP",
                                "direction": "INCOMING"
                            }
                        }
                    }
                }
            },
            "attributes": {
                "NAME": {
                    "mapping": {
                        "name": "name"
                    }
                },
                "PERSON_ID": {
                    "mapping": {
                        "name": "pid"
                    }
                },
                "ENTITY_ID": {
                    "mapping": {
                        "name": "pid"
                    }
                }
            },
            "schema": {}
        }
    ]
}

Sample XML Policy

In addition to a policy for updates in a JSON format, there is also an endpoint that allows uploading a policy for updates in an XML format. Instead of "OUs" and "attributes", an XML policy uses "elements", "attributes", and "elementAttributes". Elements can have other elements nested within them, along with their own attributes and element attributes. Other parts of the policy (mapping type, schema, versioning, etc.) behave the same way as a JSON policy. Here is a sample XML policy:

{
    "name": "something",
    "displayName": "Something",
    "createdAt": "2018-01-01T00:00Z",
    "state": "ACTIVE",
    "policy": {
        "types": {
            "MAPPED_AUTO": {
                "definition": "Does what is mapped otherwise filling in everything else."
            },
            "MAPPED_ONLY": {
                "definition": "Does only what is mapped ignoring everything else."
            }
        },
        "elements": ["MAPPED_AUTO"],
        "attributes": ["MAPPED_AUTO"],
        "elementAttributes": ["MAPPED_AUTO"]
    },
    "elements": {
        "movie": {
            "mapping": {
                "name": "Movie",
                "primaryKey": "movie-id",
                "primaryKeyPrefix": "movie"
            },
            "elements": {},
            "attributes": {
                "rating": {
                    "mapping": {
                        "name": "rating",
                        "type": "IGNORE"
                    }
                }
            },
            "elementAttributes": {
                "studio": {
                    "mapping": {
                        "name": "studio",
                        "type": "NODE_RELATIONSHIP",
                        "direction": "OUTGOING"
                    }
                }
            }
        }
    },
    "attributes": {},
    "elementAttributes": {}
}

Create Policy

Creates a named Ingest Policy for a GraphGrid Cluster.

BaseURL: /1.0/ingest/{{clusterName}}/policy/{{policyName}}
Method: POST

Parameters

Parameter	Description
clusterName string	The GraphGrid Cluster associated with this Ingest Policy.
policyName string	The name for this Ingest Policy.

note

A valid cluster name must follow the format {org}-{env}-{cluster}.

Request

curl --location --request POST "${API_BASE}/1.0/ingest/gg-test-ongdb/policy/samplePolicy" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer ${BEARER_TOKEN}" \
--data-raw '{
  "name": "Movie Policy 123",
  "displayName": "Movie Policy",
  "policy": {
   "types": {
    "MAPPED_AUTO": {
     "definition": "Does what is mapped otherwise filling in everything else."
    }
   },
   "ous": [
    "MAPPED_AUTO"
   ],
    "attributes": [
     "MAPPED_AUTO"
   ],
   "schema": [
    "SCHEMA_ONLY"
   ]
  },
  "ous": {
   "PERSON": {
    "mapping": {
     "name": "Person",
     "primaryKey": "personId",
     "primaryKeyPrefix": "person:"
    },
    "attributes": {
     "NAME": {
      "mapping": {
       "name": "personName",
       "type": "PROPERTY",
       "defaultValue": "John Smith",
       "fixed": false
      }
      },
     "likes": {
      "mapping": {
       "name": "LIKES",
       "type": "RELATIONSHIP",
       "direction": "OUTGOING"
          }
        }
      }
    },
    "MOVIE": {
      "mapping": {
        "name": "Movie",
        "primaryKey": "movieId",
        "primaryKeyPrefix": "movie:",
        "altNames": [
          "Media",
          "Visual"
        ]
      },
      "attributes": {
         "in": {
           "mapping": {
            "name": "IN",
           "type": "RELATIONSHIP",
           "direction": "INCOMING"
          }
        }
      }
    }
  },
  "schema": {
    "delta": {
      "constraints": [{
        "cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
      }],
      "indexes": [{
        "cypher": "CREATE INDEX ON :Person(name);"
      }]
    },
    "full": {
      "constraints": [{
        "cypher": "CREATE CONSTRAINT ON (person:Person) ASSERT person.personId IS UNIQUE;"
      }],
      "indexes": [{
       "cypher": "CREATE INDEX ON :Person(name);"
      }]
    }
  },
  "attributes": {
    "NAME": {
      "mapping": {
       "name": "name"
      }
    },
    "PERSON_ID": {
      "mapping": {
        "name": "pid"
      }
    },
    "MOVIE_ID": {
      "mapping": {
        "name": "pid"
      }
    }
  }
}'

Response

200 OK

Ingest Update

A database can be updated and will take effect immediately.

Sample Update

{
    "update": {
        "abc": {
            "type": "PERSON",
            "change": "SET",
            "changes": {
                "NAME": {
                    "change": "SET",
                    "value": "Person Name"
                },
                "age": {
                    "change": "SET",
                    "value": 50
                },
                "likes": {
                    "change": "SET",
                    "id": "123",
                    "type": "MOVIE"
                }
            }
        },
        "123": {
            "type": "MOVIE",
            "change": "SET",
            "changes": {
                "NAME": {
                    "change": "SET",
                    "value": "Movie Name"
                },
                "in": {
                    "change": "SET",
                    "id": "abc",
                    "type": "PERSON",
                    "direction": "INCOMING"
                }
            }
        }
    }
}

Components

Component	Description
update	A list of updates to be made to the database. Each key represents the unique id of some type of node to be modified.
type	The type of the new node. This value will be mapped to a node label, either in accordance with the policy, or according to the default convention if the policy doesn't exist or uses MAPPED_AUTO. If the policy is MAPPED_ONLY, and this type isn't mapped in it, then this block will be ignored. This holds for values mapped to property keys and relationship types as well.
change	Can be either SET (if the change is new or an update) or DELETE (if the node is to be deleted).
changes	If the type of change is SET, this is a list of changes to be made to the node. Each key is either a property key on the node, or a relationship type attached to the node.
value	If the change type is SET and it's for a property, this is the value to change the property to.
id	The unique identifier of the node on the other side of the relationship being modified or deleted.
type	he type of node on the other side of the relationship being modified or deleted.
direction	The direction of the relationship being created (default is OUTGOING).

Update

Saves an update to a GraphGrid Cluster. Optionally accepts an Ingest Policy to use during mapping, otherwise perform an automatic mapping.

BaseURL: /1.0/ingest/{{clusterName}}/update?policyName={{policyName}}
Method: POST

Parameters

Parameter	Description
clusterName string	The GraphGrid Cluster to route to for this update.
policyName string	The Ingest Policy name to be used during mapping which must correspond to an existing Ingest Policy that was created for this Cluster in Create Policy.

Request

curl --location --request POST "${API_BASE}/1.0/ingest/gg-test-ongdb/update?policyName=samplePolicy" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer ${BEARER_TOKEN}" \
--data-raw '
 {
   "update": {
     "abc": {
     "type": "PERSON",
     "change":"SET",
     "changes": {
       "NAME": {
         "change": "SET",
         "value": "Person Name"
       },
       "age": {
         "change": "SET",
         "value": 50
       },
       "likes": {
         "change": "SET",
         "id": "123",
         "type": "MOVIE"
       }
     }
   },
   "123": {
     "type": "MOVIE",
     "change": "SET",
     "changes": {
       "NAME": {
         "change": "SET",
         "value": "Movie Name"
       },
       "in": {
         "change": "SET",
         "id": "abc",
         "type": "PERSON",
    "direction": "INCOMING"
       }
     }
   }
    }
}

Response

{
    "transactionId": "BSIDAdpbK73RmcDy8wNx-1612799404527",
    "trackingId": "ZXhndy0yMDIxLTAyLTA4LTE1LTUwL21hc3Rlci9CU0lEQWRwYks3M1JtY0R5OHdOeC0xNjEyNzk5NDA0NTI3",
    "update": {
        "abc": {
            "type": "PERSON",
            "change": "SET",
            "changes": {
                "NAME": {
                    "change": "SET",
                    "id": null,
                    "type": null,
                    "value": "Person Name",
                    "direction": null,
                    "time": null,
                    "properties": null
                },
                "age": {
                    "change": "SET",
                    "id": null,
                    "type": null,
                    "value": 50,
                    "direction": null,
                    "time": null,
                    "properties": null
                },
                "likes": {
                    "change": "SET",
                    "id": "123",
                    "type": "MOVIE",
                    "value": null,
                    "direction": null,
                    "time": null,
                    "properties": null
                }
            },
            "transactionTime": null
        },
        "123": {
            "type": "MOVIE",
            "change": "SET",
            "changes": {
                "NAME": {
                    "change": "SET",
                    "id": null,
                    "type": null,
                    "value": "Movie Name",
                    "direction": null,
                    "time": null,
                    "properties": null
                },
                "in": {
                    "change": "SET",
                    "id": "abc",
                    "type": "PERSON",
                    "value": null,
                    "direction": "INCOMING",
                    "time": null,
                    "properties": null
                }
            },
            "transactionTime": null
        }
    },
    "mapping": {
        "policyName": "samplePolicy",
        "transactionRequest": null
    },
    "transactionResult": {
        "transactionState": {
            "status": "UPLOADING"
        },
        "data": {
            "uploadTime": "0ms",
            "pollTime": null,
            "writeTime": null,
            "ingestTime": null,
            "updatedAt": "2021-02-08T15:50:04.528Z",
            "createdAt": "2021-02-08T15:50:04.527Z"
        }
    },
    "xml": null
}

Update XML

Saves an update to a GraphGrid Cluster. Optionally accepts an Ingest Policy to use during mapping, otherwise perform an automatic mapping.

BaseURL: /1.0/ingest/{{clusterName}}/xmlupdate?policyName={{policyName}}
Method: POST

Parameters

Parameter	Description
clusterName string	The GraphGrid Cluster to route to for this update.
policyName string	The Ingest Policy name to be used during mapping which must correspond to an existing Ingest Policy that was created for this Cluster in Create Policy.

Request

curl --location --request POST "${API_BASE}/1.0/ingest/gg-test-ongdb/xmlupdate?policyName=XMLPolicy" \
--header 'Content-Type: application/xml' \
--header "Authorization: Bearer ${BEARER_TOKEN}" \
--data-raw '<movie movie-id="abc" rating="5.0">
   <actor>
       <person person-id="def">
           <date>19800101</date>
       </person>
   </actor>
   <format>
       <name>DVD</name>
       <year>2000</year>
   </format>
   <studio>Mountain Studios</studio>
</movie>'

Response

{
    "transactionId": "UVzhh3ZVsJTt8LJXddaV-1612815345327",
    "trackingId": "clFxZC0yMDIxLTAyLTA4LTIwLTE1L21hc3Rlci9VVnpoaDNaVnNKVHQ4TEpYZGRhVi0xNjEyODE1MzQ1MzI3",
    "update": null,
    "mapping": {
        "policyName": "XMLPolicy",
        "transactionRequest": null
    },
    "transactionResult": {
        "transactionState": {
            "status": "UPLOADING"
        },
        "data": {
            "uploadTime": "0ms",
            "pollTime": null,
            "writeTime": null,
            "ingestTime": null,
            "updatedAt": "2021-02-08T20:15:45.328Z",
            "createdAt": "2021-02-08T20:15:45.327Z"
        }
    },
    "xml": "<movie movie-id=\"abc\" rating=\"5.0\">\n   <actor>\n       <person person-id=\"def\">\n           <date>19800101</date>\n       </person>\n   </actor>\n   <format>\n       <name>DVD</name>\n       <year>2000</year>\n   </format>\n   <studio>Mountain Studios</studio>\n</movie>"
}

Get Update

Gets an update by trackingId. Can be used to check status and transaction statements made for the update.

BaseURL: /ingest/{{clusterName}}/update/{{trackingId}}
Method: GET

Parameters

Parameter	Description
clusterName string	The GraphGrid Cluster associated with this transaction update.
trackingId string	A unique ID for tracking an update which is generated in Update.

Request

curl --location --request GET "${API_BASE}/1.0/ingest/gg-test-ongdb/update/ZXhndy0yMDIxLTAyLTA4LTE1LTUwL21hc3Rlci9CU0lEQWRwYks3M1JtY0R5OHdOeC0xNjEyNzk5NDA0NTI3" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer ${BEARER_TOKEN}"

Response

{
    "transactionId": "BSIDAdpbK73RmcDy8wNx-1612799404527",
    "trackingId": "ZXhndy0yMDIxLTAyLTA4LTE1LTUwL21hc3Rlci9CU0lEQWRwYks3M1JtY0R5OHdOeC0xNjEyNzk5NDA0NTI3",
    "update": {
        "abc": {
            "type": "PERSON",
            "change": "SET",
            "changes": {
                "NAME": {
                    "change": "SET",
                    "id": null,
                    "type": null,
                    "value": "Person Name",
                    "direction": null,
                    "time": null,
                    "properties": null
                },
                "age": {
                    "change": "SET",
                    "id": null,
                    "type": null,
                    "value": 50,
                    "direction": null,
                    "time": null,
                    "properties": null
                },
                "likes": {
                    "change": "SET",
                    "id": "123",
                    "type": "MOVIE",
                    "value": null,
                    "direction": null,
                    "time": null,
                    "properties": null
                }
            },
            "transactionTime": null
        },
        "123": {
            "type": "MOVIE",
            "change": "SET",
            "changes": {
                "NAME": {
                    "change": "SET",
                    "id": null,
                    "type": null,
                    "value": "Movie Name",
                    "direction": null,
                    "time": null,
                    "properties": null
                },
                "in": {
                    "change": "SET",
                    "id": "abc",
                    "type": "PERSON",
                    "value": null,
                    "direction": "INCOMING",
                    "time": null,
                    "properties": null
                }
            },
            "transactionTime": null
        }
    },
    "mapping": {
        "policyName": "samplePolicy",
        "transactionRequest": {
            "statements": [
                {
                    "statement": "MERGE (n:`Person` {`grn`: {`grn`}})  ON CREATE SET n.createdAt = {now} ON CREATE SET n.`personName` = {`nonStaticDefaultn1`} SET n.updatedAt = {now} SET n.`personName` = {`personName`};",
                    "parameters": {
                        "personName": "Person Name",
                        "nonStaticDefaultn1": "John Smith",
                        "grn": "person::abc",
                        "now": "2021-02-08T15:50:04.533Z"
                    },
                    "metadata": null
                },
                {
                    "statement": "MERGE (n:`Person` {`grn`: {`grn`}})  ON CREATE SET n.createdAt = {now} ON CREATE SET n.`personName` = {`nonStaticDefaultn1`} SET n.updatedAt = {now} SET n.`age` = {`age`};",
                    "parameters": {
                        "nonStaticDefaultn1": "John Smith",
                        "grn": "person::abc",
                        "age": 50,
                        "now": "2021-02-08T15:50:04.533Z"
                    },
                    "metadata": null
                },
                {
                    "statement": "MERGE (n:`Person` {`grn`: {`grn`}})  ON CREATE SET n.createdAt = {now} ON CREATE SET n.`personName` = {`nonStaticDefaultn1`} SET n.updatedAt = {now} WITH n MERGE (m:`Movie` {`grn`: {grn2}})  ON CREATE SET n.createdAt = {now} ON CREATE SET n.`personName` = {`nonStaticDefaultn1`} SET n.updatedAt = {now} SET m:`Media`:`Visual` WITH n, m MERGE (n)-[r:`LIKES`]->(m) ON CREATE SET r.grn = {relationshipGrn};",
                    "parameters": {
                        "nonStaticDefaultn1": "John Smith",
                        "relationshipGrn": "grn:gg:likes:068UYqJayp7XOQb0e0IgBqS9LUtfAuDmTjdMaJZfVfR9",
                        "grn": "person::abc",
                        "grn2": "movie::123",
                        "now": "2021-02-08T15:50:04.534Z"
                    },
                    "metadata": null
                },
                {
                    "statement": "MERGE (n:`Movie` {`grn`: {`grn`}})  ON CREATE SET n.createdAt = {now} SET n.updatedAt = {now} SET n.`name` = {`name`} SET n:`Media`:`Visual`;",
                    "parameters": {
                        "grn": "movie::123",
                        "now": "2021-02-08T15:50:04.534Z",
                        "name": "Movie Name"
                    },
                    "metadata": null
                },
                {
                    "statement": "MERGE (n:`Movie` {`grn`: {`grn`}})  ON CREATE SET n.createdAt = {now} SET n.updatedAt = {now} SET n:`Media`:`Visual` WITH n MERGE (m:`Person` {`grn`: {grn2}})  ON CREATE SET n.createdAt = {now} SET n.updatedAt = {now} WITH n, m MERGE (n)<-[r:`IN`]-(m) ON CREATE SET r.grn = {relationshipGrn};",
                    "parameters": {
                        "relationshipGrn": "grn:gg:in:G58HQa9x6nx1S8aaUtEsF1efnMSkaO97lglvJz6CNoZP",
                        "nonStaticDefaultm1": "John Smith",
                        "grn": "movie::123",
                        "grn2": "person::abc",
                        "now": "2021-02-08T15:50:04.534Z"
                    },
                    "metadata": null
                }
            ]
        }
    },
    "transactionResult": {
        "transactionState": {
            "status": "COMPLETED"
        },
        "data": {
            "uploadTime": "0ms",
            "pollTime": null,
            "writeTime": "161ms",
            "ingestTime": "168ms",
            "updatedAt": "2021-02-08T15:50:04.696Z",
            "createdAt": "2021-02-08T15:50:04.527Z"
        }
    },
    "xml": null
}

Graph Projection

When the sample update is sent in using the sample policy, the resulting graph projection will look like this:

The Person node would have three properties: age = 50, personId = "person:person:abc", and personName = "Person Name". The Movie node would have two additional labels corresponding to its altNames in the policy: Media and Visual. It would also have the properties movieId = "movie:movie:123" and name = "Movie Name".

More Examples

Generate one person node with a property

{
 "update": {
   "abc": {
     "type": "PERSON",
     "change":"SET",
     "changes": {
       "NAME": {
         "change": "SET",
         "value": "Person Name"
       }
     }
  }
}

Generate one Movie node

{
    "update": {
        "123": {
            "type": "MOVIE",
            "change": "SET",
            "changes": {
                "NAME": {
                    "change": "SET",
                    "value": "Movie Name"
                }
            }
        }
    }
}

If you create a relationship, but haven't created a node beforehand, you'll end up with a shell node that only has a primary key on it.

{
    "update": {
        "123": {
            "type": "MOVIE",
            "change": "SET",
            "changes": {
                "NAME": {
                    "change": "SET",
                    "value": "Movie Name"
                },
                "in": {
                    "change": "SET",
                    "id": "abc",
                    "type": "PERSON",
                    "direction": "INCOMING"
                }
            }
        }
    }
}

This would create a shell Person node with only a primary key value (personId = "person:person:abc"). It is important to be aware of this possibility and make sure every node gets its data from the source database.

To delete the IN relationship that was created in the above sample image:

{
    "update": {
        "123": {
            "type": "MOVIE",
            "change": "SET",
            "changes": {
                "in": {
                    "change": "DELETE",
                    "id": "abc",
                    "type": "PERSON",
                    "direction": "INCOMING"
                }
            }
        }
    }
}

This would be the result:

And to delete a node with all of its relationships:

{
 "update": {
   "abc": {
     "type": "PERSON",
     "change":"DELETE",
  }
}

This would leave only the Movie node.

If an object in the source database has a property there, one might decide to turn it into a relationship to a new node. In that case, the following policy and updated json would achieve that:

{
    "name": "testingRelationshipOuPolicy",
    "displayName": "Relationship Ou Policy",
    "policy": {
        "types": {
            "MAPPED_AUTO": {
                "definition": "Does what is mapped otherwise filling in everything else."
            }
        },
        "ous": ["MAPPED_AUTO"],
        "attributes": ["MAPPED_AUTO"]
    },
    "ous": {
        "PERSON": {
            "mapping": {
                "name": "Person",
                "primaryKey": "personId",
                "primaryKeyPrefix": "person"
            },
            "attributes": {
                "NAME": {
                    "mapping": {
                        "name": "personName",
                        "type": "PROPERTY"
                    }
                },
                "likes": {
                    "mapping": {
                        "name": "LIKES",
                        "type": "RELATIONSHIP",
                        "direction": "OUTGOING",
                        "ou": "MOVIE"
                    }
                }
            }
        },
        "MOVIE": {
            "mapping": {
                "name": "Movie",
                "primaryKey": "movieId",
                "primaryKeyPrefix": "movie",
                "altNames": ["Media", "Visual"]
            },
            "attributes": {
                "in": {
                    "mapping": {
                        "name": "IN",
                        "type": "RELATIONSHIP",
                        "direction": "INCOMING"
                    }
                }
            }
        }
    },
    "attributes": {
        "NAME": {
            "mapping": {
                "name": "name"
            }
        },
        "PERSON_ID": {
            "mapping": {
                "name": "pid"
            }
        },
        "MOVIE_ID": {
            "mapping": {
                "name": "pid"
            }
        }
    }
}

The "OU" value in the "likes" mapping tells it what kind of node to put on the other end of the relationship. The update would then look like the following:

{
    "update": {
        "abc": {
            "type": "PERSON",
            "change": "SET",
            "changes": {
                "likes": {
                    "change": "SET",
                    "value": "123"
                }
            }
        }
    }
}

XML Graph Projection

If the previously defined XML policy is applied to the XML update, the resulting graph would look like this:

Screenshot

It is possible to be much more explicit about the mapping of each XML tag. Here is a sample with a much more manual policy:

Screenshot

Broker Integration Strategies

Currently, AWS SQS and RabbitMQ are supported. The broker is specified in the policy json object and can be updated by using the create policy endpoint. The broker object inside the json request body defines the broker type. AWS SQS is the default if no broker object is present.

Example

{
  "name": "Movie Policy 123",
  "displayName": "Movie Policy",
  "createdAt": null,
  "clusterName": null,
  "state": null,
  "policy": {
    "types": {
      "MAPPED_AUTO": {
        "definition": "Does what is mapped otherwise filling in everything else."
      }
    },
    "ous": [
      "MAPPED_AUTO"
    ],
    "attributes": [
      "MAPPED_AUTO"
    ],
    "schema": null,
    "broker": {
      "type": "RABBITMQ"
    }
  },
  "ous": {
    "INGEST_PERSON": {
      "mapping": {
        "name": "Ingest_Person",
        "altNames": null,
        "primaryKey": null,
        "primaryKeyPrefix": "person"
      },

This broker will be used for all updates associated with this policy. By default the service tries to process update request directly (without the use of any messaging system) and only uses the messaging system if the load exceeds a configurable threshold. Direct processing can be disabled by using the 'directProcessingEnabled=false' flag when sending update requests.
Example: /ingest/:clusterName/update?policyName=samplePolicy&directProcessingEnabled=false

Introduction​

API​

Environment​

Ingest​

Get Status​

Request​

Response​

Ingest Policy​

Sample Policy​

Components​

Schema​

SCHEMA_ONLY​

SCHEMA_AUTO​

Versioning​

Sample XML Policy​

Create Policy​

Parameters​

Request​

Response​

Ingest Update​

Sample Update​

Components​

Update​

Parameters​

Request​

Response​

Update XML​

Parameters​

Request​

Response​

Get Update​

Parameters​

Request​

Response​

Graph Projection​

More Examples​

Generate one person node with a property​

Generate one Movie node​

XML Graph Projection​

Broker Integration Strategies​

Example​

Introduction

API

Environment

Ingest

Get Status

Request

Response

Ingest Policy

Sample Policy

Components

Schema

SCHEMA_ONLY

SCHEMA_AUTO

Versioning

Sample XML Policy

Create Policy

Parameters

Request

Response

Ingest Update

Sample Update

Components

Update

Parameters

Request

Response

Update XML

Parameters

Request

Response

Get Update

Parameters

Request

Response

Graph Projection

More Examples

Generate one person node with a property

Generate one Movie node

XML Graph Projection

Broker Integration Strategies

Example