GraphGrid Config

Platform Version 2.0

API Version 1.0

Introduction

GraphGrid Config provides a centralized server for managing the configuration for GraphGrid services. Spring Cloud Config is utilized to streamline external properties, providing a centralized place to manage configuration data across all environments. For clients working in GraphGrid Cloud deployments, values are sourced from AWS param store. Packaged environments utilize Redis for property storage. Configuration properties are stored in a hierarchy based on platform version, application, and environment.

The property paths are generally: /{branch_name}/{application_name}/{environment}/{property_name} with default path structures like:/2.0/application/default/{property_name}.

For example, parameters default across an application like GraphGrid Search under /2.0/search/default/{property_name} across all GraphGrid services. However, default parameters are overwritten if environment profiles are enabled. For instance, parameters would configure to only the local profile as /2.0/application/local/{property_name}. Configuration would be specific to GraphGrid Search running for the GraphGrid Org gg in the dev environment for example as /2.0/search/gg-dev/{property_name}. The customer environment is built using {graphgrid_org_name}-{gg-dev}. For a packaged environment, the environment profile is docker.

Environment

Config requires the following integrations:

• ONgDB 1.0+

API

This Config API version is 1.0 and as such all endpoints are rooted at /1.0/config/. For example, http://localhost/1.0/ingest (requires auth) would be the base context for this Config API under the GraphGrid API deployed at http://localhost.

The API uses basic HTTP authentication with a username and password that is stored/retrieved from param-store under spring.security.user.name and spring.security.user.password for that config environment. Bearer authentication is also supported.

Get Status

Check the status of GraphGrid Config. Will return a 200 OK response if healthy.

Base URL: /1.0/config/status
Method: GET

Request

curl --location --request GET "${API_BASE}/1.0/config/status"

Response

{
    "status": "OK"
}

Get Config

Get Spring Cloud config for a given application and environment. Please note that {branch_name} is optional.

Base URL: /1.0/config/data/{application_name}/{environment}/{branch_name}

Method: GET

Additional Spring Cloud Config endpoints exist to provide data in alternate formats. Please see the Spring Cloud Config docs for more information.

curl http://graphgrid:graphgrid@localhost/1.0/config/data/application/default/2.0

Set Config Property

Create or update a configuration property.

Base URL: /1.0/config/updateConfigValue/{application_name}/{environment}/{branch_name}

Method: POST

note

{branch_name} is required for this endpoint.

Parameter	Description
key string	Config property key.
value string	Config property value.

curl --location --request POST "${API_BASE}/1.0/config/updateConfigValue/nlp/default/2.0" \
--header "Authorization: Bearer ${BEARER_TOKEN}" \
--header "Content-Type: application/json" \
--data-raw '{
    "key": "spring.nlp.sentiment.model",
    "value": "sentiment-model"
}'

Packaged Environment

When running GraphGrid as a self-contained package, Redis is used for the configuration data store. When the Redis service starts up, the contents of data/redis/setup/gg_package_redis_config.txt is ingested by Redis. Values in that file can be changed, though both the config and Redis containers should be restarted for changes to take effect.

Upon startup, the config service writes out a properties file to data/graphgrid/config/credentials/configCredentials.properties. This file contains the base URL of the config server, as well as a username and password pair to authenticate with.

Using Redis as the Storage Backend

Config uses AWS Param Store as the backend by default. To use Redis as the backend, such as in a packaged environment, follow these steps:

1. Add SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_TYPE=customredis to the config service's environment variables in the docker-compose.yml.

This tells Config to query Redis for the config values to return.

config:
  environment:
    - SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_TYPE=customredis

2. Add AWS_PARAMSTORE_ENABLED=false to all GraphGrid services' (config, fuze, search, etc.) environment variables. This tells Config to not query AWS Param Store for the config values to return. This also prevents param store autoconfiguration during configuration bootstrapping for all projects. For Example:

fuze:
  environment:
    - AWS_PARAMSTORE_ENABLED=false

3. Add a volume mount to Config's /config directory.

config:
  volumes:
    - ./local/path/config/credentials:/config

Config will write a file to this volume containing the Config server's authentication details.

4. Mount that same volume as read-only to the other GraphGrid services. For example:

fuze:
  volumes:
    - ./local/path/config/credentials:/config:ro

The services will read the file that Config wrote to know how to connect to it.

5. Ensure the redis service exists in docker-compose.yml

redis:
  image: 754290812573.dkr.ecr.us-west-2.amazonaws.com/redis:5.0.9
  command: bash -c "redis-server /setup/redis.conf --daemonize yes && sleep 1 && cat /setup/graphgrid_default_config.txt /setup/gg_package_redis_default_policies.txt | redis-cli --pipe  && tail -f /dev/null"
  networks:
    - default
  ports:
    - 6379:6379
  volumes:
    - ./data/redis/setup:/setup
    - ./data/redis/data:/redis/data

The command field here tells Redis to run the commands in /setup/graphgrid_default_config.txt to load them into the database. Note that there is a volume mounted to /setup, where is where graphgrid_default_config.txt comes from. Additionally, /redis/redis.conf tells Redis where to find the configuration file, containing settings for features such as cache eviction and persistence. sleep 1 pauses the sequence of commands for a second. When persistence is enabled, this allows Redis to finish loading its data into memory before running the setup commands.

The contents of graphgrid_default_config.txt should look like this:

HSETNX /2.0/application/default spring.rabbitmq.host "localhost"
HSETNX /2.0/application/default spring.rabbitmq.port "5672"


This file is a list of HSETNX commands, which is how the values get loaded into Redis. HSETNX will only set values in Redis if the key does not exist. If you'd like certain keys to be set every time the package starts up, you can accomplish this with HSET instead. redis.conf contains the configuration for Redis. By default, appendonly mode is enabled to provide persistence. Changes to the configuration can be made through Redis or by using the config API.

Some GraphGrid services, like Fuze, can also use Redis. Fuze's Redis keys are set with a time to live (TTL) expiry value, so that Redis can use a volatile-lru eviction policy. This helps keep memory consumption low, and keeps Redis stable. The volatile-lru eviction policy will evict less recently used keys that have an expiry value set, so other keys, like those stored for Config, will be safe.

The TTL value for key expiry is stored in the graphgrid.replication.key.timeout config property under the Fuze application. Values should be formatted using d to represent days, h for hours, and s for seconds. The default is 30 days (30d).

Like with Param Store, there is the /{branch_name}/{application_name}/{environment} hierarchy./2.0/application/default is the default. A value specific to the development environment for the nlp project would go under /2.0/nlp/development. Following that is the parameter key/name (spring.rabbitmq.host). Finally, the parameter value (localhost).

Using AWS as the Storage Backend

When running GraphGrid in a cloud environment, Config uses AWS as the backend storage by default. AWS supports hierarchical property names with path segments and methods to retrieve all properties under a given path, making AWS suitable for Spring Cloud integration. Authentication is pulled from the AWS Param Store along with a /{branch_name}/{application_name}/{environment} hierarchy.

AWS Param Store

At the time of writing this documentation, AWS CLI version 2 is the most recent major version. You can install the latest version here. For more topics and resources about using the AWS CLI, visit this reference.

Below are some basic commands for working with AWS Param store to manage configuration.

1. Get all parameters for a project for a specific service profile.

aws ssm get-parameters-by-path --path /2.0/{application_name}/{environment} --recursive --output text --query 'Parameters[*].[Name, Value]'


AWS CLI reference get-parameters-by-path

2. Get information about a parameter by using the parameter name.

aws ssm get-parameter --name /2.0/{application_name}/{environment}/{graphgrid.property}


AWS CLI reference get-parameter

3. Set a parameter

`aws ssm put-parameter --name /2.0/{application_name}/{environment}/{graphgrid.property} --value "Sample_Value" --type String --overwrite`

Set URL as param

aws ssm put-parameter --cli-input-json '{
  "Name": "/2.0/ongdb/test/spring.services.files.url",
  "Value": "http://localhost/1.0/file",
  "Type": "String"
}'

AWS CLI Reference put-parameter

To set parameters containing a URL from the AWS CLI you need to add clifollow_urlparam = false to your AWS config file (~/.aws/config) under your AWS profile. This will prevent AWS CLI from trying to resolve the URL and pass the result instead of setting the URL as the value. _Note that if your get-parameter output seems truncated (only 10 results), make sure you update to the latest AWS CLI version.