HTTP/REST

Overview

The HTTP/REST Module is suitable for either of the following methods:

  • manual polling of endpoints (read)

  • continuous polling of endpoints (subscribe)

  • publishing data to endpoints (write)

The HTTP server to be contacted is described by the Cybus::Connection resource. This resource defines hostname, scheme, port, but also additional headers or authentication info. Each REST endpoint is described by a Cybus::Endpoint resource (no pun intended) that defines access to a particular REST path, optionally specifying additional query parameters or headers for this REST path.

The currently supported authentication methods are

  • Username/password for a basicAuth scheme

  • Arbitrary header for e.g. literal Bearer Tokens

  • Oauth 2.0 Client Credentials Grant

Commissioning File Specifics

The Endpoints for the HTTP connections may either use subscribe for subscribing, read for event-driven polling, or write for publishing, as described above.

Connection Properties

scheme (string, enum)

Transport scheme to be used

This element must be one of the following enum values:

  • http

  • https

Example: "https"

host (string, required)

Domain name or IP Address of the server to connect to

Examples: "126.4.10.52", "example.org"

port (integer, required)

HTTP Port to use, usually 80 or 443

Default: 443

Example: 15200

prefix (string)

Path prefix that is applied to all endpoints

Default: ""

Example: "/api/v1"

headers (object)

HTTP Headers that are added to all requests

Default:

{}

Example: {"Content-type":"application/json"}

query (object)

Query variables that are added to all requests

Example: {"foo":"bar"}

auth (object)

Credentials for Basic Auth

Properties of the auth object:

username (string)

password (string)

oauthClientCredentials (object)

Oauth Client Credentials flow configuration

Properties of the oauthClientCredentials object:

auth_url (string, required)

client_id (string, required)

client_secret (string, required)

grant_type (string)

Default: "client_credentials"

audience (string)

probeInterval (integer)

Interval in milliseconds to check if connection is still there

Default: 10000

probeTimeout (integer)

Timeout in milliseconds after which the probe request gets abort. (Defaults to 0, because of backwards compatibility - no timeout)

Default: 0

Examples: 0, 1000, 3000

probePath (string)

The path that should be used to check the health of the server

Default: "/"

probeMethod (string, enum)

The HTTP method to use for contacting the configured Probe URL

This element must be one of the following enum values:

  • GET

  • HEAD

Default: "HEAD"

mutualAuthentication (boolean)

If true, a full certificate chain including client certificate is expected to connect properly with validated certificates.

Default: false

Examples: true, false

trustAllCertificates (boolean)

If true, all server certificates will be accepted, regardless of whether they can be validated or not. Use this option if self-signed server certificates should be accepted, or if there are other reasons which prevent this client to validate the certificates.

Default: false

Examples: true, false

caCert (string)

The root CA certificate as Base64 encoded PEM file content

clientCert (string)

The client certificate as Base64 encoded PEM CRT file content

clientPrivateKey (string)

The client private key as Base64 encoded PEM CRT file content

connectionStrategy (object)

If a connection attempt fails, retries will be performed with increasing delay (waiting time) in between. The following parameters control how these delays behave.

Properties of the connectionStrategy object:

initialDelay (integer)

Delay (waiting time) of the first connection retry (in milliseconds). For subsequent retries, the delay will be increased according to the parameter incrementFactor which has a default value of 2.

Default: 1000

Additional restrictions:

  • Minimum: 1000

maxDelay (integer)

Maximum delay (waiting time) to wait until the next retry (in milliseconds). The delay (waiting time) for any subsequent connection retry will not be larger than this value. Must be strictly greater than initialDelay.

Default: 30000

incrementFactor (integer)

The factor used to increment initialDelay up to maxDelay. For example if initialDelay is set to 1000 and maxDelay to 5000 the values for the delay would be 1000, 2000, 4000, 5000.

Default: 2

Additional restrictions:

  • Minimum: 2

Endpoint Properties

path (string, required)

API path to call

Example: "/some/api"

interval (integer)

Polling interval in ms for subscriptions

Default: 5000

cronExpression (string)

The Cron expression used to poll the endpoint. (For examples, see: https://github.com/node-cron/node-cron)

Examples: "1,2,4,5 * * * *", "1-5 * * * *", "*/2 * * * *", "* * * January,September Sunday"

query (object)

Optional query parameters to be added to the request

headers (object)

Optional headers to be added to the request

method (string)

The method used for write endpoints one of: patch, post, put

Default: "post"

Subscribing Data from a REST Server

When subscribing, the REST Server is polled at regular intervals according to the interval property of the endpoint resource. Each endpoint must contain a path and may contain additional header and query parameters.

If the request fails, a warning is logged but no action is taken.

Polling is implemented with a best-effort strategy, which means polling will take place at best with the desired interval. If the REST server response takes longer than the polling interval, the next request will not be triggered before the previous has been completed. This prevents building up of infinite queues in the event of unfortunate response time / polling interval combinations. However, this serialization is implemented per endpoint (per REST path), so that multiple requests to different paths can actually be in-flight in parallel.

Response Message

Similar to most protocol implementations in the Cybus Connectware, the server response will be wrapped in a JSON structure that follow the convention timestamp/value, like so:

{
   "timestamp": 12391238123, // ms since epoch
   "value": ...              // server response
}

If the server responds with a Content-type=application/json, the response JSON will be embedded to this object as value of the "value" property, like so:

// Original server response
{
  "machineState": "okay",
  "temperature": 23,
  "logs": [
    "message1",
    "message2"
  ]
}

// Resulting protocol module output at MQTT broker
{
   "timestamp": 12391238123, // ms since epoch
   "value": {
      "machineState": "okay",
      "temperature": 23,
      "logs": [
        "message1",
        "message2"
      ]
    }
}

If the server response is not parseable as JSON, it will be embedded into the message object as a string, hence

// Original server response
<?xml><root><some interesting="other">protocol</some></root>

// Resulting protocol module output to Cybus broker
{
   "timestamp": 12391238123, // ms since epoch
   "value": "<?xml><root><some interesting="other">protocol</some></root>"
}

Reading Data from a REST Server

Data can also be read from the server on certain events. Reading data from the server is triggered by sending a message to the /req topic of the endpoint (see also Operation results). The response from the server is written to the /res topic of the endpoint.

The message sent on the endpoint’s /req topic does not need to have any content to trigger reading of data.

Dynamic path

If the REST endpoint to be called requires a dynamic (changing) path, the configured path property can be overwritten by the message to the endpoint. This is done by providing a path property string in the message payload.

Example message with dynamic path overwriting (to be sent on the /req topic):

{
  "path": "/bar"
}

# Configured path: /foo
# Actual path    : /bar

It is also possible to append some path suffix to the configured path property by the message to the endpoint. Appending requires providing a pathAppend property string in the message payload.

Example message with dynamic path appended using pathAppend (to be sent on the /req topic):

{
  "pathAppend": "/bar"
}

# Configured path: /foo
# Actual path    : /foo/bar

Dynamic query parameters

If the REST endpoint to be called requires dynamic query parameters, these can be provided in the message payload as key-value pairs.

Example message with dynamic query parameters (to be sent on the /req topic):

{
  "query": {
    "foo": "bar",
    "user": "cybus"
  }
}

If the endpoint’s configuration already has a query property, both provided query parameters are merged. If any parameters are duplicated, the dynamic parameters from the message payload override the ones from the static configuration.

Publishing Data to a REST Server

The HTTP/REST Protocol Mapper also supports the other direction of data flow, namely publishing (pushing) data from MQTT to a REST server using the write property in the Cybus::Endpoint resource. Publishing does not work on a regular time interval but will directly forward any new message that is received from the broker on the /set topic of the endpoint.

The data of the internal MQTT message must be of type object and has to contain a property called body, which will be forwarded as the body of the HTTP request.

{
  "body": {
    "foo": "bar"
  }
}

The request mime type header of the outgoing HTTP request will be set according to the internal MQTT message value: If the message contains parseable JSON, the mime type of the HTTP request will be set to application/json, otherwise it will be set to application/octet-stream. Besides that, no further changes are performed on the request. Any additional headers as defined in the endpoint resource will be passed on as given.

The result of the HTTP request is returned on the /res topic of the endpoint, as explained here: Operation results. The data message on the result topic will have the following format:

{
  "id": 29194,
  "timestamp": 1629351968526,
  "result": {
    "value":0
  }
}

In this result message, id is the request identifier that was sent on the original request, timestamp is the Unix timestamp of when the result was received, and result is the JSON object with the actual result, but its content depends on the concrete protocol implementation.

If there was an error, the resulting JSON object does not contain a property result but instead a property error. This property is simply a string variable and contains an explanatory message of the error. Hence, in the error case the data message on the result topic will have the following format:

{
  "id": 29194,
  "timestamp":1629351968526,
  "error": "POST request on /foo/bar failed with status: 403 Forbidden"
}

Dynamic path

If the REST endpoint to be called requires a dynamic (changing) path, the configured path property can be overwritten by the message to the endpoint. This is done by providing a path property string in the message payload.

Example message with dynamic path overwriting (to be sent on the /set topic):

{
  "path": "/bar",
  "body": {
    "baz": "qux"
  }
}

# Configured path: /foo
# Actual path    : /bar

It is also possible to append some path suffix to the configured path property by the message to the endpoint. Appending requires providing a pathAppend property string in the message payload.

Example message with dynamic path appended using pathAppend (to be sent on the /set topic):

{
  "pathAppend": "/bar",
  "body": {
    "baz": "qux"
  }
}

# Configured path: /foo
# Actual path    : /foo/bar

Encoded request body

If it is required to send binary data as the request body, it is possible to specify the encoding of the message payload body parameter using a flag called bufferFromBody. The contend of the body parameter will then be converted into a buffer, respecting the encoding. Allowed encodings can be found here https://nodejs.org/api/buffer.html#buffers-and-character-encodings.

{
  "bufferFromBody": "base64",
  "body": "YmFy"
}

Connection Probing

While HTTP by itself doesn’t require a continuous connection, the HTTP protocol implementation in the Connectware is monitoring the connection state for up-to-date connectivity information. The connection state is estimated by monitoring each successful read and write, and additionally by a probing function scheduled to run if no read or write operation is in progress. The additional probing function is run after the time interval from the property probeInterval (default: 10 seconds).

By default the probing is performed by sending an OPTIONS request against the endpoint server. If this request fails with network error (e.g. ECONNRESET, ETIMEOUT, EUNREACHABLE), the connection state is assumed to be disconnected. If it succeeds, or if it at least gets a HTTP response from the endpoint, the connection is assumed to be connected.

Additionally the properties probePath and probeMethod allow configuring a custom HTTP path and HTTP method to be used by the probing function. See the section Connection Properties for more details on these properties.

Oauth 2.0 Client Credentials Grant

This module supports the Oauth 2.0 Client Credentials Grant flow for authenticating requests (see rfc6749 Section-4.4).

Tokens are automatically refreshed when they expire.

To use this authentication method the property oauthClientCredentials needs to be configured in the connection object providing client_id, client_secret, auth_url and, optionally, as some Oauth 2.0 implementations do not require it, audience and grant_type (which defaults to client_credentials).

Example configuration:

httpConnection:
  type: Cybus::Connection
  properties:
    protocol: Http
    targetState: connected
    connection:
      scheme: https
      host: one.host.io
      oauthClientCredentials:
        client_id: your_client_id
        client_secret: your_client_secret
        auth_url: oauth2.token.server

Sample Commissioning File

Download: http-example.yml

  1---
  2# ----------------------------------------------------------------------------#
  3# Commissioning File
  4# ----------------------------------------------------------------------------#
  5# Copyright: Cybus GmbH (2020)
  6# Contact: support@cybus.io
  7# ----------------------------------------------------------------------------#
  8# Source Interface Definition - HTTP/REST endpoint
  9# ----------------------------------------------------------------------------#
 10
 11description: >
 12  Sample commissioning file for communicating with HTTP/REST endpoint
 13
 14metadata:
 15  name: Http Connectivity
 16  icon: https://www.cybus.io/wp-content/uploads/2017/10/for-whom1.svg
 17  provider: cybus
 18  homepage: https://www.cybus.io
 19  version: 0.0.1
 20
 21parameters:
 22
 23  httpHost:
 24    description: IP address of the HTTP Server
 25    type: string
 26    default: 192.168.0.1
 27
 28  httpPort:
 29    description: HTTP port of the Server
 30    type: number
 31    default: 80
 32
 33  initialReconnectDelay:
 34    type: integer
 35    default: 1000
 36
 37  maxReconnectDelay:
 38    type: integer
 39    default: 30000
 40
 41  factorReconnectDelay:
 42    type: integer
 43    default: 2
 44
 45resources:
 46
 47  httpConnection:
 48    type: Cybus::Connection
 49    properties:
 50      protocol: Http
 51      connection:
 52        scheme: https
 53        host: !ref httpHost
 54        port: !ref httpPort
 55        connectionStrategy:
 56          initialDelay: !ref initialReconnectDelay
 57          maxDelay: !ref maxReconnectDelay
 58          incrementFactor: !ref factorReconnectDelay
 59
 60  base:
 61    type: Cybus::Endpoint
 62    properties:
 63      protocol: Http
 64      connection: !ref httpConnection
 65      subscribe:
 66        path: /
 67
 68  counter:
 69    type: Cybus::Endpoint
 70    properties:
 71      protocol: Http
 72      connection: !ref httpConnection
 73      subscribe:
 74        path: /counter
 75
 76  getPost:
 77    type: Cybus::Endpoint
 78    properties:
 79      protocol: Http
 80      connection: !ref httpConnection
 81      subscribe:
 82        path: /post
 83
 84  writePost:
 85    type: Cybus::Endpoint
 86    properties:
 87      protocol: Http
 88      connection: !ref httpConnection
 89      write:
 90        path: /post
 91
 92  httpMapping:
 93    type: Cybus::Mapping
 94    properties:
 95      mappings:
 96        - subscribe:
 97            endpoint: !ref base
 98          publish:
 99            topic: !sub '${Cybus::MqttRoot}/server/base'
100        - subscribe:
101            endpoint: !ref counter
102          publish:
103            topic: !sub '${Cybus::MqttRoot}/server/counter'
104        - subscribe:
105            endpoint: !ref getPost
106          publish:
107            topic: !sub '${Cybus::MqttRoot}/server/post'
108        - subscribe:
109            endpoint: !ref counter
110          publish:
111            endpoint: !ref writePost