Sending and receiving manually
This quickstart aims to clarify how to send and receive data to STRM Privacy with the Avro serialization wire format, without any STRM Privacy specific code.
STRM Privacy uses standard open-source formats (like Avro and JSON-schema), authentication systems (OAuth2.0) and transport layers (HTTP/2 and websocket and gRPC).
This quickstart uses both cURL and httpie, which often is easier to use.
First create a stream.
$ strm create stream by-hand --save
{
"ref": {
"name": "by-hand",
"projectId": "30fcd008-9696-...."
},
"enabled": true,
"limits": {
"eventRate": "999999",
"eventCount": "999999999"
},
"credentials": [
{
"clientId": "stream-w0qu00hwl644b...",
"clientSecret": "OygfdpwBqoekL..."
"projectId": "30fcd008-9696-...."
}
],
"masked_fields" : {}
}
Use strm listen web-socket by-hand to get some feedback on the events you're sending.
Request an OAuth 2.0 access token with
the Client Credentials Grant
that you need for sending events to a certain stream. In the code block below, the user (-u
) is configured as
clientId:clientSecret
. This translates into an Authorization: Basic <base64 encoded username + password>
header.
accessToken=$(curl -X POST -u "stream-w0qu00hwl644b...:OygfdpwBqoekL..." \
-d "grant_type=client_credentials" \
"https://accounts.strmprivacy.io/auth/realms/streams/protocol/openid-connect/token" | jq -r .access_token)
You can inspect the resulting accessToken
in jwt.io to
see what is stored inside the claims.
With help of this tool, it's possible to easily generate some random data for the clickstream demo schema.
This is the JSON serialization format of Avro. The client drivers use the much faster and more compact Avro binary format.
To use the random data and send it to the /event
endpoint:
cat demo.json | http post https://events.strmprivacy.io/event\
authorization:"Bearer $accessToken" \
Strm-Schema-Ref:strmprivacy/clickstream/1.0.0
HTTP/1.1 400 Bad Request
Field: 'url' in event with schema: 'strmprivacy/clickstream/1.0.0' with value: 'url'
doesn't match regex: '^(https?|ftp|file)://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]'
This is as expected. STRM Privacy gives an indication that a validation failed. This is an example of the mechanism that STRM Privacy provides to indicate to the data producers that their data does not conform to the rules of the Data Contract.
Modify the url
field in demo.json
to become any valid url (like
https://strmprivacy.io
, and try to send it again:
cat demo.json | http post https://events.strmprivacy.io/event\
authorization:"Bearer $accessToken" \
Strm-Schema-Ref:strmprivacy/clickstream/1.0.0
HTTP/1.1 204 No Content
The HTTP status code 204
is returned by the Event Gateway when the event
has been accepted and processed.
cURL instead of httpie
When using curl
instead of httpie
it is possible to observe the
HTTP/2 response, indicating the use of HTTPS/2 with its much higher
throughput.
curl -v https://events.strmprivacy.io/event \
-H "authorization: Bearer $accessToken" \
-H "Strm-Schema-Ref:strmprivacy/clickstream/1.0.0" --data-binary @demo.json
...
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
...
> POST /event HTTP/2
> Host: events.strmprivacy.io
...
> Strm-Schema-Ref:strmprivacy/clickstream/1.0.0
> content-length: 434
>
...
< HTTP/2 204
Decrypting data
First, create a decrypted stream:
strm create stream --derived-from by-hand --levels 2 --save
{
"ref": { "name": "by-hand-2", "projectId": "30fcd008-9696-...." },
"consentLevels": [ 2 ],
"consentLevelType": "CUMULATIVE",
"enabled": true,
"linkedStream": "by-hand",
"credentials": [
{ "clientId": "hx7hj7w5mwkbybdrwhale1hvq0r6qk",
"clientSecret": "#Fs4DkVtJh(2uss#062hGuzTLW9u5t",
"projectId": "30fcd008-9696-...."
}
]
}
Send an event as described above with cURL or httpie. Observe the decrypted attributes in the events received from the web-socket.
$ strm listen web-socket by-hand-2
{
"strmMeta": {
"eventContractRef": "strmprivacy/example/1.3.0",
"nonce": 15082564,
"timestamp": 1629192833072,
"keyLink": "55c2f72b-cff8-4814-ae33-e125c77e50f9",
"consentLevels": [ 0, 1, 2, 3 ]
},
"uniqueIdentifier": "unique-14",
"consistentValue": "session-740",
"someSensitiveValue": "ASB9bJrnYjxjNF5Txc+Wc2k1zvzFAmE03SYK499WK5Du",
"notSensitiveValue": "not-sensitive-39"
}
That most fields are decrypted, but the someSensitiveValue
field
not, is because of consent level 3 (see the data contract). If
the event had not contained 2
in its consent levels, we wouldn't even
have seen the event in this decrypted stream. Read more on field
decryption here.
And finally, to clean up the resources:
strm delete stream by-hand --recursive
# note that everything that has been deleted is returned from this call.
Receiving from the websocket without the strm cli.
If you want to retrieve json serialized events without using the strm listen web-socket tool, follow these steps.