Complete examples

This section helps you to set up a stream and start sending data to STRM Privacy.

Setting up a stream

This section assumes that you have created an account on the portal.

Using the programming language examples.

In order to run these examples, you need the following:

  • An input stream to send data to (if you don’t know how, go here to learn how to create streams)

  • The credentials for this stream (presented upon stream creation). Either keep note of the returned values from the strm create stream command, or use --save flag to store them in the ~/.config/strmprivacy/Stream directory.

The following demo applications show how dummy data can be sent with a certain frequency. The data that is sent is quite static and does not result in any useful patterns for analysis, however, it does show how data can be constructed and transferred to STRM Privacy.

Currently (Aug. 2021) every example language has a different configuration file format. This is inconvenient and will be fixed. We aim to standardize this to the format created with strm create stream (stream-name) --save, so that getting up-and-running becomes easier.
  • Java

  • Python

  • NodeJS

  • PHP

java driver demo avro

This example is also available on GitHub. Please see the repository for the readme.

Short steps to start sending data:

git clone https://github.com/strmprivacy/java-examples
cd java-examples
strm create stream demo --save
f=~/.config/strmprivacy/saved-entities/Stream/demo.json
billingId=$(cat $f | jq -r '.ref.billingId')
clientId=$(cat $f | jq -r '.credentials[0].clientId')
clientSecret=$(cat $f | jq -r '.credentials[0].clientSecret')
mvn package
java -jar target/java-examples-0.0.1-SNAPSHOT-jar-with-dependencies.jar \
  $billingId $clientId $clientSecret
org.eclipse.jetty.util.log                  - Logging initialized ...
io.strmprivacy.driver.client.AuthService  - Initializing a new Auth Provider
io.strmprivacy.examples.Sender            - 204
io.strmprivacy.examples.Sender            - 204
io.strmprivacy.examples.Sender            - 204

...
Sender.java
package io.strmprivacy.examples;

import io.strmprivacy.driver.client.StrmPrivacyClient;
import io.strmprivacy.driver.serializer.SerializationType;
import io.strmprivacy.schemas.demo.v1.DemoEvent;
import io.strmprivacy.schemas.demo.v1.StrmMeta;
import org.slf4j.Logger;

import java.util.Random;
import java.util.UUID;

import static java.util.Collections.singletonList;
import static org.slf4j.LoggerFactory.getLogger;

public class Sender {

    private static final Logger LOG = getLogger(Sender.class);

    private static Random RANDOM = new Random();

    public static void main(String[] args) throws InterruptedException {
        new Sender().run(args);
    }

    /**
     * Generate a DemoEvent from a Java class that corresponds with a the strmprivacy/demo/1.0.2 schema.
     * These Java classes are generated and provided by STRM Privacy, based on the
     * serialization schema.
     * <p>
     *
     * @return a {@link io.strmprivacy.schemas.StrmPrivacyEvent}
     */
    private static DemoEvent createAvroEvent() {
        int consentLevel = RANDOM.nextInt(4);

        return DemoEvent.newBuilder()
                .setStrmMeta(StrmMeta.newBuilder()
                        .setEventContractRef("strmprivacy/example/1.2.3")
                        .setConsentLevels(singletonList(consentLevel))
                        .build())
                .setUniqueIdentifier(UUID.randomUUID().toString())
                .setSomeSensitiveValue("A value that should be encrypted")
                .setConsistentValue("a-user-session")
                .setNotSensitiveValue("Hello from Java")
                .build();
    }

    /**
     * start sending hardcoded avro events.
     *
     * @param args 3 parameters: [billingId, clientId, clientSecret]
     * @throws InterruptedException
     */
    private void run(String[] args) throws InterruptedException {
        StrmPrivacyClient client = ClientBuilder.createStrmPrivacyClient(args);

        while (true) {
            var event = createAvroEvent();

            client.send(event, SerializationType.AVRO_BINARY)
                  .whenComplete((response, exception) -> {
                      if (exception != null) {
                          LOG.error("An exception occurred while trying to send an event to STRM Privacy", exception);
                      }

                      if (response.getStatus() == 204) {
                          LOG.debug("{}", response.getStatus());
                      } else if (response.getStatus() == 400) {
                          // Try to change the value for the url field in the createAvroEvent method below to something that is not a url
                          // You can see that the STRM Privacy gateway rejects the
                          // message, stating that the field does not match the regex
                          // provided in resources/schema/avro/strm.json
                          LOG.debug("Bad request: {}", response.getContentAsString());
                      }
                  });

            Thread.sleep(500);
        }
    }
}

strmprivacy driver strmprivacy schemas demo avro

This example is also available on GitHub. Please see the repository for the readme.

Short steps to start sending data

git clone https://github.com/strmprivacy/python-examples
cd python-examples
python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install -r requirements.txt
strm create stream demo --save
f=~/.config/strmprivacy/saved-entities/Stream/demo.json
billingId=$(cat $f | jq -r '.ref.billingId')
clientId=$(cat $f | jq -r '.credentials[0].clientId')
clientSecret=$(cat $f | jq -r '.credentials[0].clientSecret')
python3 examples/sender_async.py --billing-id $billingId\
  --client-id $clientId --client-secret $clientSecret
DEBUG:strmprivacy.driver.client.auth:Initializing a new Auth Provider for SenderService
DEBUG:strmprivacy.driver.client.auth:authenticate
INFO:__main__:Event sent, response 204
INFO:__main__:Event sent, response 204
INFO:__main__:Event sent, response 204
...
sender_async.py
import asyncio
import logging
import sys
import random
import uuid

from strmprivacy.driver import SerializationType
from strmprivacy_io_strmprivacy_schemas_demo_v1.io.strmprivacy.schemas.demo.v1 import DemoEvent

from client_builder import ClientBuilder

log = logging.getLogger(__name__)
log.setLevel(logging.INFO)


class Sender(object):
    """
    An Asynchronous generator that periodically creates an event and sends it to STRM Privacy
    """

    def __init__(self):
        self._client = ClientBuilder.create_strm_privacy_client()

    def __aiter__(self):
        return self

    async def __anext__(self):
        event = create_avro_event()
        return await self._client.send(event, SerializationType.AVRO_BINARY)

    async def start_timers(self):
        await self._client.start_timers()


def create_avro_event():
    event = DemoEvent()

    event.strmMeta.eventContractRef = "strmprivacy/example/1.2.3"
    event.strmMeta.consentLevels = [random.randint(0, 3)]

    event.uniqueIdentifier = str(uuid.uuid4())
    event.someSensitiveValue = "A value that should be encrypted"
    event.consistentValue = "a-user-session"
    event.notSensitiveValue = "Hello from Python"

    return event


async def main():
    sender = Sender()
    await sender.start_timers()  # re-authorization jwt tokens

    async for response in sender:
        if response == 204:  # event correctly accepted by endpoint
            log.info(f"Event sent, response {response}")
        else:
            log.error(f"Something went wrong while trying to send event to Stream Machine, response: {response}")

        await asyncio.sleep(0.2)


if __name__ == '__main__':
    logging.basicConfig(stream=sys.stderr)
    asyncio.run(main())

nodejs driver schemas demo avro

This example is also available on GitHub. Please see the repository for the readme.

Quick steps getting started:

git clone https://github.com/strmprivacy/nodejs-examples
cd nodejs-examples
strm create stream demo --save
cat ~/.config/strmprivacy/saved-entities/Stream/demo.json | jq \
    '{billingId:.ref.billingId,
      clientId:.credentials[0].clientId,
      clientSecret:.credentials[0].clientSecret}' \
      > assets/credentials.json
npm i
npm run sender
> nodejs-driver-example@1.0.0 sender
> ts-node ./src/sender.ts

Status 204
Status 204
...
sender.ts
import { Sender } from '@strmprivacy/nodejs-driver';
import { DemoEvent } from '@strmprivacy/schemas-demo-avro';

const CONFIG = require('../assets/config.json');

// Copy the template credentials file, name it `credentials.json` and fill out the values
const CREDENTIALS = require('../assets/credentials.json');

const startSender = async () => {
  // Note: the schema id is hard coded in this example, in the config.json. This will be dynamically determined in a future version
  const sender = new Sender({
    ...CONFIG,
    ...CREDENTIALS,
  });

  // Make sure to listen for error events, otherwise Node does not handle the error events (they're escalated)
  sender.on('error', (error) => {
    console.log('Sender', error);
  });

  await sender.connect().catch((e) => {
    console.error(`Connect error ${e}`, e);
  });

  setInterval(async function () {
    try {
      const r = await sender.send(createEvent(), 'AVRO_BINARY');

      console.log(`Status ${r.status}`);
      if (r.status !== 204) {
        console.error(`An error occurred while sending event:`, r);
      }
    } catch (e) {
      console.error(`Error: ${e.message}`, e);
    }
  }, 100);
};

const createEvent = () => {
  const event = new DemoEvent();
  event.strmMeta = {
    eventContractRef: 'strmprivacy/example/1.2.3',
    consentLevels: [0],
  };

  event.uniqueIdentifier = 'string';
  event.someSensitiveValue = 'A value that should be encrypted';
  event.consistentValue = 'a-user-session';
  event.notSensitiveValue = 'Hello from NodeJS';

  return event;
};

startSender();

php driver

This example is also available on GitHub. Please see the repository for the readme.

Quick steps getting started:

git clone https://github.com/strmprivacy/php-examples
cd php-examples
strm create stream demo --save
composer install
# send one DemoEvent to STRM Privacy:
php examples/send.php <billingId> <clientId> <clientSecret>
send.php
<?php

use Examples\Events\DemoEvent;
use Examples\Utils\ClientBuilder;
use StrmPrivacy\Driver\Enums\SerializationType;
use StrmPrivacy\Driver\Sender;

include_once(realpath(dirname(__FILE__)) . '/../vendor/autoload.php');

/** @var \StrmPrivacy\Driver\Sender $sender */
$sender = ClientBuilder::build($argv, Sender::class);
// or instantiate a Sender class directly:
// $sender = new \StrmPrivacy\Driver\Sender('billingId', 'clientId', 'clientSecret');

while (true) {
    $event = new DemoEvent();

    $event->eventContractRef = 'strmprivacy/example/1.2.3';
    $event->consentLevels = [0];
    $event->uniqueIdentifier = uniqid();
    $event->someSensitiveValue = 'A value that should be encrypted';
    $event->consistentValue = 'a-user-session';
    $event->notSensitiveValue = 'Hello from PHP';

    $sender->send($event, SerializationType::AVRO_BINARY);
    sleep(0.5);
}

For PHP there are no code generation tools (yet) available for schema classes. The DemoEvent class below has been manually created. For every event schema, a similar class is needed. This class needs to implement the \StrmPrivacy\Driver\Contracts\Event contract.

DemoEvent.php
<?php

namespace Examples\Events;

use AvroSchema;
use StrmPrivacy\Driver\Contracts\Event;

class DemoEvent implements Event
{
    public $consentLevels;

    public $uniqueIdentifier;

    public $consistentValue;

    public $someSensitiveValue;

    public $notSensitiveValue;

    public $eventContractRef;

    public function getStrmSchemaRef(): string
    {
        return 'strmprivacy/demo/1.0.2';
    }

    public function toArray(): array
    {
        return [
            'strmMeta' => [
                'eventContractRef' => $this->eventContractRef,
                'nonce' => null,
                'timestamp' => null,
                'keyLink' => null,
                'billingId' => null,
                'consentLevels' => $this->consentLevels,
            ],
            'uniqueIdentifier' => $this->uniqueIdentifier,
            'consistentValue' => $this->consistentValue,
            'someSensitiveValue' => $this->someSensitiveValue,
            'notSensitiveValue' => $this->notSensitiveValue,
        ];
    }

    public function getStrmSchema(): AvroSchema
    {
        $json = file_get_contents(realpath(dirname(__FILE__)) . '/../../assets/schemas/demo.avsc');

        return AvroSchema::parse($json);
    }
}

Receiving data

See strm listen web-socket for a debugging view on the events.

See exporting to Kafka or batch exporters for production event consuming.