Structuring the time series workflow
Let's assume the data streams are readily available from Device Management. In other words, there are some devices connected to Device Management that provide the following, in addition to the standard LwM2M objects:
- Product ID (String): "Cheese," "Apples," "Beer," "Bread" and "Milk."
- Product count (Integer).
The same product with the same product ID may be displayed on multiple shelves, so we need to capture this in our final metric through an aggregation stage. We want to keep track of the product count for each product ID over time, so we will store these readings in a time series database. To do this, we need a custom proxy application that takes the data streams from Device Management and pushes them into the time series database. We also need to present these readings to the analysts in a meaningful way through some form of visualization. In this tutorial, we use InfluxDB for storing the time series values and Grafana for visualization.
Capturing the structure in microservices
Microservice architectures are modular applications, where subcomponents are built up as loosely coupled and reusable services. Given a standard interface, for example, the same description for an SQL database microservice can be used in just about any application. Microservice architectures also offer straightforward deployment scenarios and controllable service scaling.
Docker is one of many solutions for building microservice-based applications. In our example service, we partition the work into four microservices:
linux_client: An example client that generates data streams.
app: A proxy web application that receives data from Device Management and forwards the data to InfluxDB.
influxdb: An instance of the InfluxDB time series database.
grafana: A data visualization engine.
Microservices themselves are described using Dockerfiles, which include information, such as how to build and run a specific service and what ports are needed to connect to it. This application has two Dockerfiles: one for the proxy web application and one for a local Linux client found in
mbed-cloud-client-example/Dockerfile, respectively. Docker captures the application structure in
docker-compose.yml files, which designate which microservices are needed in an application and how to connect them.
For our example system, we use the following compose script:
version: "3" services: linux_client: build: mbed-cloud-client-example app: build: webapp ports: - "3000:8080" volumes: - ./webapp:/usr/src/app links: - influxdb - linux_client environment: - ENV=development # We can refer to the influxdb url with http://influxdb:8086 influxdb: image: influxdb:1.4-alpine expose: - "8086" grafana: image: grafana/grafana ports: - "3001:3000" links: - influxdb
For InfluxDB and Grafana, we do not need to specify Dockerfiles, as they are public on DockerHub and Docker pulls them automatically on build. The project structure is evident in the
links keyword. For example, the
app service depends on both the
influxdb service and
linux_client service. Likewise, the
grafana service pulls data from the
influxdb service. Finally, the
grafana service is visible on port 3001.
Writing a proxy sampling application based on subscriptions
For an introduction to building web applications around Device Management see integrating web applications.
# Subscribe to all devices at runtime def Initialize(): for each device in connected_devices(): if has_product_count_resource(device): add_resource_subscription(device, PRODUCT_COUNT_PATH, product_count_callback) # Push current value and product_id to the database def product_count_callback(device_id, path, current_value): TimeSeriesDatabase.push((device_id, current_value, get_product_id(device_id)))