1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
|
# Summary
The `tedge_api` crate brings centralized definitions of both messages and
behaviour by introducing traits for runtime Plugins to hook into. One goal is
to be *agnostic* on whether a given `Plugin` is part of the same executable or
not. As all messages are easily de/serializable they are also transport
agnostic, allowing implementers and users to de/construct them from/for whatever
sources/destinations they see fit.
# Motivation
Users and developers need to have a common ground to future proof communication
and growth. A common crate to both will make sure that all sides do not grow
apart and stay compatible to each other.
# Guide-level explanation
_This section is meant to be read as if the feature was_ already _implemented._
-----
thin-edge.io is an edge focused IoT framework. At its core it serves to bridge the
gap between devices and the cloud. As it is meant to run on low-resources
devices its core architecture supports that.
As such, it is built upon a message passing router with a modular approach for
extending its functionality through plugins.
thin-edge.io can thus be understood as being a common-core of a message passing router
with a collection of plugins that ultimately define what it actually _does_.
Plugins come in two forms:
- Run-time & external, which are provided to thin-edge.io and meant to
interoperate with it specifically through stdin/stdout or some other
well-specified interface (cf. external plugins further down)
- These are for example custom built executables that a user provides and
wishes to not integrate into thin-edge.io for various reasons (e.g.
language differences or license)
- Compile-time & built-in, which are compiled into the actual thin-edge.io
binary and written in Rust.
- These offer the advantage of simplifying deployment as well as assurances
w.r.t. the messages (i.e. changes)
Both ways are _functionally equivalent_ and _support the same features_.
We aim to provide plugins for all common server and use-cases, see further down
if you wish to extend thin-edge.io yourself.
## Configuring your plugins
thin-edge.io is on its own merely a collection of plugin kinds, and requires a
configuration to do any useful work. A common configuration includes a cloud
mapper, some device management plugins as well as some data sources.
> One core idea is that you can create as many plugin instances of a single
> plugin _kind_ as you wish. For example you could have multiple "File Watcher"
> instances that each watch a different file.
Here is an fictuous configuration that would connect the device to the acme cloud.
```toml
[plugins.acme]
kind = "acme_mapper"
[plugins.acme.configuration]
tenant = "coyote"
software_mgmt_plugin = "pacman" # Here we tell the acme_mapper plugin where to
# send software management requests
service_mgmt_plugin = "systemd"
[plugins.pacman]
kind = "pacman_handler"
[plugins.pacman.configuration]
allow_destructive_operations = true # Per default the plugin does not allow
# adding/removing packages
[plugins.systemd]
kind = "systemd_handler"
configuration = {
restrict_units_to = ["nginx.service"] # Only allow interacting with the nginx service
}
```
Of note here is that everying in the `plugins.<plugin id>.configuration` space
is per plugin! Each plugin exposes its own set of configurations depending on
its needs and abilities. Nonetheless some parts are probably more common:
- Per default plugins should strive to do the 'safe' thing. Ambiguitiy should
be reduced as much as possible and if defaults are unclear should either
force the user to specify it or do an idempotent safe/pure operation. (e.g.
the pacman plugin only allows listing packages per default)
- Plugins don't guess where to send their messages to. If the `acme_mapper`
receives a 'Restart "nginx" service' message it needs to be configured to
tell it where to find the destination for it.
Once you have this configuration file, you can go on and start thin-edge.io.
## Starting Thin-Edge
On startup `tedge` will:
- Check if your configuration is syntactically correct and that all requested
kinds exist
- Startup the requested plugins and process messages
In this case, the `heartbeat` service will keep sending messages every 400ms to
both `watch_sudo_calls` and `check_service_alive` with a
`MessageKind::SignalPluginState` to which they should answer with their
status, e.g. `PluginStatus::Ok`.
## Writing your own plugins
### For External Plugins
External plugins are executables that interact with the `tedge` api through a
specific compile-time plugin, like for example `StdIoExternalPlugin` which
communicates through STDIN/STDOUT.
To use it, simply choose the plugin that fits your communication style and move
to the configuration section below.
### For Compile Plugins
Compile-time plugins are written in Rust. They require two parts: A
`PluginBuilder` and a `Plugin`.
- The `PluginBuilder` creates instances of `Plugins` and verifies if the
configuration is sane.
- The `Plugin` is the object that contains the 'business logic' of your plugin
So, to get started:
1. Implement `PluginBuilder` for the struct that you will use to instantiate your plugin.
It looks like this:
```rust
/// A plugin builder for a given plugin
#[async_trait]
pub trait PluginBuilder: Sync + Send + 'static {
/// The a name for the kind of plugins this creates, this should be unique and will prevent startup otherwise
fn kind_name(&self) -> &'static str;
/// This may be called anytime to verify whether a plugin could be instantiated with the
/// passed configuration.
async fn verify_configuration(&self, config: &PluginConfiguration) -> Result<(), PluginError>;
/// Instantiate a new instance of this plugin using the given configuration
///
/// This _must not_ block
async fn instantiate(
&self,
config: PluginConfiguration,
tedge_comms: Comms,
) -> Result<Box<dyn Plugin + 'static>, PluginError>;
}
```
Things of note here:
- `kind_name` _has_ to be unique, and will be used to refer to this kind of plugin
- `verify_configuration` allows one to check if a given configuration _could even work_
- It however is not required to prove it
- `instantiate` which actually constructs your plugin and returns a new instance of it
- One argument is the `Comms` object which holds the sender part of a
channel to the tedge core, and through which messages can be sent
2. Implement `Plugin` for your plugin struct.
`Plugin` is defined as follows:
```rust
/// A functionality extension to ThinEdge
#[async_trait]
pub trait Plugin: Sync + Send {
/// The plugin can set itself up here
async fn setup(&mut self) -> Result<(), PluginError>;
/// Handle a message specific to this plugin
async fn handle_message(&self, message: Message) -> Result<(), PluginError>;
/// Gracefully handle shutdown
async fn shutdown(&mut self) -> Result<(), PluginError>;
}
```
Plugins follow a straightforward lifecycle:
- After being instantiated the plugin will have a chance to set itself up and get ready to accept messages
- It will then continuously receive messages as they come in
- If it ever needs to be shutdown, its shutdown method will give it the opportunity to do so.
See `message::Message` for possible values.
-------
At the heart of these choices lies the idea of making sure that using
thin-edge.io is precise, simple, and hard to misuse.
- In the above example, the `heartbeat` service kind would check if the targets
are actually existing plugins it could check the heartbeat on _before_ the
application itself would start!
- Similarly, the `stdio-external` plugin kind would check that the file it is
given as `path` is accessible and executable!
Users should not be mislead by too simple error messages, and in general be
helped to achieve what they need. Warnings should be emitted for potential
issues and errors for clear misconfigurations.
Similarly, _using_ the different plugins should follow the same idea, with
clear paths forwards for all errors/warnings, if possible.
To make sure that the program itself is _hard to misuse_, the configuration is
read _once_ at startup.
-------
# Reference Explanation
The core design part of ThinEdge software are messages that get passed around.
This has these benefits:
- Simple architecture as 'dumb plumbing' -> complexity is handled by the
plugins
- Clean separation of different parts, using messages codifies what can be
exchanged and makes it explicit
An example communication flow (with an arrow being a message, except 1 and 10):
```
┌─────────────┐
│ MQTT Broker │
└┬───▲────────┘
│1 │10 azure_mqtt
┌▼───┴────────────────┐
│MQTT Plugin │
│ │
│Target: azure_cloud │
│ │
│Data: Proprietary │
│ │ azure_cloud
└───────┬─────▲───────┘ ┌──────────────────────────────┐
│ │ 3 │Azure Plugin │
│ │9 ┌─────────► │
│ │ │ │Target: service_health │
2│ ┌─┴─────────────┴─┐ 4 │ │
└───► ◄───────┤Data: GetInfo "crit_service" │
│ │ ├──────────────────────────────┤
│ CORE │ 7 │ │
│ ├───────► Target: azure_mqtt │
│ │ 8 │ │
│ ◄───────┤ Data: Proprietary │
│ │ └──────────────────────────────┘
└────────▲───┬────┘
│ 5│ service_health
│ │ ┌──────────────────────┐
6│ │ │systemd Plugin │
│ └─► │
│ │Target: <sender> │
│ │ │
└─────┤Data: OK │
│ │
└──────────────────────┘
```
## Messages
Each message carries its source and destination, an id, and a payload. The
payload is _well defined_ in the `MessagePayload` enum. This means that the
amount of different message kinds is limited and well known per-version. The
message kinds are forward compatible. Meaning that new kinds of messages may be
received but simply rejected if unknown.
Messages conversations are also asynchronous, meaning that upon receiving a
message thin-edge.io might not reply. (This does not preclude the transport
layer to assure that the message has been well received)
This has two reasons:
- It makes communication easier to implement
- It reflects the network situation
Using such an interface is still clunky without some additional help.
As all messages have an ID, it could be possible to design a way of 'awaiting'
a response. This is left as a future addition.
## Plugins
_Note: 'Plugins' is simply a working name for now. We are free to rename this
to 'Extension' or any other name we think is good._
Plugins are what makes thin-edge.io work! They receive messages and execute
their specific action if applicable.
This part is split into two parts: PluginBuilders and Plugins themselves.
### PluginBuilders
A `PluginBuilder` is a Rust struct that implements `PluginBuilder`. They are the
sole sources of plugin kinds in thin-edge.io. (This is not a limitation for
proprietary and non-rust plugins, read further to see how those are handled.)
They are registered at startup and hardcoded into the thin-edge.io binary.
### Plugins
A Plugin is a Rust struct that implements `Plugin`. It is clear that requiring
all users who wish to extend thin-edge.io to learn and use Rust is a _bad idea_.
To make sure those users also get a first class experience special kinds of
plugins are bundled per default: `stdio-external`, `http-external`. (The
specific list will surely evolve over time.) These get instantiated like any
other with the exception that it is expected that all messages are forwarded,
e.g. to another binary or http endpoint. This way any user simply needs to know
what kind of messages exists and how to send/receive them, giving them the same
situation like a pure-Rust plugin would have.
# Drawbacks
- Tying the software specification to Rust code makes it more brittle to
accidental incompatibilities
- People might get confused as to how Plugins can be written -> They see "Rust"
and think they need to know the language to make their own
- Having everything in a single process group can make it harder to correctly
segment security rules for individual plugins
# Rationale and alternatives
Playing to Rust's strengths makes for better maintainable software. Using
Traits to specify behaviour and "plain old Rust objects" also make the messages
clear. As thin-edge.io is a specialized piece of software, this should be
embraced to deliver on an amazing experience.
Alternatives could include:
- Defining the objects externally, with for example CapNProto
- Defining the objects through an API like Swagger or similar
- Not doing this
# Unresolved questions
- How to extend the `MessageKind` type?
- The enum itself is `#[non_exhaustive]`, but extending it still requires a
whole developer story
- What is the process of adding new variants?
- How does the IO interface look like for external plugins?
- Which ones should exist? Just StdIO at first, HTTP maybe later?
- How to delineate between different plugin kinds in terms of messages it should be able to handle?
- For example are services always 'on system' and if one wants to restart a
container one needs to be _specific_ about the container?
- If yes, how do mappers potentially make the difference?
# Future possibilities
As all messages are routed through the application itself, it should be
possible to add other transformations to messages as they are being handled.
- Logging of what communicated with what
- Access Control
- Overriding destinations
|