# Re-organization of the rust code base using components

* Date: __2022-06-13__
* Status: __Approved__

## Motivations

The version 0.7 of thin-edge makes more difficult than expected one of its main goal:
that is to "give developers tools to build IIoT agents from software components provided by independent vendors".

To reach this goal, thin-edge has been designed around the idea of
software components implemented as daemon processes that interact over MQTT using JSON messages.
Two main flaws of the current implementation impair the expected freedom of extension by independent contributors :

1. Some APIs have not been exposed over MQTT but only through the file system and via command line tools
   - preventing components that run in a container or on a child device to leverage all the features of thin-edge.
2. The Rust code-base has been produced by and for a team of contributors working closely,
   - making difficult for independent teams to contribute with loose communication.   

The first point is critical but will be addressed independently.
The focus is set here on the second point.

* The aim is to __organize the rust code-base so independent contributors can implement in rust their own features with minimal conflicts__.
* Contributors who add a new feature, operation or mapper should not be forced as today
  * to either work closely with the core team to update the `tedge` command and associated daemons,
  * or to implement their features as independent daemons interacting with thin-edge over MQTT,
    gaining coordination flexibility but as the price of fewer opportunities to reduce resource consumption.
* Agent developers who package device software on top of thin-edge should not be forced as today
  * to incorporate all the features added by independent contributors in the `tedge` command and associated daemons,
  * *i.e.* to include all the cloud connectors, features and protocol extensions when only some of them are actually used.  
    
## Proposal

The code of thin-edge is organized around :
* external MQTT components that are daemon processes interacting over MQTT,
* internal Rust components that are rust crates interacting over in-memory channels,
* tools to build an external MQTT component from a cherry-picked subset of internal Rust components,
* batteries-included thin-edge daemons, built from stable internal Rust components.

For that to work, thin-edge defines:
* a Rust API that rules the interaction between internal Rust components,
* an MQTT API that rules the interaction between external MQTT components,
* conversion rules between Rust in-memory messages and MQTT messages.

The crate hierarchy of the project reflects this organization:

```
crates/
├─ core/
│  ├─ # The foundations
│  ├─ tedge_api
│  ├─ tedge_mqtt_api
│  ├─ tedge_runtime
│
├─ rust_plugins/
│  ├─ # The building blocks
│  ├─ apama_plugin
│  ├─ apt_plugin
│  ├─ c8y_conf_plugin
│  ├─ c8y_log_plugin
│  ├─ c8y_plugin
│  ├─ collectd_plugin
│  ├─ mqtt_plugin
│  ├─ sm_plugin
│  ├─ ...
│  ├─ # Added by independent contributors
│  
├─ rust_based_mqtt_components/
│  ├─ # Assemblage of Rust plugins into MQTT components
│  ├─ c8y_conf_manager
│  ├─ c8y_log_manager
│  ├─ tedge_agent
│  ├─ tedge_mapper
│  ├─ ...
│  ├─ # Added by independent contributors
│ 
mqtt_components/
│  ├─ # Non Rust MQTT components
│  ├─ ...
│  ├─ # Added by independent contributors
```

## Consequences

Benefits:
* Ease collaboration. A feature implemented independently of the core team can be integrated in a thin-edge daemon.
* Ease optimisation. Cherry-picked component assemblages give opportunities to share resources and to remove unnecessary overhead. 
* Ease packaging. A rust component can be packaged in various ways and in combination with various components.  
* Ease security. A cherry-picked assemblage of components reduce the attack surface.  
* Ease testing. An internal Rust component can be tested in isolation.
* Ease QA. A cherry-picked assemblage of components is not impacted by updates of independent vendors' components.
* Ease project governance. Only the core needs strong consensus. The rust components can follow a lighter integration process. 

Drawbacks:
* This implies a major refactoring of the `0.7` code base.
* There are risks, notably because the Rust API is still under design.
* This makes things a bit more complicated to explain with two kinds of plugins / components, internal and external.