Updated dist files

Better to use screen width wisely and avoid unnecessary padding.

.gitignore

@@ -22,3 +22,5 @@ platypush/requests
 .coverage
 coverage.xml
 Session.vim
+/jsconfig.json
+/package.json

.pre-commit-config.yaml

@@ -11,6 +11,7 @@ repos:
     - id: check-xml
     - id: check-symlinks
     - id: check-added-large-files
+      args: ['--maxkb=1500']
 
 -   repo: https://github.com/Lucas-C/pre-commit-hooks-nodejs
     rev: v1.1.2

CHANGELOG.md

@@ -1,10 +1,133 @@
 # Changelog
 
 All notable changes to this project will be documented in this file.
-Given the high speed of development in the first phase, changes are being reported only starting from v0.20.2.
+Given the high speed of development in the first phase, changes are being
+reported only starting from v0.20.2.
 
 ## [Unreleased]
 
+### Added
+
+- Migrated many integrations to the new [entities
+  framework](https://git.platypush.tech/platypush/platypush/pulls/230).
+  This is a very large change to the foundations of the platform, and there's
+  still a lot of work in progress. A detailed description of all the changes
+  will follow shortly.
+
+### Fixed
+
+- Migrated the `clipboard` integration from `pyperclip` to `pyclip` (see
+  [#240](https://git.platypush.tech/platypush/platypush/issues/240)).
+  `pyperclip` is unmaintained and largely broken, and `pyclip` seems to be a
+  viable drop-in alternative.
+
+## [0.24.5] - 2023-02-22
+
+### Added
+
+- Added `hid` plugin to support discoverability and data interaction with
+  generic HID devices - like Bluetooth/USB peripherals, joysticks, dongles and
+  any other type of devices that supports the HID interface.
+  
+- Added `timeout` parameter to `websocket.send` to prevent messages sent on a
+  non-responsive websocket from getting the websocket loop stuck
+
+### Fixed
+
+- Running the Zeroconf registration logic in another thread in `backend.http`,
+  so failures in the Zeroconf logic don't affect the startup of the web server.
+  
+- (Temporarily) introduced `sqlalchemy < 2.0.0` as a requirement - a PR with a
+  migration to the new stable version of SQLAlchemy is in TODO.
+
+## [0.24.4] - 2022-12-20
+
+### Fixed
+
+- Fixed cronjobs potentially being triggered even if it wasn't their slot in
+  case of clock synchronization events.
+
+## [0.24.3] - 2022-12-17
+
+### Added
+
+- Added `[-v|--verbose]` command-line option to override the default logging
+  configuration and enable debug logging.
+- Added `--version` command-line option to print the current version and exit.
+- [[#236](https://git.platypush.tech/platypush/platypush/issues/236)] Added
+  support for `author` and `tags` attributes on feed entries.
+
+## [0.24.2] - 2022-12-10
+
+### Fixed
+
+- The `main.db` configuration should use the configured `workdir` when no
+  values are specified.
+
+- The `zwave.mqtt` is now compatible both with older (i.e. `zwavejs2mqtt`) and
+  newer (i.e. `ZwaveJS`) versions of the backend.
+
+## [0.24.1] - 2022-12-08
+
+### Fixed
+
+- Removed a parenthesized context manager that broke compatibility with
+  Python &lt; 3.10.
+
+## [0.24.0] - 2022-11-22
+
+### Added
+
+- Added [Wallabag integration](https://git.platypush.tech/platypush/platypush/issues/224).
+- Added [Mimic3 TTS integration](https://git.platypush.tech/platypush/platypush/issues/226).
+- Added `qos` attribute to `mqtt.publish` and all the plugins derived from `mqtt`.
+
+### Changed
+
+- Replaced PyJWT dependency with the Python-native `rsa` package. This will
+  make the installation much lighter, compatible with more systems and less
+  dependent on the platform-specific libraries required by `cryptography`.
+
+> **NOTE**: This is a breaking change for those who use the `backend.http` API
+> with JWT tokens. The new logic encrypts and encodes the payload in a
+> different format, therefore previously generated tokens are no longer
+> compatible.
+
+## [0.23.6] - 2022-09-19
+
+### Fixed
+
+- Fixed album_id and list of tracks on `music.tidal.get_album`.
+
+## [0.23.5] - 2022-09-18
+
+### Added
+
+- Added support for web hooks returning their hook method responses back to the
+  HTTP client.
+
+- Added [Tidal integration](https://git.platypush.tech/platypush/platypush/pulls/223)
+
+- Added support for [OPML
+  subscriptions](https://git.platypush.tech/platypush/platypush/pulls/220) to
+  the `rss` plugin.
+
+- Better support for bulk database operations on the `db` plugin.
+
+### Fixed
+
+- Now supporting YAML sections with empty configurations.
+
+## [0.23.4] - 2022-08-28
+
+### Added
+
+- Added `matrix` integration
+  ([issue](https://git.platypush.tech/platypush/platypush/issues/2),
+  [PR](https://git.platypush.tech/platypush/platypush/pulls/217)).
+
+### Changed
+
 - Removed `clipboard` backend. Enabling the `clipboard` plugin will also enable
   clipboard monitoring, with no need for an additional backend.
 

README.md

@@ -4,24 +4,16 @@ Platypush
 [![Build Status](https://ci.platypush.tech/status.svg)](https://ci.platypush.tech/latest.log)
 [![Documentation Status](https://ci.platypush.tech/docs/status.svg)](https://ci.platypush.tech/docs/latest.log)
 [![pip version](https://img.shields.io/pypi/v/platypush.svg?style=flat)](https://pypi.python.org/pypi/platypush/)
-[![License](https://img.shields.io/github/license/BlackLight/platypush.svg)](https://git.platypush.tech/platypush/platypush/-/blob/master/LICENSE.txt)
-[![Last Commit](https://img.shields.io/github/last-commit/BlackLight/platypush.svg)](https://git.platypush.tech/platypush/platypush/-/commits/master/)
+[![License](https://img.shields.io/github/license/BlackLight/platypush.svg)](https://git.platypush.tech/platypush/platypush/src/branch/master/LICENSE.txt)
+[![Last Commit](https://img.shields.io/github/last-commit/BlackLight/platypush.svg)](https://git.platypush.tech/platypush/platypush/commits/branch/master)
 [![Join chat on Matrix](https://img.shields.io/matrix/:platypush?server_fqdn=matrix.platypush.tech)](https://matrix.to/#/#platypush:matrix.platypush.tech)
-[![Contributions](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://git.platypush.tech/platypush/platypush/-/blob/master/CONTRIBUTING.md)
-[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/BlackLight/platypush.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/BlackLight/platypush/context:python)
-[![Language grade: JavaScript](https://img.shields.io/lgtm/grade/javascript/g/BlackLight/platypush.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/BlackLight/platypush/context:javascript)
+[![Contributions](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://git.platypush.tech/platypush/platypush/src/branch/master/CONTRIBUTING.md)
 
 <!-- toc -->
 
-- [Architecture](#architecture)
-  * [Plugins](#plugins)
-  * [Actions](#actions)
-  * [Backends](#backends)
-  * [Events](#events)
-  * [Hooks](#hooks)
-  * [Procedures](#procedures)
-  * [Cronjobs](#cronjobs)
-  * [The web interface](#the-web-interface)
+- [Useful links](#useful-links)
+- [Introduction](#introduction)
+    + [What it can do](#what-it-can-do)
 - [Installation](#installation)
   * [System installation](#system-installation)
     + [Install through `pip`](#install-through-pip)
@@ -33,17 +25,27 @@ Platypush
     + [Check the instructions reported in the documentation](#check-the-instructions-reported-in-the-documentation)
   * [Virtual environment installation](#virtual-environment-installation)
   * [Docker installation](#docker-installation)
+- [Architecture](#architecture)
+  * [Plugins](#plugins)
+  * [Actions](#actions)
+  * [Backends](#backends)
+  * [Events](#events)
+  * [Hooks](#hooks)
+  * [Procedures](#procedures)
+  * [Cronjobs](#cronjobs)
+  * [Entities](#entities)
+  * [The web interface](#the-web-interface)
 - [Mobile app](#mobile-app)
 - [Tests](#tests)
 - [Funding](#funding)
 
 <!-- tocstop -->
 
+## Useful links
 - Recommended read: [**Getting started with Platypush**](https://blog.platypush.tech/article/Ultimate-self-hosted-automation-with-Platypush).
 
-- The [blog](https://blog.platypush.tech) is in general a good place to get
-  more insights on what you can build with it and inspiration about possible
-  usages.
+- The [blog](https://blog.platypush.tech) is a good place to get more insights
+  and inspiration on what you can build.
 
 - The [wiki](https://git.platypush.tech/platypush/platypush/wiki) also
   contains many resources on getting started.
@@ -51,19 +53,19 @@ Platypush
 - Extensive documentation for all the available integrations and messages [is
   available](https://docs.platypush.tech/).
 
-- If you have issues/feature requests/enhancement ideas please [create an
-  issue](https://git.platypush.tech/platypush/platypush/-/issues).
-
-- A [Reddit channel](https://www.reddit.com/r/platypush) is also available for
-  more general questions.
+- If you have issues/feature requests/enhancements please [create an
+  issue](https://git.platypush.tech/platypush/platypush/issues).
 
 - A [Matrix instance](https://matrix.to/#/#platypush:matrix.platypush.tech) is
-  also available if you are looking for more interactive support.
+  available if you are looking for interactive support.
 
----
+- A [Reddit channel](https://www.reddit.com/r/platypush) is available for
+  general questions.
 
-Platypush is a general-purpose extensible platform for automation and
-integration across multiple services and devices.
+## Introduction
+
+Platypush is a general-purpose extensible platform for automation across
+multiple services and devices.
 
 It enables users to create their own self-hosted pieces of automation based on
 events (*if this happens then do that*)
@@ -72,16 +74,15 @@ everything you need to visualize and control under one roof.
 
 It takes some concepts from [IFTTT](https://ifttt.com),
 [Tasker](https://tasker.joaoapps.com/), [Microsoft
-Flow](https://flow.microsoft.com), [PushBullet](https://pushbullet.com) and
-[Home Assistant](https://www.home-assistant.io/) to provide an environment
-where the user can easily connect things together.
+Flow](https://flow.microsoft.com) and [Home
+Assistant](https://www.home-assistant.io/) to provide an environment where the
+user can easily connect things together.
 
-Its ideal home is a single-board computer like a RaspberryPi that you can
-configure to orchestrate any home automation and cloud automation in your own
-living room or garage, but it can easily run on any device that can run a
-Python interpreter, and the bar for the hardware requirements is very low as
-well - I use it to run pieces of automation on devices as powerful as a
-RaspberryPi Zero or an old Nokia N900 with Linux.
+It's built with compatibility and flexibility in mind, and it can easily run on
+any device that can run a Python interpreter - from a Raspberry Pi zero, to an
+old smartphone, to a beefy server.
+
+#### What it can do
 
 You can use Platypush to do things like:
 
@@ -112,13 +113,198 @@ You can use Platypush to do things like:
 - [Create a custom single hub for Zigbee and Z-Wave smart devices](https://blog.platypush.tech/article/Transform-a-RaspberryPi-into-a-universal-Zigbee-and-Z-Wave-bridge)
 - Build your own web dashboard with calendar, weather, news and music controls
   (basically, anything that has a Platypush web widget)
-- ...and much more (basically, anything that comes with a [Platypush plugin](https://docs.platypush.tech/en/latest/plugins.html))
+- ...and much more (basically, anything that comes with a [Platypush plugin](https://docs.platypush.tech))
+
+## Installation
+
+### System installation
+
+Platypush uses Redis to deliver and store requests and temporary messages:
+
+```yaml
+# Example for Debian-based distributions
+[sudo] apt-get install redis-server
+
+# Enable and start the service
+[sudo] systemctl enable redis
+[sudo] systemctl start redis
+```
+
+#### Install through `pip`
+
+```shell
+[sudo] pip3 install platypush
+```
+
+#### Install through a system package manager
+
+Note: currently only Arch Linux and derived distributions are supported.
+
+You can either install the
+[`platypush`](https://aur.archlinux.org/packages/platypush) package (for the
+latest stable version) or the
+[`platypush-git`](https://aur.archlinux.org/packages/platypush-git) package
+(for the latest git version) through your favourite AUR package manager. For
+example, using `yay`:
+
+```shell
+yay platypush
+# Or
+yay platypush-git
+```
+
+The Arch Linux packages on AUR are automatically updated upon new git commits
+or tags.
+
+#### Install from sources
+
+```shell
+git clone https://git.platypush.tech/platypush/platypush.git
+cd platypush
+[sudo] pip install .
+# Or
+[sudo] python3 setup.py install
+```
+
+### Installing the dependencies for your extensions
+
+After installing the base platform, you may want to check the dependencies and
+configuration required by the extensions that you wish to use. There are a few
+ways to check the dependencies required by an extension:
+
+#### Install via `extras` name
+
+All the extensions that require extra dependencies are listed in the
+[`extras_require` section under
+`setup.py`](https://git.platypush.tech/platypush/platypush/src/branch/master/setup.py#L84).
+
+#### Install via `manifest.yaml`
+
+All the plugins and backends have a `manifest.yaml` file in their source folder.
+Any extra dependencies are listed there
+
+If you followed the `extras` or `manifest.yaml` way to discover the
+dependencies, then you can install them in two ways:
+
+1. `pip` installation:
+
+```shell
+[sudo] pip3 install 'platypush[extra1,extra2,extra3]'
+```
+
+2. Sources installation:
+
+```shell
+cd $DIR_TO_PLATYPUSH
+[sudo] pip3 install '.[extra1,extra2,extra3]'
+```
+
+#### Check the instructions reported in the documentation
+
+If you follow this route then simply run the commands listed in the
+[plugin/backend documentation](https://docs.platypush.tech) to get the
+dependencies installed.
+
+After installing the dependencies, create a configuration file under
+`~/.config/platypush/config.yaml` (the application can load the configuration
+from another location through the `-c` option) containing the configuration of
+the backend and plugins that you want to use, and add any hooks and procedures
+for your use case.
+
+You can then start the service by simply running:
+
+```shell
+platypush
+```
+
+It's advised to run it as a systemd service though - simply copy the provided
+[`.service`
+file](https://git.platypush.tech/platypush/platypush/src/branch/master/examples/systemd/platypush.service)
+to `~/.config/systemd/user`, check if the path of `platypush` matches the path
+where it's installed on your system, and start the service via `systemctl`:
+
+```shell
+systemctl --user start platypush
+```
+
+### Virtual environment installation
+
+Platypush provides a script named `platyvenv` that can parse a `config.yaml`
+and automatically create a virtual environment (under
+`~/.local/share/platypush/venv/<device_id>`) with all the dependencies required
+by the configured integrations.
+
+1. Create the environment from a configuration file:
+
+    ```shell
+    platyvenv build -c /path/to/config.yaml
+    ```
+
+2. Start the service from the virtual environment:
+
+    ```shell
+        # device_id matches either the hostname or the device_id in config.yaml
+    platyvenv start device_id
+    ```
+
+3. Stop the instance:
+
+    ```shell
+    platyvenv stop device_id
+    ```
+
+4. Remove the instance:
+
+    ```shell
+    platyvenv rm device_id
+    ```
+
+[Wiki instructions](https://git.platypush.tech/platypush/platypush/wiki/Run-platypush-in-a-virtual-environment)
+
+### Docker installation
+
+You can also install Platypush in a container - the application provides a
+script named `platydock` that automatically creates a container instance from a
+`config.yaml`:
+
+1. Create the container from a configuration file:
+
+    ```shell
+    platydock build -c /path/to/config.yaml
+    ```
+
+2. Start the container:
+
+    ```shell
+        # device_id matches either the hostname or the device_id in config.yaml
+    platydock start device_id
+    ```
+
+3. Stop the instance:
+
+    ```shell
+    platydock stop device_id
+    ```
+
+4. Remove the instance:
+
+    ```shell
+    platydock rm device_id
+    ```
+
+Note that both the virtual environment and Docker container option offer the
+possibility to include extra YAML configuration files in the main `config.yaml`
+through the `include` directive (as long as they are in the same folder as the
+main `config.yaml`), as well as external Python scripts in a `scripts`
+directory in the same folder as the `config.yaml`.
+
+[Wiki instructions](https://git.platypush.tech/platypush/platypush/wiki/Run-platypush-in-a-container)
 
 ## Architecture
 
 The architecture of Platypush consists of a few simple pieces, orchestrated by
 a configuration file stored by default under
-[`~/.config/platypush/config.yaml`](https://git.platypush.tech/platypush/platypush/-/blob/master/examples/conf/config.yaml):
+[`~/.config/platypush/config.yaml`](https://git.platypush.tech/platypush/platypush/src/branch/master/examples/conf/config.yaml):
 
 ### Plugins
 
@@ -183,32 +369,14 @@ as simple JSON messages:
 
 [Full list](https://docs.platypush.tech/en/latest/backends.html)
 
-They are background services that either listen for messages on channels (like
+They are background services that listen for messages on channels (like
 an [HTTP
 backend](https://docs.platypush.tech/en/latest/platypush/backend/http.html), an
 [MQTT
 instance](https://docs.platypush.tech/en/latest/platypush/backend/mqtt.html), a
 [Kafka
 instance](https://docs.platypush.tech/en/latest/platypush/backend/kafka.html),
-a [Websocket
-service](https://docs.platypush.tech/en/latest/platypush/backend/websocket.html),
 [Pushbullet](https://docs.platypush.tech/en/latest/platypush/backend/pushbullet.html)
-etc.) or monitor a device or a service for events (like a
-[sensor](https://docs.platypush.tech/en/latest/platypush/backend/sensor.html),
-a custom [voice
-assistant](https://docs.platypush.tech/en/latest/platypush/backend/assistant.google.html),
-a bridge running on a
-[Zigbee](https://docs.platypush.tech/en/latest/platypush/backend/zigbee.mqtt.html)
-or
-[Z-Wave](https://docs.platypush.tech/en/latest/platypush/backend/zwave.html),
-an [NFC card
-reader](https://docs.platypush.tech/en/latest/platypush/backend/nfc.html), a
-[MIDI
-device](https://docs.platypush.tech/en/latest/platypush/backend/midi.html), a
-[Telegram
-channel](https://docs.platypush.tech/en/latest/platypush/backend/chat.telegram.html),
-a [Bluetooth
-scanner](https://docs.platypush.tech/en/latest/platypush/backend/bluetooth.scanner.ble.html)
 etc.).
 
 If a backend supports the execution of requests (e.g. HTTP, MQTT, Kafka,
@@ -224,7 +392,6 @@ curl -XPOST -H 'Content-Type: application/json' -d '
 }' http://host:8008/auth
 
 # Execute a request
-
 curl -XPOST -H 'Content-Type: application/json' \
     -H "Authorization: Bearer $YOUR_TOKEN" -d '
 {
@@ -269,7 +436,7 @@ triggered. Hooks are the glue that connects events to actions, exposing a
 paradigm similar to IFTTT (_if a certain event happens then run these
 actions_). They can declared as:
 
-- Sections of the [`config.yaml`](https://git.platypush.tech/platypush/platypush/-/blob/master/examples/conf/config.yaml).
+- Sections of the [`config.yaml`](https://git.platypush.tech/platypush/platypush/src/branch/master/examples/conf/config.yaml).
   Example:
 
 ```yaml
@@ -293,7 +460,7 @@ event.hook.SearchSongVoiceCommand:
 
 - Stand-alone Python scripts stored under `~/.config/platypush/scripts` and
   will be dynamically imported at start time.
-  [Example](https://git.platypush.tech/platypush/platypush/-/blob/master/examples/conf/hook.py):
+  [Example](https://git.platypush.tech/platypush/platypush/src/branch/master/examples/conf/hook.py):
 
 ```python
 from platypush.event.hook import hook
@@ -417,6 +584,37 @@ def check_bt_device(**context):
     # on_device_off logic here
 ```
 
+### Entities
+
+Entities are a fundamental building block of Platypush. Most of the
+integrations will store their state or connected devices in the form of
+entities - e.g. the sensors detected by the Z-Wave/Zigbee/Bluetooth
+integration, or the lights connected to a Hue bridge, or your cloud nodes, or
+your custom Arduino/ESP machinery, and so on.
+
+Entities provide a consistent interface to interact with your integrations
+regardless of their type and the plugin that handles them. For instance, all
+temperature sensors will expose the same interface, regardless if they are
+Bluetooth or Zigbee sensors, and all the media plugins will expose the same
+interface, regardless if they manage Chromecasts, Kodi, Plex, Jellyfin or a
+local VLC player.
+
+Once you enable the HTTP backend and a few integrations that export entities
+and register a user, you can query the detected entities via:
+
+```shell
+curl -XPOST -H 'Content-Type: application/json' \
+    -H "Authorization: Bearer $YOUR_TOKEN" \
+    -d '{"type":"request", "action":"entities.get"}' \
+    http://localhost:8008/execute
+```
+
+All the entities expose the same interface and can be manipulated through the
+same API. Also, when an entity is updated it always emits an
+[`EntityUpdateEvent`](https://docs.platypush.tech/platypush/events/entities.html#platypush.message.event.entities.EntityUpdateEvent),
+so you can easily create hooks that react to these events and act on multiple
+types of entities.
+
 ### The web interface
 
 If
@@ -429,194 +627,9 @@ music controls and search, media library and torrent management, lights,
 Zigbee/Z-Wave devices and so on. The UI is responsive and mobile-friendly.
 
 The web service also provides means for the user to create [custom
-dashboards](https://git.platypush.tech/platypush/platypush/-/blob/master/examples/conf/dashboard.xml)
+dashboards](https://git.platypush.tech/platypush/platypush/src/branch/master/examples/conf/dashboard.xml)
 that can be used to show information from multiple sources on a large screen.
 
-## Installation
-
-### System installation
-
-Platypush uses Redis to deliver and store requests and temporary messages:
-
-```yaml
-    # Example for Debian-based distributions
-[sudo] apt-get install redis-server
-
-    # Enable and start the service
-[sudo] systemctl enable redis
-[sudo] systemctl start redis
-```
-
-#### Install through `pip`
-
-```shell
-[sudo] pip3 install platypush
-```
-
-#### Install through a system package manager
-
-Note: currently only Arch Linux and derived distributions are supported.
-
-You can either install the
-[`platypush`](https://aur.archlinux.org/packages/platypush) package (for the
-latest stable version) or the
-[`platypush-git`](https://aur.archlinux.org/packages/platypush-git) package
-(for the latest git version) through your favourite AUR package manager. For
-example, using `yay`:
-
-```shell
-yay platypush
-# Or
-yay platypush-git
-```
-
-The Arch Linux packages on AUR are automatically updated upon new git commits
-or tags.
-
-#### Install from sources
-
-```shell
-git clone https://git.platypush.tech/platypush/platypush.git
-cd platypush
-[sudo] pip install .
-    # Or
-[sudo] python3 setup.py install
-```
-
-### Installing the dependencies for your extensions
-
-After installing the base platform, you may want to check the dependencies and
-configuration required by the extensions that you wish to use. There are a few
-ways to check the dependencies required by an extension:
-
-#### Install via `extras` name
-
-All the extensions that require extra dependencies are listed in the
-[`extras_require` section under
-`setup.py`](https://git.platypush.tech/platypush/platypush/-/blob/master/setup.py#L72).
-
-#### Install via `manifest.yaml`
-
-All the plugins and backends have a `manifest.yaml` file in their source folder.
-Any extra dependencies are listed there
-
-If you followed the `extras` or `manifest.yaml` way to discover the
-dependencies, then you can install them in two ways:
-
-1. `pip` installation:
-
-```shell
-[sudo] pip3 install 'platypush[extra1,extra2,extra3]'
-```
-
-2. Sources installation:
-
-```shell
-cd $DIR_TO_PLATYPUSH
-[sudo] pip3 install '.[extra1,extra2,extra3]'
-```
-
-#### Check the instructions reported in the documentation
-
-If you follow this route then simply run the commands listed in the
-[plugin/backend documentation](https://docs.platypush.tech) to get the
-dependencies installed.
-
-After installing the dependencies, create a configuration file under
-`~/.config/platypush/config.yaml` (the application can load the configuration
-from another location through the `-c` option) containing the configuration of
-the backend and plugins that you want to use, and add any hooks and procedures
-for your use case.
-
-You can then start the service by simply running:
-
-```shell
-platypush
-```
-
-It's advised to run it as a systemd service though - simply copy the provided
-[`.service`
-file](https://git.platypush.tech/platypush/platypush/-/blob/master/examples/systemd/platypush.service)
-to `~/.config/systemd/user`, check if the path of `platypush` matches the path
-where it's installed on your system, and start the service via `systemctl`:
-
-```shell
-systemctl --user start platypush
-```
-
-### Virtual environment installation
-
-Platypush provides a script named `platyvenv` that can parse a `config.yaml`
-and automatically create a virtual environment (under
-`~/.local/share/platypush/venv/<device_id>`) with all the dependencies required
-by the configured integrations.
-
-1. Create the environment from a configuration file:
-
-    ```shell
-    platyvenv build -c /path/to/config.yaml
-    ```
-
-2. Start the service from the virtual environment:
-
-    ```shell
-        # device_id matches either the hostname or the device_id in config.yaml
-    platyvenv start device_id
-    ```
-
-3. Stop the instance:
-
-    ```shell
-    platyvenv stop device_id
-    ```
-
-4. Remove the instance:
-
-    ```shell
-    platyvenv rm device_id
-    ```
-
-[Wiki instructions](https://git.platypush.tech/platypush/platypush/wiki/Run-platypush-in-a-virtual-environment)
-
-### Docker installation
-
-You can also install Platypush in a container - the application provides a
-script named `platydock` that automatically creates a container instance from a
-`config.yaml`:
-
-1. Create the container from a configuration file:
-
-    ```shell
-    platydock build -c /path/to/config.yaml
-    ```
-
-2. Start the container:
-
-    ```shell
-        # device_id matches either the hostname or the device_id in config.yaml
-    platydock start device_id
-    ```
-
-3. Stop the instance:
-
-    ```shell
-    platydock stop device_id
-    ```
-
-4. Remove the instance:
-
-    ```shell
-    platydock rm device_id
-    ```
-
-Note that both the virtual environment and Docker container option offer the
-possibility to include extra YAML configuration files in the main `config.yaml`
-through the `include` directive (as long as they are in the same folder as the
-main `config.yaml`), as well as external Python scripts in a `scripts`
-directory in the same folder as the `config.yaml`.
-
-[Wiki instructions](https://git.platypush.tech/platypush/platypush/wiki/Run-platypush-in-a-container)
-
 ## Mobile app
 
 An [official Android
@@ -628,11 +641,7 @@ of Platypush to your fingertips.
 ## Tests
 
 To run the tests simply run `pytest` either from the project root folder or the
-`tests/` folder. Or run the following command from the project root folder:
-
-```shell
-python -m tests
-```
+`tests/` folder.
 
 ---
 

docs/source/backends.rst

@@ -3,17 +3,13 @@ Backends
 ========
 
 .. toctree::
-    :maxdepth: 2
+    :maxdepth: 1
     :caption: Backends:
 
     platypush/backend/adafruit.io.rst
     platypush/backend/alarm.rst
     platypush/backend/assistant.google.rst
     platypush/backend/assistant.snowboy.rst
-    platypush/backend/bluetooth.fileserver.rst
-    platypush/backend/bluetooth.pushserver.rst
-    platypush/backend/bluetooth.scanner.rst
-    platypush/backend/bluetooth.scanner.ble.rst
     platypush/backend/button.flic.rst
     platypush/backend/camera.pi.rst
     platypush/backend/chat.telegram.rst
@@ -32,7 +28,6 @@ Backends
     platypush/backend/joystick.linux.rst
     platypush/backend/kafka.rst
     platypush/backend/light.hue.rst
-    platypush/backend/linode.rst
     platypush/backend/log.http.rst
     platypush/backend/mail.rst
     platypush/backend/midi.rst
@@ -48,20 +43,8 @@ Backends
     platypush/backend/pushbullet.rst
     platypush/backend/redis.rst
     platypush/backend/scard.rst
-    platypush/backend/sensor.accelerometer.rst
-    platypush/backend/sensor.arduino.rst
-    platypush/backend/sensor.battery.rst
-    platypush/backend/sensor.bme280.rst
-    platypush/backend/sensor.dht.rst
-    platypush/backend/sensor.distance.rst
-    platypush/backend/sensor.distance.vl53l1x.rst
-    platypush/backend/sensor.envirophat.rst
     platypush/backend/sensor.ir.zeroborg.rst
     platypush/backend/sensor.leap.rst
-    platypush/backend/sensor.ltr559.rst
-    platypush/backend/sensor.mcp3008.rst
-    platypush/backend/sensor.motion.pmw3901.rst
-    platypush/backend/sensor.serial.rst
     platypush/backend/stt.deepspeech.rst
     platypush/backend/stt.picovoice.hotword.rst
     platypush/backend/stt.picovoice.speech.rst
@@ -72,8 +55,6 @@ Backends
     platypush/backend/weather.buienradar.rst
     platypush/backend/weather.darksky.rst
     platypush/backend/weather.openweathermap.rst
-    platypush/backend/websocket.rst
     platypush/backend/wiimote.rst
-    platypush/backend/zigbee.mqtt.rst
     platypush/backend/zwave.rst
     platypush/backend/zwave.mqtt.rst

docs/source/conf.py

@@ -71,7 +71,7 @@ master_doc = 'index'
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -103,6 +103,7 @@ html_domain_indices = True
 html_theme_options = {
     'toc_title': 'Platypush documentation',
     'repository_url': 'https://git.platypush.tech/platypush/platypush',
+    'repository_provider': 'github',
     'use_repository_button': True,
     'use_issues_button': True,
     'use_fullscreen_button': True,
@@ -138,15 +139,12 @@ latex_elements = {
     # The paper size ('letterpaper' or 'a4paper').
     #
     # 'papersize': 'letterpaper',
-
     # The font size ('10pt', '11pt' or '12pt').
     #
     # 'pointsize': '10pt',
-
     # Additional stuff for the LaTeX preamble.
     #
     # 'preamble': '',
-
     # Latex figure (float) alignment
     #
     # 'figure_align': 'htbp',
@@ -156,8 +154,7 @@ latex_elements = {
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'platypush.tex', 'platypush Documentation',
-     'BlackLight', 'manual'),
+    (master_doc, 'platypush.tex', 'platypush Documentation', 'BlackLight', 'manual'),
 ]
 
 
@@ -165,10 +162,7 @@ latex_documents = [
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [
-    (master_doc, 'platypush', 'platypush Documentation',
-     [author], 1)
-]
+man_pages = [(master_doc, 'platypush', 'platypush Documentation', [author], 1)]
 
 
 # -- Options for Texinfo output ----------------------------------------------
@@ -177,9 +171,15 @@ man_pages = [
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, 'platypush', 'platypush Documentation',
-     author, 'platypush', 'One line description of project.',
-     'Miscellaneous'),
+    (
+        master_doc,
+        'platypush',
+        'platypush Documentation',
+        author,
+        'platypush',
+        'A general-purpose platform for automation.',
+        'Miscellaneous',
+    ),
 ]
 
 
@@ -188,7 +188,7 @@ texinfo_documents = [
 # -- Options for intersphinx extension ---------------------------------------
 
 # Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {'https://docs.python.org/': None}
+intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
 
 # -- Options for todo extension ----------------------------------------------
 
@@ -196,10 +196,13 @@ intersphinx_mapping = {'https://docs.python.org/': None}
 todo_include_todos = True
 
 autodoc_default_options = {
-    'inherited-members': True,
+    'members': True,
+    'show-inheritance': True,
 }
 
-autodoc_mock_imports = ['googlesamples.assistant.grpc.audio_helpers',
+autodoc_mock_imports = [
+    'gunicorn',
+    'googlesamples.assistant.grpc.audio_helpers',
     'google.assistant.embedded',
     'google.assistant.library',
     'google.assistant.library.event',
@@ -215,7 +218,7 @@ autodoc_mock_imports = ['googlesamples.assistant.grpc.audio_helpers',
     'bluetooth',
     'gevent.wsgi',
     'Adafruit_IO',
-                        'pyperclip',
+    'pyclip',
     'pydbus',
     'inputs',
     'inotify',
@@ -248,6 +251,7 @@ autodoc_mock_imports = ['googlesamples.assistant.grpc.audio_helpers',
     'pyaudio',
     'avs',
     'PyOBEX',
+    'PyOBEX.client',
     'todoist',
     'trello',
     'telegram',
@@ -291,6 +295,17 @@ autodoc_mock_imports = ['googlesamples.assistant.grpc.audio_helpers',
     'irc.connection',
     'irc.events',
     'defusedxml',
+    'nio',
+    'aiofiles',
+    'aiofiles.os',
+    'async_lru',
+    'bleak',
+    'bluetooth_numbers',
+    'TheengsDecoder',
+    'simple_websocket',
+    'uvicorn',
+    'websockets',
+    'docutils',
 ]
 
 sys.path.insert(0, os.path.abspath('../..'))

docs/source/events.rst

@@ -3,7 +3,7 @@ Events
 ======
 
 .. toctree::
-    :maxdepth: 2
+    :maxdepth: 1
     :caption: Events:
 
     platypush/events/adafruit.rst
@@ -20,6 +20,7 @@ Events
     platypush/events/custom.rst
     platypush/events/dbus.rst
     platypush/events/distance.rst
+    platypush/events/entities.rst
     platypush/events/file.rst
     platypush/events/foursquare.rst
     platypush/events/geo.rst
@@ -30,6 +31,7 @@ Events
     platypush/events/gotify.rst
     platypush/events/gpio.rst
     platypush/events/gps.rst
+    platypush/events/hid.rst
     platypush/events/http.rst
     platypush/events/http.hook.rst
     platypush/events/http.rss.rst
@@ -41,11 +43,13 @@ Events
     platypush/events/linode.rst
     platypush/events/log.http.rst
     platypush/events/mail.rst
+    platypush/events/matrix.rst
     platypush/events/media.rst
     platypush/events/midi.rst
     platypush/events/mqtt.rst
     platypush/events/music.rst
     platypush/events/music.snapcast.rst
+    platypush/events/music.tidal.rst
     platypush/events/nextcloud.rst
     platypush/events/nfc.rst
     platypush/events/ngrok.rst
@@ -72,6 +76,7 @@ Events
     platypush/events/weather.rst
     platypush/events/web.rst
     platypush/events/web.widget.rst
+    platypush/events/websocket.rst
     platypush/events/wiimote.rst
     platypush/events/zeroborg.rst
     platypush/events/zeroconf.rst

docs/source/index.rst

@@ -16,7 +16,7 @@ For more information on Platypush check out:
 .. _Blog articles: https://blog.platypush.tech
 
 .. toctree::
-    :maxdepth: 3
+    :maxdepth: 2
     :caption: Contents:
 
     backends

docs/source/platypush/backend/bluetooth.fileserver.rst

@@ -1,6 +0,0 @@
-``bluetooth.fileserver``
-==========================================
-
-.. automodule:: platypush.backend.bluetooth.fileserver
-	:members:
-

docs/source/platypush/backend/bluetooth.pushserver.rst

@@ -1,6 +0,0 @@
-``bluetooth.pushserver``
-==========================================
-
-.. automodule:: platypush.backend.bluetooth.pushserver
-	:members:
-

docs/source/platypush/backend/bluetooth.scanner.ble.rst

@@ -1,5 +0,0 @@
-``bluetooth.scanner.ble``
-===========================================
-
-.. automodule:: platypush.backend.bluetooth.scanner.ble
-    :members:

docs/source/platypush/backend/bluetooth.scanner.rst

@@ -1,5 +0,0 @@
-``bluetooth.scanner``
-=======================================
-
-.. automodule:: platypush.backend.bluetooth.scanner
-    :members:

docs/source/platypush/backend/linode.rst

@@ -1,5 +0,0 @@
-``linode``
-============================
-
-.. automodule:: platypush.backend.linode
-    :members:

docs/source/platypush/backend/sensor.accelerometer.rst

@@ -1,6 +0,0 @@
-``sensor.accelerometer``
-==========================================
-
-.. automodule:: platypush.backend.sensor.accelerometer
-    :members:
-

docs/source/platypush/backend/sensor.arduino.rst

@@ -1,5 +0,0 @@
-``sensor.arduino``
-====================================
-
-.. automodule:: platypush.backend.sensor.arduino
-    :members:

docs/source/platypush/backend/sensor.battery.rst

@@ -1,5 +0,0 @@
-``sensor.battery``
-====================================
-
-.. automodule:: platypush.backend.sensor.battery
-    :members:

docs/source/platypush/backend/sensor.bme280.rst

@@ -1,6 +0,0 @@
-``sensor.bme280``
-===================================
-
-.. automodule:: platypush.backend.sensor.bme280
-	:members:
-

docs/source/platypush/backend/sensor.dht.rst

@@ -1,5 +0,0 @@
-``sensor.dht``
-================================
-
-.. automodule:: platypush.backend.sensor.dht
-    :members:

docs/source/platypush/backend/sensor.distance.rst

@@ -1,5 +0,0 @@
-``sensor.distance``
-=====================================
-
-.. automodule:: platypush.backend.sensor.distance
-    :members:

docs/source/platypush/backend/sensor.distance.vl53l1x.rst

@@ -1,6 +0,0 @@
-``sensor.distance.vl53l1x``
-=============================================
-
-.. automodule:: platypush.backend.sensor.distance.vl53l1x
-	:members:
-

docs/source/platypush/backend/sensor.envirophat.rst

@@ -1,6 +0,0 @@
-``sensor.envirophat``
-=======================================
-
-.. automodule:: platypush.backend.sensor.envirophat
-    :members:
-

docs/source/platypush/backend/sensor.ltr559.rst

@@ -1,6 +0,0 @@
-``sensor.ltr559``
-===================================
-
-.. automodule:: platypush.backend.sensor.ltr559
-	:members:
-

docs/source/platypush/backend/sensor.mcp3008.rst

@@ -1,8 +0,0 @@
-``sensor.mcp3008``
-====================================
-
-.. automodule:: platypush.backend.sensor.mcp3008
-    :members:
-
-
-

docs/source/platypush/backend/sensor.motion.pmw3901.rst

@@ -1,5 +0,0 @@
-``sensor.motion.pmw3901``
-=========================
-
-.. automodule:: platypush.backend.sensor.motion.pmw3901
-    :members:

docs/source/platypush/backend/sensor.serial.rst

@@ -1,6 +0,0 @@
-``sensor.serial``
-===================================
-
-.. automodule:: platypush.backend.sensor.serial
-    :members:
-

docs/source/platypush/backend/websocket.rst

@@ -1,6 +0,0 @@
-``websocket``
-===============================
-
-.. automodule:: platypush.backend.websocket
-	:members:
-

docs/source/platypush/backend/zigbee.mqtt.rst

@@ -1,5 +0,0 @@
-``zigbee.mqtt``
-=================================
-
-.. automodule:: platypush.backend.zigbee.mqtt
-    :members:

docs/source/platypush/events/entities.rst

@@ -0,0 +1,5 @@
+``entities``
+============
+
+.. automodule:: platypush.message.event.entities
+    :members:

docs/source/platypush/events/hid.rst

@@ -0,0 +1,5 @@
+``hid``
+=======
+
+.. automodule:: platypush.message.event.hid
+    :members:

docs/source/platypush/events/matrix.rst

@@ -0,0 +1,5 @@
+``matrix``
+==========
+
+.. automodule:: platypush.message.event.matrix
+    :members:

docs/source/platypush/events/music.tidal.rst

@@ -0,0 +1,5 @@
+``music.tidal``
+===============
+
+.. automodule:: platypush.message.event.music.tidal
+    :members:

docs/source/platypush/events/websocket.rst

@@ -0,0 +1,5 @@
+``websocket``
+=============
+
+.. automodule:: platypush.message.event.websocket
+    :members:

docs/source/platypush/plugins/bluetooth.ble.rst

@@ -1,6 +0,0 @@
-``bluetooth.ble``
-===================================
-
-.. automodule:: platypush.plugins.bluetooth.ble
-	:members:
-

docs/source/platypush/plugins/dbus.rst

@@ -2,4 +2,4 @@
 ==========================
 
 .. automodule:: platypush.plugins.dbus
-    :members:
+    :exclude-members: DBusService, BusType

docs/source/platypush/plugins/entities.rst

@@ -0,0 +1,5 @@
+``entities``
+============
+
+.. automodule:: platypush.plugins.entities
+    :members:

docs/source/platypush/plugins/gpio.sensor.accelerometer.rst

@@ -1,6 +0,0 @@
-``gpio.sensor.accelerometer``
-===============================================
-
-.. automodule:: platypush.plugins.gpio.sensor.accelerometer
-    :members:
-

docs/source/platypush/plugins/gpio.sensor.bme280.rst

@@ -1,6 +0,0 @@
-``gpio.sensor.bme280``
-========================================
-
-.. automodule:: platypush.plugins.gpio.sensor.bme280
-	:members:
-

docs/source/platypush/plugins/gpio.sensor.dht.rst

@@ -1,5 +0,0 @@
-``gpio.sensor.dht``
-=====================================
-
-.. automodule:: platypush.plugins.gpio.sensor.dht
-    :members:

docs/source/platypush/plugins/gpio.sensor.distance.rst

@@ -1,6 +0,0 @@
-``gpio.sensor.distance``
-==========================================
-
-.. automodule:: platypush.plugins.gpio.sensor.distance
-    :members:
-

docs/source/platypush/plugins/gpio.sensor.distance.vl53l1x.rst

@@ -1,6 +0,0 @@
-``gpio.sensor.distance.vl53l1x``
-==================================================
-
-.. automodule:: platypush.plugins.gpio.sensor.distance.vl53l1x
-	:members:
-

docs/source/platypush/plugins/gpio.sensor.envirophat.rst

@@ -1,6 +0,0 @@
-``gpio.sensor.envirophat``
-============================================
-
-.. automodule:: platypush.plugins.gpio.sensor.envirophat
-    :members:
-

docs/source/platypush/plugins/gpio.sensor.ltr559.rst

@@ -1,6 +0,0 @@
-``gpio.sensor.ltr559``
-========================================
-
-.. automodule:: platypush.plugins.gpio.sensor.ltr559
-	:members:
-

docs/source/platypush/plugins/gpio.sensor.mcp3008.rst

@@ -1,7 +0,0 @@
-``gpio.sensor.mcp3008``
-=========================================
-
-.. automodule:: platypush.plugins.gpio.sensor.mcp3008
-    :members:
-
-

docs/source/platypush/plugins/gpio.sensor.motion.pmw3901.rst

@@ -1,5 +0,0 @@
-``gpio.sensor.motion.pmw3901``
-==============================
-
-.. automodule:: platypush.plugins.gpio.sensor.motion.pmw3901
-    :members:

docs/source/platypush/plugins/hid.rst

@@ -0,0 +1,5 @@
+``hid``
+=======
+
+.. automodule:: platypush.plugins.hid
+    :members:

docs/source/platypush/plugins/matrix.rst

@@ -0,0 +1,5 @@
+``matrix``
+==========
+
+.. automodule:: platypush.plugins.matrix
+    :members: MatrixPlugin

docs/source/platypush/plugins/music.tidal.rst

@@ -0,0 +1,5 @@
+``music.tidal``
+===============
+
+.. automodule:: platypush.plugins.music.tidal
+    :members:

docs/source/platypush/plugins/sensor.bme280.rst

@@ -0,0 +1,5 @@
+``sensor.bme280``
+=================
+
+.. automodule:: platypush.plugins.sensor.bme280
+    :members:

docs/source/platypush/plugins/sensor.dht.rst

@@ -0,0 +1,5 @@
+``sensor.dht``
+==============
+
+.. automodule:: platypush.plugins.sensor.dht
+    :members:

docs/source/platypush/plugins/sensor.distance.vl53l1x.rst

@@ -0,0 +1,5 @@
+``sensor.distance.vl53l1x``
+===========================
+
+.. automodule:: platypush.plugins.sensor.distance.vl53l1x
+    :members:

docs/source/platypush/plugins/sensor.envirophat.rst

@@ -0,0 +1,5 @@
+``sensor.envirophat``
+=====================
+
+.. automodule:: platypush.plugins.sensor.envirophat
+    :members:

docs/source/platypush/plugins/sensor.hcsr04.rst

@@ -0,0 +1,5 @@
+``sensor.hcsr04``
+=================
+
+.. automodule:: platypush.plugins.sensor.hcsr04
+    :members:

docs/source/platypush/plugins/sensor.lis3dh.rst

@@ -0,0 +1,5 @@
+``sensor.lis3dh``
+=================
+
+.. automodule:: platypush.plugins.sensor.lis3dh
+    :members:

docs/source/platypush/plugins/sensor.ltr559.rst

@@ -0,0 +1,5 @@
+``sensor.ltr559``
+=================
+
+.. automodule:: platypush.plugins.sensor.ltr559
+    :members:

docs/source/platypush/plugins/sensor.mcp3008.rst

@@ -0,0 +1,5 @@
+``sensor.mcp3008``
+==================
+
+.. automodule:: platypush.plugins.sensor.mcp3008
+    :members:

docs/source/platypush/plugins/sensor.pmw3901.rst

@@ -0,0 +1,5 @@
+``sensor.pmw3901``
+==================
+
+.. automodule:: platypush.plugins.sensor.pmw3901
+    :members:

docs/source/platypush/plugins/switchbot.bluetooth.rst

@@ -1,5 +0,0 @@
-``switchbot.bluetooth``
-=========================================
-
-.. automodule:: platypush.plugins.switchbot.bluetooth
-    :members:

docs/source/platypush/plugins/tts.mimic3.rst

@@ -0,0 +1,5 @@
+``tts.mimic3``
+==============
+
+.. automodule:: platypush.plugins.tts.mimic3
+    :members:

docs/source/platypush/plugins/wallabag.rst

@@ -0,0 +1,5 @@
+``wallabag``
+============
+
+.. automodule:: platypush.plugins.wallabag
+    :members:

docs/source/platypush/responses/bluetooth.rst

@@ -1,5 +0,0 @@
-``bluetooth``
-========================================
-
-.. automodule:: platypush.message.response.bluetooth
-    :members:

docs/source/platypush/responses/linode.rst

@@ -1,5 +0,0 @@
-``linode``
-=====================================
-
-.. automodule:: platypush.message.response.linode
-    :members:

docs/source/platypush/responses/system.rst

@@ -1,5 +0,0 @@
-``system``
-=====================================
-
-.. automodule:: platypush.message.response.system
-    :members:

docs/source/plugins.rst

@@ -3,7 +3,7 @@ Plugins
 =======
 
 .. toctree::
-    :maxdepth: 2
+    :maxdepth: 1
     :caption: Plugins:
 
     platypush/plugins/adafruit.io.rst
@@ -14,7 +14,6 @@ Plugins
     platypush/plugins/assistant.google.pushtotalk.rst
     platypush/plugins/autoremote.rst
     platypush/plugins/bluetooth.rst
-    platypush/plugins/bluetooth.ble.rst
     platypush/plugins/calendar.rst
     platypush/plugins/calendar.ical.rst
     platypush/plugins/camera.android.ipcam.rst
@@ -32,6 +31,7 @@ Plugins
     platypush/plugins/db.rst
     platypush/plugins/dbus.rst
     platypush/plugins/dropbox.rst
+    platypush/plugins/entities.rst
     platypush/plugins/esp.rst
     platypush/plugins/ffmpeg.rst
     platypush/plugins/file.rst
@@ -46,17 +46,9 @@ Plugins
     platypush/plugins/google.youtube.rst
     platypush/plugins/gotify.rst
     platypush/plugins/gpio.rst
-    platypush/plugins/gpio.sensor.accelerometer.rst
-    platypush/plugins/gpio.sensor.bme280.rst
-    platypush/plugins/gpio.sensor.dht.rst
-    platypush/plugins/gpio.sensor.distance.rst
-    platypush/plugins/gpio.sensor.distance.vl53l1x.rst
-    platypush/plugins/gpio.sensor.envirophat.rst
-    platypush/plugins/gpio.sensor.ltr559.rst
-    platypush/plugins/gpio.sensor.mcp3008.rst
-    platypush/plugins/gpio.sensor.motion.pmw3901.rst
     platypush/plugins/gpio.zeroborg.rst
     platypush/plugins/graphite.rst
+    platypush/plugins/hid.rst
     platypush/plugins/http.request.rst
     platypush/plugins/http.request.rss.rst
     platypush/plugins/http.webpage.rst
@@ -75,6 +67,7 @@ Plugins
     platypush/plugins/mail.smtp.rst
     platypush/plugins/mailgun.rst
     platypush/plugins/mastodon.rst
+    platypush/plugins/matrix.rst
     platypush/plugins/media.chromecast.rst
     platypush/plugins/media.gstreamer.rst
     platypush/plugins/media.jellyfin.rst
@@ -93,6 +86,7 @@ Plugins
     platypush/plugins/music.mpd.rst
     platypush/plugins/music.snapcast.rst
     platypush/plugins/music.spotify.rst
+    platypush/plugins/music.tidal.rst
     platypush/plugins/nextcloud.rst
     platypush/plugins/ngrok.rst
     platypush/plugins/nmap.rst
@@ -107,6 +101,15 @@ Plugins
     platypush/plugins/redis.rst
     platypush/plugins/rss.rst
     platypush/plugins/rtorrent.rst
+    platypush/plugins/sensor.bme280.rst
+    platypush/plugins/sensor.dht.rst
+    platypush/plugins/sensor.distance.vl53l1x.rst
+    platypush/plugins/sensor.envirophat.rst
+    platypush/plugins/sensor.hcsr04.rst
+    platypush/plugins/sensor.lis3dh.rst
+    platypush/plugins/sensor.ltr559.rst
+    platypush/plugins/sensor.mcp3008.rst
+    platypush/plugins/sensor.pmw3901.rst
     platypush/plugins/serial.rst
     platypush/plugins/shell.rst
     platypush/plugins/slack.rst
@@ -120,7 +123,6 @@ Plugins
     platypush/plugins/switch.tplink.rst
     platypush/plugins/switch.wemo.rst
     platypush/plugins/switchbot.rst
-    platypush/plugins/switchbot.bluetooth.rst
     platypush/plugins/system.rst
     platypush/plugins/tcp.rst
     platypush/plugins/tensorflow.rst
@@ -130,12 +132,14 @@ Plugins
     platypush/plugins/trello.rst
     platypush/plugins/tts.rst
     platypush/plugins/tts.google.rst
+    platypush/plugins/tts.mimic3.rst
     platypush/plugins/tv.samsung.ws.rst
     platypush/plugins/twilio.rst
     platypush/plugins/udp.rst
     platypush/plugins/user.rst
     platypush/plugins/utils.rst
     platypush/plugins/variable.rst
+    platypush/plugins/wallabag.rst
     platypush/plugins/weather.buienradar.rst
     platypush/plugins/weather.darksky.rst
     platypush/plugins/weather.openweathermap.rst

docs/source/responses.rst

@@ -3,22 +3,19 @@ Responses
 =========
 
 .. toctree::
-    :maxdepth: 2
+    :maxdepth: 1
     :caption: Responses:
 
-    platypush/responses/bluetooth.rst
     platypush/responses/camera.rst
     platypush/responses/camera.android.rst
     platypush/responses/chat.telegram.rst
     platypush/responses/google.drive.rst
-    platypush/responses/linode.rst
     platypush/responses/pihole.rst
     platypush/responses/ping.rst
     platypush/responses/printer.cups.rst
     platypush/responses/qrcode.rst
     platypush/responses/ssh.rst
     platypush/responses/stt.rst
-    platypush/responses/system.rst
     platypush/responses/tensorflow.rst
     platypush/responses/todoist.rst
     platypush/responses/translate.rst

examples/conf/config.yaml

@@ -109,8 +109,6 @@ calendar:
 backend.http:
     # Listening port
     port: 8008
-    # Websocket port
-    websocket_port: 8009
 
     # Through resource_dirs you can specify external folders whose content can be accessed on
     # the web server through a custom URL. In the case below we have a Dropbox folder containing
@@ -165,10 +163,6 @@ backend.mqtt:
 #backend.tcp:
 #    port: 3333
 
-# Websocket backend. Install required dependencies through 'pip install "platypush[http]"'
-#backend.websocket:
-#    port: 8765
-
 ## --
 ## Assistant configuration examples
 ## --

examples/nginx/nginx.sample.conf

@@ -0,0 +1,38 @@
+# An nginx configuration that can be used to reverse proxy connections to your
+# Platypush' HTTP service.
+
+server {
+  server_name my-platypush-host.domain.com;
+
+  # Proxy standard HTTP connections to your Platypush IP
+  location / {
+      proxy_pass http://my-platypush-host:8008/;
+
+      client_max_body_size 5M;
+      proxy_read_timeout 60;
+      proxy_connect_timeout 60;
+      proxy_set_header Host $http_host;
+      proxy_set_header X-Real-IP $remote_addr;
+      proxy_set_header X-Forwarded-Ssl on;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+  }
+
+  # Proxy websocket connections
+  location ~ ^/ws/(.*)$ {
+      proxy_pass http://10.0.0.2:8008/ws/$1;
+
+      proxy_set_header Upgrade $http_upgrade;
+      proxy_set_header Connection "upgrade";
+      proxy_redirect off;
+      proxy_http_version 1.1;
+      client_max_body_size 200M;
+      proxy_set_header Host $http_host;
+  }
+
+  # Optional SSL configuration - using Let's Encrypt certificates in this case
+  # listen 443 ssl;
+  # ssl_certificate /etc/letsencrypt/live/my-platypush-host.domain.com/fullchain.pem;
+  # ssl_certificate_key /etc/letsencrypt/live/my-platypush-host.domain.com/privkey.pem;
+  # include /etc/letsencrypt/options-ssl-nginx.conf;
+  # ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
+}

generate_missing_docs.py

@@ -1,4 +1,5 @@
 import os
+from typing import Iterable, Optional
 
 from platypush.backend import Backend
 from platypush.context import get_plugin
@@ -13,21 +14,11 @@ def _get_inspect_plugin():
 
 
 def get_all_plugins():
-    manifests = {mf.component_name for mf in get_manifests(Plugin)}
-    return {
-        plugin_name: plugin_info
-        for plugin_name, plugin_info in _get_inspect_plugin().get_all_plugins().output.items()
-        if plugin_name in manifests
-    }
+    return sorted([mf.component_name for mf in get_manifests(Plugin)])
 
 
 def get_all_backends():
-    manifests = {mf.component_name for mf in get_manifests(Backend)}
-    return {
-        backend_name: backend_info
-        for backend_name, backend_info in _get_inspect_plugin().get_all_backends().output.items()
-        if backend_name in manifests
-    }
+    return sorted([mf.component_name for mf in get_manifests(Backend)])
 
 
 def get_all_events():
@@ -38,142 +29,122 @@ def get_all_responses():
     return _get_inspect_plugin().get_all_responses().output
 
 
-# noinspection DuplicatedCode
+def _generate_components_doc(
+    index_name: str,
+    package_name: str,
+    components: Iterable[str],
+    doc_dir: Optional[str] = None,
+):
+    if not doc_dir:
+        doc_dir = index_name
+
+    index_file = os.path.join(
+        os.path.dirname(os.path.abspath(__file__)),
+        'docs',
+        'source',
+        f'{index_name}.rst',
+    )
+    docs_dir = os.path.join(
+        os.path.dirname(os.path.abspath(__file__)),
+        'docs',
+        'source',
+        'platypush',
+        doc_dir,
+    )
+
+    for comp in components:
+        comp_file = os.path.join(docs_dir, comp + '.rst')
+        if not os.path.exists(comp_file):
+            comp = f'platypush.{package_name}.{comp}'
+            header = '``' + '.'.join(comp.split('.')[2:]) + '``'
+            divider = '=' * len(header)
+            body = f'\n.. automodule:: {comp}\n    :members:\n'
+            out = '\n'.join([header, divider, body])
+
+            with open(comp_file, 'w') as f:
+                f.write(out)
+
+    with open(index_file, 'w') as f:
+        f.write(
+            f'''
+{index_name.title()}
+{''.join(['='] * len(index_name))}
+
+.. toctree::
+    :maxdepth: 1
+    :caption: {index_name.title()}:
+
+'''
+        )
+
+        for comp in components:
+            f.write(f'    platypush/{doc_dir}/{comp}.rst\n')
+
+    _cleanup_removed_components_docs(docs_dir, components)
+
+
+def _cleanup_removed_components_docs(docs_dir: str, components: Iterable[str]):
+    new_components = set(components)
+    existing_files = {
+        os.path.join(root, file)
+        for root, _, files in os.walk(docs_dir)
+        for file in files
+        if file.endswith('.rst')
+    }
+
+    files_to_remove = {
+        file
+        for file in existing_files
+        if os.path.basename(file).removesuffix('.rst') not in new_components
+    }
+
+    for file in files_to_remove:
+        print(f'Removing unlinked component {file}')
+        os.unlink(file)
+
+
 def generate_plugins_doc():
-    plugins_index = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'docs', 'source', 'plugins.rst')
-    plugins_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'docs', 'source', 'platypush', 'plugins')
-    all_plugins = sorted(plugin for plugin in get_all_plugins().keys())
-
-    for plugin in all_plugins:
-        plugin_file = os.path.join(plugins_dir, plugin + '.rst')
-        if not os.path.exists(plugin_file):
-            plugin = 'platypush.plugins.' + plugin
-            header = '``{}``'.format('.'.join(plugin.split('.')[2:]))
-            divider = '=' * len(header)
-            body = '\n.. automodule:: {}\n    :members:\n'.format(plugin)
-            out = '\n'.join([header, divider, body])
-
-            with open(plugin_file, 'w') as f:
-                f.write(out)
-
-    with open(plugins_index, 'w') as f:
-        f.write('''
-Plugins
-=======
-
-.. toctree::
-    :maxdepth: 2
-    :caption: Plugins:
-
-''')
-
-        for plugin in all_plugins:
-            f.write('    platypush/plugins/' + plugin + '.rst\n')
+    _generate_components_doc(
+        index_name='plugins', package_name='plugins', components=get_all_plugins()
+    )
 
 
-# noinspection DuplicatedCode
 def generate_backends_doc():
-    backends_index = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'docs', 'source', 'backends.rst')
-    backends_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'docs', 'source', 'platypush', 'backend')
-    all_backends = sorted(backend for backend in get_all_backends().keys())
-
-    for backend in all_backends:
-        backend_file = os.path.join(backends_dir, backend + '.rst')
-        if not os.path.exists(backend_file):
-            backend = 'platypush.backend.' + backend
-            header = '``{}``'.format('.'.join(backend.split('.')[2:]))
-            divider = '=' * len(header)
-            body = '\n.. automodule:: {}\n    :members:\n'.format(backend)
-            out = '\n'.join([header, divider, body])
-
-            with open(backend_file, 'w') as f:
-                f.write(out)
-
-    with open(backends_index, 'w') as f:
-        f.write('''
-Backends
-========
-
-.. toctree::
-    :maxdepth: 2
-    :caption: Backends:
-
-''')
-
-        for backend in all_backends:
-            f.write('    platypush/backend/' + backend + '.rst\n')
+    _generate_components_doc(
+        index_name='backends',
+        package_name='backend',
+        components=get_all_backends(),
+        doc_dir='backend',
+    )
 
 
-# noinspection DuplicatedCode
 def generate_events_doc():
-    from platypush.message import event as event_module
-    events_index = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'docs', 'source', 'events.rst')
-    events_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'docs', 'source', 'platypush', 'events')
-    all_events = sorted(event for event in get_all_events().keys() if event)
-
-    for event in all_events:
-        event_file = os.path.join(events_dir, event + '.rst')
-        if not os.path.exists(event_file):
-            header = '``{}``'.format(event)
-            divider = '=' * len(header)
-            body = '\n.. automodule:: {}.{}\n    :members:\n'.format(event_module.__name__, event)
-            out = '\n'.join([header, divider, body])
-
-            with open(event_file, 'w') as f:
-                f.write(out)
-
-    with open(events_index, 'w') as f:
-        f.write('''
-Events
-======
-
-.. toctree::
-    :maxdepth: 2
-    :caption: Events:
-
-''')
-
-        for event in all_events:
-            f.write('    platypush/events/' + event + '.rst\n')
+    _generate_components_doc(
+        index_name='events',
+        package_name='message.event',
+        components=sorted(event for event in get_all_events().keys() if event),
+    )
 
 
-# noinspection DuplicatedCode
 def generate_responses_doc():
-    from platypush.message import response as response_module
-    responses_index = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'docs', 'source', 'responses.rst')
-    responses_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'docs', 'source', 'platypush', 'responses')
-    all_responses = sorted(response for response in get_all_responses().keys() if response)
-
-    for response in all_responses:
-        response_file = os.path.join(responses_dir, response + '.rst')
-        if not os.path.exists(response_file):
-            header = '``{}``'.format(response)
-            divider = '=' * len(header)
-            body = '\n.. automodule:: {}.{}\n    :members:\n'.format(response_module.__name__, response)
-            out = '\n'.join([header, divider, body])
-
-            with open(response_file, 'w') as f:
-                f.write(out)
-
-    with open(responses_index, 'w') as f:
-        f.write('''
-Responses
-=========
-
-.. toctree::
-    :maxdepth: 2
-    :caption: Responses:
-
-''')
-
-        for response in all_responses:
-            f.write('    platypush/responses/' + response + '.rst\n')
+    _generate_components_doc(
+        index_name='responses',
+        package_name='message.response',
+        components=sorted(
+            response for response in get_all_responses().keys() if response
+        ),
+    )
 
 
+def main():
     generate_plugins_doc()
     generate_backends_doc()
     generate_events_doc()
     generate_responses_doc()
 
 
+if __name__ == '__main__':
+    main()
+
+
 # vim:sw=4:ts=4:et:

platypush/__init__.py

@@ -9,11 +9,13 @@ import argparse
 import logging
 import os
 import sys
+from typing import Optional
 
 from .bus.redis import RedisBus
 from .config import Config
 from .context import register_backends, register_plugins
 from .cron.scheduler import CronScheduler
+from .entities import init_entities_engine, EntitiesEngine
 from .event.processor import EventProcessor
 from .logger import Logger
 from .message.event import Event
@@ -23,9 +25,9 @@ from .message.response import Response
 from .utils import set_thread_name, get_enabled_plugins
 
 __author__ = 'Fabio Manganiello <info@fabiomanganiello.com>'
-__version__ = '0.23.3'
+__version__ = '0.24.5'
 
-logger = logging.getLogger('platypush')
+log = logging.getLogger('platypush')
 
 
 class Daemon:
@@ -59,6 +61,7 @@ class Daemon:
         no_capture_stdout=False,
         no_capture_stderr=False,
         redis_queue=None,
+        verbose=False,
     ):
         """
         Constructor
@@ -74,6 +77,7 @@ class Daemon:
             no_capture_stderr -- Set to true if you want to disable the stderr
                                  capture by the logging system
             redis_queue -- Name of the (Redis) queue used for dispatching messages (default: platypush/bus).
+            verbose -- Enable debug/verbose logging, overriding the stored configuration (default: False).
         """
 
         if pidfile:
@@ -84,7 +88,10 @@ class Daemon:
         self.redis_queue = redis_queue or self._default_redis_queue
         self.config_file = config_file
         Config.init(self.config_file)
-        logging.basicConfig(**Config.get('logging'))
+        logging_conf = Config.get('logging') or {}
+        if verbose:
+            logging_conf['level'] = logging.DEBUG
+        logging.basicConfig(**logging_conf)
 
         redis_conf = Config.get('backend.redis') or {}
         self.bus = RedisBus(
@@ -96,6 +103,7 @@ class Daemon:
         self.no_capture_stdout = no_capture_stdout
         self.no_capture_stderr = no_capture_stderr
         self.event_processor = EventProcessor()
+        self.entities_engine: Optional[EntitiesEngine] = None
         self.requests_to_process = requests_to_process
         self.processed_requests = 0
         self.cron_scheduler = None
@@ -116,6 +124,21 @@ class Daemon:
             default=None,
             help=cls.config_file.__doc__,
         )
+        parser.add_argument(
+            '--version',
+            dest='version',
+            required=False,
+            action='store_true',
+            help="Print the current version and exit",
+        )
+        parser.add_argument(
+            '--verbose',
+            '-v',
+            dest='verbose',
+            required=False,
+            action='store_true',
+            help="Enable verbose/debug logging",
+        )
         parser.add_argument(
             '--pidfile',
             '-P',
@@ -154,12 +177,18 @@ class Daemon:
         )
 
         opts, args = parser.parse_known_args(args)
+
+        if opts.version:
+            print(__version__)
+            sys.exit(0)
+
         return cls(
             config_file=opts.config,
             pidfile=opts.pidfile,
             no_capture_stdout=opts.no_capture_stdout,
             no_capture_stderr=opts.no_capture_stderr,
             redis_queue=opts.redis_queue,
+            verbose=opts.verbose,
         )
 
     def on_message(self):
@@ -178,7 +207,7 @@ class Daemon:
                 try:
                     msg.execute(n_tries=self.n_tries)
                 except PermissionError:
-                    logger.info('Dropped unauthorized request: {}'.format(msg))
+                    log.info('Dropped unauthorized request: {}'.format(msg))
 
                 self.processed_requests += 1
                 if (
@@ -187,10 +216,9 @@ class Daemon:
                 ):
                     self.stop_app()
             elif isinstance(msg, Response):
-                logger.info('Received response: {}'.format(msg))
+                msg.log()
             elif isinstance(msg, Event):
-                if not msg.disable_logging:
-                    logger.info('Received event: {}'.format(msg))
+                msg.log()
                 self.event_processor.process_event(msg)
 
         return _f
@@ -199,6 +227,7 @@ class Daemon:
         """Stops the backends and the bus"""
         from .plugins import RunnablePlugin
 
+        if self.backends:
             for backend in self.backends.values():
                 backend.stop()
 
@@ -206,19 +235,27 @@ class Daemon:
             if isinstance(plugin, RunnablePlugin):
                 plugin.stop()
 
+        if self.bus:
             self.bus.stop()
+            self.bus = None
+
         if self.cron_scheduler:
             self.cron_scheduler.stop()
+            self.cron_scheduler = None
+
+        if self.entities_engine:
+            self.entities_engine.stop()
+            self.entities_engine = None
 
     def run(self):
         """Start the daemon"""
         if not self.no_capture_stdout:
-            sys.stdout = Logger(logger.info)
+            sys.stdout = Logger(log.info)
         if not self.no_capture_stderr:
-            sys.stderr = Logger(logger.warning)
+            sys.stderr = Logger(log.warning)
 
         set_thread_name('platypush')
-        logger.info('---- Starting platypush v.{}'.format(__version__))
+        log.info('---- Starting platypush v.{}'.format(__version__))
 
         # Initialize the backends and link them to the bus
         self.backends = register_backends(bus=self.bus, global_scope=True)
@@ -230,18 +267,22 @@ class Daemon:
         # Initialize the plugins
         register_plugins(bus=self.bus)
 
+        # Initialize the entities engine
+        self.entities_engine = init_entities_engine()
+
         # Start the cron scheduler
         if Config.get_cronjobs():
             self.cron_scheduler = CronScheduler(jobs=Config.get_cronjobs())
             self.cron_scheduler.start()
 
+        assert self.bus, 'The bus is not running'
         self.bus.post(ApplicationStartedEvent())
 
         # Poll for messages on the bus
         try:
             self.bus.poll()
         except KeyboardInterrupt:
-            logger.info('SIGINT received, terminating application')
+            log.info('SIGINT received, terminating application')
         finally:
             self.stop_app()
 

platypush/backend/assistant/google/__init__.py

@@ -8,11 +8,22 @@ import os
 import time
 
 from platypush.backend.assistant import AssistantBackend
-from platypush.message.event.assistant import \
-    ConversationStartEvent, ConversationEndEvent, ConversationTimeoutEvent, \
-    ResponseEvent, NoResponseEvent, SpeechRecognizedEvent, AlarmStartedEvent, \
-    AlarmEndEvent, TimerStartedEvent, TimerEndEvent, AlertStartedEvent, \
-    AlertEndEvent, MicMutedEvent, MicUnmutedEvent
+from platypush.message.event.assistant import (
+    ConversationStartEvent,
+    ConversationEndEvent,
+    ConversationTimeoutEvent,
+    ResponseEvent,
+    NoResponseEvent,
+    SpeechRecognizedEvent,
+    AlarmStartedEvent,
+    AlarmEndEvent,
+    TimerStartedEvent,
+    TimerEndEvent,
+    AlertStartedEvent,
+    AlertEndEvent,
+    MicMutedEvent,
+    MicUnmutedEvent,
+)
 
 
 class AssistantGoogleBackend(AssistantBackend):
@@ -57,22 +68,30 @@ class AssistantGoogleBackend(AssistantBackend):
 
         * **google-assistant-library** (``pip install google-assistant-library``)
         * **google-assistant-sdk[samples]** (``pip install google-assistant-sdk[samples]``)
+        * **google-auth** (``pip install google-auth``)
+
     """
 
-    def __init__(self,
-                 credentials_file=os.path.join(
-                     os.path.expanduser('~/.config'),
-                     'google-oauthlib-tool', 'credentials.json'),
-                 device_model_id='Platypush', **kwargs):
+    _default_credentials_file = os.path.join(
+        os.path.expanduser('~/.config'), 'google-oauthlib-tool', 'credentials.json'
+    )
+
+    def __init__(
+        self,
+        credentials_file=_default_credentials_file,
+        device_model_id='Platypush',
+        **kwargs
+    ):
         """
-        :param credentials_file: Path to the Google OAuth credentials file \
-            (default: ~/.config/google-oauthlib-tool/credentials.json). \
-            See https://developers.google.com/assistant/sdk/guides/library/python/embed/install-sample#generate_credentials \
+        :param credentials_file: Path to the Google OAuth credentials file
+            (default: ~/.config/google-oauthlib-tool/credentials.json).
+            See
+            https://developers.google.com/assistant/sdk/guides/library/python/embed/install-sample#generate_credentials
             for instructions to get your own credentials file.
 
         :type credentials_file: str
 
-        :param device_model_id: Device model ID to use for the assistant \
+        :param device_model_id: Device model ID to use for the assistant
             (default: Platypush)
         :type device_model_id: str
         """
@@ -102,17 +121,23 @@ class AssistantGoogleBackend(AssistantBackend):
             self.bus.post(ConversationTimeoutEvent(assistant=self))
         elif event.type == EventType.ON_NO_RESPONSE:
             self.bus.post(NoResponseEvent(assistant=self))
-        elif hasattr(EventType, 'ON_RENDER_RESPONSE') and \
-                event.type == EventType.ON_RENDER_RESPONSE:
-            self.bus.post(ResponseEvent(assistant=self, response_text=event.args.get('text')))
+        elif (
+            hasattr(EventType, 'ON_RENDER_RESPONSE')
+            and event.type == EventType.ON_RENDER_RESPONSE
+        ):
+            self.bus.post(
+                ResponseEvent(assistant=self, response_text=event.args.get('text'))
+            )
             tts, args = self._get_tts_plugin()
 
             if tts and 'text' in event.args:
                 self.stop_conversation()
                 tts.say(text=event.args['text'], **args)
-        elif hasattr(EventType, 'ON_RESPONDING_STARTED') and \
-                event.type == EventType.ON_RESPONDING_STARTED and \
-                event.args.get('is_error_response', False) is True:
+        elif (
+            hasattr(EventType, 'ON_RESPONDING_STARTED')
+            and event.type == EventType.ON_RESPONDING_STARTED
+            and event.args.get('is_error_response', False) is True
+        ):
             self.logger.warning('Assistant response error')
         elif event.type == EventType.ON_RECOGNIZING_SPEECH_FINISHED:
             phrase = event.args['text'].lower().strip()
@@ -177,7 +202,9 @@ class AssistantGoogleBackend(AssistantBackend):
         super().run()
 
         with open(self.credentials_file, 'r') as f:
-            self.credentials = google.oauth2.credentials.Credentials(token=None, **json.load(f))
+            self.credentials = google.oauth2.credentials.Credentials(
+                token=None, **json.load(f)
+            )
 
         while not self.should_stop():
             self._has_error = False
@@ -186,12 +213,16 @@ class AssistantGoogleBackend(AssistantBackend):
                 self.assistant = assistant
                 for event in assistant.start():
                     if not self.is_detecting():
-                        self.logger.info('Assistant event received but detection is currently paused')
+                        self.logger.info(
+                            'Assistant event received but detection is currently paused'
+                        )
                         continue
 
                     self._process_event(event)
                     if self._has_error:
-                        self.logger.info('Restarting the assistant after an unrecoverable error')
+                        self.logger.info(
+                            'Restarting the assistant after an unrecoverable error'
+                        )
                         time.sleep(5)
                         break
 

platypush/backend/assistant/google/manifest.yaml

@@ -22,5 +22,6 @@ manifest:
     pip:
     - google-assistant-library
     - google-assistant-sdk[samples]
+    - google-auth
   package: platypush.backend.assistant.google
   type: backend

platypush/backend/bluetooth/__init__.py

@@ -1,99 +0,0 @@
-import os
-import time
-
-# noinspection PyPackageRequirements
-from PyOBEX import headers, requests, responses
-# noinspection PyPackageRequirements
-from PyOBEX.server import Server
-
-from platypush.backend import Backend
-from platypush.message.event.bluetooth import BluetoothDeviceConnectedEvent, BluetoothFileReceivedEvent, \
-    BluetoothDeviceDisconnectedEvent, BluetoothFilePutRequestEvent
-
-
-class BluetoothBackend(Backend, Server):
-    _sleep_on_error = 10.0
-
-    def __init__(self, address: str = '', port: int = None, directory: str = None, whitelisted_addresses=None,
-                 **kwargs):
-        Backend.__init__(self, **kwargs)
-        Server.__init__(self, address=address)
-        self.port = port
-        self.directory = os.path.join(os.path.expanduser(directory))
-        self.whitelisted_addresses = whitelisted_addresses or []
-        self._sock = None
-
-    def run(self):
-        self.logger.info('Starting bluetooth service [address={}] [port={}]'.format(
-            self.address, self.port))
-
-        while not self.should_stop():
-            try:
-                # noinspection PyArgumentList
-                self._sock = self.start_service(self.port)
-                self.serve(self._sock)
-            except Exception as e:
-                self.logger.error('Error on bluetooth connection [address={}] [port={}]: {}'.format(
-                    self.address, self.port, str(e)))
-                time.sleep(self._sleep_on_error)
-            finally:
-                self.stop()
-
-    def stop(self):
-        if self._sock:
-            self.stop_service(self._sock)
-        self._sock = None
-
-    def put(self, socket, request):
-        name = ""
-        body = ""
-
-        while True:
-            for header in request.header_data:
-                if isinstance(header, headers.Name):
-                    name = header.decode()
-                    self.logger.info("Receiving {}".format(name))
-                elif isinstance(header, headers.Length):
-                    length = header.decode()
-                    self.logger.info("Content length: {} bytes".format(length))
-                elif isinstance(header, headers.Body):
-                    body += header.decode()
-                elif isinstance(header, headers.End_Of_Body):
-                    body += header.decode()
-
-            if request.is_final():
-                break
-
-            # Ask for more data.
-            Server.send_response(self, socket, responses.Continue())
-
-            # Get the next part of the data.
-            request = self.request_handler.decode(socket)
-
-        Server.send_response(self, socket, responses.Success())
-        name = os.path.basename(name.strip("\x00"))
-        path = os.path.join(self.directory, name)
-
-        self.logger.info("Writing file {}" .format(path))
-        open(path, "wb").write(body.encode())
-        self.bus.post(BluetoothFileReceivedEvent(path=path))
-
-    def process_request(self, connection, request, *address):
-        if isinstance(request, requests.Connect):
-            self.connect(connection, request)
-            self.bus.post(BluetoothDeviceConnectedEvent(address=address[0], port=address[1]))
-        elif isinstance(request, requests.Disconnect):
-            self.disconnect(connection, request)
-            self.bus.post(BluetoothDeviceDisconnectedEvent(address=address[0], port=address[1]))
-        elif isinstance(request, requests.Put):
-            self.bus.post(BluetoothFilePutRequestEvent(address=address[0], port=address[1]))
-            self.put(connection, request)
-        else:
-            self._reject(connection)
-            self.bus.post(BluetoothFilePutRequestEvent(address=address[0], port=address[1]))
-
-    def accept_connection(self, address, port):
-        return address in self.whitelisted_addresses
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/bluetooth/fileserver/__init__.py

@@ -1,92 +0,0 @@
-import os
-import stat
-
-# noinspection PyPackageRequirements
-from PyOBEX import requests, responses, headers
-# noinspection PyPackageRequirements
-from PyOBEX.server import BrowserServer
-
-from platypush.backend.bluetooth import BluetoothBackend
-from platypush.message.event.bluetooth import BluetoothFileGetRequestEvent
-
-
-class BluetoothFileserverBackend(BluetoothBackend, BrowserServer):
-    """
-    Bluetooth OBEX file server.
-    Enable it to allow bluetooth devices to browse files on this machine.
-
-    If you run platypush as a non-root user (and you should) then you to change the group owner of the
-    service discovery protocol file (/var/run/sdp) and add your user to that group. See
-    `here <https://stackoverflow.com/questions/34599703/rfcomm-bluetooth-permission-denied-error-raspberry-pi>`_
-    for details.
-
-    Requires:
-
-        * **pybluez** (``pip install pybluez``)
-        * **pyobex** (``pip install git+https://github.com/BlackLight/PyOBEX``)
-
-    """
-
-    def __init__(self, port: int, address: str = '', directory: str = os.path.expanduser('~'),
-                 whitelisted_addresses: list = None, **kwargs):
-        """
-        :param port: Bluetooth listen port
-        :param address: Bluetooth address to bind the server to (default: any)
-        :param directory: Directory to share (default: HOME directory)
-        :param whitelisted_addresses: If set then only accept connections from the listed device addresses
-        """
-        BluetoothBackend.__init__(self, address=address, port=port, directory=directory,
-                                  whitelisted_addresses=whitelisted_addresses, **kwargs)
-
-        if not os.path.isdir(self.directory):
-            raise FileNotFoundError(self.directory)
-
-    def process_request(self, socket, request, *address):
-        if isinstance(request, requests.Get):
-            self.bus.post(BluetoothFileGetRequestEvent(address=address[0], port=address[1]))
-            self.get(socket, request)
-        else:
-            super().process_request(socket, request, *address)
-
-    def get(self, socket, request):
-        name = ""
-        req_type = ""
-
-        for header in request.header_data:
-            if isinstance(header, headers.Name):
-                name = header.decode().strip("\x00")
-                self.logger.info("Receiving request for {}".format(name))
-            elif isinstance(header, headers.Type):
-                req_type = header.decode().strip("\x00")
-                self.logger.info("Request type: {}".format(req_type))
-
-        path = os.path.abspath(os.path.join(self.directory, name))
-
-        if os.path.isdir(path) or req_type == "x-obex/folder-listing":
-            if path.startswith(self.directory):
-                filelist = os.listdir(path)
-                s = '<?xml version="1.0"?>\n<folder-listing>\n'
-
-                for i in filelist:
-                    objpath = os.path.join(path, i)
-                    if os.path.isdir(objpath):
-                        s += '  <folder name="{}" created="{}" />'.format(i, os.stat(objpath)[stat.ST_CTIME])
-                    else:
-                        s += '  <file name="{}" created="{}" size="{}" />'.format(
-                            i, os.stat(objpath)[stat.ST_CTIME], os.stat(objpath)[stat.ST_SIZE])
-
-                s += "</folder-listing>\n"
-                self.logger.debug('Bluetooth get XML output:\n' + s)
-
-                response = responses.Success()
-                response_headers = [headers.Name(name.encode("utf8")),
-                                    headers.Length(len(s)),
-                                    headers.Body(s.encode("utf8"))]
-                BrowserServer.send_response(self, socket, response, response_headers)
-            else:
-                self._reject(socket)
-        else:
-            self._reject(socket)
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/bluetooth/fileserver/manifest.yaml

@@ -1,8 +0,0 @@
-manifest:
-  events: {}
-  install:
-    pip:
-    - pybluez
-    - pyobex
-  package: platypush.backend.bluetooth.fileserver
-  type: backend

platypush/backend/bluetooth/pushserver/__init__.py

@@ -1,47 +0,0 @@
-import os
-
-# noinspection PyPackageRequirements
-from PyOBEX.server import PushServer
-
-from platypush.backend.bluetooth import BluetoothBackend
-
-
-class BluetoothPushserverBackend(BluetoothBackend, PushServer):
-    """
-    Bluetooth OBEX push server.
-    Enable it to allow bluetooth file transfers from other devices.
-
-    If you run platypush as a non-root user (and you should) then you to change the group owner of the
-    service discovery protocol file (/var/run/sdp) and add your user to that group. See
-    `here <https://stackoverflow.com/questions/34599703/rfcomm-bluetooth-permission-denied-error-raspberry-pi>`_
-    for details.
-
-    Requires:
-
-        * **pybluez** (``pip install pybluez``)
-        * **pyobex** (``pip install git+https://github.com/BlackLight/PyOBEX``)
-
-    """
-
-    _sleep_on_error = 10.0
-
-    def __init__(self, port: int, address: str = '',
-                 directory: str = os.path.join(os.path.expanduser('~'), 'bluetooth'),
-                 whitelisted_addresses: list = None, **kwargs):
-        """
-        :param port: Bluetooth listen port
-        :param address: Bluetooth address to bind the server to (default: any)
-        :param directory: Destination directory where files will be downloaded (default: ~/bluetooth)
-        :param whitelisted_addresses: If set then only accept connections from the listed device addresses
-        """
-        BluetoothBackend.__init__(self, address=address, port=port, directory=directory,
-                                  whitelisted_addresses=whitelisted_addresses, **kwargs)
-
-    def run(self):
-        if not os.path.isdir(self.directory):
-            os.makedirs(self.directory, exist_ok=True)
-
-        super().run()
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/bluetooth/pushserver/manifest.yaml

@@ -1,8 +0,0 @@
-manifest:
-  events: {}
-  install:
-    pip:
-    - pybluez
-    - pyobex
-  package: platypush.backend.bluetooth.pushserver
-  type: backend

platypush/backend/bluetooth/scanner/__init__.py

@@ -1,109 +0,0 @@
-import time
-from threading import Thread, RLock
-from typing import Dict, Optional, List
-
-from platypush.backend.sensor import SensorBackend
-from platypush.context import get_plugin
-from platypush.message.event.bluetooth import BluetoothDeviceFoundEvent, BluetoothDeviceLostEvent
-
-
-class BluetoothScannerBackend(SensorBackend):
-    """
-    This backend periodically scans for available bluetooth devices and returns events when a devices enter or exits
-    the range.
-
-    Triggers:
-
-        * :class:`platypush.message.event.bluetooth.BluetoothDeviceFoundEvent` when a new bluetooth device is found.
-        * :class:`platypush.message.event.bluetooth.BluetoothDeviceLostEvent` when a bluetooth device is lost.
-
-    Requires:
-
-        * The :class:`platypush.plugins.bluetooth.BluetoothPlugin` plugin working.
-
-    """
-
-    def __init__(self, device_id: Optional[int] = None, scan_duration: int = 10,
-                 track_devices: Optional[List[str]] = None, **kwargs):
-        """
-        :param device_id: Bluetooth adapter ID to use (default configured on the ``bluetooth`` plugin if None).
-        :param scan_duration:  How long the scan should run (default: 10 seconds).
-        :param track_devices: List of addresses of devices to actively track, even if they aren't discoverable.
-        """
-        super().__init__(plugin='bluetooth', plugin_args={
-            'device_id': device_id,
-            'duration': scan_duration,
-        }, **kwargs)
-
-        self._last_seen_devices = {}
-        self._tracking_thread: Optional[Thread] = None
-        self._bt_lock = RLock()
-        self.track_devices = set(track_devices or [])
-        self.scan_duration = scan_duration
-
-    def _add_last_seen_device(self, dev):
-        addr = dev.pop('addr')
-        if addr not in self._last_seen_devices:
-            self.bus.post(BluetoothDeviceFoundEvent(address=addr, **dev))
-        self._last_seen_devices[addr] = {'addr': addr, **dev}
-
-    def _remove_last_seen_device(self, addr: str):
-        dev = self._last_seen_devices.get(addr)
-        if not dev:
-            return
-
-        self.bus.post(BluetoothDeviceLostEvent(address=addr, **dev))
-        del self._last_seen_devices[addr]
-
-    def _addr_tracker(self, addr):
-        with self._bt_lock:
-            name = get_plugin('bluetooth').lookup_name(addr, timeout=self.scan_duration).name
-
-        if name is None:
-            self._remove_last_seen_device(addr)
-        else:
-            self._add_last_seen_device({'addr': addr, 'name': name})
-
-    def _bt_tracker(self):
-        self.logger.info('Starting Bluetooth tracker')
-        while not self.should_stop():
-            trackers = []
-            for addr in self.track_devices:
-                tracker = Thread(target=self._addr_tracker, args=(addr,))
-                tracker.start()
-                trackers.append(tracker)
-
-            for tracker in trackers:
-                tracker.join(timeout=self.scan_duration)
-
-            time.sleep(self.scan_duration)
-
-        self.logger.info('Bluetooth tracker stopped')
-
-    def get_measurement(self):
-        with self._bt_lock:
-            return super().get_measurement()
-
-    def process_data(  # lgtm [py/inheritance/signature-mismatch]
-        self, data: Dict[str, dict], new_data: Optional[Dict[str, dict]] = None, **_
-    ):
-        for addr, dev in data.items():
-            self._add_last_seen_device(dev)
-
-        for addr, dev in self._last_seen_devices.copy().items():
-            if addr not in data and addr not in self.track_devices:
-                self._remove_last_seen_device(addr)
-
-    def run(self):
-        self._tracking_thread = Thread(target=self._bt_tracker)
-        self._tracking_thread.start()
-        super().run()
-
-    def on_stop(self):
-        super().on_stop()
-        if self._tracking_thread and self._tracking_thread.is_alive():
-            self.logger.info('Waiting for the Bluetooth tracking thread to stop')
-            self._tracking_thread.join(timeout=self.scan_duration)
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/bluetooth/scanner/ble/__init__.py

@@ -1,33 +0,0 @@
-from typing import Optional
-
-from platypush.backend.bluetooth.scanner import BluetoothScannerBackend
-
-
-class BluetoothBleScannerBackend(BluetoothScannerBackend):
-    """
-    This backend periodically scans for available bluetooth low-energy devices and returns events when a devices enter
-    or exits the range.
-
-    Triggers:
-
-        * :class:`platypush.message.event.bluetooth.BluetoothDeviceFoundEvent` when a new bluetooth device is found.
-        * :class:`platypush.message.event.bluetooth.BluetoothDeviceLostEvent` when a bluetooth device is lost.
-
-    Requires:
-
-        * The :class:`platypush.plugins.bluetooth.BluetoothBlePlugin` plugin working.
-
-    """
-
-    def __init__(self, interface: Optional[int] = None, scan_duration: int = 10, **kwargs):
-        """
-        :param interface: Bluetooth adapter name to use (default configured on the ``bluetooth.ble`` plugin if None).
-        :param scan_duration:  How long the scan should run (default: 10 seconds).
-        """
-        super().__init__(plugin='bluetooth.ble', plugin_args={
-            'interface': interface,
-            'duration': scan_duration,
-        }, **kwargs)
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/bluetooth/scanner/ble/manifest.yaml

@@ -1,10 +0,0 @@
-manifest:
-  events:
-    platypush.message.event.bluetooth.BluetoothDeviceFoundEvent: when a new bluetooth
-      device is found.
-    platypush.message.event.bluetooth.BluetoothDeviceLostEvent: when a bluetooth device
-      is lost.
-  install:
-    pip: []
-  package: platypush.backend.bluetooth.scanner.ble
-  type: backend

platypush/backend/bluetooth/scanner/manifest.yaml

@@ -1,10 +0,0 @@
-manifest:
-  events:
-    platypush.message.event.bluetooth.BluetoothDeviceFoundEvent: when a new bluetooth
-      device is found.
-    platypush.message.event.bluetooth.BluetoothDeviceLostEvent: when a bluetooth device
-      is lost.
-  install:
-    pip: []
-  package: platypush.backend.bluetooth.scanner
-  type: backend

platypush/backend/camera/pi/manifest.yaml

@@ -3,5 +3,7 @@ manifest:
   install:
     pip:
     - picamera
+    - numpy
+    - Pillow
   package: platypush.backend.camera.pi
   type: backend

platypush/backend/covid19/__init__.py

@@ -4,9 +4,9 @@ from typing import Optional, Union, List, Dict, Any
 
 from sqlalchemy import create_engine, Column, Integer, String, DateTime
 from sqlalchemy.orm import sessionmaker, scoped_session
-from sqlalchemy.ext.declarative import declarative_base
 
 from platypush.backend import Backend
+from platypush.common.db import declarative_base
 from platypush.config import Config
 from platypush.context import get_plugin
 from platypush.message.event.covid19 import Covid19UpdateEvent
@@ -20,7 +20,7 @@ class Covid19Update(Base):
     """Models the Covid19Data table"""
 
     __tablename__ = 'covid19data'
-    __table_args__ = ({'sqlite_autoincrement': True})
+    __table_args__ = {'sqlite_autoincrement': True}
 
     country = Column(String, primary_key=True)
     confirmed = Column(Integer, nullable=False, default=0)
@@ -40,7 +40,12 @@ class Covid19Backend(Backend):
     """
 
     # noinspection PyProtectedMember
-    def __init__(self, country: Optional[Union[str, List[str]]], poll_seconds: Optional[float] = 3600.0, **kwargs):
+    def __init__(
+        self,
+        country: Optional[Union[str, List[str]]],
+        poll_seconds: Optional[float] = 3600.0,
+        **kwargs
+    ):
         """
         :param country: Default country (or list of countries) to retrieve the stats for. It can either be the full
             country name or the country code. Special values:
@@ -56,7 +61,9 @@ class Covid19Backend(Backend):
         super().__init__(poll_seconds=poll_seconds, **kwargs)
         self._plugin: Covid19Plugin = get_plugin('covid19')
         self.country: List[str] = self._plugin._get_countries(country)
-        self.workdir = os.path.join(os.path.expanduser(Config.get('workdir')), 'covid19')
+        self.workdir = os.path.join(
+            os.path.expanduser(Config.get('workdir')), 'covid19'
+        )
         self.dbfile = os.path.join(self.workdir, 'data.db')
         os.makedirs(self.workdir, exist_ok=True)
 
@@ -67,22 +74,30 @@ class Covid19Backend(Backend):
         self.logger.info('Stopped Covid19 backend')
 
     def _process_update(self, summary: Dict[str, Any], session: Session):
-        update_time = datetime.datetime.fromisoformat(summary['Date'].replace('Z', '+00:00'))
+        update_time = datetime.datetime.fromisoformat(
+            summary['Date'].replace('Z', '+00:00')
+        )
 
-        self.bus.post(Covid19UpdateEvent(
+        self.bus.post(
+            Covid19UpdateEvent(
                 country=summary['Country'],
                 country_code=summary['CountryCode'],
                 confirmed=summary['TotalConfirmed'],
                 deaths=summary['TotalDeaths'],
                 recovered=summary['TotalRecovered'],
                 update_time=update_time,
-        ))
+            )
+        )
 
-        session.merge(Covid19Update(country=summary['CountryCode'],
+        session.merge(
+            Covid19Update(
+                country=summary['CountryCode'],
                 confirmed=summary['TotalConfirmed'],
                 deaths=summary['TotalDeaths'],
                 recovered=summary['TotalRecovered'],
-                                    last_updated_at=update_time))
+                last_updated_at=update_time,
+            )
+        )
 
     def loop(self):
         # noinspection PyUnresolvedReferences
@@ -90,23 +105,30 @@ class Covid19Backend(Backend):
         if not summaries:
             return
 
-        engine = create_engine('sqlite:///{}'.format(self.dbfile), connect_args={'check_same_thread': False})
+        engine = create_engine(
+            'sqlite:///{}'.format(self.dbfile),
+            connect_args={'check_same_thread': False},
+        )
         Base.metadata.create_all(engine)
         Session.configure(bind=engine)
         session = Session()
 
         last_records = {
             record.country: record
-            for record in session.query(Covid19Update).filter(Covid19Update.country.in_(self.country)).all()
+            for record in session.query(Covid19Update)
+            .filter(Covid19Update.country.in_(self.country))
+            .all()
         }
 
         for summary in summaries:
             country = summary['CountryCode']
             last_record = last_records.get(country)
-            if not last_record or \
-                    summary['TotalConfirmed'] != last_record.confirmed or \
-                    summary['TotalDeaths'] != last_record.deaths or \
-                    summary['TotalRecovered'] != last_record.recovered:
+            if (
+                not last_record
+                or summary['TotalConfirmed'] != last_record.confirmed
+                or summary['TotalDeaths'] != last_record.deaths
+                or summary['TotalRecovered'] != last_record.recovered
+            ):
                 self._process_update(summary=summary, session=session)
 
         session.commit()

platypush/backend/github/__init__.py

@@ -6,15 +6,29 @@ from typing import Optional, List
 
 import requests
 from sqlalchemy import create_engine, Column, String, DateTime
-from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm import sessionmaker, scoped_session
 
 from platypush.backend import Backend
+from platypush.common.db import declarative_base
 from platypush.config import Config
-from platypush.message.event.github import GithubPushEvent, GithubCommitCommentEvent, GithubCreateEvent, \
-    GithubDeleteEvent, GithubEvent, GithubForkEvent, GithubWikiEvent, GithubIssueCommentEvent, GithubIssueEvent, \
-    GithubMemberEvent, GithubPublicEvent, GithubPullRequestEvent, GithubPullRequestReviewCommentEvent, \
-    GithubReleaseEvent, GithubSponsorshipEvent, GithubWatchEvent
+from platypush.message.event.github import (
+    GithubPushEvent,
+    GithubCommitCommentEvent,
+    GithubCreateEvent,
+    GithubDeleteEvent,
+    GithubEvent,
+    GithubForkEvent,
+    GithubWikiEvent,
+    GithubIssueCommentEvent,
+    GithubIssueEvent,
+    GithubMemberEvent,
+    GithubPublicEvent,
+    GithubPullRequestEvent,
+    GithubPullRequestReviewCommentEvent,
+    GithubReleaseEvent,
+    GithubSponsorshipEvent,
+    GithubWatchEvent,
+)
 
 Base = declarative_base()
 Session = scoped_session(sessionmaker())
@@ -71,8 +85,17 @@ class GithubBackend(Backend):
 
     _base_url = 'https://api.github.com'
 
-    def __init__(self, user: str, user_token: str, repos: Optional[List[str]] = None, org: Optional[str] = None,
-                 poll_seconds: int = 60, max_events_per_scan: Optional[int] = 10, *args, **kwargs):
+    def __init__(
+        self,
+        user: str,
+        user_token: str,
+        repos: Optional[List[str]] = None,
+        org: Optional[str] = None,
+        poll_seconds: int = 60,
+        max_events_per_scan: Optional[int] = 10,
+        *args,
+        **kwargs
+    ):
         """
         If neither ``repos`` nor ``org`` is specified then the backend will monitor all new events on user level.
 
@@ -102,11 +125,17 @@ class GithubBackend(Backend):
 
     def _request(self, uri: str, method: str = 'get') -> dict:
         method = getattr(requests, method.lower())
-        return method(self._base_url + uri, auth=(self.user, self.user_token),
-                      headers={'Accept': 'application/vnd.github.v3+json'}).json()
+        return method(
+            self._base_url + uri,
+            auth=(self.user, self.user_token),
+            headers={'Accept': 'application/vnd.github.v3+json'},
+        ).json()
 
     def _init_db(self):
-        engine = create_engine('sqlite:///{}'.format(self.dbfile), connect_args={'check_same_thread': False})
+        engine = create_engine(
+            'sqlite:///{}'.format(self.dbfile),
+            connect_args={'check_same_thread': False},
+        )
         Base.metadata.create_all(engine)
         Session.configure(bind=engine)
 
@@ -128,7 +157,11 @@ class GithubBackend(Backend):
     def _get_last_event_time(self, uri: str):
         with self.db_lock:
             record = self._get_or_create_resource(uri=uri, session=Session())
-            return record.last_updated_at.replace(tzinfo=datetime.timezone.utc) if record.last_updated_at else None
+            return (
+                record.last_updated_at.replace(tzinfo=datetime.timezone.utc)
+                if record.last_updated_at
+                else None
+            )
 
     def _update_last_event_time(self, uri: str, last_updated_at: datetime.datetime):
         with self.db_lock:
@@ -158,9 +191,18 @@ class GithubBackend(Backend):
             'WatchEvent': GithubWatchEvent,
         }
 
-        event_type = event_mapping[event['type']] if event['type'] in event_mapping else GithubEvent
-        return event_type(event_type=event['type'], actor=event['actor'], repo=event.get('repo', {}),
-                          payload=event['payload'], created_at=cls._to_datetime(event['created_at']))
+        event_type = (
+            event_mapping[event['type']]
+            if event['type'] in event_mapping
+            else GithubEvent
+        )
+        return event_type(
+            event_type=event['type'],
+            actor=event['actor'],
+            repo=event.get('repo', {}),
+            payload=event['payload'],
+            created_at=cls._to_datetime(event['created_at']),
+        )
 
     def _events_monitor(self, uri: str, method: str = 'get'):
         def thread():
@@ -175,7 +217,10 @@ class GithubBackend(Backend):
                     fired_events = []
 
                     for event in events:
-                        if self.max_events_per_scan and len(fired_events) >= self.max_events_per_scan:
+                        if (
+                            self.max_events_per_scan
+                            and len(fired_events) >= self.max_events_per_scan
+                        ):
                             break
 
                         event_time = self._to_datetime(event['created_at'])
@@ -189,12 +234,17 @@ class GithubBackend(Backend):
                     for event in fired_events:
                         self.bus.post(event)
 
-                    self._update_last_event_time(uri=uri, last_updated_at=new_last_event_time)
+                    self._update_last_event_time(
+                        uri=uri, last_updated_at=new_last_event_time
+                    )
                 except Exception as e:
-                    self.logger.warning('Encountered exception while fetching events from {}: {}'.format(
-                        uri, str(e)))
+                    self.logger.warning(
+                        'Encountered exception while fetching events from {}: {}'.format(
+                            uri, str(e)
+                        )
+                    )
                     self.logger.exception(e)
-                finally:
+
                 if self.wait_stop(timeout=self.poll_seconds):
                     break
 
@@ -206,12 +256,30 @@ class GithubBackend(Backend):
 
         if self.repos:
             for repo in self.repos:
-                monitors.append(threading.Thread(target=self._events_monitor('/networks/{repo}/events'.format(repo=repo))))
+                monitors.append(
+                    threading.Thread(
+                        target=self._events_monitor(
+                            '/networks/{repo}/events'.format(repo=repo)
+                        )
+                    )
+                )
         if self.org:
-            monitors.append(threading.Thread(target=self._events_monitor('/orgs/{org}/events'.format(org=self.org))))
+            monitors.append(
+                threading.Thread(
+                    target=self._events_monitor(
+                        '/orgs/{org}/events'.format(org=self.org)
+                    )
+                )
+            )
 
         if not (self.repos or self.org):
-            monitors.append(threading.Thread(target=self._events_monitor('/users/{user}/events'.format(user=self.user))))
+            monitors.append(
+                threading.Thread(
+                    target=self._events_monitor(
+                        '/users/{user}/events'.format(user=self.user)
+                    )
+                )
+            )
 
         for monitor in monitors:
             monitor.start()
@@ -222,4 +290,5 @@ class GithubBackend(Backend):
 
         self.logger.info('Github backend terminated')
 
+
 # vim:sw=4:ts=4:et:

platypush/backend/http/__init__.py

@@ -1,19 +1,27 @@
+import asyncio
 import os
-import subprocess
+import pathlib
+import secrets
 import threading
 
 from multiprocessing import Process
+from time import time
+from typing import List, Mapping, Optional
+from tornado.httpserver import HTTPServer
 
-try:
-    from websockets.exceptions import ConnectionClosed
-    from websockets import serve as websocket_serve
-except ImportError:
-    from websockets import ConnectionClosed, serve as websocket_serve
+from tornado.netutil import bind_sockets
+from tornado.process import cpu_count, fork_processes
+from tornado.wsgi import WSGIContainer
+from tornado.web import Application, FallbackHandler
 
 from platypush.backend import Backend
 from platypush.backend.http.app import application
-from platypush.context import get_or_create_event_loop
-from platypush.utils import get_ssl_server_context, set_thread_name
+from platypush.backend.http.app.utils import get_ws_routes
+from platypush.backend.http.app.ws.events import events_redis_topic
+
+from platypush.bus.redis import RedisBus
+from platypush.config import Config
+from platypush.utils import get_redis
 
 
 class HttpBackend(Backend):
@@ -27,8 +35,6 @@ class HttpBackend(Backend):
         backend.http:
             # Default HTTP listen port
             port: 8008
-            # Default websocket port
-            websocket_port: 8009
             # External folders that will be exposed over `/resources/<name>`
             resource_dirs:
                 photos: /mnt/hd/photos
@@ -39,9 +45,11 @@ class HttpBackend(Backend):
 
         * To execute Platypush commands via HTTP calls. In order to do so:
 
-            * Register a user to Platypush through the web panel (usually served on ``http://host:8008/``).
+            * Register a user to Platypush through the web panel (usually
+              served on ``http://host:8008/``).
 
-            * Generate a token for your user, either through the web panel (Settings -> Generate Token) or via API:
+            * Generate a token for your user, either through the web panel
+              (Settings -> Generate Token) or via API:
 
                 .. code-block:: shell
 
@@ -64,16 +72,35 @@ class HttpBackend(Backend):
                         }
                       }' http://host:8008/execute
 
-        * To interact with your system (and control plugins and backends) through the Platypush web panel,
-          by default available on ``http://host:8008/``. Any configured plugin that has an available panel
-          plugin will be automatically added to the web panel.
+        * To interact with your system (and control plugins and backends)
+          through the Platypush web panel, by default available on
+          ``http://host:8008/``. Any configured plugin that has an available
+          panel plugin will be automatically added to the web panel.
+
+        * To create asynchronous integrations with Platypush over websockets.
+          Two routes are available:
+
+            * ``/ws/events`` - Subscribe to this websocket to receive the
+              events generated by the application.
+            * ``/ws/requests`` - Subscribe to this websocket to send commands
+              to Platypush and receive the response asynchronously.
+
+          You will have to authenticate your connection to these websockets,
+          just like the ``/execute`` endpoint. In both cases, you can pass the
+          token either via ``Authorization: Bearer``, via the ``token`` query
+          string or body parameter, or leverage ``Authorization: Basic`` with
+          username and password (not advised), or use a valid ``session_token``
+          cookie from an authenticated web panel session.
 
         * To display a fullscreen dashboard with custom widgets.
 
-            * Widgets are available as Vue.js components under ``platypush/backend/http/webapp/src/components/widgets``.
+            * Widgets are available as Vue.js components under
+              ``platypush/backend/http/webapp/src/components/widgets``.
 
-            * Explore their options (some may require some plugins or backends to be configured in order to work) and
-              create a new dashboard template under ``~/.config/platypush/dashboards``- e.g. ``main.xml``:
+            * Explore their options (some may require some plugins or backends
+              to be configured in order to work) and create a new dashboard
+              template under ``~/.config/platypush/dashboards``- e.g.
+              ``main.xml``:
 
                 .. code-block:: xml
 
@@ -109,13 +136,17 @@ class HttpBackend(Backend):
                         </Row>
                     </Dashboard>
 
-            * The dashboard will be accessible under ``http://host:8008/dashboard/<name>``, where ``name=main`` if for
-              example you stored your template under ``~/.config/platypush/dashboards/main.xml``.
+            * The dashboard will be accessible under
+              ``http://host:8008/dashboard/<name>``, where ``name=main`` if for
+              example you stored your template under
+              ``~/.config/platypush/dashboards/main.xml``.
 
-        * To expose custom endpoints that can be called as web hooks by other applications and run some custom logic.
-          All you have to do in this case is to create a hook on a
-          :class:`platypush.message.event.http.hook.WebhookEvent` with the endpoint that you want to expose and store
-          it under e.g. ``~/.config/platypush/scripts/hooks.py``:
+        * To expose custom endpoints that can be called as web hooks by other
+          applications and run some custom logic. All you have to do in this case
+          is to create a hook on a
+          :class:`platypush.message.event.http.hook.WebhookEvent` with the
+          endpoint that you want to expose and store it under e.g.
+          ``~/.config/platypush/scripts/hooks.py``:
 
             .. code-block:: python
 
@@ -140,117 +171,54 @@ class HttpBackend(Backend):
     module can expose lists of routes to the main webapp through the
     ``__routes__`` object (a list of Flask blueprints).
 
-    Security: Access to the endpoints requires at least one user to be registered. Access to the endpoints is regulated
-    in the following ways (with the exception of event hooks, whose logic is up to the user):
+    Security: Access to the endpoints requires at least one user to be
+    registered. Access to the endpoints is regulated in the following ways
+    (with the exception of event hooks, whose logic is up to the user):
 
         * **Simple authentication** - i.e. registered username and password.
-        * **JWT token** provided either over as ``Authorization: Bearer`` header or ``GET`` ``?token=<TOKEN>``
-          parameter. A JWT token can be generated either through the web panel or over the ``/auth`` endpoint.
-        * **Global platform token**, usually configured on the root of the ``config.yaml`` as ``token: <VALUE>``.
-          It can provided either over on the ``X-Token`` header or as a ``GET`` ``?token=<TOKEN>`` parameter.
-        * **Session token**, generated upon login, it can be used to authenticate requests through the ``Cookie`` header
-          (cookie name: ``session_token``).
-
-    Requires:
-
-        * **gunicorn** (``pip install gunicorn``) - optional, to run the Platypush webapp over uWSGI.
-
-    By default the Platypush web server will run in a
-    process spawned on the fly by the HTTP backend. However, being a
-    Flask app, it will serve clients in a single thread and it won't
-    support many features of a full-blown web server. gunicorn allows
-    you to easily spawn the web server in a uWSGI wrapper, separate
-    from the main Platypush daemon, and the uWSGI layer can be easily
-    exposed over an nginx/lighttpd web server.
-
-    Command to run the web server over a gunicorn uWSGI wrapper::
-
-        gunicorn -w <n_workers> -b <bind_address>:8008 platypush.backend.http.uwsgi
+        * **JWT token** provided either over as ``Authorization: Bearer``
+          header or ``GET`` ``?token=<TOKEN>`` parameter. A JWT token can be
+          generated either through the web panel or over the ``/auth``
+          endpoint.
+        * **Global platform token**, usually configured on the root of the
+          ``config.yaml`` as ``token: <VALUE>``. It can provided either over on
+          the ``X-Token`` header or as a ``GET`` ``?token=<TOKEN>`` parameter.
+        * **Session token**, generated upon login, it can be used to
+          authenticate requests through the ``Cookie`` header (cookie name:
+          ``session_token``).
 
     """
 
     _DEFAULT_HTTP_PORT = 8008
-    _DEFAULT_WEBSOCKET_PORT = 8009
+    """The default listen port for the webserver."""
 
     def __init__(
         self,
-        port=_DEFAULT_HTTP_PORT,
-        websocket_port=_DEFAULT_WEBSOCKET_PORT,
-        bind_address='0.0.0.0',
-        disable_websocket=False,
-        resource_dirs=None,
-        ssl_cert=None,
-        ssl_key=None,
-        ssl_cafile=None,
-        ssl_capath=None,
-        maps=None,
-        run_externally=False,
-        uwsgi_args=None,
-        **kwargs
+        port: int = _DEFAULT_HTTP_PORT,
+        bind_address: str = '0.0.0.0',
+        resource_dirs: Optional[Mapping[str, str]] = None,
+        secret_key_file: Optional[str] = None,
+        num_workers: Optional[int] = None,
+        **kwargs,
     ):
         """
         :param port: Listen port for the web server (default: 8008)
-        :type port: int
-
-        :param websocket_port: Listen port for the websocket server (default: 8009)
-        :type websocket_port: int
-
         :param bind_address: Address/interface to bind to (default: 0.0.0.0, accept connection from any IP)
-        :type bind_address: str
-
-        :param disable_websocket: Disable the websocket interface (default: False)
-        :type disable_websocket: bool
-
-        :param ssl_cert: Set it to the path of your certificate file if you want to enable HTTPS (default: None)
-        :type ssl_cert: str
-
-        :param ssl_key: Set it to the path of your key file if you want to enable HTTPS (default: None)
-        :type ssl_key: str
-
-        :param ssl_cafile: Set it to the path of your certificate authority file if you want to enable HTTPS
-            (default: None)
-        :type ssl_cafile: str
-
-        :param ssl_capath: Set it to the path of your certificate authority directory if you want to enable HTTPS
-            (default: None)
-        :type ssl_capath: str
-
         :param resource_dirs: Static resources directories that will be
             accessible through ``/resources/<path>``. It is expressed as a map
             where the key is the relative path under ``/resources`` to expose and
             the value is the absolute path to expose.
-        :type resource_dirs: dict[str, str]
-
-        :param run_externally: If set, then the HTTP backend will not directly
-            spawn the web server. Set this option if you plan to run the webapp
-            in a separate web server (recommended), like uwsgi or uwsgi+nginx.
-        :type run_externally: bool
-
-        :param uwsgi_args: If ``run_externally`` is set and you would like the
-            HTTP backend to directly spawn and control the uWSGI application
-            server instance, then pass the list of uWSGI arguments through
-            this parameter. Some examples include::
-
-                # Start uWSGI instance listening on HTTP port 8008 with 4
-                # processes
-                ['--plugin', 'python', '--http-socket', ':8008', '--master', '--processes', '4']
-
-                # Start uWSGI instance listening on uWSGI socket on port 3031.
-                # You can then use another full-blown web server, like nginx
-                # or Apache, to communicate with the uWSGI instance
-                ['--plugin', 'python', '--socket', '127.0.0.1:3031', '--master', '--processes', '4']
-        :type uwsgi_args: list[str]
+        :param secret_key_file: Path to the file containing the secret key that will be used by Flask
+            (default: ``~/.local/share/platypush/flask.secret.key``).
+        :param num_workers: Number of worker processes to use (default: ``(cpu_count * 2) + 1``).
         """
 
         super().__init__(**kwargs)
 
         self.port = port
-        self.websocket_port = websocket_port
-        self.maps = maps or {}
-        self.server_proc = None
-        self.disable_websocket = disable_websocket
-        self.websocket_thread = None
-        self._websocket_loop = None
+        self._server_proc: Optional[Process] = None
+        self._workers: List[Process] = []
+        self._service_registry_thread = None
         self.bind_address = bind_address
 
         if resource_dirs:
@@ -261,36 +229,14 @@ class HttpBackend(Backend):
         else:
             self.resource_dirs = {}
 
-        self.active_websockets = set()
-        self.run_externally = run_externally
-        self.uwsgi_args = uwsgi_args or []
-        self.ssl_context = (
-            get_ssl_server_context(
-                ssl_cert=ssl_cert,
-                ssl_key=ssl_key,
-                ssl_cafile=ssl_cafile,
-                ssl_capath=ssl_capath,
-            )
-            if ssl_cert
-            else None
+        self.secret_key_file = os.path.expanduser(
+            secret_key_file
+            or os.path.join(Config.get('workdir'), 'flask.secret.key')  # type: ignore
         )
+        self.local_base_url = f'http://localhost:{self.port}'
+        self.num_workers = num_workers or (cpu_count() * 2) + 1
 
-        if self.uwsgi_args:
-            self.uwsgi_args = [str(_) for _ in self.uwsgi_args] + [
-                '--module',
-                'platypush.backend.http.uwsgi',
-                '--enable-threads',
-            ]
-
-        self.local_base_url = '{proto}://localhost:{port}'.format(
-            proto=('https' if ssl_cert else 'http'), port=self.port
-        )
-
-        self._websocket_lock_timeout = 10
-        self._websocket_lock = threading.RLock()
-        self._websocket_locks = {}
-
-    def send_message(self, msg, **kwargs):
+    def send_message(self, *_, **__):
         self.logger.warning('Use cURL or any HTTP client to query the HTTP backend')
 
     def on_stop(self):
@@ -298,190 +244,120 @@ class HttpBackend(Backend):
         super().on_stop()
         self.logger.info('Received STOP event on HttpBackend')
 
-        if self.server_proc:
-            if isinstance(self.server_proc, subprocess.Popen):
-                self.server_proc.kill()
-                self.server_proc.wait(timeout=10)
-                if self.server_proc.poll() is not None:
-                    self.logger.info(
-                        'HTTP server process may be still alive at termination'
-                    )
-                else:
-                    self.logger.info('HTTP server process terminated')
-            else:
-                self.server_proc.terminate()
-                self.server_proc.join(timeout=10)
-                if self.server_proc.is_alive():
-                    self.server_proc.kill()
-                if self.server_proc.is_alive():
-                    self.logger.info(
-                        'HTTP server process may be still alive at termination'
-                    )
-                else:
-                    self.logger.info('HTTP server process terminated')
+        start_time = time()
+        timeout = 5
+        workers = self._workers.copy()
 
-        if (
-            self.websocket_thread
-            and self.websocket_thread.is_alive()
-            and self._websocket_loop
-        ):
-            self._websocket_loop.stop()
-            self.logger.info('HTTP websocket service terminated')
+        for i, worker in enumerate(workers[::-1]):
+            if worker and worker.is_alive():
+                worker.terminate()
+                worker.join(timeout=max(0, start_time + timeout - time()))
 
-    def _acquire_websocket_lock(self, ws):
-        try:
-            acquire_ok = self._websocket_lock.acquire(
-                timeout=self._websocket_lock_timeout
-            )
-            if not acquire_ok:
-                raise TimeoutError('Websocket lock acquire timeout')
+            if worker and worker.is_alive():
+                worker.kill()
+                self._workers.pop(i)
 
-            addr = ws.remote_address
-            if addr not in self._websocket_locks:
-                self._websocket_locks[addr] = threading.RLock()
-        finally:
-            self._websocket_lock.release()
+        if self._server_proc:
+            self._server_proc.terminate()
+            self._server_proc.join(timeout=5)
+            self._server_proc = None
 
-        acquire_ok = self._websocket_locks[addr].acquire(
-            timeout=self._websocket_lock_timeout
-        )
-        if not acquire_ok:
-            raise TimeoutError(
-                'Websocket on address {} not ready to receive data'.format(addr)
-            )
+        if self._server_proc and self._server_proc.is_alive():
+            self._server_proc.kill()
 
-    def _release_websocket_lock(self, ws):
-        try:
-            acquire_ok = self._websocket_lock.acquire(
-                timeout=self._websocket_lock_timeout
-            )
-            if not acquire_ok:
-                raise TimeoutError('Websocket lock acquire timeout')
+        self._server_proc = None
+        self.logger.info('HTTP server terminated')
 
-            addr = ws.remote_address
-            if addr in self._websocket_locks:
-                self._websocket_locks[addr].release()
-        except Exception as e:
-            self.logger.warning(
-                'Unhandled exception while releasing websocket lock: {}'.format(str(e))
-            )
-        finally:
-            self._websocket_lock.release()
+        if self._service_registry_thread and self._service_registry_thread.is_alive():
+            self._service_registry_thread.join(timeout=5)
+            self._service_registry_thread = None
 
     def notify_web_clients(self, event):
         """Notify all the connected web clients (over websocket) of a new event"""
+        get_redis().publish(events_redis_topic, str(event))
 
-        async def send_event(ws):
-            try:
-                self._acquire_websocket_lock(ws)
-                await ws.send(str(event))
-            except Exception as e:
-                self.logger.warning('Error on websocket send_event: {}'.format(e))
-            finally:
-                self._release_websocket_lock(ws)
+    def _get_secret_key(self, _create=False):
+        if _create:
+            self.logger.info('Creating web server secret key')
+            pathlib.Path(self.secret_key_file).parent.mkdir(parents=True, exist_ok=True)
+            with open(self.secret_key_file, 'w') as f:
+                f.write(secrets.token_urlsafe(32))
 
-        loop = get_or_create_event_loop()
-        wss = self.active_websockets.copy()
-
-        for _ws in wss:
-            try:
-                loop.run_until_complete(send_event(_ws))
-            except ConnectionClosed:
-                self.logger.warning(
-                    'Websocket client {} connection lost'.format(_ws.remote_address)
-                )
-                self.active_websockets.remove(_ws)
-                if _ws.remote_address in self._websocket_locks:
-                    del self._websocket_locks[_ws.remote_address]
-
-    def websocket(self):
-        """Websocket main server"""
-        set_thread_name('WebsocketServer')
-
-        async def register_websocket(websocket, path):
-            address = (
-                websocket.remote_address
-                if websocket.remote_address
-                else '<unknown client>'
-            )
-
-            self.logger.info(
-                'New websocket connection from {} on path {}'.format(address, path)
-            )
-            self.active_websockets.add(websocket)
+            os.chmod(self.secret_key_file, 0o600)
+            return secrets.token_urlsafe(32)
 
         try:
-                await websocket.recv()
-            except ConnectionClosed:
-                self.logger.info(
-                    'Websocket client {} closed connection'.format(address)
-                )
-                self.active_websockets.remove(websocket)
-                if address in self._websocket_locks:
-                    del self._websocket_locks[address]
+            with open(self.secret_key_file, 'r') as f:
+                return f.read()
+        except IOError as e:
+            if not _create:
+                return self._get_secret_key(_create=True)
 
-        websocket_args = {}
-        if self.ssl_context:
-            websocket_args['ssl'] = self.ssl_context
+            raise e
 
-        self._websocket_loop = get_or_create_event_loop()
-        self._websocket_loop.run_until_complete(
-            websocket_serve(
-                register_websocket,
-                self.bind_address,
-                self.websocket_port,
-                **websocket_args
-            )
-        )
-        self._websocket_loop.run_forever()
-
-    def _start_web_server(self):
-        def proc():
-            self.logger.info('Starting local web server on port {}'.format(self.port))
-            kwargs = {
-                'host': self.bind_address,
-                'port': self.port,
-                'use_reloader': False,
-                'debug': False,
-            }
-
-            application.config['redis_queue'] = self.bus.redis_queue
-            if self.ssl_context:
-                kwargs['ssl_context'] = self.ssl_context
-
-            application.run(**kwargs)
-
-        return proc
-
-    def run(self):
-        super().run()
+    def _register_service(self):
         try:
             self.register_service(port=self.port)
         except Exception as e:
             self.logger.warning('Could not register the Zeroconf service')
             self.logger.exception(e)
 
-        if not self.disable_websocket:
-            self.logger.info('Initializing websocket interface')
-            self.websocket_thread = threading.Thread(target=self.websocket)
-            self.websocket_thread.start()
+    def _start_zeroconf_service(self):
+        self._service_registry_thread = threading.Thread(
+            target=self._register_service,
+            name='ZeroconfService',
+        )
+        self._service_registry_thread.start()
 
-        if not self.run_externally:
-            self.server_proc = Process(
-                target=self._start_web_server(), name='WebServer'
+    async def _post_fork_main(self, sockets):
+        assert isinstance(
+            self.bus, RedisBus
+        ), 'The HTTP backend only works if backed by a Redis bus'
+
+        application.config['redis_queue'] = self.bus.redis_queue
+        application.secret_key = self._get_secret_key()
+        container = WSGIContainer(application)
+        tornado_app = Application(
+            [
+                *[(route.path(), route) for route in get_ws_routes()],
+                (r'.*', FallbackHandler, {'fallback': container}),
+            ]
         )
-            self.server_proc.start()
-            self.server_proc.join()
-        elif self.uwsgi_args:
-            uwsgi_cmd = ['uwsgi'] + self.uwsgi_args
-            self.logger.info('Starting uWSGI with arguments {}'.format(uwsgi_cmd))
-            self.server_proc = subprocess.Popen(uwsgi_cmd)
-        else:
+
+        server = HTTPServer(tornado_app)
+        server.add_sockets(sockets)
+
+        try:
+            await asyncio.Event().wait()
+        except (asyncio.CancelledError, KeyboardInterrupt):
+            return
+
+    def _web_server_proc(self):
         self.logger.info(
-                'The web server is configured to be launched externally but '
-                + 'no uwsgi_args were provided. Make sure that you run another external service'
-                + 'for the webserver (e.g. nginx)'
+            'Starting local web server on port %s with %d service workers',
+            self.port,
+            self.num_workers,
         )
 
+        sockets = bind_sockets(self.port, address=self.bind_address, reuse_port=True)
+
+        try:
+            fork_processes(self.num_workers)
+            future = self._post_fork_main(sockets)
+            asyncio.run(future)
+        except (asyncio.CancelledError, KeyboardInterrupt):
+            return
+
+    def _start_web_server(self):
+        self._server_proc = Process(target=self._web_server_proc)
+        self._server_proc.start()
+        self._server_proc.join()
+
+    def run(self):
+        super().run()
+
+        self._start_zeroconf_service()
+        self._start_web_server()
+
 
 # vim:sw=4:ts=4:et:

platypush/backend/http/app/routes/auth.py

@@ -50,7 +50,6 @@ def auth_endpoint():
     except Exception as e:
         log.warning('Invalid payload passed to the auth endpoint: ' + str(e))
         abort(400)
-        return jsonify({'token': None})
 
     expiry_days = payload.get('expiry_days')
     expires_at = None
@@ -65,4 +64,3 @@ def auth_endpoint():
         })
     except UserException as e:
         abort(401, str(e))
-        return jsonify({'token': None})

platypush/backend/http/app/routes/dashboard.py

@@ -1,7 +1,7 @@
 from flask import Blueprint, render_template
 
 from platypush.backend.http.app import template_folder
-from platypush.backend.http.app.utils import authenticate, get_websocket_port
+from platypush.backend.http.app.utils import authenticate
 from platypush.backend.http.utils import HttpUtils
 
 dashboard = Blueprint('dashboard', __name__, template_folder=template_folder)
@@ -17,9 +17,10 @@ __routes__ = [
 @authenticate()
 def render_dashboard(name):
     """Route for the dashboard"""
-    return render_template('index.html',
+    return render_template(
+        'index.html',
         utils=HttpUtils,
-                           websocket_port=get_websocket_port())
+    )
 
 
 # vim:sw=4:ts=4:et:

platypush/backend/http/app/routes/execute.py

@@ -1,6 +1,7 @@
 import json
 
-from flask import Blueprint, abort, request, Response
+from flask import Blueprint, abort, request
+from flask.wrappers import Response
 
 from platypush.backend.http.app import template_folder
 from platypush.backend.http.app.utils import authenticate, logger, send_message
@@ -14,8 +15,8 @@ __routes__ = [
 
 
 @execute.route('/execute', methods=['POST'])
-@authenticate()
-def execute():
+@authenticate(json=True)
+def execute_route():
     """Endpoint to execute commands"""
     try:
         msg = json.loads(request.data.decode('utf-8'))

platypush/backend/http/app/routes/hook.py

@@ -1,9 +1,11 @@
 import json
 
-from flask import Blueprint, abort, request, Response
+from flask import Blueprint, abort, request, make_response
 
 from platypush.backend.http.app import template_folder
 from platypush.backend.http.app.utils import logger, send_message
+from platypush.config import Config
+from platypush.event.hook import EventCondition
 from platypush.message.event.http.hook import WebhookEvent
 
 
@@ -15,8 +17,22 @@ __routes__ = [
 ]
 
 
-@hook.route('/hook/<hook_name>', methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS'])
-def _hook(hook_name):
+def matches_condition(event: WebhookEvent, hook):
+    if isinstance(hook, dict):
+        if_ = hook['if'].copy()
+        if_['type'] = '.'.join([event.__module__, event.__class__.__qualname__])
+
+        condition = EventCondition.build(if_)
+    else:
+        condition = hook.condition
+
+    return event.matches_condition(condition)
+
+
+@hook.route(
+    '/hook/<hook_name>', methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS']
+)
+def hook_route(hook_name):
     """Endpoint for custom webhooks"""
 
     event_args = {
@@ -28,20 +44,54 @@ def _hook(hook_name):
     }
 
     if event_args['data']:
-        # noinspection PyBroadException
         try:
             event_args['data'] = json.loads(event_args['data'])
         except Exception as e:
-            logger().warning('Not a valid JSON string: {}: {}'.format(event_args['data'], str(e)))
+            logger().warning(
+                'Not a valid JSON string: %s: %s', event_args['data'], str(e)
+            )
 
     event = WebhookEvent(**event_args)
+    matching_hooks = [
+        hook
+        for hook in Config.get_event_hooks().values()
+        if matches_condition(event, hook)
+    ]
 
     try:
         send_message(event)
-        return Response(json.dumps({'status': 'ok', **event_args}), mimetype='application/json')
+        rs = default_rs = make_response(json.dumps({'status': 'ok', **event_args}))
+        headers = {}
+        status_code = 200
+
+        # If there are matching hooks, wait for their completion before returning
+        if matching_hooks:
+            rs = event.wait_response(timeout=60)
+            try:
+                rs = json.loads(rs.decode())  # type: ignore
+            except Exception:
+                pass
+
+            if isinstance(rs, dict) and '___data___' in rs:
+                # data + http_code + custom_headers return format
+                headers = rs.get('___headers___', {})
+                status_code = rs.get('___code___', status_code)
+                rs = rs['___data___']
+
+            if rs is None:
+                rs = default_rs
+                headers = {'Content-Type': 'application/json'}
+
+            rs = make_response(rs)
+        else:
+            headers = {'Content-Type': 'application/json'}
+
+        rs.status_code = status_code
+        rs.headers.update(headers)
+        return rs
     except Exception as e:
         logger().exception(e)
-        logger().error('Error while dispatching webhook event {}: {}'.format(event, str(e)))
+        logger().error('Error while dispatching webhook event %s: %s', event, str(e))
         abort(500, str(e))
 
 

platypush/backend/http/app/routes/logo.py

@@ -0,0 +1,185 @@
+from dataclasses import dataclass
+import math
+from typing import Tuple
+
+from flask import Blueprint, make_response, request
+
+
+logo = Blueprint('logo', __name__)
+
+# Declare routes list
+__routes__ = [
+    logo,
+]
+
+
+@dataclass
+class Gear:
+    """
+    A utility class used to model the gears in the application's logo.
+    """
+
+    center: Tuple[float, float]
+    outer_radius: float
+    inner_radius: float
+    color: str = "currentColor"
+    background: str = ""
+    num_spikes: int = 0
+    spike_max_base: float = 0
+    spike_min_base: float = 0
+    spike_height: float = 0
+    alpha_offset: float = 0
+
+    def to_svg(self) -> str:
+        """
+        Returns the SVG representation of a gear.
+        """
+
+        # Generate the basic circle
+        svg = f"""
+        <circle
+            cx="{self.center[0]}" cy="{self.center[1]}"
+            r="{self.outer_radius - (self.inner_radius / math.pi)}"
+            stroke-width="{self.inner_radius}"
+            stroke="{self.color}"
+            fill="none" />
+        """
+
+        # Generate the spikes
+        for i in range(self.num_spikes):
+            # Iterate for alpha -> [0, 2*pi]
+            alpha = (2 * math.pi * i) / self.num_spikes
+            # Calculate the base angle for the major base of the gear polygon
+            maj_delta_alpha = math.asin(self.spike_max_base / (2 * self.outer_radius))
+            # Calculate the points of the gear polygon's major base
+            maj_base = (
+                (
+                    self.center[0]
+                    + self.outer_radius
+                    * math.cos(alpha + maj_delta_alpha + self.alpha_offset),
+                    self.center[1]
+                    + self.outer_radius
+                    * math.sin(alpha + maj_delta_alpha + self.alpha_offset),
+                ),
+                (
+                    self.center[0]
+                    + self.outer_radius
+                    * math.cos(alpha - maj_delta_alpha + self.alpha_offset),
+                    self.center[1]
+                    + self.outer_radius
+                    * math.sin(alpha - maj_delta_alpha + self.alpha_offset),
+                ),
+            )
+
+            # Height of the gear relative to the circle's center
+            h = self.outer_radius * math.cos(maj_delta_alpha) + self.spike_height
+            # Calculate the base angle for the minor base of the gear polygon
+            min_delta_alpha = math.asin(self.spike_min_base / (2 * h))
+            # Calculate the points of the gear polygon's minor base
+            min_base = (
+                (
+                    self.center[0]
+                    + h * math.cos(alpha - min_delta_alpha + self.alpha_offset),
+                    self.center[1]
+                    + h * math.sin(alpha - min_delta_alpha + self.alpha_offset),
+                ),
+                (
+                    self.center[0]
+                    + h * math.cos(alpha + min_delta_alpha + self.alpha_offset),
+                    self.center[1]
+                    + h * math.sin(alpha + min_delta_alpha + self.alpha_offset),
+                ),
+            )
+
+            # Flatten the polygon's points
+            svg_points = " ".join(
+                [f"{point[0]},{point[1]}" for point in [*maj_base, *min_base]]
+            )
+
+            # Serialize the gear polygon to SVG
+            svg += f"""
+        <polygon points="{svg_points}" stroke="{self.color}" fill="{self.color}" />"""
+
+        return svg
+
+
+# Properties of the two gears on the logo
+gears = [
+    Gear(
+        center=(32.9, 34.5),
+        outer_radius=22.6,
+        inner_radius=12.4,
+        num_spikes=12,
+        spike_max_base=9,
+        spike_min_base=4.3,
+        spike_height=10.16,
+    ),
+    Gear(
+        center=(65.5, 70.5),
+        outer_radius=14.4,
+        inner_radius=8.5,
+        num_spikes=7,
+        spike_max_base=9,
+        spike_min_base=4.3,
+        spike_height=7.5,
+        alpha_offset=math.pi / 6.6,
+    ),
+]
+
+
+template_start = """
+<svg version="1.1"
+     width="{width}" height="{height}"
+     viewBox="0 0 100 100"
+     preserveAspectRatio="none"
+     xmlns="http://www.w3.org/2000/svg">
+  <defs>
+    <linearGradient id="triangleGradient">
+        <stop offset="0%" stop-color="#8acb45" />
+        <stop offset="50%" stop-color="#6bbb4c" />
+        <stop offset="100%" stop-color="#5cb450" />
+    </linearGradient>
+  </defs>
+
+  <rect width="100%" height="100%" fill="{bg_color}" />
+"""
+
+template_end = "\n</svg>"
+
+
+@logo.route('/logo.svg', methods=['GET'])
+def logo_path():
+    """
+    This path dynamically generates the logo image as a parametrizable vector SVG.
+
+    Parameters:
+
+        - ``size``: Size of the image in pixels (default: 256)
+        - ``bg``: Background color (default: "none")
+        - ``fg``: Foreground color (default: "currentColor")
+
+    """
+    size = request.args.get("size", 256)
+    bg = request.args.get("bg", "none")
+    fg = request.args.get("fg", "currentColor")
+    svg = template_start.format(
+        width=size,
+        height=size,
+        bg_color=bg,
+    )
+
+    for gear in gears:
+        gear.color = fg
+        gear.background = bg
+        svg += gear.to_svg()
+
+    # "Play" triangle on the logo
+    svg += """\n\t\t<polygon points="67,47 67,3 99,25.3" fill="url(#triangleGradient)" />"""
+    svg += template_end
+
+    rs = make_response(svg)
+    rs.headers.update({"Content-Type": "image/svg+xml"})
+    return rs
+
+
+# vim:sw=4:ts=4:et:

platypush/backend/http/app/routes/plugins/tts/mimic3.py

@@ -0,0 +1,46 @@
+import requests
+from urllib.parse import urljoin
+
+from flask import abort, request, Blueprint
+
+from platypush.backend.http.app import template_folder
+
+mimic3 = Blueprint('mimic3', __name__, template_folder=template_folder)
+
+# Declare routes list
+__routes__ = [
+    mimic3,
+]
+
+
+@mimic3.route('/tts/mimic3/say', methods=['GET'])
+def proxy_tts_request():
+    """
+    This route is used to proxy the POST request to the Mimic3 TTS server
+    through a GET, so it can be easily processed as a URL through a media
+    plugin.
+    """
+    required_args = {
+        'text',
+        'server_url',
+        'voice',
+    }
+
+    missing_args = required_args.difference(set(request.args.keys()))
+    if missing_args:
+        abort(400, f'Missing parameters: {missing_args}')
+
+    args = {arg: request.args[arg] for arg in required_args}
+
+    rs = requests.post(
+        urljoin(args['server_url'], '/api/tts'),
+        data=args['text'],
+        params={
+            'voice': args['voice'],
+        },
+    )
+
+    return rs.content
+
+
+# vim:sw=4:ts=4:et:

platypush/backend/http/app/routes/pwa.py

@@ -0,0 +1,113 @@
+from flask import Blueprint, jsonify, send_from_directory
+
+from platypush.config import Config
+from platypush.backend.http.app import template_folder
+
+pwa = Blueprint('pwa', __name__, template_folder=template_folder)
+
+# Declare routes list
+__routes__ = [
+    pwa,
+]
+
+
+@pwa.route('/manifest.json', methods=['GET'])
+def manifest_json():
+    """Generated manifest file for the PWA"""
+    return jsonify(
+        {
+            "name": f'Platypush @ {Config.get("device_id")}',
+            "short_name": Config.get('device_id'),
+            "icons": [
+                {
+                    "src": "/img/icons/favicon-16x16.png",
+                    "sizes": "16x16",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/favicon-32x32.png",
+                    "sizes": "32x32",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/apple-touch-icon-60x60.png",
+                    "sizes": "60x60",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/apple-touch-icon-76x76.png",
+                    "sizes": "76x76",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/apple-touch-icon-120x120.png",
+                    "sizes": "120x120",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/msapplication-icon-144x144.png",
+                    "sizes": "144x144",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/mstile-150x150.png",
+                    "sizes": "150x150",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/apple-touch-icon-152x152.png",
+                    "sizes": "152x152",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/apple-touch-icon-180x180.png",
+                    "sizes": "180x180",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/android-chrome-192x192.png",
+                    "sizes": "192x192",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/android-chrome-maskable-192x192.png",
+                    "sizes": "192x192",
+                    "type": "image/png",
+                    "purpose": "maskable",
+                },
+                {
+                    "src": "/img/icons/logo-256x256.png",
+                    "sizes": "256x256",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/android-chrome-512x512.png",
+                    "sizes": "512x512",
+                    "type": "image/png",
+                },
+                {
+                    "src": "/img/icons/android-chrome-maskable-512x512.png",
+                    "sizes": "512x512",
+                    "type": "image/png",
+                    "purpose": "maskable",
+                },
+            ],
+            "gcm_sender_id": "",
+            "gcm_user_visible_only": True,
+            "start_url": "/",
+            "permissions": ["gcm"],
+            "orientation": "portrait",
+            "display": "standalone",
+            "theme_color": "#000000",
+            "background_color": "#ffffff",
+        }
+    )
+
+
+@pwa.route('/service-worker.js', methods=['GET'])
+def service_worker_js():
+    """URL that serves the service worker for the PWA"""
+    return send_from_directory(template_folder, 'service-worker.js')
+
+
+# vim:sw=4:ts=4:et:

platypush/backend/http/app/routes/resources.py

@@ -8,16 +8,21 @@ from platypush.backend.http.app import template_folder
 
 
 img_folder = os.path.join(template_folder, 'img')
+fonts_folder = os.path.join(template_folder, 'fonts')
 icons_folder = os.path.join(template_folder, 'icons')
 resources = Blueprint('resources', __name__, template_folder=template_folder)
 favicon = Blueprint('favicon', __name__, template_folder=template_folder)
 img = Blueprint('img', __name__, template_folder=template_folder)
+icons = Blueprint('icons', __name__, template_folder=template_folder)
+fonts = Blueprint('fonts', __name__, template_folder=template_folder)
 
 # Declare routes list
 __routes__ = [
     resources,
     favicon,
     img,
+    icons,
+    fonts,
 ]
 
 
@@ -42,9 +47,11 @@ def resources_path(path):
     real_path = real_base_path
 
     file_path = [
-        s for s in re.sub(
+        s
+        for s in re.sub(
             r'^{}(.*)$'.format(base_path), '\\1', path  # lgtm [py/regex-injection]
-        ).split('/') if s
+        ).split('/')
+        if s
     ]
 
     for p in file_path[:-1]:
@@ -71,10 +78,16 @@ def imgpath(path):
     return send_from_directory(img_folder, path)
 
 
-@img.route('/icons/<path:path>', methods=['GET'])
+@icons.route('/icons/<path:path>', methods=['GET'])
 def iconpath(path):
     """Default static icons"""
     return send_from_directory(icons_folder, path)
 
 
+@fonts.route('/fonts/<path:path>', methods=['GET'])
+def fontpath(path):
+    """Default fonts"""
+    return send_from_directory(fonts_folder, path)
+
+
 # vim:sw=4:ts=4:et:

platypush/backend/http/app/utils.py

@@ -1,274 +0,0 @@
-import importlib
-import logging
-import os
-
-from functools import wraps
-from flask import abort, request, redirect, Response, current_app
-from redis import Redis
-
-# NOTE: The HTTP service will *only* work on top of a Redis bus. The default
-# internal bus service won't work as the web server will run in a different process.
-from platypush.bus.redis import RedisBus
-
-from platypush.config import Config
-from platypush.message import Message
-from platypush.message.request import Request
-from platypush.user import UserManager
-from platypush.utils import get_redis_queue_name_by_message, get_ip_or_hostname
-
-_bus = None
-_logger = None
-
-
-def bus():
-    global _bus
-    if _bus is None:
-        _bus = RedisBus(redis_queue=current_app.config.get('redis_queue'))
-    return _bus
-
-
-def logger():
-    global _logger
-    if not _logger:
-        log_args = {
-            'level': logging.INFO,
-            'format': '%(asctime)-15s|%(levelname)5s|%(name)s|%(message)s',
-        }
-
-        level = (Config.get('backend.http') or {}).get('logging') or \
-            (Config.get('logging') or {}).get('level')
-        filename = (Config.get('backend.http') or {}).get('filename')
-
-        if level:
-            log_args['level'] = getattr(logging, level.upper()) \
-                if isinstance(level, str) else level
-        if filename:
-            log_args['filename'] = filename
-
-        logging.basicConfig(**log_args)
-        _logger = logging.getLogger('platypush:web')
-
-    return _logger
-
-
-def get_message_response(msg):
-    redis = Redis(**bus().redis_args)
-    response = redis.blpop(get_redis_queue_name_by_message(msg), timeout=60)
-    if response and len(response) > 1:
-        response = Message.build(response[1])
-    else:
-        response = None
-
-    return response
-
-
-# noinspection PyProtectedMember
-def get_http_port():
-    from platypush.backend.http import HttpBackend
-    http_conf = Config.get('backend.http')
-    return http_conf.get('port', HttpBackend._DEFAULT_HTTP_PORT)
-
-
-# noinspection PyProtectedMember
-def get_websocket_port():
-    from platypush.backend.http import HttpBackend
-    http_conf = Config.get('backend.http')
-    return http_conf.get('websocket_port', HttpBackend._DEFAULT_WEBSOCKET_PORT)
-
-
-def send_message(msg, wait_for_response=True):
-    msg = Message.build(msg)
-
-    if isinstance(msg, Request):
-        msg.origin = 'http'
-
-    if Config.get('token'):
-        msg.token = Config.get('token')
-
-    bus().post(msg)
-
-    if isinstance(msg, Request) and wait_for_response:
-        response = get_message_response(msg)
-        logger().debug('Processing response on the HTTP backend: {}'.
-                       format(response))
-
-        return response
-
-
-def send_request(action, wait_for_response=True, **kwargs):
-    msg = {
-        'type': 'request',
-        'action': action
-    }
-
-    if kwargs:
-        msg['args'] = kwargs
-
-    return send_message(msg, wait_for_response=wait_for_response)
-
-
-def _authenticate_token():
-    token = Config.get('token')
-    user_manager = UserManager()
-
-    if 'X-Token' in request.headers:
-        user_token = request.headers['X-Token']
-    elif 'Authorization' in request.headers and request.headers['Authorization'].startswith('Bearer '):
-        user_token = request.headers['Authorization'][len('Bearer '):]
-    elif 'token' in request.args:
-        user_token = request.args.get('token')
-    else:
-        return False
-
-    try:
-        user_manager.validate_jwt_token(user_token)
-        return True
-    except Exception as e:
-        logger().debug(str(e))
-        return token and user_token == token
-
-
-def _authenticate_http():
-    user_manager = UserManager()
-
-    if not request.authorization:
-        return False
-
-    username = request.authorization.username
-    password = request.authorization.password
-    return user_manager.authenticate_user(username, password)
-
-
-def _authenticate_session():
-    user_manager = UserManager()
-    user_session_token = None
-    user = None
-
-    if 'X-Session-Token' in request.headers:
-        user_session_token = request.headers['X-Session-Token']
-    elif 'session_token' in request.args:
-        user_session_token = request.args.get('session_token')
-    elif 'session_token' in request.cookies:
-        user_session_token = request.cookies.get('session_token')
-
-    if user_session_token:
-        user, session = user_manager.authenticate_user_session(user_session_token)
-
-    return user is not None
-
-
-def _authenticate_csrf_token():
-    user_manager = UserManager()
-    user_session_token = None
-
-    if 'X-Session-Token' in request.headers:
-        user_session_token = request.headers['X-Session-Token']
-    elif 'session_token' in request.args:
-        user_session_token = request.args.get('session_token')
-    elif 'session_token' in request.cookies:
-        user_session_token = request.cookies.get('session_token')
-
-    if user_session_token:
-        user, session = user_manager.authenticate_user_session(user_session_token)
-    else:
-        return False
-
-    if user is None:
-        return False
-
-    return session.csrf_token is None or request.form.get('csrf_token') == session.csrf_token
-
-
-def authenticate(redirect_page='', skip_auth_methods=None, check_csrf_token=False):
-    def decorator(f):
-        @wraps(f)
-        def wrapper(*args, **kwargs):
-            user_manager = UserManager()
-            n_users = user_manager.get_user_count()
-            skip_methods = skip_auth_methods or []
-
-            # User/pass HTTP authentication
-            http_auth_ok = True
-            if n_users > 0 and 'http' not in skip_methods:
-                http_auth_ok = _authenticate_http()
-                if http_auth_ok:
-                    return f(*args, **kwargs)
-
-            # Token-based authentication
-            token_auth_ok = True
-            if 'token' not in skip_methods:
-                token_auth_ok = _authenticate_token()
-                if token_auth_ok:
-                    return f(*args, **kwargs)
-
-            # Session token based authentication
-            session_auth_ok = True
-            if n_users > 0 and 'session' not in skip_methods:
-                session_auth_ok = _authenticate_session()
-                if session_auth_ok:
-                    return f(*args, **kwargs)
-
-                return redirect('/login?redirect=' + (redirect_page or request.url), 307)
-
-            # CSRF token check
-            if check_csrf_token:
-                csrf_check_ok = _authenticate_csrf_token()
-                if not csrf_check_ok:
-                    return abort(403, 'Invalid or missing csrf_token')
-
-            if n_users == 0 and 'session' not in skip_methods:
-                return redirect('/register?redirect=' + (redirect_page or request.url), 307)
-
-            if ('http' not in skip_methods and http_auth_ok) or \
-                    ('token' not in skip_methods and token_auth_ok) or \
-                    ('session' not in skip_methods and session_auth_ok):
-                return f(*args, **kwargs)
-
-            return Response('Authentication required', 401,
-                            {'WWW-Authenticate': 'Basic realm="Login required"'})
-
-        return wrapper
-
-    return decorator
-
-
-def get_routes():
-    routes_dir = os.path.join(
-        os.path.dirname(os.path.abspath(__file__)), 'routes')
-    routes = []
-    base_module = '.'.join(__name__.split('.')[:-1])
-
-    for path, dirs, files in os.walk(routes_dir):
-        for f in files:
-            if f.endswith('.py'):
-                mod_name = '.'.join(
-                    (base_module + '.' + os.path.join(path, f).replace(
-                        os.path.dirname(__file__), '')[1:].replace(os.sep, '.')).split('.')
-                    [:(-2 if f == '__init__.py' else -1)])
-
-                try:
-                    mod = importlib.import_module(mod_name)
-                    if hasattr(mod, '__routes__'):
-                        routes.extend(mod.__routes__)
-                except Exception as e:
-                    logger().warning('Could not import routes from {}/{}: {}: {}'.
-                                     format(path, f, type(e), str(e)))
-
-    return routes
-
-
-def get_local_base_url():
-    http_conf = Config.get('backend.http') or {}
-    return '{proto}://localhost:{port}'.format(
-        proto=('https' if http_conf.get('ssl_cert') else 'http'),
-        port=get_http_port())
-
-
-def get_remote_base_url():
-    http_conf = Config.get('backend.http') or {}
-    return '{proto}://{host}:{port}'.format(
-        proto=('https' if http_conf.get('ssl_cert') else 'http'),
-        host=get_ip_or_hostname(), port=get_http_port())
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/http/app/utils/__init__.py

@@ -0,0 +1,37 @@
+from .auth import (
+    authenticate,
+    authenticate_token,
+    authenticate_user_pass,
+    get_auth_status,
+)
+from .bus import bus, get_message_response, send_message, send_request
+from .logger import logger
+from .routes import (
+    get_http_port,
+    get_ip_or_hostname,
+    get_local_base_url,
+    get_remote_base_url,
+    get_routes,
+)
+from .ws import get_ws_routes
+
+__all__ = [
+    'authenticate',
+    'authenticate_token',
+    'authenticate_user_pass',
+    'bus',
+    'get_auth_status',
+    'get_http_port',
+    'get_ip_or_hostname',
+    'get_local_base_url',
+    'get_message_response',
+    'get_remote_base_url',
+    'get_routes',
+    'get_ws_routes',
+    'logger',
+    'send_message',
+    'send_request',
+]
+
+
+# vim:sw=4:ts=4:et:

platypush/backend/http/app/utils/auth/__init__.py

@@ -0,0 +1,196 @@
+import base64
+from functools import wraps
+from typing import Optional
+
+from flask import request, redirect, jsonify
+from flask.wrappers import Response
+
+from platypush.config import Config
+from platypush.user import UserManager
+
+from ..logger import logger
+from .status import AuthStatus
+
+user_manager = UserManager()
+
+
+def get_arg(req, name: str) -> Optional[str]:
+    # The Flask way
+    if hasattr(req, 'args'):
+        return req.args.get(name)
+
+    # The Tornado way
+    if hasattr(req, 'arguments'):
+        arg = req.arguments.get(name)
+        if arg:
+            return arg[0].decode()
+
+    return None
+
+
+def get_cookie(req, name: str) -> Optional[str]:
+    cookie = req.cookies.get(name)
+    if not cookie:
+        return None
+
+    # The Flask way
+    if isinstance(cookie, str):
+        return cookie
+
+    # The Tornado way
+    return cookie.value
+
+
+def authenticate_token(req):
+    token = Config.get('token')
+    user_token = None
+
+    if 'X-Token' in req.headers:
+        user_token = req.headers['X-Token']
+    elif 'Authorization' in req.headers and req.headers['Authorization'].startswith(
+        'Bearer '
+    ):
+        user_token = req.headers['Authorization'][7:]
+    else:
+        user_token = get_arg(req, 'token')
+
+    if not user_token:
+        return False
+
+    try:
+        user_manager.validate_jwt_token(user_token)
+        return True
+    except Exception as e:
+        logger().debug(str(e))
+        return bool(token and user_token == token)
+
+
+def authenticate_user_pass(req):
+    # Flask populates request.authorization
+    if hasattr(req, 'authorization'):
+        if not req.authorization:
+            return False
+
+        username = req.authorization.username
+        password = req.authorization.password
+
+    # Otherwise, check the Authorization header
+    elif 'Authorization' in req.headers and req.headers['Authorization'].startswith(
+        'Basic '
+    ):
+        auth = req.headers['Authorization'][6:]
+        try:
+            auth = base64.b64decode(auth)
+        except ValueError:
+            pass
+
+        username, password = auth.decode().split(':', maxsplit=1)
+    else:
+        return False
+
+    return user_manager.authenticate_user(username, password)
+
+
+def authenticate_session(req):
+    user = None
+
+    # Check the X-Session-Token header
+    user_session_token = req.headers.get('X-Session-Token')
+
+    # Check the `session_token` query/body parameter
+    if not user_session_token:
+        user_session_token = get_arg(req, 'session_token')
+
+    # Check the `session_token` cookie
+    if not user_session_token:
+        user_session_token = get_cookie(req, 'session_token')
+
+    if user_session_token:
+        user, _ = user_manager.authenticate_user_session(user_session_token)
+
+    return user is not None
+
+
+def authenticate(
+    redirect_page='',
+    skip_auth_methods=None,
+    json=False,
+):
+    """
+    Authentication decorator for Flask routes.
+    """
+
+    def decorator(f):
+        @wraps(f)
+        def wrapper(*args, **kwargs):
+            auth_status = get_auth_status(
+                request,
+                skip_auth_methods=skip_auth_methods,
+            )
+
+            if auth_status == AuthStatus.OK:
+                return f(*args, **kwargs)
+
+            if json:
+                return jsonify(auth_status.to_dict()), auth_status.value.code
+
+            if auth_status == AuthStatus.NO_USERS:
+                return redirect(
+                    f'/register?redirect={redirect_page or request.url}', 307
+                )
+
+            if auth_status == AuthStatus.UNAUTHORIZED:
+                return redirect(f'/login?redirect={redirect_page or request.url}', 307)
+
+            return Response(
+                'Authentication required',
+                401,
+                {'WWW-Authenticate': 'Basic realm="Login required"'},
+            )
+
+        return wrapper
+
+    return decorator
+
+
+# pylint: disable=too-many-return-statements
+def get_auth_status(req, skip_auth_methods=None) -> AuthStatus:
+    """
+    Check against the available authentication methods (except those listed in
+    ``skip_auth_methods``) if the user is properly authenticated.
+    """
+
+    n_users = user_manager.get_user_count()
+    skip_methods = skip_auth_methods or []
+
+    # User/pass HTTP authentication
+    http_auth_ok = True
+    if n_users > 0 and 'http' not in skip_methods:
+        http_auth_ok = authenticate_user_pass(req)
+        if http_auth_ok:
+            return AuthStatus.OK
+
+    # Token-based authentication
+    token_auth_ok = True
+    if 'token' not in skip_methods:
+        token_auth_ok = authenticate_token(req)
+        if token_auth_ok:
+            return AuthStatus.OK
+
+    # Session token based authentication
+    session_auth_ok = True
+    if n_users > 0 and 'session' not in skip_methods:
+        return AuthStatus.OK if authenticate_session(req) else AuthStatus.UNAUTHORIZED
+
+    # At least a user should be created before accessing an authenticated resource
+    if n_users == 0 and 'session' not in skip_methods:
+        return AuthStatus.NO_USERS
+
+    if (  # pylint: disable=too-many-boolean-expressions
+        ('http' not in skip_methods and http_auth_ok)
+        or ('token' not in skip_methods and token_auth_ok)
+        or ('session' not in skip_methods and session_auth_ok)
+    ):
+        return AuthStatus.OK
+
+    return AuthStatus.UNAUTHORIZED

platypush/backend/http/app/utils/auth/status.py

@@ -0,0 +1,21 @@
+from collections import namedtuple
+from enum import Enum
+
+
+StatusValue = namedtuple('StatusValue', ['code', 'message'])
+
+
+class AuthStatus(Enum):
+    """
+    Models the status of the authentication.
+    """
+
+    OK = StatusValue(200, 'OK')
+    UNAUTHORIZED = StatusValue(401, 'Unauthorized')
+    NO_USERS = StatusValue(412, 'Please create a user first')
+
+    def to_dict(self):
+        return {
+            'code': self.value[0],
+            'message': self.value[1],
+        }

platypush/backend/http/app/utils/bus.py

@@ -0,0 +1,64 @@
+from redis import Redis
+
+from platypush.bus.redis import RedisBus
+from platypush.config import Config
+from platypush.context import get_backend
+from platypush.message import Message
+from platypush.message.request import Request
+from platypush.utils import get_redis_queue_name_by_message
+
+from .logger import logger
+
+_bus = None
+
+
+def bus():
+    global _bus
+    if _bus is None:
+        redis_queue = get_backend('http').bus.redis_queue  # type: ignore
+        _bus = RedisBus(redis_queue=redis_queue)
+    return _bus
+
+
+def send_message(msg, wait_for_response=True):
+    msg = Message.build(msg)
+    if msg is None:
+        return
+
+    if isinstance(msg, Request):
+        msg.origin = 'http'
+
+    if Config.get('token'):
+        msg.token = Config.get('token')
+
+    bus().post(msg)
+
+    if isinstance(msg, Request) and wait_for_response:
+        response = get_message_response(msg)
+        logger().debug('Processing response on the HTTP backend: {}'.format(response))
+
+        return response
+
+
+def send_request(action, wait_for_response=True, **kwargs):
+    msg = {'type': 'request', 'action': action}
+
+    if kwargs:
+        msg['args'] = kwargs
+
+    return send_message(msg, wait_for_response=wait_for_response)
+
+
+def get_message_response(msg):
+    redis = Redis(**bus().redis_args)
+    redis_queue = get_redis_queue_name_by_message(msg)
+    if not redis_queue:
+        return
+
+    response = redis.blpop(redis_queue, timeout=60)
+    if response and len(response) > 1:
+        response = Message.build(response[1])
+    else:
+        response = None
+
+    return response

platypush/backend/http/app/utils/logger.py

@@ -0,0 +1,31 @@
+import logging
+
+from platypush.config import Config
+
+_logger = None
+
+
+def logger():
+    global _logger
+    if not _logger:
+        log_args = {
+            'level': logging.INFO,
+            'format': '%(asctime)-15s|%(levelname)5s|%(name)s|%(message)s',
+        }
+
+        level = (Config.get('backend.http') or {}).get('logging') or (
+            Config.get('logging') or {}
+        ).get('level')
+        filename = (Config.get('backend.http') or {}).get('filename')
+
+        if level:
+            log_args['level'] = (
+                getattr(logging, level.upper()) if isinstance(level, str) else level
+            )
+        if filename:
+            log_args['filename'] = filename
+
+        logging.basicConfig(**log_args)
+        _logger = logging.getLogger('platypush:web')
+
+    return _logger

platypush/backend/http/app/utils/routes.py

@@ -0,0 +1,59 @@
+import importlib
+import inspect
+import os
+import pkgutil
+
+from platypush.backend import Backend
+from platypush.config import Config
+from platypush.utils import get_ip_or_hostname
+
+from .logger import logger
+
+
+def get_http_port():
+    from platypush.backend.http import HttpBackend
+
+    http_conf = Config.get('backend.http') or {}
+    return http_conf.get('port', HttpBackend._DEFAULT_HTTP_PORT)
+
+
+def get_routes():
+    base_pkg = '.'.join([Backend.__module__, 'http', 'app', 'routes'])
+    base_dir = os.path.join(
+        os.path.dirname(inspect.getfile(Backend)), 'http', 'app', 'routes'
+    )
+    routes = []
+
+    for _, mod_name, _ in pkgutil.walk_packages([base_dir], prefix=base_pkg + '.'):
+        try:
+            module = importlib.import_module(mod_name)
+            if hasattr(module, '__routes__'):
+                routes.extend(module.__routes__)
+        except Exception as e:
+            logger.warning('Could not import module %s', mod_name)
+            logger.exception(e)
+            continue
+
+    return routes
+
+
+def get_local_base_url():
+    http_conf = Config.get('backend.http') or {}
+    bind_address = http_conf.get('bind_address')
+    if not bind_address or bind_address == '0.0.0.0':
+        bind_address = 'localhost'
+
+    return '{proto}://{host}:{port}'.format(
+        proto=('https' if http_conf.get('ssl_cert') else 'http'),
+        host=bind_address,
+        port=get_http_port(),
+    )
+
+
+def get_remote_base_url():
+    http_conf = Config.get('backend.http') or {}
+    return '{proto}://{host}:{port}'.format(
+        proto=('https' if http_conf.get('ssl_cert') else 'http'),
+        host=get_ip_or_hostname(),
+        port=get_http_port(),
+    )

platypush/backend/http/app/utils/ws.py

@@ -0,0 +1,37 @@
+import os
+import importlib
+import inspect
+from typing import List, Type
+
+import pkgutil
+
+from ..ws import WSRoute, logger
+
+
+def get_ws_routes() -> List[Type[WSRoute]]:
+    """
+    Scans for websocket route objects.
+    """
+    from platypush.backend.http import HttpBackend
+
+    base_pkg = '.'.join([HttpBackend.__module__, 'app', 'ws'])
+    base_dir = os.path.join(os.path.dirname(inspect.getfile(HttpBackend)), 'app', 'ws')
+    routes = []
+
+    for _, mod_name, _ in pkgutil.walk_packages([base_dir], prefix=base_pkg + '.'):
+        try:
+            module = importlib.import_module(mod_name)
+        except Exception as e:
+            logger.warning('Could not import module %s', mod_name)
+            logger.exception(e)
+            continue
+
+        for _, obj in inspect.getmembers(module):
+            if (
+                inspect.isclass(obj)
+                and not inspect.isabstract(obj)
+                and issubclass(obj, WSRoute)
+            ):
+                routes.append(obj)
+
+    return routes