Merge branch 'master' into snyk-upgrade-7f1672a9074c3d844aa231ad5ba0e90d

.dockerignore

@@ -0,0 +1,6 @@
+**/.git
+**/node_modules
+**/__pycache__
+**/venv
+**/.mypy_cache
+**/build

.gitignore

@@ -10,7 +10,7 @@ package.sh
 platypush/backend/http/static/resources/*
 docs/build
 .idea/
-config
+/config
 platypush/backend/http/static/css/*/.sass-cache/
 .vscode
 platypush/backend/http/static/js/lib/vue.js
@@ -24,3 +24,5 @@ coverage.xml
 Session.vim
 /jsconfig.json
 /package.json
+/Dockerfile
+/docs/source/wiki

MANIFEST.in

@@ -1,4 +1,5 @@
 recursive-include platypush/backend/http/webapp/dist *
+recursive-include platypush/install *
 include platypush/plugins/http/webpage/mercury-parser.js
 include platypush/config/*.yaml
 global-include manifest.yaml

README.md

@@ -2,7 +2,6 @@ Platypush
 =========
 
 [![Build Status](https://ci-cd.platypush.tech/api/badges/platypush/platypush/status.svg)](https://ci-cd.platypush.tech/platypush/platypush)
-[![Documentation Status](https://ci.platypush.tech/docs/status.svg)](https://ci.platypush.tech/docs/latest.log)
 [![pip version](https://img.shields.io/pypi/v/platypush.svg?style=flat)](https://pypi.python.org/pypi/platypush/)
 [![License](https://img.shields.io/github/license/BlackLight/platypush.svg)](https://git.platypush.tech/platypush/platypush/src/branch/master/LICENSE.txt)
 [![Last Commit](https://img.shields.io/github/last-commit/BlackLight/platypush.svg)](https://git.platypush.tech/platypush/platypush/commits/branch/master)
@@ -11,20 +10,22 @@ Platypush
 
 <!-- toc -->
 
-- [Useful links](#useful-links)
 - [Introduction](#introduction)
     + [What it can do](#what-it-can-do)
 - [Installation](#installation)
-  * [System installation](#system-installation)
+  * [Prerequisites](#prerequisites)
-    + [Install through `pip`](#install-through-pip)
+    + [Docker installation](#docker-installation)
-    + [Install through a system package manager](#install-through-a-system-package-manager)
+    + [Use an external service](#use-an-external-service)
-    + [Install from sources](#install-from-sources)
+    + [Manual installation](#manual-installation)
+  * [Install through `pip`](#install-through-pip)
+  * [Install through a system package manager](#install-through-a-system-package-manager)
+  * [Install from sources](#install-from-sources)
   * [Installing the dependencies for your extensions](#installing-the-dependencies-for-your-extensions)
     + [Install via `extras` name](#install-via-extras-name)
     + [Install via `manifest.yaml`](#install-via-manifestyaml)
     + [Check the instructions reported in the documentation](#check-the-instructions-reported-in-the-documentation)
   * [Virtual environment installation](#virtual-environment-installation)
-  * [Docker installation](#docker-installation)
+  * [Docker installation](#docker-installation-1)
 - [Architecture](#architecture)
   * [Plugins](#plugins)
   * [Actions](#actions)
@@ -44,31 +45,11 @@ Platypush
     + [PWA support](#pwa-support)
 - [Mobile app](#mobile-app)
 - [Tests](#tests)
+- [Useful links](#useful-links)
 - [Funding](#funding)
 
 <!-- tocstop -->
 
-## Useful links
-- Recommended read: [**Getting started with Platypush**](https://blog.platypush.tech/article/Ultimate-self-hosted-automation-with-Platypush).
-
-- The [blog](https://blog.platypush.tech) is a good place to get more insights
-  and inspiration on what you can build.
-
-- The [wiki](https://git.platypush.tech/platypush/platypush/wiki) also
-  contains many resources on getting started.
-
-- Extensive documentation for all the available integrations and messages [is
-  available](https://docs.platypush.tech/).
-
-- If you have issues/feature requests/enhancements please [create an
-  issue](https://git.platypush.tech/platypush/platypush/issues).
-
-- A [Matrix instance](https://matrix.to/#/#platypush:matrix.platypush.tech) is
-  available if you are looking for interactive support.
-
-- A [Lemmy instance](https://lemmy.platypush.tech/c/platypush) is available for
-  general questions.
-
 ## Introduction
 
 Platypush is a general-purpose extensible platform for automation across
@@ -124,26 +105,82 @@ You can use Platypush to do things like:
 
 ## Installation
 
-### System installation
+### Prerequisites
 
-Platypush uses Redis to deliver and store requests and temporary messages:
+Platypush uses [Redis](https://redis.io/) to dispatch requests, responses,
+events and custom messages across several processes and integrations.
+
+#### Docker installation
+
+You can run Redis on the fly on your local machine using a Docker image:
+
+```bash
+# Expose a Redis server on port 6379 (default)
+docker run --rm -p 6379:6379 --name redis redis
+```
+
+#### Use an external service
+
+You can let Platypush use an external Redis service, if you wish to avoid
+running one on the same machine.
+
+In such scenario, simply start the application by passing custom values for
+`--redis-host` and `--redis-port`, or configure these values in its
+configuration file:
 
 ```yaml
-# Example for Debian-based distributions
+redis:
-[sudo] apt-get install redis-server
+  host: some-ip
+  port: some-port
+```
 
+If you wish to run multiple instances that use the same Redis server, you may
+also want to customize the name of the default queue that they use
+(`--redis-queue` command-line option) in order to avoid conflicts.
+
+#### Manual installation
+
+Unless you are running Platypush in a Docker container, or you are running
+Redis in a Docker container, or you want to use a remote Redis service, the
+Redis server should be installed on the same machine where Platypush runs:
+
+```bash
+# On Debian-based distributions
+sudo apt install redis-server
+
+# On Arch-based distributions
+# The hiredis package is also advised
+sudo pacman -S redis
+
+# On MacOS
+brew install redis
+```
+
+Once Redis is installed, you have two options:
+
+1. Run it a separate service. This depends on your operating system and
+   supervisor/service controller. For example, on systemd:
+
+```bash
 # Enable and start the service
-[sudo] systemctl enable redis
+sudo systemctl enable redis
-[sudo] systemctl start redis
+sudo systemctl start redis
 ```
 
-#### Install through `pip`
+2. Let Platypush run and control the Redis service. This is a good option if
+   you want Platypush to run its own service, separate from any other one
+   running on the same machine, and terminate it as soon as the application
+   ends. In this case, simply launch the application with the `--start-redis`
+   option (and optionally `--redis-port <any-num>` to customize the listen
+   port).
 
-```shell
+### Install through `pip`
-[sudo] pip3 install platypush
+
+```bash
+[sudo] pip install platypush
 ```
 
-#### Install through a system package manager
+### Install through a system package manager
 
 Note: currently only Arch Linux and derived distributions are supported.
 
@@ -154,7 +191,7 @@ latest stable version) or the
 (for the latest git version) through your favourite AUR package manager. For
 example, using `yay`:
 
-```shell
+```bash
 yay platypush
 # Or
 yay platypush-git
@@ -163,14 +200,12 @@ yay platypush-git
 The Arch Linux packages on AUR are automatically updated upon new git commits
 or tags.
 
-#### Install from sources
+### Install from sources
 
 ```shell
 git clone https://git.platypush.tech/platypush/platypush.git
 cd platypush
 [sudo] pip install .
-# Or
-[sudo] python3 setup.py install
 ```
 
 ### Installing the dependencies for your extensions
@@ -224,6 +259,8 @@ You can then start the service by simply running:
 platypush
 ```
 
+See `platypush --help` for a full list of options.
+
 It's advised to run it as a systemd service though - simply copy the provided
 [`.service`
 file](https://git.platypush.tech/platypush/platypush/src/branch/master/examples/systemd/platypush.service)
@@ -751,6 +788,30 @@ of Platypush to your fingertips.
 To run the tests simply run `pytest` either from the project root folder or the
 `tests/` folder.
 
+## Useful links
+- Recommended read: [**Getting started with Platypush**](https://blog.platypush.tech/article/Ultimate-self-hosted-automation-with-Platypush).
+
+- The [blog](https://blog.platypush.tech) is a good place to get more insights
+  and inspiration on what you can build.
+
+- The [wiki](https://git.platypush.tech/platypush/platypush/wiki) also
+  contains many resources on getting started.
+
+- Extensive documentation for all the available integrations and messages [is
+  available](https://docs.platypush.tech/).
+
+- If you have issues/feature requests/enhancements please [create an
+  issue](https://git.platypush.tech/platypush/platypush/issues).
+
+- A [Matrix instance](https://matrix.to/#/#platypush:matrix.platypush.tech) is
+  available if you are looking for interactive support.
+
+- An IRC channel is also available at `#platypush@irc.platypush.tech:6697` (SSL
+  only).
+
+- A [Lemmy instance](https://lemmy.platypush.tech/c/platypush) is available for
+  general questions.
+
 ---
 
 ## Funding

bin/platypush

@@ -1,13 +0,0 @@
-#!python3
-
-import sys
-
-from platypush.app import Application
-
-
-if __name__ == '__main__':
-    app = Application.build(*sys.argv[1:])
-    app.run()
-
-
-# vim:sw=4:ts=4:et:

bin/platyvenv

@@ -1,249 +0,0 @@
-#!/bin/bash
-
-##############################################################################
-# This script allows you to easily manage Platypush instances through Python #
-# virtual environment. You can build environments from a config.yaml file    #
-# and automatically managed the required dependencies, as well as start,     #
-# stop and remove them                                                       #
-#                                                                            #
-# @author: Fabio Manganiello <fabio@platypush.tech>                          #
-# @licence: MIT                                                              #
-##############################################################################
-
-
-workdir="$HOME/.local/share/platypush/venv"
-
-function build {
-    cfgfile=
-
-    while getopts ':c:' opt; do
-        case ${opt} in
-            c)
-                cfgfile=$OPTARG;;
-            \?)
-                echo "Invalid option: -$OPTARG" >&2
-                exit 1;;
-            :)
-                echo "Option -$OPTARG requires the path to a Platypush configuration file" >&2
-                exit 1;;
-        esac
-    done
-
-    if [[ -z "$cfgfile" ]]; then
-        echo "Usage: $0 build -c <path-to-platypush-config-file>" >&2
-        exit 1
-    fi
-
-    echo "Parsing configuration file"
-    pip_cmd=
-    pkg_cmd=
-    includes=()
-    cmd_exec=()
-
-    while read -r line; do
-      if echo "$line" | grep -E "^pip:\s*"; then
-        pip_cmd="$(echo "$line" | sed -r -e 's/^pip:\s*(.*)'/\\1/)"
-      elif echo "$line" | grep -E "^packages:\s*"; then
-        pkg_cmd="$(echo "$line" | sed -r -e 's/^packages:\s*(.*)'/\\1/)"
-      elif echo "$line" | grep -E "^exec:\s*"; then
-        cmd_exec+=("$(echo "$line" | sed -r -e 's/^exec:\s*(.*)'/\\1/)")
-      elif echo "$line" | grep -E "^include:\s*"; then
-        includes+=("$(echo "$line" | sed -r -e 's/^include:\s*(.*)'/\\1/)")
-      elif echo "$line" | grep -E "^device_id:\s*"; then
-        device_id="$(echo "$line" | sed -r -e 's/^device_id:\s*(.*)'/\\1/)"
-      fi
-    done <<< "$(python <<EOF
-from platypush.config import Config
-from platypush.utils.manifest import get_install_commands_from_conf
-
-deps = get_install_commands_from_conf('$(realpath "${cfgfile}")')
-print(f'device_id: {Config.get("device_id")}')
-
-if deps.get('pip'):
-    print(f'pip: {deps["pip"]}')
-
-if deps.get('packages'):
-    print(f'packages: {deps["packages"]}')
-
-for cmd in deps.get('exec', []):
-    print(f'exec: {cmd}')
-
-for include in Config._included_files:
-    print(f'include: {include}')
-EOF
-)"
-
-    envdir="${workdir}/${device_id}"
-    etcdir="${envdir}/etc/platypush"
-
-    echo "Preparing virtual environment for device $device_id"
-    mkdir -p "$envdir"
-    mkdir -p "$etcdir"
-    srcdir=$(dirname "$cfgfile")
-
-    for ((i=0; i < ${#includes[@]}; i++)); do
-        incdir=$(dirname "${includes[$i]}")
-        incdir=$(realpath --relative-to="$srcdir" "$incdir")
-        destdir="$etcdir/$incdir"
-        mkdir -p "$destdir"
-        cp "${includes[$i]}" "$destdir"
-    done
-
-    cp "$cfgfile" "$etcdir/config.yaml"
-    cfgfile="${etcdir}/config.yaml"
-
-    python3 -m venv "${envdir}"
-    cd "${envdir}" || exit 1
-    source "${envdir}/bin/activate"
-
-    echo "Installing required dependencies"
-    # shellcheck disable=SC2086
-    [ -n "${pkg_cmd}" ] && sudo ${pkg_cmd}
-    [ -n "${pip_cmd}" ] && ${pip_cmd}
-
-    for ((i=0; i < ${#cmd_exec[@]}; i++)); do
-        ${cmd_exec[$i]}
-    done
-
-    pip install --upgrade git+https://git.platypush.tech/platypush/platypush.git
-    echo "Platypush virtual environment prepared under $envdir"
-}
-
-function start {
-    if [[ -z "$1" ]]; then
-        echo "Usage: $0 start <env-name>" >&2
-        exit 1
-    fi
-
-    env=$1
-    envdir="${workdir}/${env}"
-    rundir="${envdir}/var/run"
-    pidfile="${rundir}/platypush.pid"
-    cfgfile="${envdir}/etc/platypush/config.yaml"
-
-    if [[ ! -d "$envdir" ]]; then
-        echo "No such directory: $envdir" >&2
-        exit 1
-    fi
-
-    mkdir -p "${rundir}"
-
-    if [[ -f "$pidfile" ]]; then
-        if pgrep -F "${pidfile}"; then
-            echo "Another instance (PID $(cat "${pidfile}")) is running, please stop that instance first"
-            exit 1
-        fi
-
-        echo "A PID file was found but the process does not seem to be running, starting anyway"
-        rm -f "$pidfile"
-    fi
-
-    python3 -m venv "${envdir}"
-    cd "${envdir}" || exit 1
-    source bin/activate
-    bin/platypush -c "$cfgfile" -P "$pidfile" &
-    start_time=$(date +'%s')
-    timeout=30
-
-    while :; do
-        [[ -f "$pidfile" ]] && break
-        now=$(date +'%s')
-        elapsed=$(( now-start_time ))
-        if (( elapsed >= timeout )); then
-            echo "Platypush instance '$env' did not start within $timeout seconds" >&2
-            exit 1
-        fi
-
-        echo -n '.'
-        sleep 1
-    done
-
-    pid=$(cat "$pidfile")
-    echo
-    echo "Platypush environment $env started with PID $pid"
-    wait "${pid}"
-    echo "Platypush environment $env terminated"
-}
-
-function stop {
-    if [[ -z "$1" ]]; then
-        echo "Usage: $0 stop <env-name>" >&2
-        exit 1
-    fi
-
-    env=$1
-    envdir="${workdir}/${env}"
-    rundir="${envdir}/var/run"
-    pidfile="${rundir}/platypush.pid"
-
-    if [[ ! -d "$envdir" ]]; then
-        echo "No such directory: $envdir" >&2
-        exit 1
-    fi
-
-    if [[ ! -f "$pidfile" ]]; then
-        echo "No pidfile found for instance \"${env}\""
-        exit 1
-    fi
-
-    pid=$(cat "$pidfile")
-    pids="$pid $(ps --no-headers -o pid= --ppid "$pid")"
-    # shellcheck disable=SC2086
-    kill -9 ${pids}
-    rm -f "$pidfile"
-    echo "Instance '$env' with PID $pid stopped"
-}
-
-function rme {
-    if [[ -z "$1" ]]; then
-        echo "Usage: $0 rm <env-name>" >&2
-        exit 1
-    fi
-
-    envdir="${workdir}/$1"
-    rundir="${envdir}/var/run"
-    pidfile="${rundir}/platypush.pid"
-
-    if [[ ! -d "$envdir" ]]; then
-        echo "No such directory: $envdir" >&2
-        exit 1
-    fi
-
-    if [[ -f "$pidfile" ]]; then
-        if pgrep -F "${pidfile}"; then
-            echo "Another instance (PID $(cat "$pidfile")) is running, please stop that instance first"
-            exit 1
-        fi
-
-        echo "A PID file was found but the process does not seem to be running, removing anyway"
-    fi
-
-    echo "WARNING: This operation will permanently remove the Platypush environment $1"
-    echo -n "Are you sure you want to continue? (y/N) "
-    IFS= read -r answer
-    echo "$answer" | grep -E '^[yY]' >/dev/null || exit 0
-    rm -rf "$envdir"
-    echo "$envdir removed"
-}
-
-function usage {
-    echo "Usage: $0 <build|start|stop|rm> [options]" >&2
-    exit 1
-}
-
-if (( $# < 1 )); then
-    usage
-fi
-
-action=$1
-shift
-mkdir -p "${workdir}"
-
-# shellcheck disable=SC2048,SC2086
-case ${action} in
-    'build') build $*;;
-    'start') start $*;;
-    'stop') stop $*;;
-    'rm') rme $*;;
-    *) usage;;
-esac

docs/source/_ext/add_dependencies.py

@@ -0,0 +1,186 @@
+import inspect
+import os
+import re
+import sys
+import textwrap as tw
+
+from sphinx.application import Sphinx
+
+base_path = os.path.abspath(
+    os.path.join(os.path.dirname(os.path.relpath(__file__)), '..', '..', '..')
+)
+
+sys.path.insert(0, base_path)
+
+from platypush.common.reflection import Integration  # noqa: E402
+from platypush.utils import get_plugin_name_by_class  # noqa: E402
+from platypush.utils.mock import auto_mocks  # noqa: E402
+
+
+class IntegrationEnricher:
+    @staticmethod
+    def add_events(source: list[str], manifest: Integration, idx: int) -> int:
+        if not manifest.events:
+            return idx
+
+        source.insert(
+            idx,
+            'Triggered events\n----------------\n\n'
+            + '\n'.join(
+                f'\t- :class:`{event.__module__}.{event.__qualname__}`'
+                for event in manifest.events
+            )
+            + '\n\n',
+        )
+
+        return idx + 1
+
+    @staticmethod
+    def add_actions(source: list[str], manifest: Integration, idx: int) -> int:
+        if not (manifest.actions and manifest.cls):
+            return idx
+
+        source.insert(
+            idx,
+            'Actions\n-------\n\n'
+            + '\n'.join(
+                f'\t- `{get_plugin_name_by_class(manifest.cls)}.{action} '
+                + f'<#{manifest.cls.__module__}.{manifest.cls.__qualname__}.{action}>`_'
+                for action in sorted(manifest.actions.keys())
+            )
+            + '\n\n',
+        )
+
+        return idx + 1
+
+    @staticmethod
+    def _shellify(title: str, cmd: str) -> str:
+        return (
+            f'**{title}**\n\n'
+            + '.. code-block:: bash\n\n'
+            + tw.indent(cmd, '\t')
+            + '\n\n'
+        )
+
+    @classmethod
+    def add_install_deps(
+        cls, source: list[str], manifest: Integration, idx: int
+    ) -> int:
+        deps = manifest.deps
+        parsed_deps = {
+            'before': deps.before,
+            'pip': deps.pip,
+            'after': deps.after,
+        }
+
+        if not (any(parsed_deps.values()) or deps.by_pkg_manager):
+            return idx
+
+        source.insert(idx, 'Dependencies\n------------\n\n')
+        idx += 1
+
+        if parsed_deps['before']:
+            source.insert(idx, cls._shellify('Pre-install', '\n'.join(deps.before)))
+            idx += 1
+
+        if parsed_deps['pip']:
+            source.insert(
+                idx, cls._shellify('pip', 'pip install ' + ' '.join(deps.pip))
+            )
+            idx += 1
+
+        for pkg_manager, sys_deps in deps.by_pkg_manager.items():
+            if not sys_deps:
+                continue
+
+            source.insert(
+                idx,
+                cls._shellify(
+                    pkg_manager.value.default_os.value.description,
+                    pkg_manager.value.install_doc + ' ' + ' '.join(sys_deps),
+                ),
+            )
+
+            idx += 1
+
+        if parsed_deps['after']:
+            source.insert(idx, cls._shellify('Post-install', '\n'.join(deps.after)))
+            idx += 1
+
+        return idx
+
+    @classmethod
+    def add_description(cls, source: list[str], manifest: Integration, idx: int) -> int:
+        docs = (
+            doc
+            for doc in (
+                inspect.getdoc(manifest.cls) or '',
+                manifest.constructor.doc if manifest.constructor else '',
+            )
+            if doc
+        )
+
+        if not docs:
+            return idx
+
+        docstring = '\n\n'.join(docs)
+        source.insert(idx, f"Description\n-----------\n\n{docstring}\n\n")
+        return idx + 1
+
+    @classmethod
+    def add_conf_snippet(
+        cls, source: list[str], manifest: Integration, idx: int
+    ) -> int:
+        source.insert(
+            idx,
+            tw.dedent(
+                f"""
+                Configuration
+                -------------
+
+                .. code-block:: yaml
+
+{tw.indent(manifest.config_snippet, '                  ')}
+                """
+            ),
+        )
+
+        return idx + 1
+
+    def __call__(self, _: Sphinx, doc: str, source: list[str]):
+        if not (source and re.match(r'^platypush/(backend|plugins)/.*', doc)):
+            return
+
+        src = [src.split('\n') for src in source][0]
+        if len(src) < 3:
+            return
+
+        manifest_file = os.path.join(
+            base_path,
+            *doc.split(os.sep)[:-1],
+            *doc.split(os.sep)[-1].split('.'),
+            'manifest.yaml',
+        )
+
+        if not os.path.isfile(manifest_file):
+            return
+
+        with auto_mocks():
+            manifest = Integration.from_manifest(manifest_file)
+            idx = self.add_description(src, manifest, idx=3)
+            idx = self.add_conf_snippet(src, manifest, idx=idx)
+            idx = self.add_install_deps(src, manifest, idx=idx)
+            idx = self.add_events(src, manifest, idx=idx)
+            idx = self.add_actions(src, manifest, idx=idx)
+
+        src.insert(idx, '\n\nModule reference\n----------------\n\n')
+        source[0] = '\n'.join(src)
+
+
+def setup(app: Sphinx):
+    app.connect('source-read', IntegrationEnricher())
+    return {
+        'version': '0.1',
+        'parallel_read_safe': True,
+        'parallel_write_safe': True,
+    }

docs/source/_static/scripts/custom.js

@@ -0,0 +1,196 @@
+const processList = (list, level, addTitle) => {
+  const title = list.parentElement.querySelector('a')
+  list.classList.add('grid')
+  if (addTitle)
+    title.classList.add('grid-title')
+
+  list.querySelectorAll(`li.toctree-l${level}`).forEach((item) => {
+    const link = item.querySelector('a')
+    if (link) {
+      item.style.cursor = 'pointer'
+      item.addEventListener('click', () => link.click())
+    }
+
+    const name = item.querySelector('a').innerText
+    const img = document.createElement('img')
+    img.src = `https://static.platypush.tech/icons/${name.toLowerCase()}-64.png`
+    img.alt = ' '
+    item.prepend(img)
+  })
+}
+
+const addClipboard = (parent) => {
+  const pre = parent.tagName === 'PRE' ? parent : parent.querySelector('pre')
+  if (!pre)
+    return
+
+  const clipboard = document.createElement('i')
+  const setClipboard = (img, text) => {
+    clipboard.innerHTML = `<img src="https://static.platypush.tech/icons/${img}-64.png" alt="${text}">`
+  }
+
+  clipboard.classList.add('clipboard')
+  setClipboard('clipboard-bw', 'Copy')
+  clipboard.onclick = () => {
+    if (navigator && navigator.clipboard && navigator.clipboard.writeText) {
+      setClipboard('ok', 'Copied!')
+      setTimeout(() => setClipboard('clipboard-bw', 'Copy'), 2000)
+      return navigator.clipboard.writeText(pre.innerText.trim())
+    }
+
+    return Promise.reject('The Clipboard API is not available.');
+  }
+
+  pre.style.position = 'relative'
+  pre.appendChild(clipboard)
+}
+
+const Tabs = () => {
+  let selectedTab = null
+  let parent = null
+  let data = {}
+
+  const init = (obj) => {
+    data = obj
+    if (Object.keys(data).length && selectedTab == null)
+      selectedTab = Object.keys(data)[0]
+  }
+
+  const select = (name) => {
+    if (!parent) {
+      console.warn('Cannot select tab: parent not set')
+      return
+    }
+
+    if (!data[name]) {
+      console.warn(`Cannot select tab: invalid name: ${name}`)
+      return
+    }
+
+    const tabsBody = parent.querySelector('.body')
+    selectedTab = name
+    tabsBody.innerHTML = data[selectedTab]
+    parent.querySelectorAll('.tabs li').forEach(
+      (tab) => tab.classList.remove('selected')
+    )
+
+    const tab = [...parent.querySelectorAll('.tabs li')].find(
+      (t) => t.innerText === name
+    )
+
+    if (!tab) {
+      console.warn(`Cannot select tab: invalid name: ${name}`)
+      return
+    }
+
+    addClipboard(tabsBody)
+    tab.classList.add('selected')
+  }
+
+  const mount = (p) => {
+    const tabs = document.createElement('div')
+    tabs.classList.add('tabs')
+    parent = p
+
+    const tabsList = document.createElement('ul')
+    Object.keys(data).forEach((title) => {
+      const tab = document.createElement('li')
+      tab.innerText = title
+      tab.onclick = (event) => {
+        event.stopPropagation()
+        select(title)
+      },
+
+      tabsList.appendChild(tab)
+    })
+
+    const tabsBody = document.createElement('div')
+    tabsBody.classList.add('body')
+
+    tabs.appendChild(tabsList)
+    tabs.appendChild(tabsBody)
+    parent.innerHTML = ''
+    parent.appendChild(tabs)
+    select(selectedTab)
+  }
+
+  return {
+    init,
+    select,
+    mount,
+  }
+}
+
+const depsTabs = Tabs()
+
+const convertDepsToTabs = () => {
+  const depsContainer = document.getElementById('dependencies')
+  if (!depsContainer)
+    return
+
+  const blocks = [...depsContainer.querySelectorAll('.highlight-bash')].map((block) => block.outerHTML)
+  const titles = [...depsContainer.querySelectorAll('p strong')].map((title) => title.innerText)
+
+  if (!(blocks.length && titles.length && blocks.length === titles.length))
+    return
+
+  const title = depsContainer.querySelector('h2')
+  const tabsData = titles.reduce((obj, title, i) => {
+    obj[title] = blocks[i]
+    return obj
+  }, {})
+
+  depsTabs.init(tabsData)
+  depsTabs.mount(depsContainer)
+  depsContainer.prepend(title)
+}
+
+const generateComponentsGrid = () => {
+  const tocWrappers = document.querySelectorAll('.toctree-wrapper.compound')
+
+  if (!tocWrappers.length) {
+    return
+  }
+
+  if (window.location.pathname.endsWith('/index.html')) {
+    if (tocWrappers.length < 2) {
+      return
+    }
+
+    const referenceLists = [
+      ...tocWrappers[1].querySelectorAll('ul li.toctree-l1 ul')
+    ].slice(0, 4)
+
+    referenceLists.forEach((list) => processList(list, 2, true))
+  } else if (window.location.pathname.endsWith('/plugins.html') || window.location.pathname.endsWith('/backends.html')) {
+    if (tocWrappers.length < 1) {
+      return
+    }
+
+    const list = tocWrappers[0].querySelector('ul')
+    if (list)
+      processList(list, 1, false)
+  }
+}
+
+const addClipboardToCodeBlocks = () => {
+  document.querySelectorAll('pre').forEach((pre) => addClipboard(pre))
+}
+
+const renderActionsList = () => {
+  const actionsList = document.getElementById('actions')?.querySelector('ul')
+  if (!actionsList)
+    return
+
+  [...actionsList.querySelectorAll('li')].forEach((li) => {
+    const link = li.querySelector('a')
+    link.innerHTML = `<code class="docutils literal notranslate"><span class="pre">${link.innerText}</span></code>`
+  })
+}
+
+document.addEventListener("DOMContentLoaded", function() {
+  generateComponentsGrid()
+  convertDepsToTabs()
+  addClipboardToCodeBlocks()
+  renderActionsList()
+})

docs/source/_static/styles/custom.css

@@ -0,0 +1,130 @@
+a, a:visited {
+  /* Don't change the color for visited links */
+  color: var(--pst-color-link) !important;
+}
+
+ul.grid {
+  display: grid;
+
+  @media screen and (max-width: 500px) {
+    grid-template-columns: repeat(1, 1fr);
+  }
+
+  @media screen and (min-width: 501px) and (max-width: 699px) {
+    grid-template-columns: repeat(2, 1fr);
+  }
+
+  @media screen and (min-width: 700px) {
+    grid-template-columns: repeat(3, 1fr);
+  }
+}
+
+a.grid-title {
+  width: 100%;
+  display: block;
+  margin: 1.5em 0;
+  font-size: 1.5em !important;
+  border-bottom: 1px solid #ccc;
+}
+
+ul.grid li {
+  display: flex;
+  align-items: center;
+  margin: 0 10px 10px 0;
+  padding: 10px;
+  border: 1px solid #ccc;
+  border-radius: 15px;
+}
+
+ul.grid img {
+  width: 32px;
+  margin-right: 5px;
+}
+
+ul.grid li code {
+  width: 100%;
+}
+
+ul.grid li code .pre {
+  width: 100%;
+  display: block;
+  white-space: pre-wrap;
+}
+
+ul.grid li:hover {
+  background: linear-gradient(0deg, #e0ffe8, #e3ffff);
+}
+
+ul.grid li a {
+  width: calc(100% - 35px);
+  display: flex;
+  justify-content: center;
+}
+
+ul.grid li a code {
+  background: none;
+  border: none;
+}
+
+ul.grid .icon {
+  width: 32px;
+}
+
+/* Clipboard button */
+.clipboard {
+  position: absolute;
+  display: inline-block;
+  width: 32px;
+  top: 0.5em;
+  right: 0.5em;
+  cursor: pointer;
+}
+
+/* Tabs */
+.tabs {
+  margin: 0 0 1em 0;
+  padding: 0;
+  list-style: none;
+}
+
+.tabs ul {
+  display: flex;
+  flex-direction: row;
+  justify-content: space-between;
+  align-items: center;
+  margin: 0 0 1em 0;
+  padding: 0;
+  list-style: none;
+  border-bottom: 1px solid #ccc;
+}
+
+.tabs ul li {
+  display: inline-flex;
+  max-width: 25%;
+  margin: 0;
+  padding: 0.25em 0.5em;
+  list-style: none;
+  cursor: pointer;
+  flex-grow: 1;
+  justify-content: center;
+  align-items: center;
+  border-radius: 0.75em 0.75em 0 0;
+  border: 1px solid #ddd;
+}
+
+.tabs ul li.selected {
+  background: rgb(200,255,208);
+}
+
+.tabs ul li:hover {
+  background: rgb(190,246,218);
+}
+
+.tabs .body {
+  margin-top: -1em;
+  padding: 1em;
+  border: 1px solid #ccc;
+  border-top: none;
+  border-radius: 0 0 0.75em 0.75em;
+}
+

docs/source/backends.rst

@@ -8,12 +8,9 @@ Backends
 
     platypush/backend/adafruit.io.rst
     platypush/backend/alarm.rst
-    platypush/backend/assistant.google.rst
-    platypush/backend/assistant.snowboy.rst
     platypush/backend/button.flic.rst
     platypush/backend/camera.pi.rst
     platypush/backend/chat.telegram.rst
-    platypush/backend/covid19.rst
     platypush/backend/file.monitor.rst
     platypush/backend/foursquare.rst
     platypush/backend/github.rst
@@ -21,17 +18,13 @@ Backends
     platypush/backend/google.pubsub.rst
     platypush/backend/gps.rst
     platypush/backend/http.rst
-    platypush/backend/http.poll.rst
-    platypush/backend/inotify.rst
     platypush/backend/joystick.rst
     platypush/backend/joystick.jstest.rst
     platypush/backend/joystick.linux.rst
     platypush/backend/kafka.rst
-    platypush/backend/light.hue.rst
     platypush/backend/log.http.rst
     platypush/backend/mail.rst
     platypush/backend/midi.rst
-    platypush/backend/mqtt.rst
     platypush/backend/music.mopidy.rst
     platypush/backend/music.mpd.rst
     platypush/backend/music.snapcast.rst
@@ -50,11 +43,8 @@ Backends
     platypush/backend/stt.picovoice.speech.rst
     platypush/backend/tcp.rst
     platypush/backend/todoist.rst
-    platypush/backend/travisci.rst
     platypush/backend/trello.rst
     platypush/backend/weather.buienradar.rst
     platypush/backend/weather.darksky.rst
     platypush/backend/weather.openweathermap.rst
     platypush/backend/wiimote.rst
-    platypush/backend/zwave.rst
-    platypush/backend/zwave.mqtt.rst

docs/source/conf.py

@@ -15,17 +15,14 @@ import sys
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
 sys.path.insert(0, os.path.abspath("./_ext"))
 
 
 # -- Project information -----------------------------------------------------
 
 project = 'Platypush'
-copyright = '2017-2021, Fabio Manganiello'
+copyright = '2017-2023, Fabio Manganiello'
-author = 'Fabio Manganiello'
+author = 'Fabio Manganiello <fabio@manganiello.tech>'
 
 # The short X.Y version
 version = ''
@@ -43,6 +40,7 @@ release = ''
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    'myst_parser',
     'sphinx.ext.autodoc',
     'sphinx.ext.intersphinx',
     'sphinx.ext.todo',
@@ -52,6 +50,7 @@ extensions = [
     'sphinx.ext.githubpages',
     'sphinx_rtd_theme',
     'sphinx_marshmallow',
+    'add_dependencies',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -60,8 +59,8 @@ templates_path = ['_templates']
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-# source_suffix = ['.rst', '.md']
+source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+# source_suffix = '.rst'
 
 # The master toctree document.
 master_doc = 'index'
@@ -113,7 +112,14 @@ html_theme_options = {
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-# html_static_path = ['_static']
+html_static_path = ['_static']
+html_css_files = [
+    'styles/custom.css',
+]
+
+html_js_files = [
+    'scripts/custom.js',
+]
 
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.
@@ -165,9 +171,9 @@ latex_documents = [
 man_pages = [(master_doc, 'platypush', 'platypush Documentation', [author], 1)]
 
 
-# -- Options for Texinfo output ----------------------------------------------
+# -- Options for TexInfo output ----------------------------------------------
 
-# Grouping the document tree into Texinfo files. List of tuples
+# Grouping the document tree into TexInfo files. List of tuples
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
@@ -190,136 +196,30 @@ texinfo_documents = [
 # Example configuration for intersphinx: refer to the Python standard library.
 intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
 
-# -- Options for todo extension ----------------------------------------------
-
-# If true, `todo` and `todoList` produce output, else they produce nothing.
-todo_include_todos = True
-
 autodoc_default_options = {
     'members': True,
     'show-inheritance': True,
 }
 
-autodoc_mock_imports = [
-    'gunicorn',
-    'googlesamples.assistant.grpc.audio_helpers',
-    'google.assistant.embedded',
-    'google.assistant.library',
-    'google.assistant.library.event',
-    'google.assistant.library.file_helpers',
-    'google.oauth2.credentials',
-    'oauth2client',
-    'apiclient',
-    'tenacity',
-    'smartcard',
-    'Leap',
-    'oauth2client',
-    'rtmidi',
-    'bluetooth',
-    'gevent.wsgi',
-    'Adafruit_IO',
-    'pyclip',
-    'pydbus',
-    'inputs',
-    'inotify',
-    'omxplayer',
-    'plexapi',
-    'cwiid',
-    'sounddevice',
-    'soundfile',
-    'numpy',
-    'cv2',
-    'nfc',
-    'ndef',
-    'bcrypt',
-    'google',
-    'feedparser',
-    'kafka',
-    'googlesamples',
-    'icalendar',
-    'httplib2',
-    'mpd',
-    'serial',
-    'pyHS100',
-    'grpc',
-    'envirophat',
-    'gps',
-    'picamera',
-    'pmw3901',
-    'PIL',
-    'croniter',
-    'pyaudio',
-    'avs',
-    'PyOBEX',
-    'PyOBEX.client',
-    'todoist',
-    'trello',
-    'telegram',
-    'telegram.ext',
-    'pyfirmata2',
-    'cups',
-    'graphyte',
-    'cpuinfo',
-    'psutil',
-    'openzwave',
-    'deepspeech',
-    'wave',
-    'pvporcupine ',
-    'pvcheetah',
-    'pyotp',
-    'linode_api4',
-    'pyzbar',
-    'tensorflow',
-    'keras',
-    'pandas',
-    'samsungtvws',
-    'paramiko',
-    'luma',
-    'zeroconf',
-    'dbus',
-    'gi',
-    'gi.repository',
-    'twilio',
-    'Adafruit_Python_DHT',
-    'RPi.GPIO',
-    'RPLCD',
-    'imapclient',
-    'pysmartthings',
-    'aiohttp',
-    'watchdog',
-    'pyngrok',
-    'irc',
-    'irc.bot',
-    'irc.strings',
-    'irc.client',
-    'irc.connection',
-    'irc.events',
-    'defusedxml',
-    'nio',
-    'aiofiles',
-    'aiofiles.os',
-    'async_lru',
-    'bleak',
-    'bluetooth_numbers',
-    'TheengsDecoder',
-    'simple_websocket',
-    'uvicorn',
-    'websockets',
-    'docutils',
-    'aioxmpp',
-]
-
 sys.path.insert(0, os.path.abspath('../..'))
 
+from platypush.utils.mock.modules import mock_imports  # noqa
 
-def skip(app, what, name, obj, skip, options):
+autodoc_mock_imports = [*mock_imports]
+
+
+# _ = app
+# __ = what
+# ___ = obj
+# ____ = options
+def _skip(_, __, name, ___, skip, ____):
     if name == "__init__":
         return False
     return skip
 
 
 def setup(app):
-    app.connect("autodoc-skip-member", skip)
+    app.connect("autodoc-skip-member", _skip)
 
 
 # vim:sw=4:ts=4:et:

docs/source/events.rst

@@ -16,7 +16,6 @@ Events
     platypush/events/chat.slack.rst
     platypush/events/chat.telegram.rst
     platypush/events/clipboard.rst
-    platypush/events/covid19.rst
     platypush/events/custom.rst
     platypush/events/dbus.rst
     platypush/events/distance.rst
@@ -70,7 +69,6 @@ Events
     platypush/events/tensorflow.rst
     platypush/events/todoist.rst
     platypush/events/torrent.rst
-    platypush/events/travisci.rst
     platypush/events/trello.rst
     platypush/events/video.rst
     platypush/events/weather.rst

docs/source/index.rst

@@ -1,23 +1,50 @@
 Platypush
 #########
 
-Welcome to the Platypush reference of available plugins, backends and event types.
+Description
+===========
 
-For more information on Platypush check out:
+This is the main documentation hub for Platypush. It includes both the wiki and
+the complete reference of the available integrations.
 
-* The `main page`_ of the project
+Platypush is a general-purpose automation framework that can be used to cover
-* The `Gitea page`_ of the project
+all the cases where you'd use a home automation hub, a media center, a smart
-* The `online wiki`_ for quickstart and examples
+assistant, some IFTTT recipes, and a variety of other products and services.
-* The `Blog articles`_ for inspiration on use-cases possible projects
 
-.. _main page: https://platypush.tech
+It draws inspiration from the following projects, and it aims to cover all of
-.. _Gitea page: https://git.platypush.tech/platypush/platypush
+their use-cases:
-.. _online wiki: https://git.platypush.tech/platypush/platypush/wiki
+
-.. _Blog articles: https://blog.platypush.tech
+* `Home Assistant <https://www.home-assistant.io/>`_
+* `Homebridge <https://homebridge.io/>`_
+* `OpenHAB <https://www.openhab.org/>`_
+* `IFTTT <https://ifttt.com/>`_
+* `Tasker <https://tasker.joaoapps.com/>`_
+
+Useful links
+============
+
+* The `main page <https://platypush.tech>`_ of the project.
+* The `Gitea page <https://git.platypush.tech/platypush/platypush>`_.
+* The `blog <https://blog.platypush.tech>`_, for articles showing how to use
+  Platypush in real-world scenarios.
+
+Wiki
+====
+
+.. toctree::
+    :maxdepth: 3
+
+    wiki/index
+    wiki/Installation
+    wiki/Configuration
+    wiki/Installing-extensions
+    wiki/A-configuration-example
+
+Reference
+=========
 
 .. toctree::
     :maxdepth: 2
-    :caption: Contents:
 
     backends
     plugins

docs/source/platypush/backend/assistant.google.rst

@@ -1,6 +0,0 @@
-``assistant.google``
-======================================
-
-.. automodule:: platypush.backend.assistant.google
-    :members:
-

docs/source/platypush/backend/assistant.snowboy.rst

@@ -1,6 +0,0 @@
-``assistant.snowboy``
-=======================================
-
-.. automodule:: platypush.backend.assistant.snowboy
-    :members:
-

docs/source/platypush/backend/covid19.rst

@@ -1,5 +0,0 @@
-``covid19``
-=============================
-
-.. automodule:: platypush.backend.covid19
-    :members:

docs/source/platypush/backend/http.poll.rst

@@ -1,6 +0,0 @@
-``http.poll``
-===============================
-
-.. automodule:: platypush.backend.http.poll
-    :members:
-

docs/source/platypush/backend/inotify.rst

@@ -1,6 +0,0 @@
-``inotify``
-=============================
-
-.. automodule:: platypush.backend.inotify
-    :members:
-

docs/source/platypush/backend/light.hue.rst

@@ -1,6 +0,0 @@
-``light.hue``
-===============================
-
-.. automodule:: platypush.backend.light.hue
-	:members:
-

docs/source/platypush/backend/mqtt.rst

@@ -1,6 +0,0 @@
-``mqtt``
-==========================
-
-.. automodule:: platypush.backend.mqtt
-    :members:
-

docs/source/platypush/backend/travisci.rst

@@ -1,5 +0,0 @@
-``travisci``
-==============================
-
-.. automodule:: platypush.backend.travisci
-    :members:

docs/source/platypush/backend/zwave.mqtt.rst

@@ -1,5 +0,0 @@
-``zwave.mqtt``
-================================
-
-.. automodule:: platypush.backend.zwave.mqtt
-    :members:

docs/source/platypush/backend/zwave.rst

@@ -1,5 +0,0 @@
-``zwave``
-===========================
-
-.. automodule:: platypush.backend.zwave
-    :members:

docs/source/platypush/events/covid19.rst

@@ -1,5 +0,0 @@
-``covid19``
-===================================
-
-.. automodule:: platypush.message.event.covid19
-    :members:

docs/source/platypush/events/travisci.rst

@@ -1,5 +0,0 @@
-``travisci``
-====================================
-
-.. automodule:: platypush.message.event.travisci
-    :members:

docs/source/platypush/events/xmpp.rst

@@ -1,5 +1,5 @@
-``event.xmpp``
+``xmpp``
-==============
+========
 
 .. automodule:: platypush.message.event.xmpp
     :members:

docs/source/platypush/plugins/application.rst

@@ -0,0 +1,5 @@
+``application``
+===============
+
+.. automodule:: platypush.plugins.application
+    :members:

docs/source/platypush/plugins/assistant.echo.rst

@@ -1,6 +0,0 @@
-``assistant.echo``
-====================================
-
-.. automodule:: platypush.plugins.assistant.echo
-    :members:
-

docs/source/platypush/plugins/assistant.google.pushtotalk.rst

@@ -1,6 +0,0 @@
-``assistant.google.pushtotalk``
-=================================================
-
-.. automodule:: platypush.plugins.assistant.google.pushtotalk
-    :members:
-

docs/source/platypush/plugins/covid19.rst

@@ -1,5 +0,0 @@
-``covid19``
-=============================
-
-.. automodule:: platypush.plugins.covid19
-    :members:

docs/source/platypush/plugins/http.request.rss.rst

@@ -1,6 +0,0 @@
-``http.request.rss``
-======================================
-
-.. automodule:: platypush.plugins.http.request.rss
-	:members:
-

docs/source/platypush/plugins/travisci.rst

@@ -1,5 +0,0 @@
-``travisci``
-==============================
-
-.. automodule:: platypush.plugins.travisci
-    :members:

docs/source/platypush/plugins/zwave.rst

@@ -1,5 +0,0 @@
-``zwave``
-===========================
-
-.. automodule:: platypush.plugins.zwave
-    :members:

docs/source/plugins.rst

@@ -8,10 +8,9 @@ Plugins
 
     platypush/plugins/adafruit.io.rst
     platypush/plugins/alarm.rst
+    platypush/plugins/application.rst
     platypush/plugins/arduino.rst
-    platypush/plugins/assistant.echo.rst
     platypush/plugins/assistant.google.rst
-    platypush/plugins/assistant.google.pushtotalk.rst
     platypush/plugins/autoremote.rst
     platypush/plugins/bluetooth.rst
     platypush/plugins/calendar.rst
@@ -26,7 +25,6 @@ Plugins
     platypush/plugins/chat.telegram.rst
     platypush/plugins/clipboard.rst
     platypush/plugins/config.rst
-    platypush/plugins/covid19.rst
     platypush/plugins/csv.rst
     platypush/plugins/db.rst
     platypush/plugins/dbus.rst
@@ -50,7 +48,6 @@ Plugins
     platypush/plugins/graphite.rst
     platypush/plugins/hid.rst
     platypush/plugins/http.request.rst
-    platypush/plugins/http.request.rss.rst
     platypush/plugins/http.webpage.rst
     platypush/plugins/ifttt.rst
     platypush/plugins/inputs.rst
@@ -128,7 +125,6 @@ Plugins
     platypush/plugins/tensorflow.rst
     platypush/plugins/todoist.rst
     platypush/plugins/torrent.rst
-    platypush/plugins/travisci.rst
     platypush/plugins/trello.rst
     platypush/plugins/tts.rst
     platypush/plugins/tts.google.rst
@@ -148,5 +144,4 @@ Plugins
     platypush/plugins/xmpp.rst
     platypush/plugins/zeroconf.rst
     platypush/plugins/zigbee.mqtt.rst
-    platypush/plugins/zwave.rst
     platypush/plugins/zwave.mqtt.rst

examples/conf/config.yaml

@@ -1,379 +0,0 @@
-#################################################################################
-# Sample platypush configuration file.
-# Edit it and copy it to /etc/platypush/config.yaml for system installation or to
-# ~/.config/platypush/config.yaml for user installation (recommended).
-#################################################################################
-
-# --
-# include directive example
-# --
-#
-# You can split your configuration over multiple files
-# and use the include directive to import them in your configuration.
-# Relative paths are also supported, and computed using the config.yaml
-# installation directory as base folder. Symlinks are also supported.
-#
-# Using multiple files is encouraged in the case of large configurations
-# that can easily end up in a messy config.yaml file, as they help you
-# keep your configuration more organized.
-#include:
-#    - include/logging.yaml
-#    - include/media.yaml
-#    - include/sensors.yaml
-
-# platypush logs on stdout by default. You can use the logging section to specify
-# an alternative file or change the logging level.
-#logging:
-#    filename: ~/.local/log/platypush/platypush.log
-#    level: INFO
-
-# The device_id is used by many components of platypush and it should uniquely
-# identify a device in your network. If nothing is specified then the hostname
-# will be used.
-#device_id: my_device
-
-## --
-## Plugin configuration examples
-## --
-#
-# Plugins configuration is very straightforward. Each plugin is mapped to
-# a plugin class. The methods of the class with @action annotation will
-# be exported as runnable actions, while the __init__ parameters are
-# configuration attributes that you can initialize in your config.yaml.
-# Plugin classes are documented at https://docs.platypush.tech/en/latest/plugins.html
-#
-# In this example we'll configure the light.hue plugin, see
-# https://docs.platypush.tech/en/latest/platypush/plugins/light.hue.html
-# for reference. You can easily install the required dependencies for the plugin through
-# pip install 'platypush[hue]'
-light.hue:
-    # IP address or hostname of the Hue bridge
-    bridge: 192.168.1.10
-    # Groups that will be handled by default if nothing is specified on the request
-    groups:
-        - Living Room
-
-# Example configuration of music.mpd plugin, see
-# https://docs.platypush.tech/en/latest/platypush/plugins/music.mpd.html
-# You can easily install the dependencies through pip install 'platypush[mpd]'
-music.mpd:
-    host: localhost
-    port: 6600
-
-# Example configuration of media.chromecast plugin, see
-# https://docs.platypush.tech/en/latest/platypush/plugins/media.chromecast.html
-# You can easily install the dependencies through pip install 'platypush[chromecast]'
-media.chromecast:
-    chromecast: Living Room TV
-
-# Plugins with empty configuration can also be explicitly enabled by specifying
-# enabled=True or disabled=False (it's a good practice if you want the
-# corresponding web panel to be enabled, if available)
-camera.pi:
-    enabled: True
-
-# Support for calendars - in this case Google and Facebook calendars
-# Installing the dependencies: pip install 'platypush[ical,google]'
-calendar:
-    calendars:
-        - type: platypush.plugins.google.calendar.GoogleCalendarPlugin
-        - type: platypush.plugins.calendar.ical.CalendarIcalPlugin
-          url: https://www.facebook.com/events/ical/upcoming/?uid=your_user_id&key=your_key
-
-## --
-## Backends configuration examples
-## --
-#
-# Backends are basically threads that run in the background and listen for something
-# to happen and either trigger events or provide additional services on top of platypush.
-# Just like plugins, backends are classes whose configuration matches one-to-one the
-# supported parameters on the __init__ methods. You can check the documentation for the
-# available backends here: https://docs.platypush.tech/en/latest/backends.html.
-# Moreover, most of the backends will generate events that you can react to through custom
-# event hooks. Check here for the events documentation:
-# https://docs.platypush.tech/en/latest/events.html
-#
-# You may usually want to enable the HTTP backend, as it provides many useful features on
-# top of platypush. Among those:
-#
-# - Expose the /execute endpoint, that allows you to send requests to platypush through a
-#   JSON-RPC interface.
-# - Web panel, one of the key additiona features of platypush. Many plugins will expose web
-#   panel tabs for e.g. accessing and controlling lights, music, media and sensors.
-# - Dashboard: platypush can be configured to show a custom dashboard on your screens with
-#   e.g. music platypush and weather info, news, upcoming calendar events and photo carousel.
-# - Streaming support - the HTTP backend makes it possible to stream local media to other
-#   devices - e.g. Chromecasts and external browsers.
-#
-# To install the HTTP backend dependencies simply run 'pip install "platypush[http]"'
-backend.http:
-    # Listening port
-    port: 8008
-
-    # Through resource_dirs you can specify external folders whose content can be accessed on
-    # the web server through a custom URL. In the case below we have a Dropbox folder containing
-    # our pictures and we mount it to the '/carousel' endpoint.
-    resource_dirs:
-        carousel: /mnt/hd/photos/carousel
-
-# The HTTP poll backend is a versatile backend that can monitor for HTTP-based resources and
-# trigger events whenever new entries are available. In the example below we show how to use
-# the backend to listen for changes on a set of RSS feeds. New content will be stored by default
-# on a SQLite database under ~/.local/share/platypush/feeds/rss.db.
-# Install the required dependencies through 'pip install "platypush[rss,db]"'
-backend.http.poll:
-    requests:
-        - type: platypush.backend.http.request.rss.RssUpdates # HTTP poll type (RSS)
-          # Remote URL
-          url: http://www.theguardian.com/rss/world
-          # Custom title
-          title: The Guardian - World News
-          # How often we should check for changes
-          poll_seconds: 600
-          # Maximum number of new entries to be processed
-          max_entries: 10
-
-        - type: platypush.backend.http.request.rss.RssUpdates
-          url: http://www.physorg.com/rss-feed
-          title: Phys.org
-          poll_seconds: 600
-          max_entries: 10
-
-        - type: platypush.backend.http.request.rss.RssUpdates
-          url: http://feeds.feedburner.com/Techcrunch
-          title: Tech Crunch
-          poll_seconds: 600
-          max_entries: 10
-
-        - type: platypush.backend.http.request.rss.RssUpdates
-          url: http://www.nytimes.com/services/xml/rss/nyt/HomePage.xml
-          title: The New York Times
-          poll_seconds: 300
-          max_entries: 10
-
-# MQTT backend. Installed required dependencies through 'pip install "platypush[mqtt]"'
-backend.mqtt:
-    # Remote MQTT server IP or hostname
-    host: mqtt-server
-    # By default the backend will listen for messages on the platypush_bus_mq/device_id
-    # topic, but you can change the prefix using the topic attribute
-#    topic: MyBus
-
-# Raw TCP socket backend. It can run commands sent as JSON over telnet or netcat
-#backend.tcp:
-#    port: 3333
-
-## --
-## Assistant configuration examples
-## --
-#
-# Both Google Assistant and Alexa voice assistant interfaces are supported by platypush.
-# You can easily make your custom voice assistant with a RaspberryPi and a USB microphone,
-# or on your laptop. Note however that the Alexa integration is still experimental
-# (mostly because of glitches and bugs on the avs package provided by Amazon), while the
-# Google Assistant support should be more robust. The recommended way of triggering a
-# hotword ('OK Google', 'Alexa' or any custom hotword you like) is through the snowboy
-# backend (install it through 'pip install "platypush[hotword]"'). You can download custom
-# voice model files (.umdl) from https://snowboy.kitt.ai.
-backend.assistant.snowboy:
-    # Microphone audio gain
-    audio_gain: 1.1
-
-    models:
-        # "Computer" hotword model
-        computer:
-            # UMDL file path
-            voice_model_file: ~/.local/share/snowboy/models/computer.umdl
-            # Plugin to use (Google Assistant)
-            assistant_plugin: assistant.google.pushtotalk
-            # Language assistant (Italian)
-            assistant_language: it-IT
-            # Sound to play when the hotword is detected
-            detect_sound: ~/.local/share/sounds/hotword.wav
-            # Model sensitivity
-            sensitivity: 0.4
-        # "OK Google" hotword model
-        ok_google:
-            voice_model_file: ~/.local/share/snowboy/models/OK Google.pmdl
-            assistant_plugin: assistant.google.pushtotalk
-            assistant_language: en-US
-            detect_sound: ~/.local/share/sounds/sci-fi/PremiumBeat_0013_cursor_selection_16.wav
-            sensitivity: 0.4
-        # "Alexa" voice model
-        alexa:
-            voice_model_file: ~/.local/share/snowboy/models/Alexa.pmdl
-            assistant_plugin: assistant.echo
-            assistant_language: en-US
-            detect_sound: ~/.local/share/sounds/sci-fi/PremiumBeat_0013_cursor_selection_16.wav
-            sensitivity: 0.5
-
-# Install Alexa dependencies with 'pip install "platypush[alexa]"'
-assistant.echo:
-    audio_player: mplayer
-
-# Install Google Assistant dependencies with 'pip install "platypush[google-assistant-legacy]"'
-assistant.google:
-    enabled: True
-
-backend.assistant.google:
-    enabled: True
-
-## --
-## Procedure examples
-## --
-#
-# Procedures are lists of actions that can be executed synchronously (default) or in parallel
-# (procedure.async. prefix). Basic flow control operators (if/else/for) are also available.
-# You can also access Python variables and evaluate Python expressions by using the ${} expressions.
-# The 'context' special variable is a name->value dictionary containing the items returned from
-# previous actions - for example if an action returned '{"status": "ok", "temperature":21.5}' then
-# the following actions can access those variables through ${status} and ${temperature} respectively,
-# and you can also add things like '- if ${temperature > 20.0}' or '- for ${temp in temperature_values}'.
-# Alternatively, you can access those variable also through ${context.get('status')} or ${context.get('temperature')}.
-# Other special variables that you can use in your procedures:
-#
-# - output: Will contain the parsed output of the previous action
-# - errors: Will contain the errors of the previous action
-# - event: If the procedure is executed within an event hook, it contains the event that triggered the hook
-#
-# An example procedure that can be called when you arrive home. You can run this procedure by sending a JSON
-# message like this on whichever backend you like (HTTP, websocket, TCP, Redis, MQTT, Node-RED, Pushbullet...)
-# {"type":"request", "action":"procedure.at_home"}
-# You can for instance install Tasker+AutoLocation on your mobile and send this message whenever you enter
-# your home area.
-procedure.at_home:
-    # Set the db variable HOME to 1
-    - action: variable.set
-      args:
-          HOME: 1
-
-    # Check the luminosity level from a connected LTR559 sensor
-    - action: gpio.sensor.ltr559.get_data
-
-    # If it's below a certain threshold turn on the lights
-    - if ${int(light or 0) < 110}:
-        - action: light.hue.on
-
-    # Say a welcome home message. Install dependencies through 'pip install "platypush[google-tts]"'
-    - action: tts.google.say
-      args:
-          text: Welcome home
-
-    # Start the music
-    - action: music.mpd.play
-
-# Procedure that will be execute when you're outside of home
-procedure.outside_home:
-    # Unset the db variable HOME
-    - action: variable.unset
-      args:
-          name: HOME
-
-    # Stop the music
-    - action: music.mpd.stop
-
-    # Turn off the lights
-    - action: light.hue.off
-
-    # Start the camera streaming. Install the Pi Camera dependencies through
-    # 'pip install "platypush[picamera]"'
-    - action: camera.pi.start_streaming
-      args:
-          listen_port: 2222
-
-# Procedures can also take optional arguments. The example below show a
-# generic procedure to send a request to another platypush host over MQTT
-# given target, action and args
-procedure.send_request(target, action, args):
-    - action: mqtt.send_message
-      args:
-          topic: platypush_bus_mq/${target}
-          host: mqtt-server
-          port: 1883
-          msg:
-              type: request
-              target: ${target}
-              action: ${action}
-              args: ${args}
-
-## --
-## Event hook examples
-## --
-#
-# Event hooks are procedures that are run when a certain condition is met.
-# Check the documentation of the backends to see which events they can trigger.
-# An event hook consists of two parts: an 'if' field that specifies on which
-# event the hook will be triggered (type and attributes content), and a 'then'
-# field that uses the same syntax as procedures to specify a list of actions to
-# execute when the event is matched.
-#
-# The example below plays the music on mpd/mopidy when your voice assistant
-# triggers a speech recognized event with "play the music" content.
-event.hook.PlayMusicAssistantCommand:
-    if:
-        type: platypush.message.event.assistant.SpeechRecognizedEvent
-        # Note that basic regexes are supported, so the hook will be triggered
-        # both if you say "play the music" and "play music"
-        phrase: "play (the)? music"
-    then:
-        - action: music.mpd.play
-
-# This will turn on the lights when you say "turn on the lights"
-event.hook.TurnOnLightsCommand:
-    if:
-        type: platypush.message.event.assistant.SpeechRecognizedEvent
-        phrase: "turn on (the)? lights?"
-    then:
-        - action: light.hue.on
-
-# This will play a song by a specified artist
-event.hook.SearchSongVoiceCommand:
-    if:
-        type: platypush.message.event.assistant.SpeechRecognizedEvent
-        # Note that you can use the ${} operator in event matching to
-        # extract part of the matched string into context variables that
-        # can be accessed in your event hook.
-        phrase: "play ${title} by ${artist}"
-    then:
-        - action: music.mpd.clear
-        - action: music.mpd.search
-          args:
-              filter:
-                  artist: ${artist}
-                  title: ${title}
-
-        # Play the first search result
-        - action: music.mpd.play
-          args:
-              resource: ${output[0]['file']}
-
-# This event will scrobble newly listened tracks on mpd/mopidy to last.fm
-event.hook.ScrobbleNewTrack:
-    if:
-        type: platypush.message.event.music.NewPlayingTrackEvent
-    then:
-        - action: lastfm.scrobble
-          args:
-              artist: ${track['artist']}
-              title: ${track['title']}
-
-        - action: lastfm.update_now_playing
-          args:
-              artist: ${track['artist']}
-              title: ${track['title']}
-
-## --
-## Cron examples
-## --
-#
-# Cronjobs allow you to execute procedures at periodic intervals.
-# Standard UNIX cron syntax is supported, plus an optional 6th indicator
-# at the end of the expression to run jobs with second granularity.
-# The example below executes a script at intervals of 1 minute.
-cron.TestCron:
-    cron_expression: '* * * * *'
-    actions:
-        - action: shell.exec
-          args:
-              cmd: ~/bin/myscript.sh
-

examples/config/config.yaml

@@ -0,0 +1 @@
+../../platypush/config/config.yaml

examples/config/hook.py

@@ -12,7 +12,10 @@ from platypush.utils import run
 from platypush.event.hook import hook
 
 # Event types that you want to react to
-from platypush.message.event.assistant import ConversationStartEvent, SpeechRecognizedEvent
+from platypush.message.event.assistant import (
+    ConversationStartEvent,
+    SpeechRecognizedEvent,
+)
 
 
 @hook(SpeechRecognizedEvent, phrase='play ${title} by ${artist}')
@@ -23,10 +26,13 @@ def on_music_play_command(event, title=None, artist=None, **context):
     Note that in this specific case we can leverage the token-extraction feature of SpeechRecognizedEvent through
     ${} that operates on regex-like principles to extract any text that matches the pattern into context variables.
     """
-    results = run('music.mpd.search', filter={
+    results = run(
-        'artist': artist,
+        'music.mpd.search',
-        'title': title,
+        filter={
-    })
+            'artist': artist,
+            'title': title,
+        },
+    )
 
     if results:
         run('music.mpd.play', results[0]['file'])

examples/docker/Dockerfile

@@ -1,19 +0,0 @@
-# Sample Dockerfile. Use platydock -c /path/to/custom/config.yaml
-# to generate your custom Dockerfile.
-
-
-FROM python:3.11-alpine
-
-RUN mkdir -p /install /app
-COPY . /install
-RUN apk add --update --no-cache redis
-RUN apk add --update --no-cache --virtual build-base g++ rust
-RUN pip install -U pip
-RUN cd /install && pip install .
-RUN apk del build-base g++ rust
-
-EXPOSE 8008
-VOLUME /app/config
-VOLUME /app/workdir
-
-CMD python -m platypush --start-redis --config-file /app/config/config.yaml --workdir /app/workdir

examples/nginx/nginx.sample.conf

@@ -1,12 +1,17 @@
 # An nginx configuration that can be used to reverse proxy connections to your
 # Platypush' HTTP service.
 
-server {
+upstream platypush {
-  server_name my-platypush-host.domain.com;
+  # The address and port where the HTTP backend is listening
+	server 127.0.0.1:8008;
+}
 
-  # Proxy standard HTTP connections to your Platypush IP
+server {
+  server_name platypush.example.com;
+
+  # Proxy standard HTTP connections
   location / {
-      proxy_pass http://my-platypush-host:8008/;
+      proxy_pass http://platypush;
 
       client_max_body_size 5M;
       proxy_read_timeout 60;
@@ -18,21 +23,33 @@ server {
   }
 
   # Proxy websocket connections
-  location ~ ^/ws/(.*)$ {
+  location /ws/ {
-      proxy_pass http://10.0.0.2:8008/ws/$1;
+      proxy_pass http://platypush;
 
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_redirect off;
       proxy_http_version 1.1;
-      client_max_body_size 200M;
+      client_max_body_size 5M;
       proxy_set_header Host $http_host;
   }
 
   # Optional SSL configuration - using Let's Encrypt certificates in this case
   # listen 443 ssl;
-  # ssl_certificate /etc/letsencrypt/live/my-platypush-host.domain.com/fullchain.pem;
+  # ssl_certificate /etc/letsencrypt/live/platypush.example.com/fullchain.pem;
-  # ssl_certificate_key /etc/letsencrypt/live/my-platypush-host.domain.com/privkey.pem;
+  # ssl_certificate_key /etc/letsencrypt/live/platypush.example.com/privkey.pem;
   # include /etc/letsencrypt/options-ssl-nginx.conf;
   # ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
 }
+
+# Uncomment if you are using SSL and you want to force an HTTPS upgrade to
+# clients connecting over the port 80
+# server {
+#     if ($host = platypush.example.com) {
+#         return 301 https://$host$request_uri;
+#     }
+# 
+#     server_name platypush.example.com;
+#     listen 80;
+#     return 404;
+# }

generate_missing_docs.py

@@ -1,32 +1,57 @@
+import importlib
+import inspect
 import os
+import sys
 from typing import Iterable, Optional
 
+import pkgutil
+
 from platypush.backend import Backend
-from platypush.context import get_plugin
+from platypush.message.event import Event
+from platypush.message.response import Response
 from platypush.plugins import Plugin
-from platypush.utils.manifest import get_manifests
+from platypush.utils.manifest import Manifests
-
+from platypush.utils.mock import auto_mocks
-
-def _get_inspect_plugin():
-    p = get_plugin('inspect')
-    assert p, 'Could not load the `inspect` plugin'
-    return p
 
 
 def get_all_plugins():
-    return sorted([mf.component_name for mf in get_manifests(Plugin)])
+    return sorted([mf.component_name for mf in Manifests.by_base_class(Plugin)])
 
 
 def get_all_backends():
-    return sorted([mf.component_name for mf in get_manifests(Backend)])
+    return sorted([mf.component_name for mf in Manifests.by_base_class(Backend)])
 
 
 def get_all_events():
-    return _get_inspect_plugin().get_all_events().output
+    return _get_modules(Event)
 
 
 def get_all_responses():
-    return _get_inspect_plugin().get_all_responses().output
+    return _get_modules(Response)
+
+
+def _get_modules(base_type: type):
+    ret = set()
+    base_dir = os.path.dirname(inspect.getfile(base_type))
+    package = base_type.__module__
+
+    for _, mod_name, _ in pkgutil.walk_packages([base_dir], prefix=package + '.'):
+        try:
+            module = importlib.import_module(mod_name)
+        except Exception:
+            print('Could not import module', mod_name, file=sys.stderr)
+            continue
+
+        for _, obj_type in inspect.getmembers(module):
+            if (
+                inspect.isclass(obj_type)
+                and issubclass(obj_type, base_type)
+                # Exclude the base_type itself
+                and obj_type != base_type
+            ):
+                ret.add(obj_type.__module__.replace(package + '.', '', 1))
+
+    return list(ret)
 
 
 def _generate_components_doc(
@@ -122,7 +147,7 @@ def generate_events_doc():
     _generate_components_doc(
         index_name='events',
         package_name='message.event',
-        components=sorted(event for event in get_all_events().keys() if event),
+        components=sorted(event for event in get_all_events() if event),
     )
 
 
@@ -130,17 +155,16 @@ def generate_responses_doc():
     _generate_components_doc(
         index_name='responses',
         package_name='message.response',
-        components=sorted(
+        components=sorted(response for response in get_all_responses() if response),
-            response for response in get_all_responses().keys() if response
-        ),
     )
 
 
 def main():
-    generate_plugins_doc()
+    with auto_mocks():
-    generate_backends_doc()
+        generate_plugins_doc()
-    generate_events_doc()
+        generate_backends_doc()
-    generate_responses_doc()
+        generate_events_doc()
+        generate_responses_doc()
 
 
 if __name__ == '__main__':

platypush/__init__.py

@@ -5,12 +5,14 @@ Platypush
 .. license: MIT
 """
 
-from .app import Application, main
+from .app import Application
 from .config import Config
 from .context import get_backend, get_bus, get_plugin
 from .message.event import Event
 from .message.request import Request
 from .message.response import Response
+from .runner import main
+from .utils import run
 
 
 __author__ = 'Fabio Manganiello <fabio@manganiello.tech>'
@@ -25,6 +27,7 @@ __all__ = [
     'get_bus',
     'get_plugin',
     'main',
+    'run',
 ]
 
 

platypush/__main__.py

@@ -1,10 +1,3 @@
-import sys
+from platypush.runner import main
 
-from platypush.app import main
+main()
-
-
-if __name__ == '__main__':
-    main(*sys.argv[1:])
-
-
-# vim:sw=4:ts=4:et:

platypush/app/__init__.py

@@ -0,0 +1,4 @@
+from ._app import Application, main
+
+
+__all__ = ["Application", "main"]

platypush/app/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from ._app import main
+
+sys.exit(main(*sys.argv[1:]))

platypush/app/_app.py

@@ -1,23 +1,26 @@
-import argparse
+from contextlib import contextmanager
 import logging
 import os
+import signal
 import subprocess
 import sys
-from typing import Optional
+from typing import Optional, Sequence
 
-from .bus import Bus
+from platypush.bus import Bus
-from .bus.redis import RedisBus
+from platypush.bus.redis import RedisBus
-from .config import Config
+from platypush.cli import parse_cmdline
-from .context import register_backends, register_plugins
+from platypush.commands import CommandStream
-from .cron.scheduler import CronScheduler
+from platypush.config import Config
-from .entities import init_entities_engine, EntitiesEngine
+from platypush.context import register_backends, register_plugins
-from .event.processor import EventProcessor
+from platypush.cron.scheduler import CronScheduler
-from .logger import Logger
+from platypush.entities import init_entities_engine, EntitiesEngine
-from .message.event import Event
+from platypush.event.processor import EventProcessor
-from .message.event.application import ApplicationStartedEvent
+from platypush.logger import Logger
-from .message.request import Request
+from platypush.message.event import Event
-from .message.response import Response
+from platypush.message.event.application import ApplicationStartedEvent
-from .utils import get_enabled_plugins, get_redis_conf
+from platypush.message.request import Request
+from platypush.message.response import Response
+from platypush.utils import get_enabled_plugins, get_redis_conf
 
 log = logging.getLogger('platypush')
 
@@ -25,9 +28,6 @@ log = logging.getLogger('platypush')
 class Application:
     """Main class for the Platypush application."""
 
-    # Default bus queue name
-    _default_redis_queue = 'platypush/bus'
-
     # Default Redis port
     _default_redis_port = 6379
 
@@ -42,6 +42,8 @@ class Application:
         config_file: Optional[str] = None,
         workdir: Optional[str] = None,
         logsdir: Optional[str] = None,
+        cachedir: Optional[str] = None,
+        device_id: Optional[str] = None,
         pidfile: Optional[str] = None,
         requests_to_process: Optional[int] = None,
         no_capture_stdout: bool = False,
@@ -51,6 +53,7 @@ class Application:
         start_redis: bool = False,
         redis_host: Optional[str] = None,
         redis_port: Optional[int] = None,
+        ctrl_sock: Optional[str] = None,
     ):
         """
         :param config_file: Configuration file override (default: None).
@@ -60,6 +63,12 @@ class Application:
             ``filename`` setting under the ``logging`` section of the
             configuration file is used. If not set, logging will be sent to
             stdout and stderr.
+        :param cachedir: Overrides the ``cachedir`` setting in the configuration
+            file (default: None).
+        :param device_id: Override the device ID used to identify this
+            instance. If not passed here, it is inferred from the configuration
+            (device_id field). If not present there either, it is inferred from
+            the hostname.
         :param pidfile: File where platypush will store its PID upon launch,
            useful if you're planning to integrate the application within a
            service or a launcher script (default: None).
@@ -82,24 +91,31 @@ class Application:
             the settings in the ``redis`` section of the configuration file.
         :param redis_port: Port of the local Redis server. It overrides the
             settings in the ``redis`` section of the configuration file.
+        :param ctrl_sock: If set, it identifies a path to a UNIX domain socket
+            that the application can use to send control messages (e.g. STOP
+            and RESTART) to its parent.
         """
 
         self.pidfile = pidfile
-        if pidfile:
-            with open(pidfile, 'w') as f:
-                f.write(str(os.getpid()))
-
         self.bus: Optional[Bus] = None
-        self.redis_queue = redis_queue or self._default_redis_queue
+        self.redis_queue = redis_queue or RedisBus.DEFAULT_REDIS_QUEUE
         self.config_file = config_file
         self._verbose = verbose
         self._logsdir = (
             os.path.abspath(os.path.expanduser(logsdir)) if logsdir else None
         )
-        Config.init(self.config_file)
 
-        if workdir:
+        Config.init(
-            Config.set('workdir', os.path.abspath(os.path.expanduser(workdir)))
+            self.config_file,
+            device_id=device_id,
+            workdir=os.path.abspath(os.path.expanduser(workdir)) if workdir else None,
+            cachedir=os.path.abspath(os.path.expanduser(cachedir))
+            if cachedir
+            else None,
+            ctrl_sock=os.path.abspath(os.path.expanduser(ctrl_sock))
+            if ctrl_sock
+            else None,
+        )
 
         self.no_capture_stdout = no_capture_stdout
         self.no_capture_stderr = no_capture_stderr
@@ -113,6 +129,7 @@ class Application:
         self.redis_port = redis_port
         self.redis_conf = {}
         self._redis_proc: Optional[subprocess.Popen] = None
+        self.cmd_stream = CommandStream(ctrl_sock)
 
         self._init_bus()
         self._init_logging()
@@ -153,16 +170,27 @@ class Application:
 
         port = self._redis_conf['port']
         log.info('Starting local Redis instance on %s', port)
-        self._redis_proc = subprocess.Popen(  # pylint: disable=consider-using-with
+        redis_cmd_args = [
-            [
+            'redis-server',
-                'redis-server',
+            '--bind',
-                '--bind',
+            'localhost',
-                'localhost',
+            '--port',
-                '--port',
+            str(port),
-                str(port),
+        ]
-            ],
+
-            stdout=subprocess.PIPE,
+        try:
-        )
+            self._redis_proc = subprocess.Popen(  # pylint: disable=consider-using-with
+                redis_cmd_args,
+                stdout=subprocess.PIPE,
+            )
+        except Exception as e:
+            log.error(
+                'Failed to start local Redis instance: "%s": %s',
+                ' '.join(redis_cmd_args),
+                e,
+            )
+
+            sys.exit(1)
 
         log.info('Waiting for Redis to start')
         for line in self._redis_proc.stdout:  # type: ignore
@@ -176,137 +204,17 @@ class Application:
             self._redis_proc = None
 
     @classmethod
-    def build(cls, *args: str):
+    def from_cmdline(cls, args: Sequence[str]) -> "Application":
         """
         Build the app from command line arguments.
         """
-        from . import __version__
+        opts = parse_cmdline(args)
-
-        parser = argparse.ArgumentParser()
-
-        parser.add_argument(
-            '--config',
-            '-c',
-            dest='config',
-            required=False,
-            default=None,
-            help='Custom location for the configuration file',
-        )
-
-        parser.add_argument(
-            '--workdir',
-            '-w',
-            dest='workdir',
-            required=False,
-            default=None,
-            help='Custom working directory to be used for the application',
-        )
-
-        parser.add_argument(
-            '--logsdir',
-            '-l',
-            dest='logsdir',
-            required=False,
-            default=None,
-            help='Store logs in the specified directory. By default, the '
-            '`[logging.]filename` configuration option will be used. If not '
-            'set, logging will be sent to stdout and stderr.',
-        )
-
-        parser.add_argument(
-            '--version',
-            dest='version',
-            required=False,
-            action='store_true',
-            help="Print the current version and exit",
-        )
-
-        parser.add_argument(
-            '--verbose',
-            '-v',
-            dest='verbose',
-            required=False,
-            action='store_true',
-            help="Enable verbose/debug logging",
-        )
-
-        parser.add_argument(
-            '--pidfile',
-            '-P',
-            dest='pidfile',
-            required=False,
-            default=None,
-            help="File where platypush will "
-            + "store its PID, useful if you're planning to "
-            + "integrate it in a service",
-        )
-
-        parser.add_argument(
-            '--no-capture-stdout',
-            dest='no_capture_stdout',
-            required=False,
-            action='store_true',
-            help="Set this flag if you have max stack depth "
-            + "exceeded errors so stdout won't be captured by "
-            + "the logging system",
-        )
-
-        parser.add_argument(
-            '--no-capture-stderr',
-            dest='no_capture_stderr',
-            required=False,
-            action='store_true',
-            help="Set this flag if you have max stack depth "
-            + "exceeded errors so stderr won't be captured by "
-            + "the logging system",
-        )
-
-        parser.add_argument(
-            '--redis-queue',
-            dest='redis_queue',
-            required=False,
-            default=cls._default_redis_queue,
-            help="Name of the Redis queue to be used to internally deliver messages "
-            "(default: platypush/bus)",
-        )
-
-        parser.add_argument(
-            '--start-redis',
-            dest='start_redis',
-            required=False,
-            action='store_true',
-            help="Set this flag if you want to run and manage Redis internally "
-            "from the app rather than using an external server. It requires the "
-            "redis-server executable to be present in the path",
-        )
-
-        parser.add_argument(
-            '--redis-host',
-            dest='redis_host',
-            required=False,
-            default=None,
-            help="Overrides the host specified in the redis section of the "
-            "configuration file",
-        )
-
-        parser.add_argument(
-            '--redis-port',
-            dest='redis_port',
-            required=False,
-            default=None,
-            help="Overrides the port specified in the redis section of the "
-            "configuration file",
-        )
-
-        opts, _ = parser.parse_known_args(args)
-        if opts.version:
-            print(__version__)
-            sys.exit(0)
-
         return cls(
             config_file=opts.config,
             workdir=opts.workdir,
+            cachedir=opts.cachedir,
             logsdir=opts.logsdir,
+            device_id=opts.device_id,
             pidfile=opts.pidfile,
             no_capture_stdout=opts.no_capture_stdout,
             no_capture_stderr=opts.no_capture_stderr,
@@ -315,6 +223,7 @@ class Application:
             start_redis=opts.start_redis,
             redis_host=opts.redis_host,
             redis_port=opts.redis_port,
+            ctrl_sock=opts.ctrl_sock,
         )
 
     def on_message(self):
@@ -340,7 +249,7 @@ class Application:
                     self.requests_to_process
                     and self.processed_requests >= self.requests_to_process
                 ):
-                    self.stop_app()
+                    self.stop()
             elif isinstance(msg, Response):
                 msg.log()
             elif isinstance(msg, Event):
@@ -349,36 +258,68 @@ class Application:
 
         return _f
 
-    def stop_app(self):
+    def stop(self):
         """Stops the backends and the bus."""
-        from .plugins import RunnablePlugin
+        from platypush.plugins import RunnablePlugin
 
-        if self.backends:
+        log.info('Stopping the application')
-            for backend in self.backends.values():
+        backends = (self.backends or {}).copy().values()
-                backend.stop()
+        runnable_plugins = [
+            plugin
+            for plugin in get_enabled_plugins().values()
+            if isinstance(plugin, RunnablePlugin)
+        ]
 
-        for plugin in get_enabled_plugins().values():
+        for backend in backends:
-            if isinstance(plugin, RunnablePlugin):
+            backend.stop()
-                plugin.stop()
+
+        for plugin in runnable_plugins:
+            plugin.stop()
+
+        for backend in backends:
+            backend.wait_stop()
+
+        for plugin in runnable_plugins:
+            plugin.wait_stop()
+
+        if self.entities_engine:
+            self.entities_engine.stop()
+            self.entities_engine.wait_stop()
+            self.entities_engine = None
+
+        if self.cron_scheduler:
+            self.cron_scheduler.stop()
+            self.cron_scheduler.wait_stop()
+            self.cron_scheduler = None
 
         if self.bus:
             self.bus.stop()
             self.bus = None
 
-        if self.cron_scheduler:
-            self.cron_scheduler.stop()
-            self.cron_scheduler = None
-
-        if self.entities_engine:
-            self.entities_engine.stop()
-            self.entities_engine = None
-
         if self.start_redis:
             self._stop_redis()
 
-    def run(self):
+        log.info('Exiting application')
-        """Start the daemon."""
+
-        from . import __version__
+    @contextmanager
+    def _open_pidfile(self):
+        if self.pidfile:
+            try:
+                with open(self.pidfile, 'w') as f:
+                    f.write(str(os.getpid()))
+            except OSError as e:
+                log.warning('Failed to create PID file %s: %s', self.pidfile, e)
+
+        yield
+
+        if self.pidfile:
+            try:
+                os.remove(self.pidfile)
+            except OSError as e:
+                log.warning('Failed to remove PID file %s: %s', self.pidfile, e)
+
+    def _run(self):
+        from platypush import __version__
 
         if not self.no_capture_stdout:
             sys.stdout = Logger(log.info)
@@ -417,16 +358,30 @@ class Application:
             self.bus.poll()
         except KeyboardInterrupt:
             log.info('SIGINT received, terminating application')
+            # Ignore other SIGINT signals
+            signal.signal(signal.SIGINT, signal.SIG_IGN)
         finally:
-            self.stop_app()
+            self.stop()
+
+    def run(self):
+        """Run the application."""
+
+        with self._open_pidfile():
+            self._run()
 
 
 def main(*args: str):
     """
     Application entry point.
     """
-    app = Application.build(*args)
+    app = Application.from_cmdline(args)
-    app.run()
+
+    try:
+        app.run()
+    except KeyboardInterrupt:
+        pass
+
+    return 0
 
 
 # vim:sw=4:ts=4:et:

platypush/backend/__init__.py

@@ -6,27 +6,28 @@ import time
 from threading import Thread, Event as ThreadEvent, get_ident
 from typing import Optional, Dict
 
+from platypush import __version__
 from platypush.bus import Bus
 from platypush.common import ExtensionWithManifest
 from platypush.config import Config
 from platypush.context import get_backend
+from platypush.event import EventGenerator
+from platypush.message import Message
+from platypush.message.event import Event
 from platypush.message.event.zeroconf import (
     ZeroconfServiceAddedEvent,
     ZeroconfServiceRemovedEvent,
 )
-from platypush.utils import (
-    set_timeout,
-    clear_timeout,
-    get_redis_queue_name_by_message,
-    get_backend_name_by_class,
-)
-
-from platypush import __version__
-from platypush.event import EventGenerator
-from platypush.message import Message
-from platypush.message.event import Event
 from platypush.message.request import Request
 from platypush.message.response import Response
+from platypush.utils import (
+    clear_timeout,
+    get_backend_name_by_class,
+    get_redis,
+    get_redis_queue_name_by_message,
+    get_remaining_timeout,
+    set_timeout,
+)
 
 
 class Backend(Thread, EventGenerator, ExtensionWithManifest):
@@ -68,6 +69,7 @@ class Backend(Thread, EventGenerator, ExtensionWithManifest):
         self.device_id = Config.get('device_id')
         self.thread_id = None
         self._stop_event = ThreadEvent()
+        self._stop_thread: Optional[Thread] = None
         self._kwargs = kwargs
         self.logger = logging.getLogger(
             'platypush:backend:' + get_backend_name_by_class(self.__class__)
@@ -299,30 +301,38 @@ class Backend(Thread, EventGenerator, ExtensionWithManifest):
             self._stop_event.set()
             self.unregister_service()
             self.on_stop()
+            self._stop_thread = None
 
-        Thread(target=_async_stop).start()
+        if not (self._stop_thread and self._stop_thread.is_alive()):
+            self._stop_thread = Thread(target=_async_stop)
+            self._stop_thread.start()
 
     def should_stop(self):
+        """
+        :return: True if the backend thread should be stopped, False otherwise.
+        """
         return self._stop_event.is_set()
 
     def wait_stop(self, timeout=None) -> bool:
-        return self._stop_event.wait(timeout)
+        """
+        Waits for the backend thread to stop.
 
-    def _get_redis(self):
+        :param timeout: The maximum time to wait for the backend thread to stop (default: None)
-        import redis
+        :return: True if the backend thread has stopped, False otherwise.
+        """
+        start = time.time()
 
-        redis_backend = get_backend('redis')
+        if self._stop_thread:
-        if not redis_backend:
+            try:
-            self.logger.warning(
+                self._stop_thread.join(
-                'Redis backend not configured - some '
+                    get_remaining_timeout(timeout=timeout, start=start)
-                'web server features may not be working properly'
+                )
-            )
+            except AttributeError:
-            redis_args = {}
+                pass
-        else:
-            redis_args = redis_backend.redis_args
 
-        redis = redis.Redis(**redis_args)
+        return self._stop_event.wait(
-        return redis
+            get_remaining_timeout(timeout=timeout, start=start)
+        )
 
     def get_message_response(self, msg):
         queue = get_redis_queue_name_by_message(msg)
@@ -331,7 +341,7 @@ class Backend(Thread, EventGenerator, ExtensionWithManifest):
             return None
 
         try:
-            redis = self._get_redis()
+            redis = get_redis()
             response = redis.blpop(queue, timeout=60)
             if response and len(response) > 1:
                 response = Message.build(response[1])
@@ -431,6 +441,8 @@ class Backend(Thread, EventGenerator, ExtensionWithManifest):
         """
         Unregister the Zeroconf service configuration if available.
         """
+        from redis import exceptions
+
         if self.zeroconf and self.zeroconf_info:
             try:
                 self.zeroconf.unregister_service(self.zeroconf_info)
@@ -448,17 +460,22 @@ class Backend(Thread, EventGenerator, ExtensionWithManifest):
                 except TimeoutError:
                     pass
 
-            if self.zeroconf_info:
+            try:
-                self.bus.post(
+                if self.zeroconf_info:
-                    ZeroconfServiceRemovedEvent(
+                    self.bus.post(
-                        service_type=self.zeroconf_info.type,
+                        ZeroconfServiceRemovedEvent(
-                        service_name=self.zeroconf_info.name,
+                            service_type=self.zeroconf_info.type,
+                            service_name=self.zeroconf_info.name,
+                        )
                     )
-                )
+                else:
-            else:
+                    self.bus.post(
-                self.bus.post(
+                        ZeroconfServiceRemovedEvent(
-                    ZeroconfServiceRemovedEvent(service_type=None, service_name=None)
+                            service_type=None, service_name=None
-                )
+                        )
+                    )
+            except exceptions.ConnectionError:
+                pass
 
             self.zeroconf_info = None
             self.zeroconf = None

platypush/backend/adafruit/io/__init__.py

@@ -2,23 +2,17 @@ from typing import Optional
 
 from platypush.backend import Backend
 from platypush.context import get_plugin
-from platypush.message.event.adafruit import ConnectedEvent, DisconnectedEvent, \
+from platypush.message.event.adafruit import (
-    FeedUpdateEvent
+    ConnectedEvent,
+    DisconnectedEvent,
+    FeedUpdateEvent,
+)
 
 
 class AdafruitIoBackend(Backend):
     """
     Backend that listens to messages received over the Adafruit IO message queue
 
-    Triggers:
-
-        * :class:`platypush.message.event.adafruit.ConnectedEvent` when the
-            backend connects to the Adafruit queue
-        * :class:`platypush.message.event.adafruit.DisconnectedEvent` when the
-            backend disconnects from the Adafruit queue
-        * :class:`platypush.message.event.adafruit.FeedUpdateEvent` when an
-            update event is received on a monitored feed
-
     Requires:
 
         * The :class:`platypush.plugins.adafruit.io.AdafruitIoPlugin` plugin to
@@ -33,6 +27,7 @@ class AdafruitIoBackend(Backend):
 
         super().__init__(*args, **kwargs)
         from Adafruit_IO import MQTTClient
+
         self.feeds = feeds
         self._client: Optional[MQTTClient] = None
 
@@ -41,6 +36,7 @@ class AdafruitIoBackend(Backend):
             return
 
         from Adafruit_IO import MQTTClient
+
         plugin = get_plugin('adafruit.io')
         if not plugin:
             raise RuntimeError('Adafruit IO plugin not configured')
@@ -80,8 +76,11 @@ class AdafruitIoBackend(Backend):
     def run(self):
         super().run()
 
-        self.logger.info(('Initialized Adafruit IO backend, listening on ' +
+        self.logger.info(
-                          'feeds {}').format(self.feeds))
+            ('Initialized Adafruit IO backend, listening on ' + 'feeds {}').format(
+                self.feeds
+            )
+        )
 
         while not self.should_stop():
             try:
@@ -94,4 +93,5 @@ class AdafruitIoBackend(Backend):
                 self.logger.exception(e)
                 self._client = None
 
+
 # vim:sw=4:ts=4:et:

platypush/backend/alarm/__init__.py

@@ -11,7 +11,11 @@ from dateutil.tz import gettz
 
 from platypush.backend import Backend
 from platypush.context import get_bus, get_plugin
-from platypush.message.event.alarm import AlarmStartedEvent, AlarmDismissedEvent, AlarmSnoozedEvent
+from platypush.message.event.alarm import (
+    AlarmStartedEvent,
+    AlarmDismissedEvent,
+    AlarmSnoozedEvent,
+)
 from platypush.plugins.media import MediaPlugin, PlayerState
 from platypush.procedure import Procedure
 
@@ -28,10 +32,17 @@ class Alarm:
     _alarms_count = 0
     _id_lock = threading.RLock()
 
-    def __init__(self, when: str, actions: Optional[list] = None, name: Optional[str] = None,
+    def __init__(
-                 audio_file: Optional[str] = None, audio_plugin: Optional[str] = None,
+        self,
-                 audio_volume: Optional[Union[int, float]] = None,
+        when: str,
-                 snooze_interval: float = 300.0, enabled: bool = True):
+        actions: Optional[list] = None,
+        name: Optional[str] = None,
+        audio_file: Optional[str] = None,
+        audio_plugin: Optional[str] = None,
+        audio_volume: Optional[Union[int, float]] = None,
+        snooze_interval: float = 300.0,
+        enabled: bool = True,
+    ):
         with self._id_lock:
             self._alarms_count += 1
             self.id = self._alarms_count
@@ -42,20 +53,26 @@ class Alarm:
 
         if audio_file:
             self.audio_file = os.path.abspath(os.path.expanduser(audio_file))
-            assert os.path.isfile(self.audio_file), 'No such audio file: {}'.format(self.audio_file)
+            assert os.path.isfile(self.audio_file), 'No such audio file: {}'.format(
+                self.audio_file
+            )
 
         self.audio_plugin = audio_plugin
         self.audio_volume = audio_volume
         self.snooze_interval = snooze_interval
         self.state: Optional[AlarmState] = None
         self.timer: Optional[threading.Timer] = None
-        self.actions = Procedure.build(name=name, _async=False, requests=actions or [], id=self.id)
+        self.actions = Procedure.build(
+            name=name, _async=False, requests=actions or [], id=self.id
+        )
 
         self._enabled = enabled
         self._runtime_snooze_interval = snooze_interval
 
     def get_next(self) -> float:
-        now = datetime.datetime.now().replace(tzinfo=gettz())  # lgtm [py/call-to-non-callable]
+        now = datetime.datetime.now().replace(
+            tzinfo=gettz()
+        )  # lgtm [py/call-to-non-callable]
 
         try:
             cron = croniter.croniter(self.when, now)
@@ -63,10 +80,14 @@ class Alarm:
         except (AttributeError, croniter.CroniterBadCronError):
             try:
                 timestamp = datetime.datetime.fromisoformat(self.when).replace(
-                    tzinfo=gettz())  # lgtm [py/call-to-non-callable]
+                    tzinfo=gettz()
+                )  # lgtm [py/call-to-non-callable]
             except (TypeError, ValueError):
-                timestamp = (datetime.datetime.now().replace(tzinfo=gettz()) +  # lgtm [py/call-to-non-callable]
+                timestamp = datetime.datetime.now().replace(
-                             datetime.timedelta(seconds=int(self.when)))
+                    tzinfo=gettz()
+                ) + datetime.timedelta(  # lgtm [py/call-to-non-callable]
+                    seconds=int(self.when)
+                )
 
             return timestamp.timestamp() if timestamp >= now else None
 
@@ -88,7 +109,9 @@ class Alarm:
         self._runtime_snooze_interval = interval or self.snooze_interval
         self.state = AlarmState.SNOOZED
         self.stop_audio()
-        get_bus().post(AlarmSnoozedEvent(name=self.name, interval=self._runtime_snooze_interval))
+        get_bus().post(
+            AlarmSnoozedEvent(name=self.name, interval=self._runtime_snooze_interval)
+        )
 
     def start(self):
         if self.timer:
@@ -159,7 +182,9 @@ class Alarm:
                     break
 
                 if not sleep_time:
-                    sleep_time = self.get_next() - time.time() if self.get_next() else 10
+                    sleep_time = (
+                        self.get_next() - time.time() if self.get_next() else 10
+                    )
 
                 time.sleep(sleep_time)
 
@@ -179,18 +204,15 @@ class Alarm:
 class AlarmBackend(Backend):
     """
     Backend to handle user-configured alarms.
-
-    Triggers:
-
-        * :class:`platypush.message.event.alarm.AlarmStartedEvent` when an alarm starts.
-        * :class:`platypush.message.event.alarm.AlarmSnoozedEvent` when an alarm is snoozed.
-        * :class:`platypush.message.event.alarm.AlarmTimeoutEvent` when an alarm times out.
-        * :class:`platypush.message.event.alarm.AlarmDismissedEvent` when an alarm is dismissed.
-
     """
 
-    def __init__(self, alarms: Optional[Union[list, Dict[str, Any]]] = None, audio_plugin: str = 'media.mplayer',
+    def __init__(
-                 *args, **kwargs):
+        self,
+        alarms: Optional[Union[list, Dict[str, Any]]] = None,
+        audio_plugin: str = 'media.mplayer',
+        *args,
+        **kwargs
+    ):
         """
         :param alarms: List or name->value dict with the configured alarms. Example:
 
@@ -231,13 +253,29 @@ class AlarmBackend(Backend):
             alarms = [{'name': name, **alarm} for name, alarm in alarms.items()]
 
         self.audio_plugin = audio_plugin
-        alarms = [Alarm(**{'audio_plugin': self.audio_plugin, **alarm}) for alarm in alarms]
+        alarms = [
+            Alarm(**{'audio_plugin': self.audio_plugin, **alarm}) for alarm in alarms
+        ]
         self.alarms: Dict[str, Alarm] = {alarm.name: alarm for alarm in alarms}
 
-    def add_alarm(self, when: str, actions: list, name: Optional[str] = None, audio_file: Optional[str] = None,
+    def add_alarm(
-                  audio_volume: Optional[Union[int, float]] = None, enabled: bool = True) -> Alarm:
+        self,
-        alarm = Alarm(when=when, actions=actions, name=name, enabled=enabled, audio_file=audio_file,
+        when: str,
-                      audio_plugin=self.audio_plugin, audio_volume=audio_volume)
+        actions: list,
+        name: Optional[str] = None,
+        audio_file: Optional[str] = None,
+        audio_volume: Optional[Union[int, float]] = None,
+        enabled: bool = True,
+    ) -> Alarm:
+        alarm = Alarm(
+            when=when,
+            actions=actions,
+            name=name,
+            enabled=enabled,
+            audio_file=audio_file,
+            audio_plugin=self.audio_plugin,
+            audio_volume=audio_volume,
+        )
 
         if alarm.name in self.alarms:
             self.logger.info('Overwriting existing alarm {}'.format(alarm.name))
@@ -274,10 +312,15 @@ class AlarmBackend(Backend):
         alarm.snooze(interval=interval)
 
     def get_alarms(self) -> List[Alarm]:
-        return sorted([alarm for alarm in self.alarms.values()], key=lambda alarm: alarm.get_next())
+        return sorted(
+            self.alarms.values(),
+            key=lambda alarm: alarm.get_next(),
+        )
 
     def get_running_alarm(self) -> Optional[Alarm]:
-        running_alarms = [alarm for alarm in self.alarms.values() if alarm.state == AlarmState.RUNNING]
+        running_alarms = [
+            alarm for alarm in self.alarms.values() if alarm.state == AlarmState.RUNNING
+        ]
         return running_alarms[0] if running_alarms else None
 
     def __enter__(self):
@@ -285,9 +328,11 @@ class AlarmBackend(Backend):
             alarm.stop()
             alarm.start()
 
-        self.logger.info('Initialized alarm backend with {} alarms'.format(len(self.alarms)))
+        self.logger.info(
+            'Initialized alarm backend with {} alarms'.format(len(self.alarms))
+        )
 
-    def __exit__(self, exc_type, exc_val, exc_tb):
+    def __exit__(self, *_, **__):
         for alarm in self.alarms.values():
             alarm.stop()
 
@@ -295,7 +340,9 @@ class AlarmBackend(Backend):
 
     def loop(self):
         for name, alarm in self.alarms.copy().items():
-            if not alarm.timer or (not alarm.timer.is_alive() and alarm.state == AlarmState.SHUTDOWN):
+            if not alarm.timer or (
+                not alarm.timer.is_alive() and alarm.state == AlarmState.SHUTDOWN
+            ):
                 del self.alarms[name]
 
         time.sleep(10)

platypush/backend/assistant/__init__.py

@@ -1,39 +0,0 @@
-from abc import ABC
-import threading
-from typing import Optional, Dict, Any, Tuple
-
-from platypush.backend import Backend
-from platypush.context import get_plugin
-from platypush.plugins.tts import TtsPlugin
-
-
-class AssistantBackend(Backend):
-    def __init__(self, tts_plugin: Optional[str] = None, tts_args: Optional[Dict[str, Any]] = None, **kwargs):
-        """
-        Default assistant backend constructor.
-
-        :param tts_plugin: If set, and if the assistant returns the processed response as text, then the processed
-            response will be played through the selected text-to-speech plugin (can be e.g. "``tts``",
-            "``tts.google``" or any other implementation of :class:`platypush.plugins.tts.TtsPlugin`).
-        :param tts_args: Extra parameters to pass to the ``say`` method of the selected TTS plugin (e.g.
-            language, voice or gender).
-        """
-        super().__init__(**kwargs)
-        self._detection_paused = threading.Event()
-        self.tts_plugin = tts_plugin
-        self.tts_args = tts_args or {}
-
-    def pause_detection(self):
-        self._detection_paused.set()
-
-    def resume_detection(self):
-        self._detection_paused.clear()
-
-    def is_detecting(self):
-        return not self._detection_paused.is_set()
-
-    def _get_tts_plugin(self) -> Tuple[Optional[TtsPlugin], Dict[str, Any]]:
-        return get_plugin(self.tts_plugin) if self.tts_plugin else None, self.tts_args
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/assistant/google/__init__.py

@@ -1,225 +0,0 @@
-import json
-import os
-import time
-
-from platypush.backend.assistant import AssistantBackend
-from platypush.message.event.assistant import (
-    ConversationStartEvent,
-    ConversationEndEvent,
-    ConversationTimeoutEvent,
-    ResponseEvent,
-    NoResponseEvent,
-    SpeechRecognizedEvent,
-    AlarmStartedEvent,
-    AlarmEndEvent,
-    TimerStartedEvent,
-    TimerEndEvent,
-    AlertStartedEvent,
-    AlertEndEvent,
-    MicMutedEvent,
-    MicUnmutedEvent,
-)
-
-
-class AssistantGoogleBackend(AssistantBackend):
-    """
-    Google Assistant backend.
-
-    It listens for voice commands and post conversation events on the bus.
-
-    **WARNING**: The Google Assistant library used by this backend has officially been deprecated:
-    https://developers.google.com/assistant/sdk/reference/library/python/. This backend still works on most of the
-    devices where I use it, but its correct functioning is not guaranteed as the assistant library is no longer
-    maintained.
-
-    Triggers:
-
-        * :class:`platypush.message.event.assistant.ConversationStartEvent` \
-            when a new conversation starts
-        * :class:`platypush.message.event.assistant.SpeechRecognizedEvent` \
-            when a new voice command is recognized
-        * :class:`platypush.message.event.assistant.NoResponse` \
-            when a conversation returned no response
-        * :class:`platypush.message.event.assistant.ResponseEvent` \
-            when the assistant is speaking a response
-        * :class:`platypush.message.event.assistant.ConversationTimeoutEvent` \
-            when a conversation times out
-        * :class:`platypush.message.event.assistant.ConversationEndEvent` \
-            when a new conversation ends
-        * :class:`platypush.message.event.assistant.AlarmStartedEvent` \
-            when an alarm starts
-        * :class:`platypush.message.event.assistant.AlarmEndEvent` \
-            when an alarm ends
-        * :class:`platypush.message.event.assistant.TimerStartedEvent` \
-            when a timer starts
-        * :class:`platypush.message.event.assistant.TimerEndEvent` \
-            when a timer ends
-        * :class:`platypush.message.event.assistant.MicMutedEvent` \
-            when the microphone is muted.
-        * :class:`platypush.message.event.assistant.MicUnmutedEvent` \
-            when the microphone is un-muted.
-
-    Requires:
-
-        * **google-assistant-library** (``pip install google-assistant-library``)
-        * **google-assistant-sdk[samples]** (``pip install google-assistant-sdk[samples]``)
-        * **google-auth** (``pip install google-auth``)
-
-    """
-
-    _default_credentials_file = os.path.join(
-        os.path.expanduser('~/.config'), 'google-oauthlib-tool', 'credentials.json'
-    )
-
-    def __init__(
-        self,
-        credentials_file=_default_credentials_file,
-        device_model_id='Platypush',
-        **kwargs
-    ):
-        """
-        :param credentials_file: Path to the Google OAuth credentials file
-            (default: ~/.config/google-oauthlib-tool/credentials.json).
-            See
-            https://developers.google.com/assistant/sdk/guides/library/python/embed/install-sample#generate_credentials
-            for instructions to get your own credentials file.
-
-        :type credentials_file: str
-
-        :param device_model_id: Device model ID to use for the assistant
-            (default: Platypush)
-        :type device_model_id: str
-        """
-
-        super().__init__(**kwargs)
-        self.credentials_file = credentials_file
-        self.device_model_id = device_model_id
-        self.credentials = None
-        self.assistant = None
-        self._has_error = False
-        self._is_muted = False
-
-        self.logger.info('Initialized Google Assistant backend')
-
-    def _process_event(self, event):
-        from google.assistant.library.event import EventType, AlertType
-
-        self.logger.info('Received assistant event: {}'.format(event))
-        self._has_error = False
-
-        if event.type == EventType.ON_CONVERSATION_TURN_STARTED:
-            self.bus.post(ConversationStartEvent(assistant=self))
-        elif event.type == EventType.ON_CONVERSATION_TURN_FINISHED:
-            if not event.args.get('with_follow_on_turn'):
-                self.bus.post(ConversationEndEvent(assistant=self))
-        elif event.type == EventType.ON_CONVERSATION_TURN_TIMEOUT:
-            self.bus.post(ConversationTimeoutEvent(assistant=self))
-        elif event.type == EventType.ON_NO_RESPONSE:
-            self.bus.post(NoResponseEvent(assistant=self))
-        elif (
-            hasattr(EventType, 'ON_RENDER_RESPONSE')
-            and event.type == EventType.ON_RENDER_RESPONSE
-        ):
-            self.bus.post(
-                ResponseEvent(assistant=self, response_text=event.args.get('text'))
-            )
-            tts, args = self._get_tts_plugin()
-
-            if tts and 'text' in event.args:
-                self.stop_conversation()
-                tts.say(text=event.args['text'], **args)
-        elif (
-            hasattr(EventType, 'ON_RESPONDING_STARTED')
-            and event.type == EventType.ON_RESPONDING_STARTED
-            and event.args.get('is_error_response', False) is True
-        ):
-            self.logger.warning('Assistant response error')
-        elif event.type == EventType.ON_RECOGNIZING_SPEECH_FINISHED:
-            phrase = event.args['text'].lower().strip()
-            self.logger.info('Speech recognized: {}'.format(phrase))
-            self.bus.post(SpeechRecognizedEvent(assistant=self, phrase=phrase))
-        elif event.type == EventType.ON_ALERT_STARTED:
-            if event.args.get('alert_type') == AlertType.ALARM:
-                self.bus.post(AlarmStartedEvent(assistant=self))
-            elif event.args.get('alert_type') == AlertType.TIMER:
-                self.bus.post(TimerStartedEvent(assistant=self))
-            else:
-                self.bus.post(AlertStartedEvent(assistant=self))
-        elif event.type == EventType.ON_ALERT_FINISHED:
-            if event.args.get('alert_type') == AlertType.ALARM:
-                self.bus.post(AlarmEndEvent(assistant=self))
-            elif event.args.get('alert_type') == AlertType.TIMER:
-                self.bus.post(TimerEndEvent(assistant=self))
-            else:
-                self.bus.post(AlertEndEvent(assistant=self))
-        elif event.type == EventType.ON_ASSISTANT_ERROR:
-            self._has_error = True
-            if event.args.get('is_fatal'):
-                self.logger.error('Fatal assistant error')
-            else:
-                self.logger.warning('Assistant error')
-        if event.type == EventType.ON_MUTED_CHANGED:
-            self._is_muted = event.args.get('is_muted')
-            event = MicMutedEvent() if self._is_muted else MicUnmutedEvent()
-            self.bus.post(event)
-
-    def start_conversation(self):
-        """Starts an assistant conversation"""
-        if self.assistant:
-            self.assistant.start_conversation()
-
-    def stop_conversation(self):
-        """Stops an assistant conversation"""
-        if self.assistant:
-            self.assistant.stop_conversation()
-
-    def set_mic_mute(self, muted):
-        if not self.assistant:
-            self.logger.warning('Assistant not running')
-            return
-
-        self.assistant.set_mic_mute(muted)
-
-    def is_muted(self) -> bool:
-        return self._is_muted
-
-    def send_text_query(self, query):
-        if not self.assistant:
-            self.logger.warning('Assistant not running')
-            return
-
-        self.assistant.send_text_query(query)
-
-    def run(self):
-        import google.oauth2.credentials
-        from google.assistant.library import Assistant
-
-        super().run()
-
-        with open(self.credentials_file, 'r') as f:
-            self.credentials = google.oauth2.credentials.Credentials(
-                token=None, **json.load(f)
-            )
-
-        while not self.should_stop():
-            self._has_error = False
-
-            with Assistant(self.credentials, self.device_model_id) as assistant:
-                self.assistant = assistant
-                for event in assistant.start():
-                    if not self.is_detecting():
-                        self.logger.info(
-                            'Assistant event received but detection is currently paused'
-                        )
-                        continue
-
-                    self._process_event(event)
-                    if self._has_error:
-                        self.logger.info(
-                            'Restarting the assistant after an unrecoverable error'
-                        )
-                        time.sleep(5)
-                        break
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/assistant/google/manifest.yaml

@@ -1,27 +0,0 @@
-manifest:
-  events:
-    platypush.message.event.assistant.AlarmEndEvent: when an alarm ends
-    platypush.message.event.assistant.AlarmStartedEvent: when an alarm starts
-    platypush.message.event.assistant.ConversationEndEvent: when a new conversation
-      ends
-    platypush.message.event.assistant.ConversationStartEvent: when a new conversation
-      starts
-    platypush.message.event.assistant.ConversationTimeoutEvent: when a conversation
-      times out
-    platypush.message.event.assistant.MicMutedEvent: when the microphone is muted.
-    platypush.message.event.assistant.MicUnmutedEvent: when the microphone is un-muted.
-    platypush.message.event.assistant.NoResponse: when a conversation returned no
-      response
-    platypush.message.event.assistant.ResponseEvent: when the assistant is speaking
-      a response
-    platypush.message.event.assistant.SpeechRecognizedEvent: when a new voice command
-      is recognized
-    platypush.message.event.assistant.TimerEndEvent: when a timer ends
-    platypush.message.event.assistant.TimerStartedEvent: when a timer starts
-  install:
-    pip:
-    - google-assistant-library
-    - google-assistant-sdk[samples]
-    - google-auth
-  package: platypush.backend.assistant.google
-  type: backend

platypush/backend/assistant/snowboy/__init__.py

@@ -1,193 +0,0 @@
-import os
-import threading
-
-from platypush.backend.assistant import AssistantBackend
-from platypush.context import get_plugin
-from platypush.message.event.assistant import HotwordDetectedEvent
-
-
-class AssistantSnowboyBackend(AssistantBackend):
-    """
-    Backend for detecting custom voice hotwords through Snowboy.  The purpose of
-    this component is only to detect the hotword specified in your Snowboy voice
-    model. If you want to trigger proper assistant conversations or custom
-    speech recognition, you should create a hook in your configuration on
-    HotwordDetectedEvent to trigger the conversation on whichever assistant
-    plugin you're using (Google, Alexa...)
-
-    Triggers:
-
-        * :class:`platypush.message.event.assistant.HotwordDetectedEvent` \
-            whenever the hotword has been detected
-
-    Requires:
-
-        * **snowboy** (``pip install snowboy``)
-
-    Manual installation for snowboy and its Python bindings if the command above fails::
-
-        $ [sudo] apt-get install libatlas-base-dev swig
-        $ [sudo] pip install pyaudio
-        $ git clone https://github.com/Kitt-AI/snowboy
-        $ cd snowboy/swig/Python3
-        $ make
-        $ cd ../..
-        $ python3 setup.py build
-        $ [sudo] python setup.py install
-
-    You will also need a voice model for the hotword detection. You can find
-    some under the ``resources/models`` directory of the Snowboy repository,
-    or train/download other models from https://snowboy.kitt.ai.
-    """
-
-    def __init__(self, models, audio_gain=1.0, **kwargs):
-        """
-        :param models: Map (name -> configuration) of voice models to be used by
-            the assistant.  See https://snowboy.kitt.ai/ for training/downloading
-            models. Sample format::
-
-                ok_google:    # Hotword model name
-                    voice_model_file: /path/models/OK Google.pmdl  # Voice model file location
-                    sensitivity: 0.5            # Model sensitivity, between 0 and 1 (default: 0.5)
-                    assistant_plugin: assistant.google.pushtotalk  # When the hotword is detected trigger the Google
-                                                                   # push-to-talk assistant plugin (optional)
-                    assistant_language: en-US   # The assistant will conversate in English when this hotword is
-                        detected (optional)
-                    detect_sound: /path/to/bell.wav   # Sound file to be played when the hotword is detected (optional)
-
-                ciao_google:  # Hotword model name
-                    voice_model_file: /path/models/Ciao Google.pmdl  # Voice model file location
-                    sensitivity: 0.5            # Model sensitivity, between 0 and 1 (default: 0.5)
-                    assistant_plugin: assistant.google.pushtotalk    # When the hotword is detected trigger the Google
-                                                                     # push-to-talk assistant plugin (optional)
-                    assistant_language: it-IT   # The assistant will conversate in Italian when this hotword is
-                                                # detected (optional)
-                    detect_sound: /path/to/bell.wav   # Sound file to be played when the hotword is detected (optional)
-
-        :type models: dict
-
-        :param audio_gain: Audio gain, between 0 and 1. Default: 1
-        :type audio_gain: float
-        """
-
-        try:
-            import snowboydecoder
-        except ImportError:
-            import snowboy.snowboydecoder as snowboydecoder
-
-        super().__init__(**kwargs)
-
-        self.models = {}
-        self._init_models(models)
-        self.audio_gain = audio_gain
-
-        self.detector = snowboydecoder.HotwordDetector(
-            [model['voice_model_file'] for model in self.models.values()],
-            sensitivity=[model['sensitivity'] for model in self.models.values()],
-            audio_gain=self.audio_gain,
-        )
-
-        self.logger.info(
-            'Initialized Snowboy hotword detection with {} voice model configurations'.format(
-                len(self.models)
-            )
-        )
-
-    def _init_models(self, models):
-        if not models:
-            raise AttributeError('Please specify at least one voice model')
-
-        self.models = {}
-        for name, conf in models.items():
-            if name in self.models:
-                raise AttributeError('Duplicate model key {}'.format(name))
-
-            model_file = conf.get('voice_model_file')
-            detect_sound = conf.get('detect_sound')
-
-            if not model_file:
-                raise AttributeError(
-                    'No voice_model_file specified for model {}'.format(name)
-                )
-
-            model_file = os.path.abspath(os.path.expanduser(model_file))
-            assistant_plugin_name = conf.get('assistant_plugin')
-
-            if detect_sound:
-                detect_sound = os.path.abspath(os.path.expanduser(detect_sound))
-
-            if not os.path.isfile(model_file):
-                raise FileNotFoundError(
-                    'Voice model file {} does not exist or it not a regular file'.format(
-                        model_file
-                    )
-                )
-
-            self.models[name] = {
-                'voice_model_file': model_file,
-                'sensitivity': conf.get('sensitivity', 0.5),
-                'detect_sound': detect_sound,
-                'assistant_plugin': get_plugin(assistant_plugin_name)
-                if assistant_plugin_name
-                else None,
-                'assistant_language': conf.get('assistant_language'),
-                'tts_plugin': conf.get('tts_plugin'),
-                'tts_args': conf.get('tts_args', {}),
-            }
-
-    def hotword_detected(self, hotword):
-        """
-        Callback called on hotword detection
-        """
-        try:
-            import snowboydecoder
-        except ImportError:
-            import snowboy.snowboydecoder as snowboydecoder
-
-        def sound_thread(sound):
-            snowboydecoder.play_audio_file(sound)
-
-        def callback():
-            if not self.is_detecting():
-                self.logger.info(
-                    'Hotword detected but assistant response currently paused'
-                )
-                return
-
-            self.bus.post(HotwordDetectedEvent(hotword=hotword))
-            model = self.models[hotword]
-
-            detect_sound = model.get('detect_sound')
-            assistant_plugin = model.get('assistant_plugin')
-            assistant_language = model.get('assistant_language')
-            tts_plugin = model.get('tts_plugin')
-            tts_args = model.get('tts_args')
-
-            if detect_sound:
-                threading.Thread(target=sound_thread, args=(detect_sound,)).start()
-
-            if assistant_plugin:
-                assistant_plugin.start_conversation(
-                    language=assistant_language,
-                    tts_plugin=tts_plugin,
-                    tts_args=tts_args,
-                )
-
-        return callback
-
-    def on_stop(self):
-        super().on_stop()
-        if self.detector:
-            self.detector.terminate()
-            self.detector = None
-
-    def run(self):
-        super().run()
-        self.detector.start(
-            detected_callback=[
-                self.hotword_detected(hotword) for hotword in self.models.keys()
-            ]
-        )
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/assistant/snowboy/manifest.yaml

@@ -1,9 +0,0 @@
-manifest:
-  events:
-    platypush.message.event.assistant.HotwordDetectedEvent: whenever the hotword has
-      been detected
-  install:
-    pip:
-    - snowboy
-  package: platypush.backend.assistant.snowboy
-  type: backend

platypush/backend/button/flic/__init__.py

@@ -12,15 +12,12 @@ class ButtonFlicBackend(Backend):
     Backend that listen for events from the Flic (https://flic.io/) bluetooth
     smart buttons.
 
-    Triggers:
-
-        * :class:`platypush.message.event.button.flic.FlicButtonEvent` when a button is pressed.
-            The event will also contain the press sequence
-            (e.g. ``["ShortPressEvent", "LongPressEvent", "ShortPressEvent"]``)
-
     Requires:
 
-        * **fliclib** (https://github.com/50ButtonsEach/fliclib-linux-hci). For the backend to work properly you need to have the ``flicd`` daemon from the fliclib running, and you have to first pair the buttons with your device using any of the scanners provided by the library.
+        * **fliclib** (https://github.com/50ButtonsEach/fliclib-linux-hci). For
+          the backend to work properly you need to have the ``flicd`` daemon
+          from the fliclib running, and you have to first pair the buttons with
+          your device using any of the scanners provided by the library.
 
     """
 
@@ -29,16 +26,23 @@ class ButtonFlicBackend(Backend):
     ShortPressEvent = "ShortPressEvent"
     LongPressEvent = "LongPressEvent"
 
-    def __init__(self, server='localhost', long_press_timeout=_long_press_timeout,
+    def __init__(
-                 btn_timeout=_btn_timeout, **kwargs):
+        self,
+        server='localhost',
+        long_press_timeout=_long_press_timeout,
+        btn_timeout=_btn_timeout,
+        **kwargs
+    ):
         """
         :param server: flicd server host (default: localhost)
         :type server: str
 
-        :param long_press_timeout: How long you should press a button for a press action to be considered "long press" (default: 0.3 secohds)
+        :param long_press_timeout: How long you should press a button for a
+            press action to be considered "long press" (default: 0.3 secohds)
         :type long_press_timeout: float
 
-        :param btn_timeout: How long since the last button release before considering the user interaction completed (default: 0.5 seconds)
+        :param btn_timeout: How long since the last button release before
+            considering the user interaction completed (default: 0.5 seconds)
         :type btn_timeout: float
         """
 
@@ -55,15 +59,16 @@ class ButtonFlicBackend(Backend):
         self._btn_addr = None
         self._down_pressed_time = None
         self._cur_sequence = []
-
+        self.logger.info('Initialized Flic buttons backend on %s', self.server)
-        self.logger.info('Initialized Flic buttons backend on {}'.format(self.server))
 
     def _got_button(self):
         def _f(bd_addr):
             cc = ButtonConnectionChannel(bd_addr)
-            cc.on_button_up_or_down = \
+            cc.on_button_up_or_down = (
-                lambda channel, click_type, was_queued, time_diff: \
+                lambda channel, click_type, was_queued, time_diff: self._on_event()(
-                self._on_event()(bd_addr, channel, click_type, was_queued, time_diff)
+                    bd_addr, channel, click_type, was_queued, time_diff
+                )
+            )
             self.client.add_connection_channel(cc)
 
         return _f
@@ -72,23 +77,27 @@ class ButtonFlicBackend(Backend):
         def _f(items):
             for bd_addr in items["bd_addr_of_verified_buttons"]:
                 self._got_button()(bd_addr)
+
         return _f
 
     def _on_btn_timeout(self):
         def _f():
-            self.logger.info('Flic event triggered from {}: {}'.format(
+            self.logger.info(
-                self._btn_addr, self._cur_sequence))
+                'Flic event triggered from %s: %s', self._btn_addr, self._cur_sequence
+            )
 
-            self.bus.post(FlicButtonEvent(
+            self.bus.post(
-                btn_addr=self._btn_addr, sequence=self._cur_sequence))
+                FlicButtonEvent(btn_addr=self._btn_addr, sequence=self._cur_sequence)
+            )
 
             self._cur_sequence = []
 
         return _f
 
     def _on_event(self):
-        # noinspection PyUnusedLocal
+        # _ = channel
-        def _f(bd_addr, channel, click_type, was_queued, time_diff):
+        # __ = time_diff
+        def _f(bd_addr, _, click_type, was_queued, __):
             if was_queued:
                 return
 
@@ -120,4 +129,3 @@ class ButtonFlicBackend(Backend):
 
 
 # vim:sw=4:ts=4:et:
-

platypush/backend/camera/pi/__init__.py

@@ -15,10 +15,6 @@ class CameraPiBackend(Backend):
     the :class:`platypush.plugins.camera.pi` plugin. Note that the Redis backend
     must be configured and running to enable camera control.
 
-    Requires:
-
-        * **picamera** (``pip install picamera``)
-
     This backend is **DEPRECATED**. Use the plugin :class:`platypush.plugins.camera.pi.CameraPiPlugin` instead to run
     Pi camera actions. If you want to start streaming the camera on application start then simply create an event hook
     on :class:`platypush.message.event.application.ApplicationStartedEvent` that runs ``camera.pi.start_streaming``.
@@ -33,15 +29,32 @@ class CameraPiBackend(Backend):
             return self.value == other
 
     # noinspection PyUnresolvedReferences,PyPackageRequirements
-    def __init__(self, listen_port, bind_address='0.0.0.0', x_resolution=640, y_resolution=480,
+    def __init__(
-                 redis_queue='platypush/camera/pi',
+        self,
-                 start_recording_on_startup=True,
+        listen_port,
-                 framerate=24, hflip=False, vflip=False,
+        bind_address='0.0.0.0',
-                 sharpness=0, contrast=0, brightness=50,
+        x_resolution=640,
-                 video_stabilization=False, iso=0, exposure_compensation=0,
+        y_resolution=480,
-                 exposure_mode='auto', meter_mode='average', awb_mode='auto',
+        redis_queue='platypush/camera/pi',
-                 image_effect='none', color_effects=None, rotation=0,
+        start_recording_on_startup=True,
-                 crop=(0.0, 0.0, 1.0, 1.0), **kwargs):
+        framerate=24,
+        hflip=False,
+        vflip=False,
+        sharpness=0,
+        contrast=0,
+        brightness=50,
+        video_stabilization=False,
+        iso=0,
+        exposure_compensation=0,
+        exposure_mode='auto',
+        meter_mode='average',
+        awb_mode='auto',
+        image_effect='none',
+        color_effects=None,
+        rotation=0,
+        crop=(0.0, 0.0, 1.0, 1.0),
+        **kwargs
+    ):
         """
         See https://www.raspberrypi.org/documentation/usage/camera/python/README.md
         for a detailed reference about the Pi camera options.
@@ -58,7 +71,9 @@ class CameraPiBackend(Backend):
         self.bind_address = bind_address
         self.listen_port = listen_port
         self.server_socket = socket.socket()
-        self.server_socket.bind((self.bind_address, self.listen_port))   # lgtm [py/bind-socket-all-network-interfaces]
+        self.server_socket.bind(
+            (self.bind_address, self.listen_port)
+        )  # lgtm [py/bind-socket-all-network-interfaces]
         self.server_socket.listen(0)
 
         import picamera
@@ -87,10 +102,7 @@ class CameraPiBackend(Backend):
         self._recording_thread = None
 
     def send_camera_action(self, action, **kwargs):
-        action = {
+        action = {'action': action.value, **kwargs}
-            'action': action.value,
-            **kwargs
-        }
 
         self.redis.send_message(msg=json.dumps(action), queue_name=self.redis_queue)
 
@@ -127,7 +139,9 @@ class CameraPiBackend(Backend):
             else:
                 while not self.should_stop():
                     connection = self.server_socket.accept()[0].makefile('wb')
-                    self.logger.info('Accepted client connection on port {}'.format(self.listen_port))
+                    self.logger.info(
+                        'Accepted client connection on port {}'.format(self.listen_port)
+                    )
 
                     try:
                         self.camera.start_recording(connection, format=format)
@@ -138,12 +152,16 @@ class CameraPiBackend(Backend):
                         try:
                             self.stop_recording()
                         except Exception as e:
-                            self.logger.warning('Could not stop recording: {}'.format(str(e)))
+                            self.logger.warning(
+                                'Could not stop recording: {}'.format(str(e))
+                            )
 
                         try:
                             connection.close()
                         except Exception as e:
-                            self.logger.warning('Could not close connection: {}'.format(str(e)))
+                            self.logger.warning(
+                                'Could not close connection: {}'.format(str(e))
+                            )
 
                         self.send_camera_action(self.CameraAction.START_RECORDING)
 
@@ -152,12 +170,13 @@ class CameraPiBackend(Backend):
             return
 
         self.logger.info('Starting camera recording')
-        self._recording_thread = Thread(target=recording_thread,
+        self._recording_thread = Thread(
-                                        name='PiCameraRecorder')
+            target=recording_thread, name='PiCameraRecorder'
+        )
         self._recording_thread.start()
 
     def stop_recording(self):
-        """ Stops recording """
+        """Stops recording"""
 
         self.logger.info('Stopping camera recording')
 

platypush/backend/camera/pi/manifest.yaml

@@ -1,9 +1,21 @@
 manifest:
   events: {}
   install:
+    apk:
+      - py3-numpy
+      - py3-pillow
+    dnf:
+      - python-numpy
+      - python-pillow
+    pacman:
+      - python-numpy
+      - python-pillow
+    apt:
+      - python3-numpy
+      - python3-pillow
     pip:
-    - picamera
+      - picamera
-    - numpy
+      - numpy
-    - Pillow
+      - Pillow
   package: platypush.backend.camera.pi
   type: backend

platypush/backend/chat/telegram/__init__.py

@@ -4,9 +4,17 @@ from typing import Type, Optional, Union, List
 
 from platypush.backend import Backend
 from platypush.context import get_plugin
-from platypush.message.event.chat.telegram import MessageEvent, CommandMessageEvent, TextMessageEvent, \
+from platypush.message.event.chat.telegram import (
-    PhotoMessageEvent, VideoMessageEvent, ContactMessageEvent, DocumentMessageEvent, LocationMessageEvent, \
+    MessageEvent,
-    GroupChatCreatedEvent
+    CommandMessageEvent,
+    TextMessageEvent,
+    PhotoMessageEvent,
+    VideoMessageEvent,
+    ContactMessageEvent,
+    DocumentMessageEvent,
+    LocationMessageEvent,
+    GroupChatCreatedEvent,
+)
 from platypush.plugins.chat.telegram import ChatTelegramPlugin
 
 
@@ -14,24 +22,15 @@ class ChatTelegramBackend(Backend):
     """
     Telegram bot that listens for messages and updates.
 
-    Triggers:
-
-        * :class:`platypush.message.event.chat.telegram.TextMessageEvent` when a text message is received.
-        * :class:`platypush.message.event.chat.telegram.PhotoMessageEvent` when a photo is received.
-        * :class:`platypush.message.event.chat.telegram.VideoMessageEvent` when a video is received.
-        * :class:`platypush.message.event.chat.telegram.LocationMessageEvent` when a location is received.
-        * :class:`platypush.message.event.chat.telegram.ContactMessageEvent` when a contact is received.
-        * :class:`platypush.message.event.chat.telegram.DocumentMessageEvent` when a document is received.
-        * :class:`platypush.message.event.chat.telegram.CommandMessageEvent` when a command message is received.
-        * :class:`platypush.message.event.chat.telegram.GroupCreatedEvent` when the bot is invited to a new group.
-
     Requires:
 
         * The :class:`platypush.plugins.chat.telegram.ChatTelegramPlugin` plugin configured
 
     """
 
-    def __init__(self, authorized_chat_ids: Optional[List[Union[str, int]]] = None, **kwargs):
+    def __init__(
+        self, authorized_chat_ids: Optional[List[Union[str, int]]] = None, **kwargs
+    ):
         """
         :param authorized_chat_ids: Optional list of chat_id/user_id which are authorized to send messages to
             the bot. If nothing is specified then no restrictions are applied.
@@ -39,40 +38,52 @@ class ChatTelegramBackend(Backend):
 
         super().__init__(**kwargs)
         self.authorized_chat_ids = set(authorized_chat_ids or [])
-        self._plugin: ChatTelegramPlugin = get_plugin('chat.telegram')
+        self._plugin: ChatTelegramPlugin = get_plugin('chat.telegram')  # type: ignore
 
     def _authorize(self, msg):
         if not self.authorized_chat_ids:
             return
 
         if msg.chat.type == 'private' and msg.chat.id not in self.authorized_chat_ids:
-            self.logger.info('Received message from unauthorized chat_id {}'.format(msg.chat.id))
+            self.logger.info(
-            self._plugin.send_message(chat_id=msg.chat.id, text='You are not allowed to send messages to this bot')
+                'Received message from unauthorized chat_id %s', msg.chat.id
+            )
+            self._plugin.send_message(
+                chat_id=msg.chat.id,
+                text='You are not allowed to send messages to this bot',
+            )
             raise PermissionError
 
     def _msg_hook(self, cls: Type[MessageEvent]):
         # noinspection PyUnusedLocal
-        def hook(update, context):
+        def hook(update, _):
             msg = update.effective_message
 
             try:
                 self._authorize(msg)
-                self.bus.post(cls(chat_id=update.effective_chat.id,
+                self.bus.post(
-                                  message=self._plugin.parse_msg(msg).output,
+                    cls(
-                                  user=self._plugin.parse_user(update.effective_user).output))
+                        chat_id=update.effective_chat.id,
+                        message=self._plugin.parse_msg(msg).output,
+                        user=self._plugin.parse_user(update.effective_user).output,
+                    )
+                )
             except PermissionError:
                 pass
 
         return hook
 
     def _group_hook(self):
-        # noinspection PyUnusedLocal
         def hook(update, context):
             msg = update.effective_message
             if msg.group_chat_created:
-                self.bus.post(GroupChatCreatedEvent(chat_id=update.effective_chat.id,
+                self.bus.post(
-                                                    message=self._plugin.parse_msg(msg).output,
+                    GroupChatCreatedEvent(
-                                                    user=self._plugin.parse_user(update.effective_user).output))
+                        chat_id=update.effective_chat.id,
+                        message=self._plugin.parse_msg(msg).output,
+                        user=self._plugin.parse_user(update.effective_user).output,
+                    )
+                )
             elif msg.photo:
                 self._msg_hook(PhotoMessageEvent)(update, context)
             elif msg.video:
@@ -92,27 +103,33 @@ class ChatTelegramBackend(Backend):
         return hook
 
     def _command_hook(self):
-        # noinspection PyUnusedLocal
+        def hook(update, _):
-        def hook(update, context):
             msg = update.effective_message
-            m = re.match('\s*/([0-9a-zA-Z_-]+)\s*(.*)', msg.text)
+            m = re.match(r'\s*/([0-9a-zA-Z_-]+)\s*(.*)', msg.text)
+            if not m:
+                self.logger.warning('Invalid command: %s', msg.text)
+                return
+
             cmd = m.group(1).lower()
-            args = [arg for arg in re.split('\s+', m.group(2)) if len(arg)]
+            args = [arg for arg in re.split(r'\s+', m.group(2)) if len(arg)]
 
             try:
                 self._authorize(msg)
-                self.bus.post(CommandMessageEvent(chat_id=update.effective_chat.id,
+                self.bus.post(
-                                                  command=cmd,
+                    CommandMessageEvent(
-                                                  cmdargs=args,
+                        chat_id=update.effective_chat.id,
-                                                  message=self._plugin.parse_msg(msg).output,
+                        command=cmd,
-                                                  user=self._plugin.parse_user(update.effective_user).output))
+                        cmdargs=args,
+                        message=self._plugin.parse_msg(msg).output,
+                        user=self._plugin.parse_user(update.effective_user).output,
+                    )
+                )
             except PermissionError:
                 pass
 
         return hook
 
     def run(self):
-        # noinspection PyPackageRequirements
         from telegram.ext import MessageHandler, Filters
 
         super().run()
@@ -120,12 +137,24 @@ class ChatTelegramBackend(Backend):
         dispatcher = telegram.dispatcher
 
         dispatcher.add_handler(MessageHandler(Filters.group, self._group_hook()))
-        dispatcher.add_handler(MessageHandler(Filters.text, self._msg_hook(TextMessageEvent)))
+        dispatcher.add_handler(
-        dispatcher.add_handler(MessageHandler(Filters.photo, self._msg_hook(PhotoMessageEvent)))
+            MessageHandler(Filters.text, self._msg_hook(TextMessageEvent))
-        dispatcher.add_handler(MessageHandler(Filters.video, self._msg_hook(VideoMessageEvent)))
+        )
-        dispatcher.add_handler(MessageHandler(Filters.contact, self._msg_hook(ContactMessageEvent)))
+        dispatcher.add_handler(
-        dispatcher.add_handler(MessageHandler(Filters.location, self._msg_hook(LocationMessageEvent)))
+            MessageHandler(Filters.photo, self._msg_hook(PhotoMessageEvent))
-        dispatcher.add_handler(MessageHandler(Filters.document, self._msg_hook(DocumentMessageEvent)))
+        )
+        dispatcher.add_handler(
+            MessageHandler(Filters.video, self._msg_hook(VideoMessageEvent))
+        )
+        dispatcher.add_handler(
+            MessageHandler(Filters.contact, self._msg_hook(ContactMessageEvent))
+        )
+        dispatcher.add_handler(
+            MessageHandler(Filters.location, self._msg_hook(LocationMessageEvent))
+        )
+        dispatcher.add_handler(
+            MessageHandler(Filters.document, self._msg_hook(DocumentMessageEvent))
+        )
         dispatcher.add_handler(MessageHandler(Filters.command, self._command_hook()))
 
         self.logger.info('Initialized Telegram backend')

platypush/backend/chat/telegram/manifest.yaml

@@ -5,7 +5,7 @@ manifest:
     platypush.message.event.chat.telegram.ContactMessageEvent: when a contact is received.
     platypush.message.event.chat.telegram.DocumentMessageEvent: when a document is
       received.
-    platypush.message.event.chat.telegram.GroupCreatedEvent: when the bot is invited
+    platypush.message.event.chat.telegram.GroupChatCreatedEvent: when the bot is invited
       to a new group.
     platypush.message.event.chat.telegram.LocationMessageEvent: when a location is
       received.

platypush/backend/covid19/__init__.py

@@ -1,137 +0,0 @@
-import datetime
-import os
-from typing import Optional, Union, List, Dict, Any
-
-from sqlalchemy import create_engine, Column, Integer, String, DateTime
-from sqlalchemy.orm import sessionmaker, scoped_session
-
-from platypush.backend import Backend
-from platypush.common.db import declarative_base
-from platypush.config import Config
-from platypush.context import get_plugin
-from platypush.message.event.covid19 import Covid19UpdateEvent
-from platypush.plugins.covid19 import Covid19Plugin
-
-Base = declarative_base()
-Session = scoped_session(sessionmaker())
-
-
-class Covid19Update(Base):
-    """Models the Covid19Data table"""
-
-    __tablename__ = 'covid19data'
-    __table_args__ = {'sqlite_autoincrement': True}
-
-    country = Column(String, primary_key=True)
-    confirmed = Column(Integer, nullable=False, default=0)
-    deaths = Column(Integer, nullable=False, default=0)
-    recovered = Column(Integer, nullable=False, default=0)
-    last_updated_at = Column(DateTime, nullable=False, default=datetime.datetime.utcnow)
-
-
-class Covid19Backend(Backend):
-    """
-    This backend polls new data about the Covid-19 pandemic diffusion and triggers events when new data is available.
-
-    Triggers:
-
-        - :class:`platypush.message.event.covid19.Covid19UpdateEvent` when new data is available.
-
-    """
-
-    # noinspection PyProtectedMember
-    def __init__(
-        self,
-        country: Optional[Union[str, List[str]]],
-        poll_seconds: Optional[float] = 3600.0,
-        **kwargs
-    ):
-        """
-        :param country: Default country (or list of countries) to retrieve the stats for. It can either be the full
-            country name or the country code. Special values:
-
-            - ``world``: Get worldwide stats.
-            - ``all``: Get all the available stats.
-
-        Default: either the default configured on the :class:`platypush.plugins.covid19.Covid19Plugin` plugin or
-        ``world``.
-
-        :param poll_seconds: How often the backend should check for new check-ins (default: one hour).
-        """
-        super().__init__(poll_seconds=poll_seconds, **kwargs)
-        self._plugin: Covid19Plugin = get_plugin('covid19')
-        self.country: List[str] = self._plugin._get_countries(country)
-        self.workdir = os.path.join(
-            os.path.expanduser(Config.get('workdir')), 'covid19'
-        )
-        self.dbfile = os.path.join(self.workdir, 'data.db')
-        os.makedirs(self.workdir, exist_ok=True)
-
-    def __enter__(self):
-        self.logger.info('Started Covid19 backend')
-
-    def __exit__(self, exc_type, exc_val, exc_tb):
-        self.logger.info('Stopped Covid19 backend')
-
-    def _process_update(self, summary: Dict[str, Any], session: Session):
-        update_time = datetime.datetime.fromisoformat(
-            summary['Date'].replace('Z', '+00:00')
-        )
-
-        self.bus.post(
-            Covid19UpdateEvent(
-                country=summary['Country'],
-                country_code=summary['CountryCode'],
-                confirmed=summary['TotalConfirmed'],
-                deaths=summary['TotalDeaths'],
-                recovered=summary['TotalRecovered'],
-                update_time=update_time,
-            )
-        )
-
-        session.merge(
-            Covid19Update(
-                country=summary['CountryCode'],
-                confirmed=summary['TotalConfirmed'],
-                deaths=summary['TotalDeaths'],
-                recovered=summary['TotalRecovered'],
-                last_updated_at=update_time,
-            )
-        )
-
-    def loop(self):
-        # noinspection PyUnresolvedReferences
-        summaries = self._plugin.summary(self.country).output
-        if not summaries:
-            return
-
-        engine = create_engine(
-            'sqlite:///{}'.format(self.dbfile),
-            connect_args={'check_same_thread': False},
-        )
-        Base.metadata.create_all(engine)
-        Session.configure(bind=engine)
-        session = Session()
-
-        last_records = {
-            record.country: record
-            for record in session.query(Covid19Update)
-            .filter(Covid19Update.country.in_(self.country))
-            .all()
-        }
-
-        for summary in summaries:
-            country = summary['CountryCode']
-            last_record = last_records.get(country)
-            if (
-                not last_record
-                or summary['TotalConfirmed'] != last_record.confirmed
-                or summary['TotalDeaths'] != last_record.deaths
-                or summary['TotalRecovered'] != last_record.recovered
-            ):
-                self._process_update(summary=summary, session=session)
-
-        session.commit()
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/covid19/manifest.yaml

@@ -1,7 +0,0 @@
-manifest:
-  events:
-    platypush.message.event.covid19.Covid19UpdateEvent: when new data is available.
-  install:
-    pip: []
-  package: platypush.backend.covid19
-  type: backend

platypush/backend/file/monitor/__init__.py

@@ -10,17 +10,6 @@ from .entities.resources import MonitoredResource, MonitoredPattern, MonitoredRe
 class FileMonitorBackend(Backend):
     """
     This backend monitors changes to local files and directories using the Watchdog API.
-
-    Triggers:
-
-        * :class:`platypush.message.event.file.FileSystemCreateEvent` if a resource is created.
-        * :class:`platypush.message.event.file.FileSystemDeleteEvent` if a resource is removed.
-        * :class:`platypush.message.event.file.FileSystemModifyEvent` if a resource is modified.
-
-    Requires:
-
-        * **watchdog** (``pip install watchdog``)
-
     """
 
     class EventHandlerFactory:
@@ -29,20 +18,28 @@ class FileMonitorBackend(Backend):
         """
 
         @staticmethod
-        def from_resource(resource: Union[str, Dict[str, Any], MonitoredResource]) -> EventHandler:
+        def from_resource(
+            resource: Union[str, Dict[str, Any], MonitoredResource]
+        ) -> EventHandler:
             if isinstance(resource, str):
                 resource = MonitoredResource(resource)
             elif isinstance(resource, dict):
                 if 'regexes' in resource or 'ignore_regexes' in resource:
                     resource = MonitoredRegex(**resource)
-                elif 'patterns' in resource or 'ignore_patterns' in resource or 'ignore_directories' in resource:
+                elif (
+                    'patterns' in resource
+                    or 'ignore_patterns' in resource
+                    or 'ignore_directories' in resource
+                ):
                     resource = MonitoredPattern(**resource)
                 else:
                     resource = MonitoredResource(**resource)
 
             return EventHandler.from_resource(resource)
 
-    def __init__(self, paths: Iterable[Union[str, Dict[str, Any], MonitoredResource]], **kwargs):
+    def __init__(
+        self, paths: Iterable[Union[str, Dict[str, Any], MonitoredResource]], **kwargs
+    ):
         """
         :param paths: List of paths to monitor. Paths can either be expressed in any of the following ways:
 
@@ -113,7 +110,9 @@ class FileMonitorBackend(Backend):
 
         for path in paths:
             handler = self.EventHandlerFactory.from_resource(path)
-            self._observer.schedule(handler, handler.resource.path, recursive=handler.resource.recursive)
+            self._observer.schedule(
+                handler, handler.resource.path, recursive=handler.resource.recursive
+            )
 
     def run(self):
         super().run()

platypush/backend/file/monitor/manifest.yaml

@@ -4,7 +4,15 @@ manifest:
     platypush.message.event.file.FileSystemDeleteEvent: if a resource is removed.
     platypush.message.event.file.FileSystemModifyEvent: if a resource is modified.
   install:
+    apk:
+      - py3-watchdog
+    apt:
+      - python3-watchdog
+    dnf:
+      - python-watchdog
+    pacman:
+      - python-watchdog
     pip:
-    - watchdog
+      - watchdog
   package: platypush.backend.file.monitor
   type: backend

platypush/backend/foursquare/__init__.py

@@ -14,10 +14,6 @@ class FoursquareBackend(Backend):
 
         * The :class:`platypush.plugins.foursquare.FoursquarePlugin` plugin configured and enabled.
 
-    Triggers:
-
-        - :class:`platypush.message.event.foursquare.FoursquareCheckinEvent` when a new check-in occurs.
-
     """
 
     _last_created_at_varname = '_foursquare_checkin_last_created_at'
@@ -30,8 +26,12 @@ class FoursquareBackend(Backend):
         self._last_created_at = None
 
     def __enter__(self):
-        self._last_created_at = int(get_plugin('variable').get(self._last_created_at_varname).
+        self._last_created_at = int(
-                                    output.get(self._last_created_at_varname) or 0)
+            get_plugin('variable')
+            .get(self._last_created_at_varname)
+            .output.get(self._last_created_at_varname)
+            or 0
+        )
         self.logger.info('Started Foursquare backend')
 
     def loop(self):
@@ -46,7 +46,9 @@ class FoursquareBackend(Backend):
 
         self.bus.post(FoursquareCheckinEvent(checkin=last_checkin))
         self._last_created_at = last_checkin_created_at
-        get_plugin('variable').set(**{self._last_created_at_varname: self._last_created_at})
+        get_plugin('variable').set(
+            **{self._last_created_at_varname: self._last_created_at}
+        )
 
 
 # vim:sw=4:ts=4:et:

platypush/backend/github/__init__.py

@@ -60,27 +60,6 @@ class GithubBackend(Backend):
         - ``notifications``
         - ``read:org`` if you want to access repositories on organization level.
 
-    Triggers:
-
-        - :class:`platypush.message.event.github.GithubPushEvent` when a new push is created.
-        - :class:`platypush.message.event.github.GithubCommitCommentEvent` when a new commit comment is created.
-        - :class:`platypush.message.event.github.GithubCreateEvent` when a tag or branch is created.
-        - :class:`platypush.message.event.github.GithubDeleteEvent` when a tag or branch is deleted.
-        - :class:`platypush.message.event.github.GithubForkEvent` when a user forks a repository.
-        - :class:`platypush.message.event.github.GithubWikiEvent` when new activity happens on a repository wiki.
-        - :class:`platypush.message.event.github.GithubIssueCommentEvent` when new activity happens on an issue comment.
-        - :class:`platypush.message.event.github.GithubIssueEvent` when new repository issue activity happens.
-        - :class:`platypush.message.event.github.GithubMemberEvent` when new repository collaborators activity happens.
-        - :class:`platypush.message.event.github.GithubPublicEvent` when a repository goes public.
-        - :class:`platypush.message.event.github.GithubPullRequestEvent` when new pull request related activity happens.
-        - :class:`platypush.message.event.github.GithubPullRequestReviewCommentEvent` when activity happens on a pull
-            request commit.
-        - :class:`platypush.message.event.github.GithubReleaseEvent` when a new release happens.
-        - :class:`platypush.message.event.github.GithubSponsorshipEvent` when new sponsorship related activity happens.
-        - :class:`platypush.message.event.github.GithubWatchEvent` when someone stars/starts watching a repository.
-        - :class:`platypush.message.event.github.GithubEvent` for any event that doesn't fall in the above categories
-            (``event_type`` will be set accordingly).
-
     """
 
     _base_url = 'https://api.github.com'

platypush/backend/google/fit/__init__.py

@@ -13,28 +13,30 @@ class GoogleFitBackend(Backend):
     measurements, new fitness activities etc.) on the specified data streams and
     fire an event upon new data.
 
-    Triggers:
-
-        * :class:`platypush.message.event.google.fit.GoogleFitEvent` when a new
-            data point is received on one of the registered streams.
-
     Requires:
 
         * The **google.fit** plugin
-            (:class:`platypush.plugins.google.fit.GoogleFitPlugin`) enabled.
+          (:class:`platypush.plugins.google.fit.GoogleFitPlugin`) enabled.
-        * The **db** plugin (:class:`platypush.plugins.db`) configured
+
     """
 
     _default_poll_seconds = 60
     _default_user_id = 'me'
     _last_timestamp_varname = '_GOOGLE_FIT_LAST_TIMESTAMP_'
 
-    def __init__(self, data_sources, user_id=_default_user_id,
+    def __init__(
-                 poll_seconds=_default_poll_seconds, *args, **kwargs):
+        self,
+        data_sources,
+        user_id=_default_user_id,
+        poll_seconds=_default_poll_seconds,
+        *args,
+        **kwargs
+    ):
         """
         :param data_sources: Google Fit data source IDs to monitor. You can
             get a list of the available data sources through the
-            :meth:`platypush.plugins.google.fit.get_data_sources` action
+            :meth:`platypush.plugins.google.fit.GoogleFitPlugin.get_data_sources`
+            action
         :type data_sources: list[str]
 
         :param user_id: Google user ID to track (default: 'me')
@@ -53,23 +55,31 @@ class GoogleFitBackend(Backend):
 
     def run(self):
         super().run()
-        self.logger.info('Started Google Fit backend on data sources {}'.format(
+        self.logger.info(
-            self.data_sources))
+            'Started Google Fit backend on data sources {}'.format(self.data_sources)
+        )
 
         while not self.should_stop():
             try:
                 for data_source in self.data_sources:
                     varname = self._last_timestamp_varname + data_source
-                    last_timestamp = float(get_plugin('variable').
+                    last_timestamp = float(
-                                           get(varname).output.get(varname) or 0)
+                        get_plugin('variable').get(varname).output.get(varname) or 0
+                    )
 
                     new_last_timestamp = last_timestamp
-                    self.logger.info('Processing new entries from data source {}, last timestamp: {}'.
+                    self.logger.info(
-                                     format(data_source,
+                        'Processing new entries from data source {}, last timestamp: {}'.format(
-                                            str(datetime.datetime.fromtimestamp(last_timestamp))))
+                            data_source,
+                            str(datetime.datetime.fromtimestamp(last_timestamp)),
+                        )
+                    )
 
-                    data_points = get_plugin('google.fit').get_data(
+                    data_points = (
-                        user_id=self.user_id, data_source_id=data_source).output
+                        get_plugin('google.fit')
+                        .get_data(user_id=self.user_id, data_source_id=data_source)
+                        .output
+                    )
                     new_data_points = 0
 
                     for dp in data_points:
@@ -78,25 +88,34 @@ class GoogleFitBackend(Backend):
                             del dp['dataSourceId']
 
                         if dp_time > last_timestamp:
-                            self.bus.post(GoogleFitEvent(
+                            self.bus.post(
-                                user_id=self.user_id, data_source_id=data_source,
+                                GoogleFitEvent(
-                                data_type=dp.pop('dataTypeName'),
+                                    user_id=self.user_id,
-                                start_time=dp_time,
+                                    data_source_id=data_source,
-                                end_time=dp.pop('endTime'),
+                                    data_type=dp.pop('dataTypeName'),
-                                modified_time=dp.pop('modifiedTime'),
+                                    start_time=dp_time,
-                                values=dp.pop('values'),
+                                    end_time=dp.pop('endTime'),
-                                **{camel_case_to_snake_case(k): v
+                                    modified_time=dp.pop('modifiedTime'),
-                                    for k, v in dp.items()}
+                                    values=dp.pop('values'),
-                            ))
+                                    **{
+                                        camel_case_to_snake_case(k): v
+                                        for k, v in dp.items()
+                                    }
+                                )
+                            )
 
                             new_data_points += 1
 
                         new_last_timestamp = max(dp_time, new_last_timestamp)
 
                     last_timestamp = new_last_timestamp
-                    self.logger.info('Got {} new entries from data source {}, last timestamp: {}'.
+                    self.logger.info(
-                                     format(new_data_points, data_source,
+                        'Got {} new entries from data source {}, last timestamp: {}'.format(
-                                            str(datetime.datetime.fromtimestamp(last_timestamp))))
+                            new_data_points,
+                            data_source,
+                            str(datetime.datetime.fromtimestamp(last_timestamp)),
+                        )
+                    )
 
                     get_plugin('variable').set(**{varname: last_timestamp})
             except Exception as e:

platypush/backend/google/pubsub/__init__.py

@@ -10,18 +10,8 @@ from platypush.message.event.google.pubsub import GooglePubsubMessageEvent
 class GooglePubsubBackend(Backend):
     """
     Subscribe to a list of topics on a Google Pub/Sub instance. See
-        :class:`platypush.plugins.google.pubsub.GooglePubsubPlugin` for a reference on how to generate your
+    :class:`platypush.plugins.google.pubsub.GooglePubsubPlugin` for a reference on how to generate your
-        project and credentials file.
+    project and credentials file.
-
-    Triggers:
-
-        * :class:`platypush.message.event.google.pubsub.GooglePubsubMessageEvent` when a new message is received on
-          a subscribed topic.
-
-    Requires:
-
-        * **google-cloud-pubsub** (``pip install google-cloud-pubsub``)
-
     """
 
     def __init__(

platypush/backend/gps/__init__.py

@@ -9,17 +9,6 @@ class GpsBackend(Backend):
     """
     This backend can interact with a GPS device and listen for events.
 
-    Triggers:
-
-        * :class:`platypush.message.event.gps.GPSVersionEvent` when a GPS device advertises its version data
-        * :class:`platypush.message.event.gps.GPSDeviceEvent` when a GPS device is connected or updated
-        * :class:`platypush.message.event.gps.GPSUpdateEvent` when a GPS device has new data
-
-    Requires:
-
-        * **gps** (``pip install gps``)
-        * **gpsd** daemon running (``apt-get install gpsd`` or ``pacman -S gpsd`` depending on your distro)
-
     Once installed gpsd you need to run it and associate it to your device. Example if your GPS device communicates
     over USB and is available on /dev/ttyUSB0::
 
@@ -52,41 +41,68 @@ class GpsBackend(Backend):
 
         with self._session_lock:
             if not self._session:
-                self._session = gps.gps(host=self.gpsd_server, port=self.gpsd_port, reconnect=True)
+                self._session = gps.gps(
+                    host=self.gpsd_server, port=self.gpsd_port, reconnect=True
+                )
                 self._session.stream(gps.WATCH_ENABLE | gps.WATCH_NEWSTYLE)
 
         return self._session
 
     def _gps_report_to_event(self, report):
         if report.get('class').lower() == 'version':
-            return GPSVersionEvent(release=report.get('release'),
+            return GPSVersionEvent(
-                                   rev=report.get('rev'),
+                release=report.get('release'),
-                                   proto_major=report.get('proto_major'),
+                rev=report.get('rev'),
-                                   proto_minor=report.get('proto_minor'))
+                proto_major=report.get('proto_major'),
+                proto_minor=report.get('proto_minor'),
+            )
         if report.get('class').lower() == 'devices':
             for device in report.get('devices', []):
-                if device.get('path') not in self._devices or device != self._devices.get('path'):
+                if device.get(
+                    'path'
+                ) not in self._devices or device != self._devices.get('path'):
                     # noinspection DuplicatedCode
                     self._devices[device.get('path')] = device
-                    return GPSDeviceEvent(path=device.get('path'), activated=device.get('activated'),
+                    return GPSDeviceEvent(
-                                          native=device.get('native'), bps=device.get('bps'),
+                        path=device.get('path'),
-                                          parity=device.get('parity'), stopbits=device.get('stopbits'),
+                        activated=device.get('activated'),
-                                          cycle=device.get('cycle'), driver=device.get('driver'))
+                        native=device.get('native'),
+                        bps=device.get('bps'),
+                        parity=device.get('parity'),
+                        stopbits=device.get('stopbits'),
+                        cycle=device.get('cycle'),
+                        driver=device.get('driver'),
+                    )
         if report.get('class').lower() == 'device':
             # noinspection DuplicatedCode
             self._devices[report.get('path')] = report
-            return GPSDeviceEvent(path=report.get('path'), activated=report.get('activated'),
+            return GPSDeviceEvent(
-                                  native=report.get('native'), bps=report.get('bps'),
+                path=report.get('path'),
-                                  parity=report.get('parity'), stopbits=report.get('stopbits'),
+                activated=report.get('activated'),
-                                  cycle=report.get('cycle'), driver=report.get('driver'))
+                native=report.get('native'),
+                bps=report.get('bps'),
+                parity=report.get('parity'),
+                stopbits=report.get('stopbits'),
+                cycle=report.get('cycle'),
+                driver=report.get('driver'),
+            )
         if report.get('class').lower() == 'tpv':
-            return GPSUpdateEvent(device=report.get('device'), latitude=report.get('lat'), longitude=report.get('lon'),
+            return GPSUpdateEvent(
-                                  altitude=report.get('alt'), mode=report.get('mode'), epv=report.get('epv'),
+                device=report.get('device'),
-                                  eph=report.get('eph'), sep=report.get('sep'))
+                latitude=report.get('lat'),
+                longitude=report.get('lon'),
+                altitude=report.get('alt'),
+                mode=report.get('mode'),
+                epv=report.get('epv'),
+                eph=report.get('eph'),
+                sep=report.get('sep'),
+            )
 
     def run(self):
         super().run()
-        self.logger.info('Initialized GPS backend on {}:{}'.format(self.gpsd_server, self.gpsd_port))
+        self.logger.info(
+            'Initialized GPS backend on {}:{}'.format(self.gpsd_server, self.gpsd_port)
+        )
         last_event = None
 
         while not self.should_stop():
@@ -94,15 +110,31 @@ class GpsBackend(Backend):
                 session = self._get_session()
                 report = session.next()
                 event = self._gps_report_to_event(report)
-                if event and (last_event is None or
+                if event and (
-                              abs((last_event.args.get('latitude') or 0) - (event.args.get('latitude') or 0)) >= self._lat_lng_tolerance or
+                    last_event is None
-                              abs((last_event.args.get('longitude') or 0) - (event.args.get('longitude') or 0)) >= self._lat_lng_tolerance or
+                    or abs(
-                              abs((last_event.args.get('altitude') or 0) - (event.args.get('altitude') or 0)) >= self._alt_tolerance):
+                        (last_event.args.get('latitude') or 0)
+                        - (event.args.get('latitude') or 0)
+                    )
+                    >= self._lat_lng_tolerance
+                    or abs(
+                        (last_event.args.get('longitude') or 0)
+                        - (event.args.get('longitude') or 0)
+                    )
+                    >= self._lat_lng_tolerance
+                    or abs(
+                        (last_event.args.get('altitude') or 0)
+                        - (event.args.get('altitude') or 0)
+                    )
+                    >= self._alt_tolerance
+                ):
                     self.bus.post(event)
                     last_event = event
             except Exception as e:
                 if isinstance(e, StopIteration):
-                    self.logger.warning('GPS service connection lost, check that gpsd is running')
+                    self.logger.warning(
+                        'GPS service connection lost, check that gpsd is running'
+                    )
                 else:
                     self.logger.exception(e)
 

platypush/backend/gps/manifest.yaml

@@ -6,11 +6,15 @@ manifest:
     platypush.message.event.gps.GPSVersionEvent: when a GPS device advertises its
       version data
   install:
-    pip:
+    apk:
-      - gps
-    pacman:
       - gpsd
     apt:
       - gpsd
+    dnf:
+      - gpsd
+    pacman:
+      - gpsd
+    pip:
+      - gps
   package: platypush.backend.gps
   type: backend

platypush/backend/http/__init__.py

@@ -2,13 +2,17 @@ import asyncio
 import os
 import pathlib
 import secrets
+import signal
 import threading
 
+from functools import partial
 from multiprocessing import Process
 from time import time
-from typing import List, Mapping, Optional
+from typing import Mapping, Optional
-from tornado.httpserver import HTTPServer
 
+import psutil
+
+from tornado.httpserver import HTTPServer
 from tornado.netutil import bind_sockets
 from tornado.process import cpu_count, fork_processes
 from tornado.wsgi import WSGIContainer
@@ -18,9 +22,9 @@ from platypush.backend import Backend
 from platypush.backend.http.app import application
 from platypush.backend.http.app.utils import get_streaming_routes, get_ws_routes
 from platypush.backend.http.app.ws.events import WSEventProxy
-
 from platypush.bus.redis import RedisBus
 from platypush.config import Config
+from platypush.utils import get_remaining_timeout
 
 
 class HttpBackend(Backend):
@@ -188,12 +192,15 @@ class HttpBackend(Backend):
 
     """
 
-    _DEFAULT_HTTP_PORT = 8008
+    DEFAULT_HTTP_PORT = 8008
     """The default listen port for the webserver."""
 
+    _STOP_TIMEOUT = 5
+    """How long we should wait (in seconds) before killing the worker processes."""
+
     def __init__(
         self,
-        port: int = _DEFAULT_HTTP_PORT,
+        port: int = DEFAULT_HTTP_PORT,
         bind_address: str = '0.0.0.0',
         resource_dirs: Optional[Mapping[str, str]] = None,
         secret_key_file: Optional[str] = None,
@@ -227,7 +234,6 @@ class HttpBackend(Backend):
 
         self.port = port
         self._server_proc: Optional[Process] = None
-        self._workers: List[Process] = []
         self._service_registry_thread = None
         self.bind_address = bind_address
 
@@ -254,35 +260,37 @@ class HttpBackend(Backend):
         """On backend stop"""
         super().on_stop()
         self.logger.info('Received STOP event on HttpBackend')
-
+        start = time()
-        start_time = time()
+        remaining_time: partial[float] = partial(  # type: ignore
-        timeout = 5
+            get_remaining_timeout, timeout=self._STOP_TIMEOUT, start=start
-        workers = self._workers.copy()
+        )
-
-        for i, worker in enumerate(workers[::-1]):
-            if worker and worker.is_alive():
-                worker.terminate()
-                worker.join(timeout=max(0, start_time + timeout - time()))
-
-            if worker and worker.is_alive():
-                worker.kill()
-                self._workers.pop(i)
 
         if self._server_proc:
-            self._server_proc.terminate()
+            if self._server_proc.pid:
-            self._server_proc.join(timeout=5)
+                try:
-            self._server_proc = None
+                    os.kill(self._server_proc.pid, signal.SIGINT)
+                except OSError:
+                    pass
+
+            if self._server_proc and self._server_proc.is_alive():
+                self._server_proc.join(timeout=remaining_time() / 2)
+                try:
+                    self._server_proc.terminate()
+                    self._server_proc.join(timeout=remaining_time() / 2)
+                except AttributeError:
+                    pass
 
         if self._server_proc and self._server_proc.is_alive():
             self._server_proc.kill()
 
         self._server_proc = None
-        self.logger.info('HTTP server terminated')
 
         if self._service_registry_thread and self._service_registry_thread.is_alive():
-            self._service_registry_thread.join(timeout=5)
+            self._service_registry_thread.join(timeout=remaining_time())
             self._service_registry_thread = None
 
+        self.logger.info('HTTP server terminated')
+
     def notify_web_clients(self, event):
         """Notify all the connected web clients (over websocket) of a new event"""
         WSEventProxy.publish(event)  # noqa: E1120
@@ -344,7 +352,10 @@ class HttpBackend(Backend):
         try:
             await asyncio.Event().wait()
         except (asyncio.CancelledError, KeyboardInterrupt):
-            return
+            pass
+        finally:
+            server.stop()
+            await server.close_all_connections()
 
     def _web_server_proc(self):
         self.logger.info(
@@ -371,7 +382,65 @@ class HttpBackend(Backend):
                 future = self._post_fork_main(sockets)
                 asyncio.run(future)
             except (asyncio.CancelledError, KeyboardInterrupt):
-                return
+                pass
+            finally:
+                self._stop_workers()
+
+    def _stop_workers(self):
+        """
+        Stop all the worker processes.
+
+        We have to run this manually on server termination because of a
+        long-standing issue with Tornado not being able to wind down the forked
+        workers when the server terminates:
+        https://github.com/tornadoweb/tornado/issues/1912.
+        """
+        parent_pid = (
+            self._server_proc.pid
+            if self._server_proc and self._server_proc.pid
+            else None
+        )
+
+        if not parent_pid:
+            return
+
+        try:
+            cur_proc = psutil.Process(parent_pid)
+        except psutil.NoSuchProcess:
+            return
+
+        # Send a SIGTERM to all the children
+        children = cur_proc.children()
+        for child in children:
+            if child.pid != parent_pid and child.is_running():
+                try:
+                    os.kill(child.pid, signal.SIGTERM)
+                except OSError as e:
+                    self.logger.warning(
+                        'Could not send SIGTERM to PID %d: %s', child.pid, e
+                    )
+
+        # Initialize the timeout
+        start = time()
+        remaining_time: partial[int] = partial(  # type: ignore
+            get_remaining_timeout, timeout=self._STOP_TIMEOUT, start=start, cls=int
+        )
+
+        # Wait for all children to terminate (with timeout)
+        for child in children:
+            if child.pid != parent_pid and child.is_running():
+                try:
+                    child.wait(timeout=remaining_time())
+                except TimeoutError:
+                    pass
+
+        # Send a SIGKILL to any child process that is still running
+        for child in children:
+            if child.pid != parent_pid and child.is_running():
+                try:
+                    child.kill()
+                except OSError:
+                    pass
 
     def _start_web_server(self):
         self._server_proc = Process(target=self._web_server_proc)

platypush/backend/http/app/mixins/__init__.py

@@ -117,6 +117,7 @@ class PubSubMixin:
         """
         try:
             with self.pubsub as pubsub:
+                pubsub.subscribe(*self._subscriptions)
                 for msg in pubsub.listen():
                     channel = msg.get('channel', b'').decode()
                     if msg.get('type') != 'message' or not (

platypush/backend/http/app/routes/plugins/tts/mimic3.py

@@ -1,46 +0,0 @@
-import requests
-from urllib.parse import urljoin
-
-from flask import abort, request, Blueprint
-
-from platypush.backend.http.app import template_folder
-
-mimic3 = Blueprint('mimic3', __name__, template_folder=template_folder)
-
-# Declare routes list
-__routes__ = [
-    mimic3,
-]
-
-
-@mimic3.route('/tts/mimic3/say', methods=['GET'])
-def proxy_tts_request():
-    """
-    This route is used to proxy the POST request to the Mimic3 TTS server
-    through a GET, so it can be easily processed as a URL through a media
-    plugin.
-    """
-    required_args = {
-        'text',
-        'server_url',
-        'voice',
-    }
-
-    missing_args = required_args.difference(set(request.args.keys()))
-    if missing_args:
-        abort(400, f'Missing parameters: {missing_args}')
-
-    args = {arg: request.args[arg] for arg in required_args}
-
-    rs = requests.post(
-        urljoin(args['server_url'], '/api/tts'),
-        data=args['text'],
-        params={
-            'voice': args['voice'],
-        },
-    )
-
-    return rs.content
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/http/app/streaming/_base.py

@@ -3,7 +3,6 @@ from http.client import responses
 import json
 from logging import getLogger
 from typing import Optional
-from typing_extensions import override
 
 from tornado.web import RequestHandler, stream_request_body
 
@@ -22,7 +21,6 @@ class StreamingRoute(RequestHandler, PubSubMixin, ABC):
         super().__init__(*args, **kwargs)
         self.logger = getLogger(__name__)
 
-    @override
     def prepare(self):
         """
         Request preparation logic. It performs user authentication if
@@ -38,7 +36,6 @@ class StreamingRoute(RequestHandler, PubSubMixin, ABC):
             'Client %s connected to %s', self.request.remote_ip, self.request.path
         )
 
-    @override
     def write_error(self, status_code: int, error: Optional[str] = None, **_):
         """
         Make sure that errors are always returned in JSON format.

platypush/backend/http/app/streaming/plugins/camera.py

@@ -1,7 +1,6 @@
 from enum import Enum
 import json
 from typing import Optional
-from typing_extensions import override
 
 from tornado.web import stream_request_body
 from platypush.context import get_plugin
@@ -37,7 +36,6 @@ class CameraRoute(StreamingRoute):
         self._request_type = RequestType.UNKNOWN
         self._extension: str = ''
 
-    @override
     @classmethod
     def path(cls) -> str:
         return r"/camera/([a-zA-Z0-9_./]+)/([a-zA-Z0-9_]+)\.?([a-zA-Z0-9_]+)?"
@@ -95,7 +93,6 @@ class CameraRoute(StreamingRoute):
 
         return kwargs
 
-    @override
     @classmethod
     def _get_redis_queue(cls, camera: CameraPlugin, *_, **__) -> str:
         plugin_name = get_plugin_name_by_class(camera.__class__)

platypush/backend/http/app/streaming/plugins/sound.py

@@ -1,7 +1,6 @@
 from contextlib import contextmanager
 import json
 from typing import Generator, Optional
-from typing_extensions import override
 
 from tornado.web import stream_request_body
 
@@ -24,7 +23,6 @@ class SoundRoute(StreamingRoute):
         self._audio_headers_written: bool = False
         """Send the audio file headers before we send the first audio frame."""
 
-    @override
     @classmethod
     def path(cls) -> str:
         return r"/sound/stream\.?([a-zA-Z0-9_]+)?"
@@ -44,7 +42,6 @@ class SoundRoute(StreamingRoute):
         yield
         send_request('sound.stop_recording')
 
-    @override
     @classmethod
     def _get_redis_queue(cls, *_, device: Optional[str] = None, **__) -> str:
         return '/'.join([cls._redis_queue_prefix, *([device] if device else [])])

platypush/backend/http/app/utils/__init__.py

@@ -4,7 +4,7 @@ from .auth import (
     authenticate_user_pass,
     get_auth_status,
 )
-from .bus import bus, get_message_response, send_message, send_request
+from .bus import bus, send_message, send_request
 from .logger import logger
 from .routes import (
     get_http_port,
@@ -25,7 +25,6 @@ __all__ = [
     'get_http_port',
     'get_ip_or_hostname',
     'get_local_base_url',
-    'get_message_response',
     'get_remote_base_url',
     'get_routes',
     'get_streaming_routes',

platypush/backend/http/app/utils/bus.py

@@ -1,11 +1,9 @@
-from redis import Redis
-
 from platypush.bus.redis import RedisBus
 from platypush.config import Config
 from platypush.context import get_backend
 from platypush.message import Message
 from platypush.message.request import Request
-from platypush.utils import get_redis_conf, get_redis_queue_name_by_message
+from platypush.utils import get_redis_conf, get_message_response
 
 from .logger import logger
 
@@ -67,24 +65,3 @@ def send_request(action, wait_for_response=True, **kwargs):
         msg['args'] = kwargs
 
     return send_message(msg, wait_for_response=wait_for_response)
-
-
-def get_message_response(msg):
-    """
-    Get the response to the given message.
-
-    :param msg: The message to get the response for.
-    :return: The response to the given message.
-    """
-    redis = Redis(**bus().redis_args)
-    redis_queue = get_redis_queue_name_by_message(msg)
-    if not redis_queue:
-        return None
-
-    response = redis.blpop(redis_queue, timeout=60)
-    if response and len(response) > 1:
-        response = Message.build(response[1])
-    else:
-        response = None
-
-    return response

platypush/backend/http/app/utils/routes.py

@@ -14,7 +14,7 @@ def get_http_port():
     from platypush.backend.http import HttpBackend
 
     http_conf = Config.get('backend.http') or {}
-    return http_conf.get('port', HttpBackend._DEFAULT_HTTP_PORT)
+    return http_conf.get('port', HttpBackend.DEFAULT_HTTP_PORT)
 
 
 def get_routes():

platypush/backend/http/app/ws/_base.py

@@ -1,7 +1,6 @@
 from abc import ABC, abstractmethod
 from logging import getLogger
 from threading import Thread
-from typing_extensions import override
 
 from tornado.ioloop import IOLoop
 from tornado.websocket import WebSocketHandler
@@ -24,7 +23,6 @@ class WSRoute(WebSocketHandler, Thread, PubSubMixin, ABC):
         Thread.__init__(self)
         self._io_loop = IOLoop.current()
 
-    @override
     def open(self, *_, **__):
         auth_status = get_auth_status(self.request)
         if auth_status != AuthStatus.OK:
@@ -37,11 +35,9 @@ class WSRoute(WebSocketHandler, Thread, PubSubMixin, ABC):
         self.name = f'ws:{self.app_name()}@{self.request.remote_ip}'
         self.start()
 
-    @override
     def data_received(self, *_, **__):
         pass
 
-    @override
     def on_message(self, message):
         return message
 
@@ -63,12 +59,10 @@ class WSRoute(WebSocketHandler, Thread, PubSubMixin, ABC):
             self.write_message, self._serialize(msg)
         )
 
-    @override
     def run(self) -> None:
         super().run()
         self.subscribe(*self._subscriptions)
 
-    @override
     def on_close(self):
         super().on_close()
         for channel in self._subscriptions.copy():

platypush/backend/http/app/ws/cmd.py

@@ -0,0 +1,66 @@
+from base64 import b64decode
+import json
+from typing import Optional
+
+from platypush.common.cmd_stream import redis_topic
+
+from . import WSRoute, logger
+
+
+class WSCommandOutput(WSRoute):
+    """
+    Websocket route that pushes the output of an executed command to the client
+    as it is generated. Mapped to ``/ws/shell``.
+    """
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, subscriptions=[redis_topic], **kwargs)
+        self._id = None
+
+    @classmethod
+    def app_name(cls) -> str:
+        return 'shell'
+
+    @classmethod
+    def path(cls) -> str:
+        return f'/ws/{cls.app_name()}'
+
+    def _parse_msg(self, msg: bytes) -> Optional[bytes]:
+        parsed_msg = json.loads(msg)
+        cmd_id = parsed_msg.get('id')
+        output = parsed_msg.get('output')
+        if output is None:  # End-of-stream
+            raise StopIteration()
+
+        if cmd_id != self._id:
+            return None
+
+        return b64decode(output)
+
+    def open(self, *args, **kwargs):
+        self._id = next(iter(self.request.arguments['id']), b'').decode() or None
+        super().open(*args, **kwargs)
+
+    def run(self) -> None:
+        super().run()
+        for msg in self.listen():
+            try:
+                output = self._parse_msg(msg.data)
+                if output is None:
+                    continue
+
+                self.send(output)
+            except StopIteration:
+                break
+            except Exception as e:
+                logger.warning('Failed to parse message: %s', e)
+                logger.exception(e)
+                continue
+
+        self._io_loop.add_callback(self._ws_close)
+
+    def _ws_close(self):
+        if not self.ws_connection:
+            return
+
+        self.ws_connection.close(1000, 'Command terminated')

platypush/backend/http/app/ws/events.py

@@ -1,5 +1,3 @@
-from typing_extensions import override
-
 from platypush.backend.http.app.mixins import MessageType
 from platypush.message.event import Event
 
@@ -16,7 +14,6 @@ class WSEventProxy(WSRoute):
         super().__init__(*args, subscriptions=[self.events_channel], **kwargs)
 
     @classmethod
-    @override
     def app_name(cls) -> str:
         return 'events'
 
@@ -25,12 +22,10 @@ class WSEventProxy(WSRoute):
     def events_channel(cls) -> str:
         return cls.get_channel('events')
 
-    @override
     @classmethod
     def publish(cls, data: MessageType, *_) -> None:
         super().publish(data, cls.events_channel)
 
-    @override
     def on_message(self, message):
         try:
             event = Event.build(message)
@@ -42,7 +37,6 @@ class WSEventProxy(WSRoute):
 
         send_message(event, wait_for_response=False)
 
-    @override
     def run(self) -> None:
         for msg in self.listen():
             try:

platypush/backend/http/app/ws/requests.py

@@ -1,6 +1,5 @@
 from threading import Thread, current_thread
 from typing import Set
-from typing_extensions import override
 
 from platypush.backend.http.app.utils import send_message
 from platypush.message.request import Request
@@ -21,7 +20,6 @@ class WSRequestsProxy(WSRoute):
         self._requests: Set[Thread] = set()
 
     @classmethod
-    @override
     def app_name(cls) -> str:
         return 'requests'
 

platypush/backend/http/manifest.yaml

@@ -1,7 +1,6 @@
 manifest:
   events: {}
   install:
-    pip:
+    pip: []
-    - gunicorn
   package: platypush.backend.http
   type: backend

platypush/backend/http/poll/__init__.py

@@ -1,88 +0,0 @@
-import importlib
-import time
-
-from platypush.backend import Backend
-from platypush.backend.http.request import HttpRequest
-
-
-class HttpPollBackend(Backend):
-    """
-    WARNING: This integration is deprecated, since it was practically only used for RSS subscriptions.
-    RSS feeds integration has been replaced by :class:`platypush.plugins.rss.RSSPlugin`.
-
-    This backend will poll multiple HTTP endpoints/services and return events
-    the bus whenever something new happened. Supported types:
-    :class:`platypush.backend.http.request.JsonHttpRequest` (for polling updates on
-    a JSON endpoint), :class:`platypush.backend.http.request.rss.RssUpdates`
-    (for polling updates on an RSS feed). Example configuration::
-
-        backend.http.poll:
-            requests:
-                -
-                    # Poll for updates on a JSON endpoint
-                    method: GET
-                    type: platypush.backend.http.request.JsonHttpRequest
-                    args:
-                        url: https://host.com/api/v1/endpoint
-                        headers:
-                            Token: TOKEN
-                        params:
-                            updatedSince: 1m
-                        timeout: 5  # Times out after 5 seconds (default)
-                    poll_seconds: 60  # Check for updates on this endpoint every 60 seconds (default)
-                    path: ${response['items']}  # Path in the JSON to check for new items.
-                                                # Python expressions are supported.
-                                                # Note that 'response' identifies the JSON root.
-                                                # Default value: JSON root.
-                -
-                    # Poll for updates on an RSS feed
-                    type: platypush.backend.http.request.rss.RssUpdates
-                    url: https://www.theguardian.com/rss/world
-                    title: The Guardian - World News
-                    poll_seconds: 120
-                    max_entries: 10
-
-    Triggers: an update event for the relevant HTTP source if it contains new items. For example:
-
-        * :class:`platypush.message.event.http.rss.NewFeedEvent` if a feed contains new items
-        * :class:`platypush.message.event.http.HttpEvent` if a JSON endpoint contains new items
-    """
-
-    def __init__(self, requests, *args, **kwargs):
-        """
-        :param requests: Configuration of the requests to make (see class description for examples)
-        :type requests: dict
-        """
-
-        super().__init__(*args, **kwargs)
-        self.requests = []
-
-        for request in requests:
-            if isinstance(request, dict):
-                req_type = request['type']
-                (module, name) = ('.'.join(req_type.split('.')[:-1]), req_type.split('.')[-1])
-                module = importlib.import_module(module)
-                request = getattr(module, name)(**request)
-            elif not isinstance(request, HttpRequest):
-                raise RuntimeError('Request should either be a dict or a ' +
-                                   'HttpRequest object, {} found'.format(type(request)))
-
-            request.bus = self.bus
-            self.requests.append(request)
-
-    def run(self):
-        super().run()
-
-        while not self.should_stop():
-            for request in self.requests:
-                if time.time() - request.last_request_timestamp > request.poll_seconds:
-                    try:
-                        request.execute()
-                    except Exception as e:
-                        self.logger.error('Error while executing request: {}'.format(request))
-                        self.logger.exception(e)
-
-            time.sleep(0.1)  # Prevent a tight loop
-
-
-# vim:sw=4:ts=4:et:

platypush/backend/http/poll/manifest.yaml

@@ -1,6 +0,0 @@
-manifest:
-  events: {}
-  install:
-    pip: []
-  package: platypush.backend.http.poll
-  type: backend

platypush/backend/http/request/rss/__init__.py

@@ -40,15 +40,6 @@ class RssUpdates(HttpRequest):
                   poll_seconds: 86400  # Poll once a day
                   digest_format: html  # Generate an HTML feed with the new items
 
-    Triggers:
-
-        - :class:`platypush.message.event.http.rss.NewFeedEvent` when new items are parsed from a feed or a new digest
-          is available.
-
-    Requires:
-
-        * **feedparser** (``pip install feedparser``)
-
     """
 
     user_agent = (

platypush/backend/http/webapp/dist/index.html

@@ -1 +1 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1"><!--[if IE]><link rel="icon" href="/favicon.ico"><![endif]--><link rel="stylesheet" href="/fonts/poppins.css"><title>platypush</title><script defer="defer" src="/static/js/chunk-vendors.22fb39a8.js"></script><script defer="defer" src="/static/js/app.ebd1a697.js"></script><link href="/static/css/chunk-vendors.d510eff2.css" rel="stylesheet"><link href="/static/css/app.0a781c41.css" rel="stylesheet"><link rel="icon" type="image/svg+xml" href="/img/icons/favicon.svg"><link rel="icon" type="image/png" sizes="32x32" href="/img/icons/favicon-32x32.png"><link rel="icon" type="image/png" sizes="16x16" href="/img/icons/favicon-16x16.png"><link rel="manifest" href="/manifest.json"><meta name="theme-color" content="#ffffff"><meta name="apple-mobile-web-app-capable" content="no"><meta name="apple-mobile-web-app-status-bar-style" content="default"><meta name="apple-mobile-web-app-title" content="Platypush"><link rel="apple-touch-icon" href="/img/icons/apple-touch-icon-152x152.png"><link rel="mask-icon" href="/img/icons/safari-pinned-tab.svg" color="#ffffff"><meta name="msapplication-TileImage" content="/img/icons/msapplication-icon-144x144.png"><meta name="msapplication-TileColor" content="#000000"></head><body><noscript><strong>We're sorry but platypush doesn't work properly without JavaScript enabled. Please enable it to continue.</strong></noscript><div id="app"></div></body></html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1"><!--[if IE]><link rel="icon" href="/favicon.ico"><![endif]--><link rel="stylesheet" href="/fonts/poppins.css"><title>platypush</title><script defer="defer" src="/static/js/chunk-vendors.650eb8e8.js"></script><script defer="defer" src="/static/js/app.6a696f12.js"></script><link href="/static/css/chunk-vendors.a2412607.css" rel="stylesheet"><link href="/static/css/app.eab47dab.css" rel="stylesheet"><link rel="icon" type="image/svg+xml" href="/img/icons/favicon.svg"><link rel="icon" type="image/png" sizes="32x32" href="/img/icons/favicon-32x32.png"><link rel="icon" type="image/png" sizes="16x16" href="/img/icons/favicon-16x16.png"><link rel="manifest" href="/manifest.json"><meta name="theme-color" content="#ffffff"><meta name="apple-mobile-web-app-capable" content="no"><meta name="apple-mobile-web-app-status-bar-style" content="default"><meta name="apple-mobile-web-app-title" content="Platypush"><link rel="apple-touch-icon" href="/img/icons/apple-touch-icon-152x152.png"><link rel="mask-icon" href="/img/icons/safari-pinned-tab.svg" color="#ffffff"><meta name="msapplication-TileImage" content="/img/icons/msapplication-icon-144x144.png"><meta name="msapplication-TileColor" content="#000000"></head><body><noscript><strong>We're sorry but platypush doesn't work properly without JavaScript enabled. Please enable it to continue.</strong></noscript><div id="app"></div></body></html>