Better documentation for the Redis server + LINT fixes.

1. Added documentation to the README on the possible options to run the Redis service. 2. Show a relevant message to the user if the application is run with `--start-redis` and Redis couldn't start. 3. Some LINT/black chores on some files that hadn't been touched in a while.
2023-08-02 22:17:11 +02:00 · 2023-08-02 22:17:11 +02:00 · 53aeb0b3b1
commit 53aeb0b3b1
parent 99018598a5
5 changed files with 172 additions and 65 deletions
--- a/README.md
+++ b/README.md
@ -15,16 +15,19 @@ Platypush
 - [Introduction](#introduction)
    + [What it can do](#what-it-can-do)
 - [Installation](#installation)
-  * [System installation](#system-installation)
-    + [Install through `pip`](#install-through-pip)
-    + [Install through a system package manager](#install-through-a-system-package-manager)
-    + [Install from sources](#install-from-sources)
+  * [Prerequisites](#prerequisites)
+    + [Docker installation](#docker-installation)
+    + [Use an external service](#use-an-external-service)
+    + [Manual installation](#manual-installation)
+  * [Install through `pip`](#install-through-pip)
+  * [Install through a system package manager](#install-through-a-system-package-manager)
+  * [Install from sources](#install-from-sources)
  * [Installing the dependencies for your extensions](#installing-the-dependencies-for-your-extensions)
    + [Install via `extras` name](#install-via-extras-name)
    + [Install via `manifest.yaml`](#install-via-manifestyaml)
    + [Check the instructions reported in the documentation](#check-the-instructions-reported-in-the-documentation)
  * [Virtual environment installation](#virtual-environment-installation)
-  * [Docker installation](#docker-installation)
+  * [Docker installation](#docker-installation-1)
 - [Architecture](#architecture)
  * [Plugins](#plugins)
  * [Actions](#actions)
@ -127,26 +130,82 @@ You can use Platypush to do things like:

 ## Installation

-### System installation
+### Prerequisites

-Platypush uses Redis to deliver and store requests and temporary messages:
+Platypush uses [Redis](https://redis.io/) to dispatch requests, responses,
+events and custom messages across several processes and integrations.
+
+#### Docker installation
+
+You can run Redis on the fly on your local machine using a Docker image:
+
+```bash
+# Expose a Redis server on port 6379 (default)
+docker run --rm -p 6379:6379 --name redis redis
+```
+
+#### Use an external service
+
+You can let Platypush use an external Redis service, if you wish to avoid
+running one on the same machine.
+
+In such scenario, simply start the application by passing custom values for
+`--redis-host` and `--redis-port`, or configure these values in its
+configuration file:

 ```yaml
-# Example for Debian-based distributions
-[sudo] apt-get install redis-server
+redis:
+  host: some-ip
+  port: some-port
+```

+If you wish to run multiple instances that use the same Redis server, you may
+also want to customize the name of the default queue that they use
+(`--redis-queue` command-line option) in order to avoid conflicts.
+
+#### Manual installation
+
+Unless you are running Platypush in a Docker container, or you are running
+Redis in a Docker container, or you want to use a remote Redis service, the
+Redis server should be installed on the same machine where Platypush runs:
+
+```bash
+# On Debian-based distributions
+sudo apt install redis-server
+
+# On Arch-based distributions
+# The hiredis package is also advised
+sudo pacman -S redis
+
+# On MacOS
+brew install redis
+```
+
+Once Redis is installed, you have two options:
+
+1. Run it a separate service. This depends on your operating system and
+   supervisor/service controller. For example, on systemd:
+
+```bash
 # Enable and start the service
-[sudo] systemctl enable redis
-[sudo] systemctl start redis
+sudo systemctl enable redis
+sudo systemctl start redis
 ```

-#### Install through `pip`
+2. Let Platypush run and control the Redis service. This is a good option if
+   you want Platypush to run its own service, separate from any other one
+   running on the same machine, and terminate it as soon as the application
+   ends. In this case, simply launch the application with the `--start-redis`
+   option (and optionally `--redis-port <any-num>` to customize the listen
+   port).

-```shell
-[sudo] pip3 install platypush
+### Install through `pip`
+
+```bash
+[sudo] pip install platypush
 ```

-#### Install through a system package manager
+### Install through a system package manager

 Note: currently only Arch Linux and derived distributions are supported.

@ -157,7 +216,7 @@ latest stable version) or the
 (for the latest git version) through your favourite AUR package manager. For
 example, using `yay`:

-```shell
+```bash
 yay platypush
 # Or
 yay platypush-git
@ -166,14 +225,12 @@ yay platypush-git
 The Arch Linux packages on AUR are automatically updated upon new git commits
 or tags.

-#### Install from sources
+### Install from sources

 ```shell
 git clone https://git.platypush.tech/platypush/platypush.git
 cd platypush
 [sudo] pip install .
-# Or
-[sudo] python3 setup.py install
 ```

 ### Installing the dependencies for your extensions
@ -227,6 +284,8 @@ You can then start the service by simply running:
 platypush
 ```

+See `platypush --help` for a full list of options.
+
 It's advised to run it as a systemd service though - simply copy the provided
 [`.service`
 file](https://git.platypush.tech/platypush/platypush/src/branch/master/examples/systemd/platypush.service)
--- a/platypush/app.py
+++ b/platypush/app.py
@ -153,16 +153,27 @@ class Application:

        port = self._redis_conf['port']
        log.info('Starting local Redis instance on %s', port)
-        self._redis_proc = subprocess.Popen(  # pylint: disable=consider-using-with
-            [
+        redis_cmd_args = [
            'redis-server',
            '--bind',
            'localhost',
            '--port',
            str(port),
-            ],
+        ]
+
+        try:
+            self._redis_proc = subprocess.Popen(  # pylint: disable=consider-using-with
+                redis_cmd_args,
                stdout=subprocess.PIPE,
            )
+        except Exception as e:
+            log.error(
+                'Failed to start local Redis instance: "%s": %s',
+                ' '.join(redis_cmd_args),
+                e,
+            )
+
+            sys.exit(1)

        log.info('Waiting for Redis to start')
        for line in self._redis_proc.stdout:  # type: ignore
--- a/platypush/bus/init.py
+++ b/platypush/bus/init.py
@ -10,8 +10,10 @@ from platypush.message.event import Event
 logger = logging.getLogger('platypush:bus')


-class Bus(object):
-    """ Main local bus where the daemon will listen for new messages """
+class Bus:
+    """
+    Main local bus where the daemon will listen for new messages.
+    """

    _MSG_EXPIRY_TIMEOUT = 60.0  # Consider a message on the bus as expired after one minute without being picked up

@ -31,31 +33,37 @@ class Bus(object):
        try:
            return self.bus.get(timeout=0.1)
        except Empty:
-            return
+            return None

    def stop(self):
        self._should_stop.set()

    def _msg_executor(self, msg):
        def event_handler(event: Event, handler: Callable[[Event], None]):
-            logger.info('Triggering event handler {}'.format(handler.__name__))
+            logger.info('Triggering event handler %s', handler.__name__)
            handler(event)

        def executor():
            if isinstance(msg, Event):
-                if type(msg) in self.event_handlers:
-                    handlers = self.event_handlers[type(msg)]
-                else:
-                    handlers = {*[hndl for event_type, hndl in self.event_handlers.items()
-                                  if isinstance(msg, event_type)]}
+                handlers = self.event_handlers.get(
+                    type(msg),
+                    {
+                        *[
+                            hndl
+                            for event_type, hndl in self.event_handlers.items()
+                            if isinstance(msg, event_type)
+                        ]
+                    },
+                )

                for hndl in handlers:
                    threading.Thread(target=event_handler, args=(msg, hndl))

            try:
+                if self.on_message:
                    self.on_message(msg)
            except Exception as e:
-                logger.error('Error on processing message {}'.format(msg))
+                logger.error('Error on processing message %s', msg)
                logger.exception(e)

        return executor
@ -76,17 +84,24 @@ class Bus(object):
            if msg is None:
                continue

-            timestamp = msg.timestamp if hasattr(msg, 'timestamp') else msg.get('timestamp')
+            timestamp = (
+                msg.timestamp if hasattr(msg, 'timestamp') else msg.get('timestamp')
+            )
            if timestamp and time.time() - timestamp > self._MSG_EXPIRY_TIMEOUT:
-                logger.debug('{} seconds old message on the bus expired, ignoring it: {}'.
-                             format(int(time.time()-msg.timestamp), msg))
+                logger.debug(
+                    '%f seconds old message on the bus expired, ignoring it: %s',
+                    time.time() - msg.timestamp,
+                    msg,
+                )
                continue

            threading.Thread(target=self._msg_executor(msg)).start()

        logger.info('Bus service stopped')

-    def register_handler(self, event_type: Type[Event], handler: Callable[[Event], None]) -> Callable[[], None]:
+    def register_handler(
+        self, event_type: Type[Event], handler: Callable[[Event], None]
+    ) -> Callable[[], None]:
        """
        Register an event handler to the bus.

@ -104,7 +119,9 @@ class Bus(object):

        return unregister

-    def unregister_handler(self, event_type: Type[Event], handler: Callable[[Event], None]) -> None:
+    def unregister_handler(
+        self, event_type: Type[Event], handler: Callable[[Event], None]
+    ) -> None:
        """
        Remove an event handler.

--- a/platypush/common/init.py
+++ b/platypush/common/init.py
@ -1,6 +1,7 @@
 import inspect
 import logging
 import os
+from typing import Any, Callable

 from platypush.utils.manifest import Manifest

@ -9,7 +10,11 @@ from ._types import StoppableThread
 logger = logging.getLogger('platypush')


-def exec_wrapper(f, *args, **kwargs):
+def exec_wrapper(f: Callable[..., Any], *args, **kwargs):
+    """
+    Utility function that runs a callable with its arguments, wraps its
+    response into a ``Response`` object and handles errors/exceptions.
+    """
    from platypush import Response

    try:
@ -23,7 +28,13 @@ def exec_wrapper(f, *args, **kwargs):
        return Response(errors=[str(e)])


+# pylint: disable=too-few-public-methods
 class ExtensionWithManifest:
+    """
+    This class models an extension with an associated manifest.yaml in the same
+    folder.
+    """
+
    def __init__(self, *_, **__):
        self._manifest = self.get_manifest()

@ -33,9 +44,7 @@ class ExtensionWithManifest:
        )
        assert os.path.isfile(
            manifest_file
-        ), 'The extension {} has no associated manifest.yaml'.format(
-            self.__class__.__name__
-        )
+        ), f'The extension {self.__class__.__name__} has no associated manifest.yaml'

        return Manifest.from_file(manifest_file)

--- a/platypush/common/gstreamer/init.py
+++ b/platypush/common/gstreamer/init.py
@ -3,18 +3,22 @@ import threading

 from typing import Optional

-# noinspection PyPackageRequirements
 import gi
+
 gi.require_version('Gst', '1.0')
 gi.require_version('GstApp', '1.0')

-# noinspection PyPackageRequirements,PyUnresolvedReferences
+# flake8: noqa
 from gi.repository import GLib, Gst, GstApp

 Gst.init(None)


 class Pipeline:
+    """
+    A GStreamer pipeline.
+    """
+
    def __init__(self):
        self.logger = logging.getLogger('gst-pipeline')
        self.pipeline = Gst.Pipeline()
@ -66,6 +70,7 @@ class Pipeline:

    def play(self):
        self.pipeline.set_state(Gst.State.PLAYING)
+        assert self.loop, 'No GLib loop is running'
        self.loop.start()

    def pause(self):
@ -92,7 +97,7 @@ class Pipeline:
    def on_buffer(self, sink):
        sample = GstApp.AppSink.pull_sample(sink)
        buffer = sample.get_buffer()
-        size, offset, maxsize = buffer.get_sizes()
+        size, offset, _ = buffer.get_sizes()
        self.data = buffer.extract_dup(offset, size)
        self.data_ready.set()
        return False
@ -101,9 +106,8 @@ class Pipeline:
        self.logger.info('End of stream event received')
        self.stop()

-    # noinspection PyUnusedLocal
-    def on_error(self, bus, msg):
-        self.logger.warning('GStreamer pipeline error: {}'.format(msg.parse_error()))
+    def on_error(self, _, msg):
+        self.logger.warning('GStreamer pipeline error: %s', msg.parse_error())
        self.stop()

    def get_source(self):
@ -113,12 +117,11 @@ class Pipeline:
        return self.sink

    def get_state(self) -> Gst.State:
-        state = self.source.current_state
-        if not state:
+        if not (self.source and self.source.current_state):
            self.logger.warning('Unable to get pipeline state')
            return Gst.State.NULL

-        return state
+        return self.source.current_state

    def is_playing(self) -> bool:
        return self.get_state() == Gst.State.PLAYING
@ -127,6 +130,10 @@ class Pipeline:
        return self.get_state() == Gst.State.PAUSED

    def get_position(self) -> Optional[float]:
+        if not self.source:
+            self.logger.warning('Unable to get pipeline state')
+            return Gst.State.NULL
+
        pos = self.source.query_position(Gst.Format(Gst.Format.TIME))
        if not pos[0]:
            return None
@ -134,6 +141,7 @@ class Pipeline:
        return pos[1] / 1e9

    def get_duration(self) -> Optional[float]:
+        assert self.source, 'No active source found'
        pos = self.source.query_duration(Gst.Format(Gst.Format.TIME))
        if not pos[0]:
            return None
@ -157,9 +165,7 @@ class Pipeline:

    def seek(self, position: float):
        assert self.source, 'No source specified'
-        if position < 0:
-            position = 0
-
+        position = max(0, position)
        duration = self.get_duration()
        if duration and position > duration:
            position = duration
@ -169,11 +175,16 @@ class Pipeline:


 class Loop(threading.Thread):
+    """
+    Wraps the execution of a GLib main loop into its own thread.
+    """
+
    def __init__(self):
        super().__init__()
        self._loop = GLib.MainLoop()

    def run(self):
+        assert self._loop, 'No GLib loop is running'
        self._loop.run()

    def is_running(self) -> bool: