Commit graph

2966 commits

Author SHA1 Message Date
98e9abde18
Extended manifest files with Python system packages (if available).
- If a Python optional dependency is available as a system package on
  the target system, try and install it that route rather than pip. It's
  usually faster and it decreases the risk of breaking system packages.

- Added support for apk dependencies in manifest files. This brings the
  number of distros officially supported by all the extensions to four:

  - Alpine
  - Arch
  - Debian
  - Ubuntu
2023-08-16 22:43:51 +02:00
00863a176e
Added application plugin. 2023-08-15 11:20:24 +02:00
f51beb271e
Large refactor + stability fixes for the external process control logic. 2023-08-15 11:12:21 +02:00
46245e851f
Synchronize with the currently running stop thread (if any) in Backend.wait_stop. 2023-08-15 02:08:47 +02:00
a8a7ceb2ac
Implemented HttpBackend._stop_workers.
The Tornado WSGI container won't guarantee the termination of the
spawned workers upon termination, so the code of the backend has to take
care of it and terminate all the children processes of the server
process when it terminates.

This also means that `psutil` is now a required base dependency, as we
need to expand the process subtree under the webserver launcher.
2023-08-15 00:13:34 +02:00
04921c759f
Added wait_stop method to the entities engine and cron scheduler. 2023-08-14 23:49:47 +02:00
ceb9d6d1ba
Removed redundant Backend._get_redis() method.
It was used only once, and it could easily be replaced by
`platypush.utils.get_redis()`.
2023-08-14 23:37:38 +02:00
7157936b87
Added get_remaining_timeout utility function. 2023-08-14 23:17:50 +02:00
dc1a152433
Added get_default_pid_file utility method. 2023-08-14 10:47:25 +02:00
c11bc69a66
Handle KeyboardInterrupt and process return code in the main. 2023-08-13 23:55:40 +02:00
8819a0ed4c
Added CommandStream to the main Application.
The stream will be used to communicate command messages from the
application to its runner.
2023-08-13 23:51:21 +02:00
06111587f7
Added CommandStream class.
This abstraction is used to write and read commands over a UNIX socket.
2023-08-13 23:37:51 +02:00
c89f992867
Added StopCommand and RestartCommand. 2023-08-13 23:36:36 +02:00
b1f244a812
Added ControllableProcess class.
This class can be used to easily control the execution of underlying
processes.
2023-08-13 23:33:08 +02:00
efef9d7bc0
Added commands module. 2023-08-13 23:21:36 +02:00
97adc3f775
Moved application argument parser to an external platypush.cli module. 2023-08-13 22:27:16 +02:00
1819ee75ef
s/Application.stop_app/Application.stop/g 2023-08-13 22:23:15 +02:00
827b564006
Using a single constant for DEFAULT_REDIS_QUEUE.
Also, catch `AttributeError` on `self._proc.terminate` in the
`HttpBackend`, since the process may already have been terminated and
set to null by another worker process.
2023-08-13 22:20:16 +02:00
4062ddbcf0
More improvements to the inspect plugin.
- Support for distinct `type` field on constructor and method arguments.

- Added `has_varargs` field.

- Added `required` field.

- Better logic for parsing arguments `default` values.
2023-08-08 20:47:27 +02:00
9beb0a7af3
Skip :type: annotations from docstring args documentation. 2023-08-08 20:47:27 +02:00
2a30f060b4
Parse the arguments list from obj_type.__init__.__doc__ too. 2023-08-08 20:47:27 +02:00
1474685cc0
[Automatic] Updated UI files 2023-08-06 22:51:57 +00:00
26b4dcba46
Reduced padding of ConfirmDialog body element. 2023-08-07 00:50:10 +02:00
5184543a3e
Normalize the size of the camera frame container from window size.
If the window is smaller than the camera resolution, then we should
scale down the container size accordingly.
2023-08-07 00:38:19 +02:00
dd313c6371
[Automatic] Updated UI files 2023-08-06 21:50:25 +00:00
2f007888fc
[Automatic] Updated UI files 2023-08-06 21:48:33 +00:00
609a8c6052
[Automatic] Updated UI files 2023-08-06 21:46:44 +00:00
276a9a7421
[Automatic] Updated UI files 2023-08-06 21:44:51 +00:00
715cdcfcff
More consistent style for the buttons on the tokens settings page. 2023-08-06 23:42:50 +02:00
bd49e460d5
More style improvements for the user settings panel. 2023-08-06 23:36:06 +02:00
6eed558342
More improvements on the users settings panel. 2023-08-06 23:14:54 +02:00
1d47b35219
Updated UI dependencies 2023-08-06 20:38:46 +02:00
237e0c47cb
Redesigned config panel UI. 2023-08-06 18:49:03 +02:00
53aeb0b3b1
Better documentation for the Redis server + LINT fixes.
1. Added documentation to the README on the possible options to run the
   Redis service.

2. Show a relevant message to the user if the application is run with
   `--start-redis` and Redis couldn't start.

3. Some LINT/black chores on some files that hadn't been touched in a
   while.
2023-08-02 22:17:11 +02:00
e4756eb639
Replaced deprecated flask.escape with markupsafe.escape. 2023-07-25 00:26:15 +02:00
91cd08cdff
Adapted tests to use a locally started Redis instance. 2023-07-24 10:37:07 +02:00
7a20fec52f
[#60] Added --workdir and --logsdir command-line options.
Also, the application is now using `XDG_CONFIG_HOME` and
`XDG_DATA_HOME` if available to lookup the configuration file and
working directory.

Closes: #60
2023-07-24 03:29:08 +02:00
c1d66abb89
[#60] Added --workdir and --logsdir command-line options.
Closes: #60
2023-07-24 03:20:18 +02:00
5fd40af285 Revert "Added explicit --help option."
This reverts commit 1304be0718.
2023-07-24 02:38:12 +02:00
1304be0718 Added explicit --help option.
argparse.ArgumentParser doesn't seem to add the option automatically
anymore.
2023-07-24 02:11:42 +02:00
e9a568fdd2
Unified interface to retrieve the Redis bus configuration.
A common `utils.get_redis_conf` has been created to handle the cascade
fallback logic used to retrive the default Redis configuration.
2023-07-24 01:04:13 +02:00
77ffefdccb
Added a Config.set method.
A useful interface to change configuration values at runtime.
2023-07-24 00:48:07 +02:00
dd862db29a
[chore] LINT fixes 2023-07-24 00:35:03 +02:00
3e3d47aa44
Check first the configuration of the Redis plugin and then that of the backend. 2023-07-24 00:26:49 +02:00
315a89fb65
Added a default parameter to Config.get. 2023-07-23 23:31:57 +02:00
37dcaba7a1
Refactored structure of main application and startup scripts.
The main application class has been moved from __init__ to the app
module.

__init__ will contain instead the relevant global variables and the
modules and objects exposed to external integrations - such as
`get_plugin` and `get_backend`, or the `main` itself.

This will make future integrations much easier - the global __init__
doesn't contain any business logic now, it can import anything without
fearing circular dependencies, and it can limit its exposed objects to
those that we want to expose to 3rd-party integrations and scripts.

It will also make it easier to extend the main entry point with
additional logic - such as a supervisor or an embedded Redis server.
2023-07-23 23:12:36 +02:00
0dc380fa94
Removed dependency from prctl.
Also, black'd and LINT-fixed some files that hadn't been touched in a
while.
2023-07-23 19:17:30 +02:00
15a34ebac9
Upgraded Docker base image.
Debian base image bumped from Bullseye -> Bookworm.
2023-07-23 17:25:52 +02:00
cfdecd5703
[#267] Don't show the PWA popup again after the user dismisses it.
Plus, upgraded a bunch of frontend libraries.

Closes: #267
2023-07-23 02:44:17 +02:00
454228c56e
Upgraded UI dependencies 2023-07-23 02:42:14 +02:00
dependabot[bot]
33699fcac8
Bump word-wrap from 1.2.3 to 1.2.5 in /platypush/backend/http/webapp
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.5.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.5)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-22 21:48:53 +00:00
dependabot[bot]
505cba44b8
Bump webpack from 5.70.0 to 5.88.2 in /platypush/backend/http/webapp
Bumps [webpack](https://github.com/webpack/webpack) from 5.70.0 to 5.88.2.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Commits](https://github.com/webpack/webpack/compare/v5.70.0...v5.88.2)

---
updated-dependencies:
- dependency-name: webpack
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-22 21:46:59 +00:00
1d98f54c54
Bump version: 0.50.2 → 0.50.3 2023-07-22 23:36:04 +02:00
66981bd00b
Updated email addresses and black'd some old source files. 2023-07-22 23:02:44 +02:00
3eda0c6f17
[#268] Implemented XMPP integration. 2023-07-22 22:36:36 +02:00
63afd90701
Serialize exceptions in the default JSON serializers.
This allows us to easily pass errors in internal JSON payloads, and
ensure that these are serialized consistently across all the
integrations.
2023-07-22 15:14:05 +02:00
cef9d0e007
Always serialize exception names when logging/returning request exceptions. 2023-07-22 15:13:17 +02:00
d49e5b1f6a
Expose Config.workdir property.
This is a useful proxy to avoid using `Config.get('workdir')` string
lookup.
2023-07-22 15:11:58 +02:00
c846c61493
Refactored Config.__init__.
The constructor of the `Config` class had grown too big. It's much more
manageable if split into multiple sub-constructor helpers.
2023-07-15 13:38:07 +02:00
0a3d6add83
Support ./config.yaml as a config file location. 2023-07-15 01:29:41 +02:00
fa1364d854
Improved logging for requests, responses and events.
- The following logging namespaces are now used, to make it easier to
  filter only log lines related to the logged application message:

  - `platypush:events`
  - `platypush:requests`
  - `platypush:responses`

- Those messages are always logged as JSON, with no prefixes nor
  suffixes.

- Requests are always logged when executed - no more delegation to the
  upstream backend.

- Responses are always logged when fully populated (including `id`,
  `origin`, `target` etc.), instead of being logged when still partially
  populated. This makes it particularly easy to link request/response
  IDs directly from the logs.
2023-07-15 01:11:45 +02:00
cb04af0bbd
Catch TypeError when execution an action.
Most of TypeError are due to the user passing wrong data. It usually
doesn't mean that we have to fail hard and reload the plugin, nor retry
the call with the same parameters.
2023-07-14 22:20:27 +02:00
27cf1bec52
Removed some optional top-level imports.
Optional top-level imports in Tornado route declarations will trigger
`ImportError`. While this will just mean that those routes will be
skipped, it will also generate a lot of noise on the logs.
2023-07-01 03:13:38 +02:00
e0af2daaff
Bump version: 0.50.1 → 0.50.2 2023-06-30 23:12:40 +02:00
42a990a132
Format string typo fix 2023-06-30 23:07:00 +02:00
9a7fc031d9
A more robust logic to detect plugins by class type in get_plugin. 2023-06-30 23:01:09 +02:00
f3b2386ed6
Bump version: 0.50.0 → 0.50.1 2023-06-28 02:44:13 +02:00
08c3a1e11a
Bump version: 0.24.5 → 0.50.0 2023-06-28 02:23:09 +02:00
5e5403287e
The inspect plugin should not fail hard if the cache couldn't be loaded.
This can happen for many reasons - not only if the cache file is not
accessible, but also if the structure/signature of some pickled objects
has changed. In that case, we should invalidate the current cache and
re-initialize it instead of failing.
2023-06-28 01:33:12 +02:00
ba827b0248
Added sound plugin documentation.
- Added example of how to stream audio over HTTP.

- Added Portaudio to the apt/pacman dependencies.
2023-06-28 00:10:36 +02:00
a103ea49f1
Fixed ffmpeg/audio consumer synchronization upon timeout. 2023-06-27 15:12:15 +02:00
77f7cd8b90
Don't use typing.Self (yet).
typing.Self has only been introduced in Python 3.10.
2023-06-27 14:17:04 +02:00
f2540437b7
Sound plugin rewritten almost from scratch. 2023-06-27 13:31:38 +02:00
2fb6e4d7d0
Updated webapp dist files 2023-06-16 15:48:23 +02:00
2f4229d7b1
pylint fixes for the camera plugin. 2023-06-16 15:40:05 +02:00
9aa8e4538a
Better termination logic for the ffmpeg audio converter. 2023-06-16 11:47:37 +02:00
a6351dddd4
Extracted AudioRecorder out of SoundPlugin. 2023-06-16 03:12:55 +02:00
da93f1b3b0
[Chore] pylint 2023-06-14 01:44:36 +02:00
c8786b75de
sound.recordplay merged into sound.record. 2023-06-12 22:15:02 +02:00
be794316a8
Merged sound.stream_recording and sound.record. 2023-06-12 13:06:02 +02:00
a415c5b231
Merged outfile/fifo logic in sound.stream_recording. 2023-06-12 12:33:14 +02:00
e238fcb6e4
Refactoring the sound plugin to use ffmpeg as a stream converter. 2023-06-11 12:48:49 +02:00
4587b262b0
Stream camera frames over HTTP using a Redis pub/sub mechanism. 2023-06-05 20:40:12 +02:00
d7208c6bbc
Refactored Tornado routes for native pub/sub support.
The Redis pub/sub mechanism is now a native feature for Tornado routes
through the `PubSubMixin`.

(Plus, lint/black chore for the sound plugin)
2023-05-30 21:58:27 +02:00
8b5eb82497
Camera stream writer fixes.
- The readiness condition should be `multiprocessing.Condition`, not
  `threading.Condition` - in most of the cases it will be checked in a
  multiprocess environment.

- Fixed parameter name for `write`.
2023-05-30 11:06:48 +02:00
4fffabd82a
Revert "Removed camera.gstreamer."
This reverts commit b4d714df8a.
2023-05-29 22:13:24 +02:00
4bf9c01ac9
Moved camera routes.
Camera routes migrated from Flask blueprints to Tornado handlers.
2023-05-27 22:24:45 +02:00
b4d714df8a
Removed camera.gstreamer.
Too much of a pain in the ass to handle, too many format options to
think of, too many combinations of pipelines to support, and if I don't
think of those beforehand then I'll have to offload all of that
complexity on the user.
2023-05-27 22:19:50 +02:00
1c40e5e843
Black'd the camera plugin and writer.
Also, proper fix for the multi-inheritance problem of
the ffmpeg writers.
2023-05-23 20:42:59 +02:00
a2f8e2f0d2
Fixed bottom margin for ConfirmDialog buttons 2023-05-22 16:44:53 +02:00
a7aabd7c52
Fixed handling of :meth: docstring annotations with relative paths. 2023-05-22 16:32:30 +02:00
7eca1c27c9
Blackened the qrcode and pushbullet plugins 2023-05-22 02:33:54 +02:00
aaac6488d6
Updated webapp dist files 2023-05-22 02:24:24 +02:00
d7405ad05d
Added multiple parsers for the entities referenced in docstrings.
The `inspect` plugin can now detect references to plugins, backends,
events, responses and schemas in docstrings and replace them either with
links to the documentation or auto-generated examples.
2023-05-22 02:20:58 +02:00
4f11d7cf74
Updated webapp dist files 2023-05-21 03:15:16 +02:00
b91c1eba6d
Parse :return: definitions from action docstrings too. 2023-05-21 03:05:19 +02:00
229b8f2985
Many improvements for the execute panel. 2023-05-21 03:04:52 +02:00
edf1dbee1d
Updated webapp dist files 2023-05-20 15:54:18 +02:00
5efc90ca17
Better PWA installation prompt.
Display a popup modal instead of a confirm box to prompt the user to
install the PWA app.

`confirm` blocks the JavaScript engine when run in `beforeMount` and
therefore the browser won't be able to proceed with `event.prompt()`.
2023-05-20 15:52:37 +02:00
666ea9ea6b
Added use_werkzeug_server option to backend.http.
There are situations where you may not want to run the HTTP server in a
full blown WSGI-over-Tornado container - unit/integration tests and
embedded single-core devices are among those cases.

In those scenarios, we should allow the user to be able to run the
backend using the built-in Werkzeug server provided by Flask.
2023-05-20 15:26:58 +02:00
013274bcbc [#259] Proper support for the progressive webapp (#262)
Closes: 259
Reviewed-on: platypush/platypush#262
2023-05-20 02:42:33 +02:00
27d4a20418
Use reflection to infer the arguments of a Python user procedure 2023-05-17 17:17:59 +02:00
0a1209fe6e
Updated webapp dist files 2023-05-17 10:56:37 +02:00
33e2879413
Various UI improvements for the execute tab. 2023-05-17 10:41:02 +02:00
91daec579d
Reverted to the previous style for entities on mobile.
Better to use screen width wisely and avoid unnecessary padding.
2023-05-17 01:13:09 +02:00
61ea3d79e4
Large refactor for the inspect plugin.
More common logic has been extracted and all the methods and classes
have been documented and black'd.
2023-05-17 00:05:22 +02:00
2cba504e3b
Improvements for the autocomplete component. 2023-05-14 15:07:54 +02:00
8447f9a854
Improved rendering of actions/arguments documentation.
The frontend now calls `utils.rst_to_html` to render the docstrings as
HTML instead of dumping them as raw text.

Also, actions and arguments are now cached to improve performance.
2023-05-14 15:06:34 +02:00
5f2d6dfeb5
Added utils.rst_to_html action. 2023-05-14 15:05:24 +02:00
3c83e7f412
A faster implementation for the inspect.get_* methods.
Plugin/backend lookup is now done by inspecting the manifest files
instead of searching all the subpackages.
2023-05-13 13:44:46 +02:00
72797e73ff
Changed the Tornado paradigm to start the WSGI workers.
Use `bind_sockets`/`fork_processes` instead of reinventing the wheel
with our own multiprocessing handling.
2023-05-13 12:35:20 +02:00
ac4fe4447e
Revert "Added a multi-worker approach to the Tornado WSGI container."
This reverts commit 71401a4936.

Temporarily reverted this commit because the `reuse_address` on the
application's `listen` method has only been implemented in Tornado 6.2 -
and Debian stable still shipts Tornado 6.1.
2023-05-13 02:36:20 +02:00
71401a4936
Added a multi-worker approach to the Tornado WSGI container.
The WSGI container is a good option to wrap a multi-modal webapp
(Flask + websocket routes), but it's constrained to a single-process
approach and queued/pre-buffered requests. That makes performance poor
when handling requests that may take a few seconds to complete.
2023-05-13 01:26:18 +02:00
b7b93edbae
Updated dist files 2023-05-12 03:52:33 +02:00
a15191d4ca
Updated dist files 2023-05-12 03:51:59 +02:00
d4f8e51caf
A less blocking implementation of the entities loading UI logic. 2023-05-12 03:49:20 +02:00
62d846ddda
Updated dist files 2023-05-12 03:42:17 +02:00
23a5e90e2e
Updated dist files 2023-05-12 03:26:55 +02:00
6cd9cb6e76
Better entities caching on the frontend. 2023-05-12 03:18:22 +02:00
79871e0fa1
Fixed devServer routes in vue.config.js.
Use `127.0.0.1` instead of `localhost` for the Vue proxy.
2023-05-12 02:57:14 +02:00
cfedcd701e
Performance improvements when loading the Tensorflow plugin.
The Tensorflow module may take a few seconds to load the first time and
slow down the first scan of the plugins.

All the Tensorflow imports should therefore be placed close to where
they are used instead of being defined at the top of the module.
2023-05-11 19:48:22 +02:00
f49ad4c349
Updated dist files 2023-05-10 02:52:24 +02:00
6b0f0883ee
A proper way to proxy websocket calls using the Vue devServer. 2023-05-10 02:27:01 +02:00
78c12212c6
[#260] A simple entities caching mechanism using the browser storage. 2023-05-10 02:26:06 +02:00
74ab884b7a
Proper redirects upon /execute failure.
If a call to `/execute` fails with a 401 or 412 status, then redirect
the user to `/register` or `/login`.
2023-05-10 02:24:50 +02:00
41233138ff
Blackened inspect module and extracted model defs to adjacent module. 2023-05-09 21:58:02 +02:00
ab2425ebd0
[#260] Removed legacy backend.websocket.
It has now been replaced by the `/ws/events` and `/ws/requests`
websocket routes under `backend.http`.
2023-05-09 02:46:43 +02:00
a069d23bb7
[#260] Added `/ws/requests` websocket route. 2023-05-09 02:40:32 +02:00
7716a416e9
[#260] Support for sending events via websocket over /ws/events. 2023-05-09 02:18:58 +02:00
edb7197f71
[#260] Implemented authentication for websocket routes.
Plus, refactored the `backend.http.app.utils` module by breaking it down
into multiple components, as the module was starting to get too large.
2023-05-09 00:03:11 +02:00
2d4b179879
Include the path info in the logging messages in WSRoute. 2023-05-08 12:35:54 +02:00
3fc622e296
Remove legacy references to the websocket HTTP port and backend in platydock. 2023-05-08 12:25:09 +02:00
f5fcccb0bd
Refactored the new websocket routes.
Defined a `platypush.backend.http.ws` package with all the routes, a
base `WSRoute` class that all the websocket routes can extend, and a
logic in the HTTP backend to automatically scan the package to register
exposed websocket routes.
2023-05-08 11:45:14 +02:00
56dc8d0972
Migrated the webapp to Tornado.
It was just too painful to find a combination of versions of  gunicorn,
gevent, eventlet, pyuwsgi etc. that could work on all of my systems.

On the other hand, Tornado works out of the box with no headaches.

Also in this commit:

- Updated a bunch of outdated/required integration dependencies.
- Black'd and LINTed a couple of old plugins.
2023-05-08 02:06:45 +02:00
f81e9061a3
lifespan=on is actually not required on Flask config level. 2023-05-07 16:30:41 +02:00
692180c653
Back to uvicorn workers from eventlet.
The eventlet API has way too many dependency issues with gunicorn.

Still TODO: Fix or at least mitigate the WSGI workers timeout issue when
they handle websocket connections.
2023-05-07 15:37:58 +02:00
8102178ba4
Updated dist files 2023-05-07 13:00:48 +02:00
c0a948f8ce
Removed remaining references to websocket port. 2023-05-07 12:54:13 +02:00
bdbbd24e6f
Only include /ws/events as a proxied websocket, without messing with the /ws route exposed by the Vue debugger 2023-05-07 12:22:51 +02:00
059fff8558
Updated dist files 2023-05-07 12:19:13 +02:00
f9b0bc905e
Migrated websocket service.
The websocket service is no longer provided by a different service,
controlled by a different thread running on another port.

Instead, it's now exposed directly over Flask routes, using
WSGI+eventlet+simple_websocket.

Also, the SSL context options have been removed from `backend.http`, for
sake of simplicity. If you want to enable SSL, you can serve Platypush
through a reverse proxy like nginx.
2023-05-07 12:08:28 +02:00
3aefc9607d
Migrated from waitress to gunicorn.
`waitress`, unlike `gunicorn`, doesn't provide an easy way to plug into
a WSGI socket that can be used for the websocket interface.
2023-05-07 00:42:57 +02:00
9951d62511
Added logic to automatically generate the secret key for Flask. 2023-05-06 22:04:48 +02:00
d1f0e1976c
Exclude squashfs/loopback mounts from system.disk_info. 2023-05-06 18:53:16 +02:00
e33a391d25
Updated dist files 2023-05-06 12:37:00 +02:00
4f78d61223
Improved UI on mobile. 2023-05-06 12:34:27 +02:00
6e939bbe62
Close modals and dropdown when ESC is pressed. 2023-05-05 20:46:42 +02:00
e9e59c857a
Updated dist files 2023-05-05 02:51:33 +02:00
59bf1c2aa0
Added close button to the modal's header. 2023-05-05 02:48:41 +02:00
8af3ae17b8
A more efficient way of detecting the entity groups to display.
Instead of iterating over each of the entities in a grouping to find out
which groups should be displayed based on the selector's policy, the
selector can directly keep its `selectedGroups` attribute in sync with
the index.
2023-05-05 02:33:34 +02:00