If the client that forwarded the request is no longer available (either
because an exception or a timeout was raised) then its I/O buffer and
event loop may be closed.
In this case, the response callback should handle and report the
exception, and still set the event, so that any other threads waiting
for the response can move on.
Added an `add_dependencies` plugin to the Sphinx build process that
parses the manifest files of the scanned backends and plugins and
automatically generates the documentation for the required dependencies
and triggered events.
This means that those dependencies are no longer required to be listed
in the docstring of the class itself.
Also in this commit:
- Black/LINT for some integrations that hadn't been touched in a long
time.
- Deleted some leftovers from previous refactors (deprecated
`backend.mqtt`, `backend.zwave.mqtt`, `backend.http.request.rss`).
- Deleted deprecated `inotify` backend - replaced by `file.monitor` (see
#289).
By default, the `phue` library will store the file containing the token
and the bridge configuration under `~/.python_hue`.
That's outside of our application folder, and it can't easily be copied
around or added to Docker volumes.
We should instead have it under `<WORKDIR>/light.hue/config.json`, in
line with what the other plugins do, and if `~/.python_hue` is available
but `<WORKDIR>/light.hue/config.json` isn't then we should copy the
legacy file to the new one.
I've tried my best to keep it around, but the endpoints seem to be
broken, they no longer have a link to their API v3 documentation, and
the API Explorer that was supposed to be in the dashboard is gone.
The @action decorator should capture all the exceptions,
log them and return them on `Response.errors`.
This ensures that uncaught exceptions from plugin
actions won't unwind out of control, and also that they
are logged and treated consistently across all the
integrations.
If we include the class name by default then we won't have to
explicitly modify the client_id in the implementation classes
in order to prevent clashes.
- Do `abspath`+`expanduser` on the configuration file path before
checking if it exists.
- If the path doesn't exist, but the user explicitly passed a
configuration file, then copy/create the default configuration
under the specified directory.
The new configuration:
- Enables `backend.http` by default
- Removes the extra `config.auto.yaml` dependency
- Includes many more examples, lots of updates for existing examples,
and extensive comments.
Following some common UNIX conventions, if no configuration file is
specified and none exists under the default locations, then a new
configuration directory should be created under:
```
- if root: /etc/platypush
- else:
- if XDG_CONFIG_HOME:
- $XDG_CONFIG_HOME/platypush
- else:
- ~/.config/platypush
```
The two scripts now share the same command interface, behaviour and base
class.
Also, Platydock now builds a Docker image instead of just printing a
Dockerfile, unless the `--print` option is passed.
Instead of having a custom `get_installed` callable field, with
replicated code for each package manager, the field has now been
promoted to a class method containing the common logic, and the
instances now expect a `list` field (base command to list the installed
packages using the specified package manager) and a `parse_list_line`
callback field (to extract the base package name given a raw line from
the command above).
Also, we shouldn't run the list command if we're running within a Docker
context - the host and container environments will be different.
This is useful to determine which is the default set of scripts that
should be used by the installer depending on the detected installed
package manager.
If the /install folder on the container doesn't contain a copy of the
source files, then the git repository will be cloned under that folder.
The user can specify via `-r/--ref` option which tag/branch/commit they
want to install.
Created `platypush/install` folder that contains:
- Dockerfiles for the supported distros
- Lists of required base dependencies for the supported distros
- Install and run scripts
- Added Debian to supported base images
Platydock now will only print out a Dockerfile given a configuration
file.
No more maintaining the state of containers, storing separate workdirs
and configuration directories etc. - that introduced way too much
overhead over Docker.
The database settings could also be overridden in the configuration file
besides the command line.
We should therefore pass the path to the runtime configuration file, so
the Alembic process can initialize its configuration from the same file
and use the same settings.
If the path of the default database engine is overridden via `--workdir`
option then it won't be visible to the new `python` subprocess spawned
for Alembic.
We should load the latest timestamps from the db when the thread starts
instead of doing it in the constructor.
The constructor may be invoked when the entities engine hasn't been
initialized yet, and result in deadlocks.
The `variable` plugin may break in the constructor the first time the
application is started.
That's because it tries to initialize the cache of stored variables, but
the local database hasn't yet been initialized.
That's because plugins are registered _before_ the entities engine is
initialized, as the entities engine assumes that it already has plugins
to scan for entities.
Therefore, the initialization of the `variable` plugin's cache should be
lazy (only done upon the first call to `get`/`set` etc.), in order to
prevent deadlock situations where the plugin waits for the engine to
start, but the engine will be initialized only after the plugin is
ready.
And the lazy initialization logic should also ensure that the entities
engine has been properly started (and emit a `TimeoutError` if that's
not the case), in order to prevent race conditions.
- If a Python optional dependency is available as a system package on
the target system, try and install it that route rather than pip. It's
usually faster and it decreases the risk of breaking system packages.
- Added support for apk dependencies in manifest files. This brings the
number of distros officially supported by all the extensions to four:
- Alpine
- Arch
- Debian
- Ubuntu
The Tornado WSGI container won't guarantee the termination of the
spawned workers upon termination, so the code of the backend has to take
care of it and terminate all the children processes of the server
process when it terminates.
This also means that `psutil` is now a required base dependency, as we
need to expand the process subtree under the webserver launcher.
Also, catch `AttributeError` on `self._proc.terminate` in the
`HttpBackend`, since the process may already have been terminated and
set to null by another worker process.
- Support for distinct `type` field on constructor and method arguments.
- Added `has_varargs` field.
- Added `required` field.
- Better logic for parsing arguments `default` values.
1. Added documentation to the README on the possible options to run the
Redis service.
2. Show a relevant message to the user if the application is run with
`--start-redis` and Redis couldn't start.
3. Some LINT/black chores on some files that hadn't been touched in a
while.
Also, the application is now using `XDG_CONFIG_HOME` and
`XDG_DATA_HOME` if available to lookup the configuration file and
working directory.
Closes: #60
The main application class has been moved from __init__ to the app
module.
__init__ will contain instead the relevant global variables and the
modules and objects exposed to external integrations - such as
`get_plugin` and `get_backend`, or the `main` itself.
This will make future integrations much easier - the global __init__
doesn't contain any business logic now, it can import anything without
fearing circular dependencies, and it can limit its exposed objects to
those that we want to expose to 3rd-party integrations and scripts.
It will also make it easier to extend the main entry point with
additional logic - such as a supervisor or an embedded Redis server.
- The following logging namespaces are now used, to make it easier to
filter only log lines related to the logged application message:
- `platypush:events`
- `platypush:requests`
- `platypush:responses`
- Those messages are always logged as JSON, with no prefixes nor
suffixes.
- Requests are always logged when executed - no more delegation to the
upstream backend.
- Responses are always logged when fully populated (including `id`,
`origin`, `target` etc.), instead of being logged when still partially
populated. This makes it particularly easy to link request/response
IDs directly from the logs.
Most of TypeError are due to the user passing wrong data. It usually
doesn't mean that we have to fail hard and reload the plugin, nor retry
the call with the same parameters.
Optional top-level imports in Tornado route declarations will trigger
`ImportError`. While this will just mean that those routes will be
skipped, it will also generate a lot of noise on the logs.
This can happen for many reasons - not only if the cache file is not
accessible, but also if the structure/signature of some pickled objects
has changed. In that case, we should invalidate the current cache and
re-initialize it instead of failing.
- The readiness condition should be `multiprocessing.Condition`, not
`threading.Condition` - in most of the cases it will be checked in a
multiprocess environment.
- Fixed parameter name for `write`.
Too much of a pain in the ass to handle, too many format options to
think of, too many combinations of pipelines to support, and if I don't
think of those beforehand then I'll have to offload all of that
complexity on the user.
The `inspect` plugin can now detect references to plugins, backends,
events, responses and schemas in docstrings and replace them either with
links to the documentation or auto-generated examples.
Display a popup modal instead of a confirm box to prompt the user to
install the PWA app.
`confirm` blocks the JavaScript engine when run in `beforeMount` and
therefore the browser won't be able to proceed with `event.prompt()`.
There are situations where you may not want to run the HTTP server in a
full blown WSGI-over-Tornado container - unit/integration tests and
embedded single-core devices are among those cases.
In those scenarios, we should allow the user to be able to run the
backend using the built-in Werkzeug server provided by Flask.
The frontend now calls `utils.rst_to_html` to render the docstrings as
HTML instead of dumping them as raw text.
Also, actions and arguments are now cached to improve performance.
This reverts commit 71401a4936.
Temporarily reverted this commit because the `reuse_address` on the
application's `listen` method has only been implemented in Tornado 6.2 -
and Debian stable still shipts Tornado 6.1.
The WSGI container is a good option to wrap a multi-modal webapp
(Flask + websocket routes), but it's constrained to a single-process
approach and queued/pre-buffered requests. That makes performance poor
when handling requests that may take a few seconds to complete.
The Tensorflow module may take a few seconds to load the first time and
slow down the first scan of the plugins.
All the Tensorflow imports should therefore be placed close to where
they are used instead of being defined at the top of the module.
Defined a `platypush.backend.http.ws` package with all the routes, a
base `WSRoute` class that all the websocket routes can extend, and a
logic in the HTTP backend to automatically scan the package to register
exposed websocket routes.
It was just too painful to find a combination of versions of gunicorn,
gevent, eventlet, pyuwsgi etc. that could work on all of my systems.
On the other hand, Tornado works out of the box with no headaches.
Also in this commit:
- Updated a bunch of outdated/required integration dependencies.
- Black'd and LINTed a couple of old plugins.
The eventlet API has way too many dependency issues with gunicorn.
Still TODO: Fix or at least mitigate the WSGI workers timeout issue when
they handle websocket connections.
The websocket service is no longer provided by a different service,
controlled by a different thread running on another port.
Instead, it's now exposed directly over Flask routes, using
WSGI+eventlet+simple_websocket.
Also, the SSL context options have been removed from `backend.http`, for
sake of simplicity. If you want to enable SSL, you can serve Platypush
through a reverse proxy like nginx.
Instead of iterating over each of the entities in a grouping to find out
which groups should be displayed based on the selector's policy, the
selector can directly keep its `selectedGroups` attribute in sync with
the index.
`delete` will actually remove the record from the database (same as
`unset`'s new behaviour), while `unset` will set it to null without
deleting it (same as the `unset`'s previous behaviour).
Added `waitress` dependency. For performance and security reasons, it's
better to always run the Flask application inside of a uWSGI server.
`waitress` also makes things easier by avoiding to ask the user to
manually provide the external executable arguments, as it was the case
with `uwsgi` and `gunicorn`.
It was broken by the previous refactor of the entities panel, which no
longer triggers the `watch` callback on the upstream `entityGroups`.
The new approach listens for entity updates on the frontend bus and
dynamically creates the entity groupings in `selectedGroups` if they are
missing.
Unlike the other entity groupings, which are 4-layered (`grouping ->
group -> entity_id -> entity`), the grouping by ID only needs 3 layers
(`grouping -> entity_id -> entity`).
- Don't recalculate entity groups every time. Instead, keep them in sync
every time an entity is added or removed.
- Removed `computedChildren` from the entity component - no null nodes
are guaranteed to be passed now, so there's no need for another
iteration on the list of children.
- `childrenByParentId` now only looks in the scope of the entity's
children instead of searching all the entities.
The animation has a big impact on page loading performance when the
system includes a high number of entities that all need their loading
animation to be render.
Multiple style improvements for the entity components. Among these:
- A more consistent style for entity values and toggler buttons.
- Fixed overflowing/underflowing entities on smaller/larger screen
sizes.
- Simplified the stylesheets for many entities as many component classes
have now been moved to `common.scss`.
The entity name and value in the component header may be arbitrarily
long and rendered on small screens.
We therefore need to ensure that the text won't overflow the screen
width.
`entities.transform_entities` will pass back an empty list instead of an
empty dict if no entities were found, and the function should be able to
handle it.
Added Alembic environment and `run_db_migrations` logic to the entities
engine so database schema changes can be processed as soon as the
application is started.
- Support for nested attributes on event hook conditions. Things like
these are now possible:
```
from platypush.event.hook import hook
from platypush.message.event.entities import EntityUpdateEvent
@hook(EntityUpdateEvent, entity={"external_id": "system:cpu"})
def on_cpu_update_event(event: EntityUpdateEvent, **_):
print(event.args["entity"]["percent"])
```
- The scoring/regex extraction/partial string match logic in
`_matches_argument` is actually only needed for
`SpeechRecognizedEvent`. Other events don't need these features, and
event hooks may be actually triggered unexpectedly in case of partial
matches. Therefore, the "complex" `_matches_argument` has been moved
as an override only for `SpeechRecognizedEvent`, and all the other
events will perform simple key-value matching.
SQLAlchemy should automatically begin a transaction on
connection/session creation. Plus, `.begin()` messes up things with
SQLAlchemy 2, which has `autobegin` enabled with no easy way of
disabling it.
`Mapped` has been introduced only in SQLAlchemy 1.4, while Debian stable
still ships 1.3.
Removing the type annotation doesn't come with a big cost, but it keeps
Platypush compatible with Debian stable.
- `percent_field` should be declared on `platypush.schemas.dataclasses`
level, since it's not specific to the `system` plugin.
- Added a common `SystemBaseSchema` that takes care of calling
`_asdict()` if the object is passed as a `psutil` object instead of a
dict.
Plus, `platypush.schemas.system` has now been split into multiple
submodules to avoid a single-file mega-module with all the system
schemas definitions.
Also, there is now a single `Cpu` entity being exported, with a nested
hierarchy structured like:
```
cpu
-> cpu_info
-> cpu_times
-> idle
-> user
-> system
-> ...
-> cpu_load
-> ...
```
There are probably more optimal ways of achieving this other than
passing a reference to the full list of entities to each of the
entities, such as running a BFS to recursively expand all the entities
within the child hierarchy of an entity.
This is needed because the entity needs to know which entities aren't
direct children, but are two or more layers down in the hierarchy, so
they should be passed to their own child entities.
Remove `backend.sensor.distance` and `gpio.sensor.distance`. They are
now replaced by the `sensor.hcsr04` integration, which is compatible
with the new `SensorPlugin` API.
Removed `backend.sensor.dht` and `gpio.sensor.dht`. They have been
merged into the new `sensor.dht` integration, which supports the new
`SensorPlugin` API.
Removed `backend.sensor.accelerometer` and `gpio.sensor.accelerometer`.
The logic has now been merged in the new `sensor.lis3dh` integration,
which is compatible with the new `SensorPlugin` API.
Removed legacy `backend.sensor.motion.pmw3901` and
`gpio.sensor.motion.pmw3901`. They have been merged in the new
`sensor.pmw3901` integration, compatible with the new `SensorPlugin`
API.
No more `_value` in the JSON output instead of the `value` property.
If a column is marked as private, and there's an associated property
mapped to its public name, then we should use and serialize that value.
Removed the old `backend.sensor.bme280` and the old `gpio.sensor.bme280`
plugin. They have now been merged into the new `sensor.bme280` runnable
plugin, which extends the `SensorPlugin` API and supports entities.
`backend.serial` has been removed and the polling logic merged into the
`serial` plugin.
The `serial` plugin now supports the new entity engine as well.
`str.title` capitalizes any alphabetic letter after any non-alphabetic
letter. That's a problem for Platypush plugins' naming convention,
because plugins like `sensor.distance.vl53l1x` may be broken into
`sensor.distance.vl53.l1.x`.
Workaround for the circular dependency between
`platypush.entities.bluetooth` and `platypush.plugins.bluetooth.model`.
Unentangling the circular dependency would require way too much work,
since the entity model provides several helpers and properties that
depend on the plugin's model.
The workaround in this commit is to simply push those imports down in
the methods that use them, so they won't be imported until those methods
are called, as well as removing some type annotations that depended on
those objects.
- Support for cloud instances as native entities.
- Using Marshmallow dataclasses+schemas instead of custom `Response`
objects.
- Merge `linode` backend into `linode` plugin.
No replacements have been made for the OBEX backends (push and file
services). PyOBEX is too broken and unmaintained, and there are too many
poorly documented steps required to get an unprivileged user to run an
SDP service.
1. Check the manufacturer parsed via Bleak/Theengs
2. Check the MAC address prefix in the oui numbers table
3. Check from the reported `manufacturer_data`
- Merged together Bluetooth legacy and BLE plugins and scanners.
- Introduced Theengs as a dependency to infer BLE device types and
create sub-entities appropriately.
- Using `BluetoothDevice` and `BluetoothService` entities as the bread
and butter for all the Bluetooth plugin's components.
- Using a shared cache of devices and services between the legacy and
BLE integrations, with merging/coalescing logic included.
- Extended list of discoverable services to include all those officially
supported by the Bluetooth specs.
- Instantiate a separate pool of workers to discover services.
- Refactor of the Bluetooth events - all of them are now instantiated
from a single `BluetoothDevice` object.
Not the upserted entities themselves, no matter if expunged or made transient.
Reminder to my future self: returning the flushed entities and then using them
outside of the session or in another thread opens a big can of worms when using
SQLAlchemy.
That's the best way to ensure that all the columns are fetched eagerly and
prevent errors later when trying to access lazily loaded attributes outside
of the session/thread.
- Added `wait_start()` method that other threads can use to synchronize
with the engine and wait before performing db operations.
- Callback logic wrapped in a try/except block to prevent custom
integrations with buggy callbacks from crashing the engine.
ImportErrors on these entity modules will be ignored when dynamically
loading them, since they have optional external dependencies and we
shouldn't throw an error if we can't import them.
- Support for an optional callback on `publish_entities` to get notified
when the published object are flushed to the db.
- Use `lazy='selectin'` for the entity parent -> children relationship -
it is more efficient and it ensures that all the data the application
needs is loaded upfront.
- `Entity.entity_key` rolled back to `<external_id, plugin>`. The
fallback logic on `<id, plugin>` created more problems than those it
as supposed to solve.
- Added `expire_on_commit=False` to the entities engine session to make
sure that we don't get errors on detached/expired instances.
- Better logic to recursively link parent/children entities, so partial
updates won't get lost.
- Removed `EntitiesCache` - it was too much to maintain while keeping
consistent with the ORM, and it was a perennial fight against
SQLAlchemy's own cache.
- Removed `EntityNotifier` - with no need to merge cached entities, the
`notify` method has become much simpler and it's simply been merged
in the `EntitiesRepository`.
Exceptions that cause the termination of the plugin's loop should always
be logged as such, unless the plugin is supposed to stop and various
exceptions may occur upon teardown.
- Out `gattlib` + `pybluez`, in `bleak`. It's not platform-dependent, it doesn't
require libboost and other heavy build dependencies, and it doesn't require the
user that runs the service from having special privileges to access raw
Bluetooth sockets.
- Better integration with Platypush native entities. The devices are now mapped
to write-only `EnumSwitch` entities, and the status returns the serialized
representation of those entities instead of the previous intermediate
representation.
- Better synchronization logic on stop for `AsyncRunnablePlugin`.
- Fixed several thread names by dropping `prctl.set_name` in favour of
specifying the name directly on thread creation.
- Several LINT fixes.
This removes warnings on `config.get`, where the `logging` configuration
key may also contain the current logging stream and we end up with a
JSONDecodeError when trying to serialize it.
- Added support for `Battery` entities
- Fixed saturation range for `Light` entities
- Parsing `min`/`max`/`unit` from the status attributes, if available
- Toggle collapsed state also if clicked on the gap between the entity
name and the right edge, instead of opening the entity modal. The
entity configuration modal should open only when clicking on the
entity name or icon (and these should be highlighted on hover as links
as well).
- The collapsed state update should be propagated to the wrapped
component as well, if applicable.
The performance of the page is heavily degraded by components loaded
dynamically via defineAsyncComponent that recursively carry behind the
whole Vue machinery.
By wrapping defineAsyncComponent calls in shallowRef we make sure that
we only wire the root level of the newly created dynamic component.
Renamed to `notifyWarning` and `notifyError` respectively.
Those names can often clash with other properties defined on components
that extend the mixin (like entities).
- Support for device options as children configuration entities
- Refactored switches management, removed legacy `switches` plugin
integration, and supporting multiple binary switches for one device
The Zeroconf registration part may randomly get stuck, resulting in the
web server not being properly started.
It's therefore better to run the Zeroconf registration process
asynchronously, for it's not strictly required for the web server to
execute.
The Zeroconf registration part may randomly get stuck, resulting in the
web server not being properly started.
It's therefore better to run the Zeroconf registration process
asynchronously, for it's not strictly required for the web server to
execute.
When a cronjob receives a TIME_SYNC event (because the system clock has
changed/drifted and the cronjobs are expected to recalculate their next
run slot) it should also clear the event.
Otherwise, the next `wait` will be skipped and the cronjob will be
executed even if it wasn't scheduled.
This may make things a bit less optimal, but it's probably the only
possible solution that preserves my sanity.
Managing upserts of cached instances that were previously made transient
and expunged from the session is far from easy, and the management of
recursive parent/children relationships only add one more layer of
complexity (and that management is already complex enough in its current
implementation).
The `disable_logging` attribute was only available on events and
responses, and it could only either entirely disable or enable logging
for all the events of a certain type.
The new flag allows more customization by setting the default logging
level used for any message of a certain type (or `None` to disable
logging). This makes it possible to e.g. set some verbose events to
debug level, and the user can see them if they configure the application
in debug mode.
It also delegates the logging logic to the message itself, instead of
having different parts of the application handling their own logic.
The `disable_logging` attribute was only available on events and
responses, and it could only either entirely disable or enable logging
for all the events of a certain type.
The new flag allows more customization by setting the default logging
level used for any message of a certain type (or `None` to disable
logging). This makes it possible to e.g. set some verbose events to
debug level, and the user can see them if they configure the application
in debug mode.
It also delegates the logging logic to the message itself, instead of
having different parts of the application handling their own logic.
ZWaveJS has broken back-compatibility with zwavejs2mqtt when it comes to
events format.
Only a partial representation of the node and value objects is
forwarded, and that's often not sufficient to infer the full state of
the node with its values.
The `_dispatch_event` logic has therefore been modified to accommodate
both the implementation.
This means that we have to go conservative in order to preserve
back-compatibility and not over-complicate things, even if it (slightly)
comes at the expense of performance.
ZWaveJS has broken back-compatibility with zwavejs2mqtt when it comes to
events format.
Only a partial representation of the node and value objects is
forwarded, and that's often not sufficient to infer the full state of
the node with its values.
The `_dispatch_event` logic has therefore been modified to accommodate
both the implementation.
This means that we have to go conservative in order to preserve
back-compatibility and not over-complicate things, even if it (slightly)
comes at the expense of performance.
Since Parenthesized context managers are only supported on very recent
versions of Python (thanks black for breaking back-compatibility), we
should still use the old multiline syntax - it's not worth breaking
compatibility with Python >= 3.6 and < 3.10 just to avoid typing a
backslash.
The most recent versions of ZwaveJS-UI don't send the `hexId` of the
node on node change events. We have therefore to infer it from the
reported `dbLink`.
The most recent versions of ZwaveJS-UI don't send the `hexId` of the
node on node change events. We have therefore to infer it from the
reported `dbLink`.
The parent->child relationship is now modelled on the database itself,
so we no longer need value names specifically formatted as
`[DeviceName] ValueName` - the UI will take care of it.