Commit graph

1466 commits

Author SHA1 Message Date
5dd7345c0b
Sync the latest parse timestamps in main instead of __init__ in rss.
We should load the latest timestamps from the db when the thread starts
instead of doing it in the constructor.

The constructor may be invoked when the entities engine hasn't been
initialized yet, and result in deadlocks.
2023-08-18 15:51:11 +02:00
bf7d060b81
Added @ensure_initialized decorator to actions in variable.
The `variable` plugin may break in the constructor the first time the
application is started.

That's because it tries to initialize the cache of stored variables, but
the local database hasn't yet been initialized.

That's because plugins are registered _before_ the entities engine is
initialized, as the entities engine assumes that it already has plugins
to scan for entities.

Therefore, the initialization of the `variable` plugin's cache should be
lazy (only done upon the first call to `get`/`set` etc.), in order to
prevent deadlock situations where the plugin waits for the engine to
start, but the engine will be initialized only after the plugin is
ready.

And the lazy initialization logic should also ensure that the entities
engine has been properly started (and emit a `TimeoutError` if that's
not the case), in order to prevent race conditions.
2023-08-17 02:47:30 +02:00
98e9abde18
Extended manifest files with Python system packages (if available).
- If a Python optional dependency is available as a system package on
  the target system, try and install it that route rather than pip. It's
  usually faster and it decreases the risk of breaking system packages.

- Added support for apk dependencies in manifest files. This brings the
  number of distros officially supported by all the extensions to four:

  - Alpine
  - Arch
  - Debian
  - Ubuntu
2023-08-16 22:43:51 +02:00
00863a176e
Added application plugin. 2023-08-15 11:20:24 +02:00
f51beb271e
Large refactor + stability fixes for the external process control logic. 2023-08-15 11:12:21 +02:00
4062ddbcf0
More improvements to the inspect plugin.
- Support for distinct `type` field on constructor and method arguments.

- Added `has_varargs` field.

- Added `required` field.

- Better logic for parsing arguments `default` values.
2023-08-08 20:47:27 +02:00
9beb0a7af3
Skip :type: annotations from docstring args documentation. 2023-08-08 20:47:27 +02:00
2a30f060b4
Parse the arguments list from obj_type.__init__.__doc__ too. 2023-08-08 20:47:27 +02:00
e9a568fdd2
Unified interface to retrieve the Redis bus configuration.
A common `utils.get_redis_conf` has been created to handle the cascade
fallback logic used to retrive the default Redis configuration.
2023-07-24 01:04:13 +02:00
0dc380fa94
Removed dependency from prctl.
Also, black'd and LINT-fixed some files that hadn't been touched in a
while.
2023-07-23 19:17:30 +02:00
66981bd00b
Updated email addresses and black'd some old source files. 2023-07-22 23:02:44 +02:00
3eda0c6f17
[#268] Implemented XMPP integration. 2023-07-22 22:36:36 +02:00
cb04af0bbd
Catch TypeError when execution an action.
Most of TypeError are due to the user passing wrong data. It usually
doesn't mean that we have to fail hard and reload the plugin, nor retry
the call with the same parameters.
2023-07-14 22:20:27 +02:00
27cf1bec52
Removed some optional top-level imports.
Optional top-level imports in Tornado route declarations will trigger
`ImportError`. While this will just mean that those routes will be
skipped, it will also generate a lot of noise on the logs.
2023-07-01 03:13:38 +02:00
5e5403287e
The inspect plugin should not fail hard if the cache couldn't be loaded.
This can happen for many reasons - not only if the cache file is not
accessible, but also if the structure/signature of some pickled objects
has changed. In that case, we should invalidate the current cache and
re-initialize it instead of failing.
2023-06-28 01:33:12 +02:00
ba827b0248
Added sound plugin documentation.
- Added example of how to stream audio over HTTP.

- Added Portaudio to the apt/pacman dependencies.
2023-06-28 00:10:36 +02:00
a103ea49f1
Fixed ffmpeg/audio consumer synchronization upon timeout. 2023-06-27 15:12:15 +02:00
77f7cd8b90
Don't use typing.Self (yet).
typing.Self has only been introduced in Python 3.10.
2023-06-27 14:17:04 +02:00
f2540437b7
Sound plugin rewritten almost from scratch. 2023-06-27 13:31:38 +02:00
2f4229d7b1
pylint fixes for the camera plugin. 2023-06-16 15:40:05 +02:00
9aa8e4538a
Better termination logic for the ffmpeg audio converter. 2023-06-16 11:47:37 +02:00
a6351dddd4
Extracted AudioRecorder out of SoundPlugin. 2023-06-16 03:12:55 +02:00
da93f1b3b0
[Chore] pylint 2023-06-14 01:44:36 +02:00
c8786b75de
sound.recordplay merged into sound.record. 2023-06-12 22:15:02 +02:00
be794316a8
Merged sound.stream_recording and sound.record. 2023-06-12 13:06:02 +02:00
a415c5b231
Merged outfile/fifo logic in sound.stream_recording. 2023-06-12 12:33:14 +02:00
e238fcb6e4
Refactoring the sound plugin to use ffmpeg as a stream converter. 2023-06-11 12:48:49 +02:00
4587b262b0
Stream camera frames over HTTP using a Redis pub/sub mechanism. 2023-06-05 20:40:12 +02:00
d7208c6bbc
Refactored Tornado routes for native pub/sub support.
The Redis pub/sub mechanism is now a native feature for Tornado routes
through the `PubSubMixin`.

(Plus, lint/black chore for the sound plugin)
2023-05-30 21:58:27 +02:00
8b5eb82497
Camera stream writer fixes.
- The readiness condition should be `multiprocessing.Condition`, not
  `threading.Condition` - in most of the cases it will be checked in a
  multiprocess environment.

- Fixed parameter name for `write`.
2023-05-30 11:06:48 +02:00
4fffabd82a
Revert "Removed camera.gstreamer."
This reverts commit b4d714df8a.
2023-05-29 22:13:24 +02:00
4bf9c01ac9
Moved camera routes.
Camera routes migrated from Flask blueprints to Tornado handlers.
2023-05-27 22:24:45 +02:00
b4d714df8a
Removed camera.gstreamer.
Too much of a pain in the ass to handle, too many format options to
think of, too many combinations of pipelines to support, and if I don't
think of those beforehand then I'll have to offload all of that
complexity on the user.
2023-05-27 22:19:50 +02:00
1c40e5e843
Black'd the camera plugin and writer.
Also, proper fix for the multi-inheritance problem of
the ffmpeg writers.
2023-05-23 20:42:59 +02:00
a7aabd7c52
Fixed handling of :meth: docstring annotations with relative paths. 2023-05-22 16:32:30 +02:00
7eca1c27c9
Blackened the qrcode and pushbullet plugins 2023-05-22 02:33:54 +02:00
d7405ad05d
Added multiple parsers for the entities referenced in docstrings.
The `inspect` plugin can now detect references to plugins, backends,
events, responses and schemas in docstrings and replace them either with
links to the documentation or auto-generated examples.
2023-05-22 02:20:58 +02:00
b91c1eba6d
Parse :return: definitions from action docstrings too. 2023-05-21 03:05:19 +02:00
27d4a20418
Use reflection to infer the arguments of a Python user procedure 2023-05-17 17:17:59 +02:00
61ea3d79e4
Large refactor for the inspect plugin.
More common logic has been extracted and all the methods and classes
have been documented and black'd.
2023-05-17 00:05:22 +02:00
8447f9a854
Improved rendering of actions/arguments documentation.
The frontend now calls `utils.rst_to_html` to render the docstrings as
HTML instead of dumping them as raw text.

Also, actions and arguments are now cached to improve performance.
2023-05-14 15:06:34 +02:00
5f2d6dfeb5
Added utils.rst_to_html action. 2023-05-14 15:05:24 +02:00
3c83e7f412
A faster implementation for the inspect.get_* methods.
Plugin/backend lookup is now done by inspecting the manifest files
instead of searching all the subpackages.
2023-05-13 13:44:46 +02:00
cfedcd701e
Performance improvements when loading the Tensorflow plugin.
The Tensorflow module may take a few seconds to load the first time and
slow down the first scan of the plugins.

All the Tensorflow imports should therefore be placed close to where
they are used instead of being defined at the top of the module.
2023-05-11 19:48:22 +02:00
41233138ff
Blackened inspect module and extracted model defs to adjacent module. 2023-05-09 21:58:02 +02:00
56dc8d0972
Migrated the webapp to Tornado.
It was just too painful to find a combination of versions of  gunicorn,
gevent, eventlet, pyuwsgi etc. that could work on all of my systems.

On the other hand, Tornado works out of the box with no headaches.

Also in this commit:

- Updated a bunch of outdated/required integration dependencies.
- Black'd and LINTed a couple of old plugins.
2023-05-08 02:06:45 +02:00
d1f0e1976c
Exclude squashfs/loopback mounts from system.disk_info. 2023-05-06 18:53:16 +02:00
373788377b
Created two separate actions under variable to delete/unset.
`delete` will actually remove the record from the database (same as
`unset`'s new behaviour), while `unset` will set it to null without
deleting it (same as the `unset`'s previous behaviour).
2023-05-05 02:21:18 +02:00
2c254e8eb9
numpy and PIL should be required dependencies for all camera plugins. 2023-05-04 23:44:42 +02:00
3febfabdd7
Bluetooth LE blacklisted device notices moved info -> debug. 2023-05-01 22:10:06 +02:00
9d82ce6ea9
Noisy beacons notice back to debug level.
There's just too many of them and it ends up polluting the logs.
2023-05-01 21:25:42 +02:00
5d4bffa119
Fixed retrieval of entities plugin. 2023-04-30 10:42:05 +02:00
3d7755159f
Improved compatibility for traceback.format_exception.
The new syntax, that only requires an `Exception` instance to be passed
to the function, is only compatible with Python >= 3.10.
2023-04-30 00:38:17 +02:00
6b28d16ccf
Exclude more noisy Bluetooth beacons.
Exclude any beacons from devices with no name, no children other than
services, and with none of those services being public/known.
2023-04-29 23:34:24 +02:00
f764d1b4fb
Noisy Bluetooth beacons notices should be logged on info level. 2023-04-29 23:18:12 +02:00
e96885a805
Delete the entity on variable.unset instead of setting it to null. 2023-04-29 18:21:57 +02:00
a3888be216
The robustness check in case of missing fields should also apply to other system entities. 2023-04-29 16:08:38 +02:00
8c9768b05e
Robustness check for system disk entities.
When the system information is still loading it may happen that the
device associated to the disk hasn't been loaded yet.
2023-04-29 16:04:57 +02:00
a20065c649
Exposed _entities utility property in Plugin.
It can be used by other plugins to easily access the `entities` plugin,
along the lines of `db` and `redis`.
2023-04-29 15:50:31 +02:00
68d8befa34
Removed some vestigial commented code. 2023-04-29 15:28:44 +02:00
23b851e9d7
variable.status robustness fix.
`entities.transform_entities` will pass back an empty list instead of an
empty dict if no entities were found, and the function should be able to
handle it.
2023-04-29 15:24:58 +02:00
e919bf95ad
Print the full stack trace if a plugin failed in entities.scan 2023-04-29 15:14:13 +02:00
f40f956507
Migrated variable table to the new entities framework. 2023-04-29 11:36:55 +02:00
8fe61217ce
Added _db and _redis properties to the Plugin class.
Plugins can now access the database and Redis APIs directly without
having to run their own `get_plugin` validation logic.
2023-04-29 11:35:57 +02:00
87db5ca5f3 Exclude all iBeacon devices by default (it's not only Apple, it's everyone) 2023-04-26 14:17:59 +02:00
10d587efd0
FIX: Possible assert evaluation error.
Some versions/configurations of Python may throw `Boolean value of this
clause is not defined` here.
2023-04-26 02:25:28 +02:00
cb288deb71
Exclude more noisy BLE beacons.
Excluding Apple iBeacons and devices with no name and no services.
2023-04-25 16:19:11 +02:00
dd60b8924d
Wrap the PRAGMA statement in sqlalchemy.text.
SQLAlchemy 2 no longer supports raw strings passed to `.execute()`
methods.
2023-04-25 10:41:37 +02:00
4cc88fcf5f
Rewritten the variable plugin to use SQLAlchemy's ORM.
This removes the need for raw SQL statements and CREATE TABLE statements
that may be engine-specific.
2023-04-25 10:35:12 +02:00
e1cd22121a
Removed connection.begin() pattern from the db plugin.
SQLAlchemy should automatically begin a transaction on
connection/session creation. Plus, `.begin()` messes up things with
SQLAlchemy 2, which has `autobegin` enabled with no easy way of
disabling it.
2023-04-25 10:31:49 +02:00
37722d12cd
No need for session.begin in db.create_all. 2023-04-24 23:55:50 +02:00
6fa179e769
LINT fixes 2023-04-24 23:49:31 +02:00
91df18f7b5
Better way to import declarative_base from SQLAlchemy.
Import `declarative_base` in a way that is compatible with any
SQLAlchemy version between 1.3 and 2.x.
2023-04-24 23:21:39 +02:00
87889142e0
Fixed compatibility with SQLAlchemy >= 2.0 in the db plugin. 2023-04-24 22:52:17 +02:00
dc3392c11d
Disk I/O stats are not always available and should therefore be optional. 2023-04-23 22:25:24 +02:00
0cd28f1040
libbluetooth-dev is a required dependency to build pybluez on Debian-derived distros 2023-04-23 18:59:37 +02:00
9c1855e4c0
Fixed docstring for zigbee.mqtt plugin. 2023-04-23 13:03:10 +02:00
27b1048789
Converted system.processes to the new data model. 2023-04-23 02:08:43 +02:00
387616ea96
Convert system.connected_users to the new data model. 2023-04-23 01:12:07 +02:00
763d9e06ec
Increased default poll_interval for system plugin to 60 seconds. 2023-04-23 00:42:44 +02:00
a72c32cb00
Added battery entity support to system plugin. 2023-04-23 00:41:21 +02:00
b3440ab96b
Added support for fan sensors on the system plugin. 2023-04-23 00:08:27 +02:00
45d5f439be
Added support for system temperature sensor entities. 2023-04-22 22:42:11 +02:00
1b048e1952
s/net_connections/network_connections/g 2023-04-22 17:19:24 +02:00
374f936c1f
Merged network_stats into NetworkInterface model. 2023-04-22 17:19:24 +02:00
e213941791
s/net_io_counters/network_info/g 2023-04-22 17:19:23 +02:00
977b55dea9
Merged network addresses into NetworkInterface model. 2023-04-22 17:19:23 +02:00
ebe79ac29a
Refactored system schema dataclasses.
- `percent_field` should be declared on `platypush.schemas.dataclasses`
  level, since it's not specific to the `system` plugin.
- Added a common `SystemBaseSchema` that takes care of calling
  `_asdict()` if the object is passed as a `psutil` object instead of a
  dict.
2023-04-22 17:19:23 +02:00
2d618188c8
Print the full exception stack trace if .status fails. 2023-04-22 17:19:23 +02:00
b3a0896485
Converted NetworkConnection schema/response. 2023-04-22 17:19:22 +02:00
98a300c4b1
Added NetworkInterface entities to system plugin.
Plus, `platypush.schemas.system` has now been split into multiple
submodules to avoid a single-file mega-module with all the system
schemas definitions.
2023-04-21 00:45:15 +02:00
44b8fd4b34
Support for disk entities in the system integration. 2023-04-20 16:26:51 +02:00
153d03d43f
Moved CPU percentage on the level of the CPU entity instead of a child entity. 2023-04-19 01:48:05 +02:00
4ebfbf3851
Added memory stats entities. 2023-04-19 01:31:11 +02:00
0073239a40
Support for CPU load_average entity. 2023-04-18 18:26:02 +02:00
1cee0459cf
Added CpuFrequency entity to system. 2023-04-18 01:49:36 +02:00
a5b0a524f6
Added CpuStats entity to system. 2023-04-18 01:19:06 +02:00
b4fbd3e915
Added percent entity to cpu. 2023-04-17 02:25:04 +02:00
b9286f50b0
Added support for CpuTimes as an entity of the system plugin.
Also, there is now a single `Cpu` entity being exported, with a nested
hierarchy structured like:

```
cpu
  -> cpu_info
  -> cpu_times
    -> idle
    -> user
    -> system
    -> ...
  -> cpu_load
    -> ...
```
2023-04-17 02:25:03 +02:00
b43017ef01
Refactoring the system plugin to support entities. 2023-04-17 02:25:02 +02:00
74aeca5c34
Trigger a sensor event only if abs(old_data - new_data) > tolerance
Not if abs(old_data - new_data) >= tolerance, otherwise events will
always be triggered when tolerance=0, even if the data hasn't changed.
2023-04-17 02:25:01 +02:00
a499b7bc2f
Deprecated poll_seconds in light.hue.
For sake of naming consistency with other plugins, we should use
`poll_interval` instead.
2023-04-03 01:36:12 +02:00
10955dad72
Fixed some documentation glitches in switchbot. 2023-04-03 01:36:12 +02:00
6e5f746dbe
Removed deprecated gpio.sensor base plugin.
Now all the plugins that used to implement it have been moved to
`SensorPlugin`.
2023-04-03 01:36:12 +02:00
8852cb8db4
Fixed new class name for sensor.mcp3008 plugin. 2023-04-03 01:36:12 +02:00
d5ddc0c65e
Migrated arduino integration to the new SensorPlugin API. 2023-04-03 01:36:12 +02:00
ac2ec58f89
Migrated mcp3008 integration to the new SensorPlugin API. 2023-04-03 01:36:11 +02:00
45e5ca47e7 Fallback for sensor._has_changes 2023-04-02 15:38:49 +02:00
962c55937d
Migrated sensor.distance integration.
Remove `backend.sensor.distance` and `gpio.sensor.distance`. They are
now replaced by the `sensor.hcsr04` integration, which is compatible
with the new `SensorPlugin` API.
2023-04-02 14:20:12 +02:00
92578a17c9
Added small docstring portion 2023-04-02 13:55:00 +02:00
beff88986a
Migrated dht integration.
Removed `backend.sensor.dht` and `gpio.sensor.dht`. They have been
merged into the new `sensor.dht` integration, which supports the new
`SensorPlugin` API.
2023-04-02 13:38:53 +02:00
8f604445a2
Migrated old sensor.accelerometer integration.
Removed `backend.sensor.accelerometer` and `gpio.sensor.accelerometer`.
The logic has now been merged in the new `sensor.lis3dh` integration,
which is compatible with the new `SensorPlugin` API.
2023-04-02 13:22:28 +02:00
44cf25271c
Migrated pmw3901 integration.
Removed legacy `backend.sensor.motion.pmw3901` and
`gpio.sensor.motion.pmw3901`. They have been merged in the new
`sensor.pmw3901` integration, compatible with the new `SensorPlugin`
API.
2023-04-02 12:36:08 +02:00
fcdda40c4a
Update the _last_measurement only if some events were processed from the new data. 2023-04-02 12:09:45 +02:00
88784985e1
Should be abs(old_data - new_data) >= tolerance.
Not `abs(old_data - new_data) > tolerance`.
2023-04-02 12:08:40 +02:00
7697c1c6ad
Migrated envirophat to the new SensorPlugin API.
Removed `backend.sensor.envirophat` and `gpio.sensor.envirophat` plugin.
They have now been merged into the new `sensor.envirophat` plugin.
2023-04-02 02:49:08 +02:00
d964167631
s/TimeDurationSensor/TimeDuration/g 2023-04-02 00:57:48 +02:00
f24d0773d1
No need for sensor.vl53l1x.transform_entities to call the parent. 2023-04-01 23:54:43 +02:00
99572f9731
Sanity check to prevent empty objects from being propagated to sensor.transform_entities 2023-04-01 23:41:28 +02:00
3f3726c50a
Fixed another occurrence of "Subscripted generics cannot be used" etc. error 2023-04-01 23:34:22 +02:00
e2e73d0fdb
Fix another Python < 3.10 subscripted generic issue. 2023-04-01 23:23:51 +02:00
c1d0f21ead
Migrated ltr559 integration to the new API.
Merged `backend.sensor.ltr559` and `gpio.sensor.ltr559` into the new
`sensor.ltr559` plugin, which extends the new `SensorPlugin` API.
2023-04-01 23:16:03 +02:00
8e0f88ea16
Don't swap the argument of SensorPlugin.publish_entities with a list if not required 2023-04-01 23:06:37 +02:00
0047d85b9d
Dirty fix for "Subscripted generics cannot be used with class and instance checks" on Python < 3.10 2023-04-01 22:52:24 +02:00
98ec018292
Replaced NoneType reference.
`types.NoneType` is not always available on all Python versions, so we
have to make our own type for it.
2023-04-01 22:42:13 +02:00
5dabfed365
Migrated sensor.bme280 to the new SensorPlugin interface.
Removed the old `backend.sensor.bme280` and the old `gpio.sensor.bme280`
plugin. They have now been merged into the new `sensor.bme280` runnable
plugin, which extends the `SensorPlugin` API and supports entities.
2023-04-01 22:31:24 +02:00
6f237a1500
Support the deprecated poll_seconds option on RunnablePlugin 2023-04-01 22:02:59 +02:00
c23e8867e2
Added enabled_sensors to the sensor plugin 2023-04-01 21:56:56 +02:00
7912a59ff8
vl53l1x plugin migrated to the new SensorPlugin interface. 2023-04-01 19:31:13 +02:00
6a5a5de03e
serial plugin migrated to the new SensorPlugin interface. 2023-04-01 19:29:56 +02:00
bf4db76830
Legacy sensor backend replaced by an extended sensor runnable plugin. 2023-04-01 19:24:35 +02:00
7bdd877e49
Support the binary flag both on serial.read and serial.write. 2023-03-31 14:31:45 +02:00
1efaff878e
Rewritten serial plugin.
`backend.serial` has been removed and the polling logic merged into the
`serial` plugin.

The `serial` plugin now supports the new entity engine as well.
2023-03-31 14:31:45 +02:00
a3e8c7c155 Rewritten vl53l1x integration as a runnable plugin with entity support 2023-03-31 14:25:05 +02:00
6fb362a6fb gpio.sensor.distance.vl53l1x -> sensor.distance.vl53l1x 2023-03-31 14:21:48 +02:00
2781eb1fb1
Merge branch 'master' into 29-generic-entities-support 2023-03-27 00:36:50 +02:00
7a368ebbb8
[#240] Migrated clipboard plugin from pyperclip to pyclip.
Closes: #240
2023-03-26 23:52:15 +02:00
bce2fdee25
Replaced deprecated asyncio.wait([]) with asyncio.gather(*[]). 2023-03-26 23:15:53 +02:00
c0251ef2f7
s/instance/instance_name/g in LinodeInstanceStatusChanged.
For sake of consistency - we also have `instance_id` and having the
instance name assigned to the `instance` attribute is quite ambiguous.
2023-03-26 22:58:20 +02:00
6d674fef21
Fixed small JSON syntax error in the docstring of ntfy.send_message. 2023-03-26 22:53:42 +02:00
295758bb20
Added frontend components for cloud instances. 2023-03-26 12:27:17 +02:00
bc2730c841
Rewritten linode integration.
- Support for cloud instances as native entities.
- Using Marshmallow dataclasses+schemas instead of custom `Response`
  objects.
- Merge `linode` backend into `linode` plugin.
2023-03-26 11:23:33 +02:00
3c355352c5
Using the new StoppableThread API. 2023-03-24 16:39:30 +01:00
998793e94f
Added OBEX_FILE_TRANSFER constant to directory stub. 2023-03-24 15:41:20 +01:00
2f49ddf33a
Fallback logic that uses DBus to disconnect from a BT device.
This logic will be used if the connection wasn't opened by the current
process and therefore a call to DBus is required to terminate it.
2023-03-24 01:57:05 +01:00
913ef6f8cd
Refresh BluetoothDevice.reachable when a device is found/lost. 2023-03-24 01:56:19 +01:00
d46d4e2300
Added support for Bluetooth devices blacklist.
Based on device address, name or manufacturer.
2023-03-24 01:52:39 +01:00
0cebcf4f9b
switchbot.bluetooth integration migrated to a bluetooth plugin. 2023-03-23 17:46:54 +01:00
4fac110bb8
Added bluetooth.set method, whose execution is delegated to the plugins. 2023-03-23 17:45:02 +01:00
cd219f44c4
Pass the list of plugins when creating Bluetooth managers. 2023-03-23 17:42:16 +01:00