continuous-integration/drone/push Build is passingDetails
- Merged `trello` plugin and backend into a single plugin.
- Removed legacy `Response` objects, replaced with data classes and
schemas.
- Fixed the Websocket connection flow to reflect the new authentication
protocol.
Closes: #307
continuous-integration/drone/push Build is failingDetails
Instead of relying on the official Google YouTube API (limited, subject
to breaking changes with short/no notice depending on Google's strategy
against scrapers, and with an initial setup that has a high cost), we'll
just stick to Piped from now on.
It's free, it doesn't require API keys, it's unlikely to change, it's
not subject to Google's hostile practices against developers, and
anybody can run an instance.
`youtube-dl` is mostly dead and there are several forks available, thus
we need to give the user the ability to pick which `youtube-dl`
executable fork they want to use.
Among these, `yt-dlp` is probably the most maintained today and it's
also included in many default repos, so it's been added as an extra
requirement for all the media plugins.
Closes: #268
- The default PopcornTime API host has changed, as popcorn-time.ga is no
longer available.
- The iMDb API now requires a paid tier even for a basic query. The
official iMDb API layer (and the API key requirement) has thus been
replaced with a dear ol' scraping of the frontend endpoint.
- Pass of Black/LINT.
The plugin now leverages the `sound` plugin for playback, like all other
`tts` plugins now do, instead of an external `media` plugin.
This also removes the need for the `/tts/mimic3/say` endpoint.
- Added `input_format`/`output_format` options to both input and output
audio streams.
- Replaced the previous (confusing) occurrences of `ffmpeg_format` and
`format`.
- Added custom `dtype` option for `sound.play`.
- Added `join` flag (default: false) to `sound.play` to wait for the
playback to finish.
Different versions of the `sounddevice` dependency may or may not return
the `index` parameter when querying the available sound devices.
Thus, the code should be ready for both cases.
The integration was based on my old fork of the AVS service, which is no
longer functional given the changes the the Amazon's backend side.
A new `avs-device-sdk` is now available, but it seems that it requires
lengthy compilation processes which are RaspberryPi-specific.
Further investigation is needed for a new Alexa plugin - see #334.
It only existed as a back-compatibility layer with armv6, since there
was no build of the assistant library that worked on Raspberry Pi Zero.
But that API layer has been discontinued by Google and it's no longer
functional, so only the `assistant.google` integration (on x86_64 and
armv7) is currently supported.
continuous-integration/drone/push Build is passingDetails
- `iputils` should be an explicit system dependency for `ping`.
Some minimal systems (like some Docker images) may not have the `ping`
command installed out of the box.
- `hid` and `marshmallow_dataclass` should be among the auto-mocked
modules.
Added a `wrapped` "hidden" parameter to the function returned by the
`@action` decorator.
We need this to access the underlying decorated function when e.g. we
need to access its specs or decorators.
- The `inspect` plugin and the Sphinx inspection extensions now use the
same underlying logic.
- Moved all the common inspection logic under
`platypush.common.reflection`.
- Faster scanning of the available integrations and components through a
pool of threads.
- Added `doc_url` parameters.
- Migrated events and responses metadata scanning logic.
- Now expanding some custom Sphinx tag instead of returning errors when
running outside of the Sphinx context - it includes `:class:`,
`:meth:` and `.. schema::`.
1. Improved documentation. Every plugin now reports the exact steps to
get the integration up and running with the right API scopes.
2. All Google plugins now have a standard process to get (and reuse) the
client secret. Except for PubSub, Translate and Maps (which have
their own flows), all the Google plugins now read the client secrets
from `<WORKDIR>/credentials/google/client_secret.json` by default.
3. Black/LINT for some of those plugins, which hadn't been touched in a
while.
4. The interface to pass API scopes is now leaner. It's now possible to
pass a scope directly as e.g. `calendar.readonly` rather than
`https://www.googleapis.com/auth/calendar.readonly`.
5. Improved the logic to retrieve the right scope tokens file. If e.g.
an integration requires the role `A`, and a credentials file exists
for the roles `A` and `B`, then this file will be used rather than
prompting the user to authenticate again.
The old type configuration
(`platypush.plugins.calendar.name.CalendarNamePlugin`) is a bit clunky.
Instead, since the type will always be a plugin, we should encourage
the use of `calendar.name` directly to identify the type.
continuous-integration/drone/push Build is passingDetails
If the client that forwarded the request is no longer available (either
because an exception or a timeout was raised) then its I/O buffer and
event loop may be closed.
In this case, the response callback should handle and report the
exception, and still set the event, so that any other threads waiting
for the response can move on.
continuous-integration/drone/push Build is passingDetails
Added an `add_dependencies` plugin to the Sphinx build process that
parses the manifest files of the scanned backends and plugins and
automatically generates the documentation for the required dependencies
and triggered events.
This means that those dependencies are no longer required to be listed
in the docstring of the class itself.
Also in this commit:
- Black/LINT for some integrations that hadn't been touched in a long
time.
- Deleted some leftovers from previous refactors (deprecated
`backend.mqtt`, `backend.zwave.mqtt`, `backend.http.request.rss`).
- Deleted deprecated `inotify` backend - replaced by `file.monitor` (see
#289).
By default, the `phue` library will store the file containing the token
and the bridge configuration under `~/.python_hue`.
That's outside of our application folder, and it can't easily be copied
around or added to Docker volumes.
We should instead have it under `<WORKDIR>/light.hue/config.json`, in
line with what the other plugins do, and if `~/.python_hue` is available
but `<WORKDIR>/light.hue/config.json` isn't then we should copy the
legacy file to the new one.
continuous-integration/drone/push Build is passingDetails
I've tried my best to keep it around, but the endpoints seem to be
broken, they no longer have a link to their API v3 documentation, and
the API Explorer that was supposed to be in the dashboard is gone.
The @action decorator should capture all the exceptions,
log them and return them on `Response.errors`.
This ensures that uncaught exceptions from plugin
actions won't unwind out of control, and also that they
are logged and treated consistently across all the
integrations.
If we include the class name by default then we won't have to
explicitly modify the client_id in the implementation classes
in order to prevent clashes.
We should load the latest timestamps from the db when the thread starts
instead of doing it in the constructor.
The constructor may be invoked when the entities engine hasn't been
initialized yet, and result in deadlocks.
continuous-integration/drone/push Build is passingDetails
The `variable` plugin may break in the constructor the first time the
application is started.
That's because it tries to initialize the cache of stored variables, but
the local database hasn't yet been initialized.
That's because plugins are registered _before_ the entities engine is
initialized, as the entities engine assumes that it already has plugins
to scan for entities.
Therefore, the initialization of the `variable` plugin's cache should be
lazy (only done upon the first call to `get`/`set` etc.), in order to
prevent deadlock situations where the plugin waits for the engine to
start, but the engine will be initialized only after the plugin is
ready.
And the lazy initialization logic should also ensure that the entities
engine has been properly started (and emit a `TimeoutError` if that's
not the case), in order to prevent race conditions.
continuous-integration/drone/push Build is passingDetails
- If a Python optional dependency is available as a system package on
the target system, try and install it that route rather than pip. It's
usually faster and it decreases the risk of breaking system packages.
- Added support for apk dependencies in manifest files. This brings the
number of distros officially supported by all the extensions to four:
- Alpine
- Arch
- Debian
- Ubuntu
continuous-integration/drone/push Build is passingDetails
- Support for distinct `type` field on constructor and method arguments.
- Added `has_varargs` field.
- Added `required` field.
- Better logic for parsing arguments `default` values.
Most of TypeError are due to the user passing wrong data. It usually
doesn't mean that we have to fail hard and reload the plugin, nor retry
the call with the same parameters.
Optional top-level imports in Tornado route declarations will trigger
`ImportError`. While this will just mean that those routes will be
skipped, it will also generate a lot of noise on the logs.
This can happen for many reasons - not only if the cache file is not
accessible, but also if the structure/signature of some pickled objects
has changed. In that case, we should invalidate the current cache and
re-initialize it instead of failing.
- The readiness condition should be `multiprocessing.Condition`, not
`threading.Condition` - in most of the cases it will be checked in a
multiprocess environment.
- Fixed parameter name for `write`.
Too much of a pain in the ass to handle, too many format options to
think of, too many combinations of pipelines to support, and if I don't
think of those beforehand then I'll have to offload all of that
complexity on the user.
The `inspect` plugin can now detect references to plugins, backends,
events, responses and schemas in docstrings and replace them either with
links to the documentation or auto-generated examples.
The frontend now calls `utils.rst_to_html` to render the docstrings as
HTML instead of dumping them as raw text.
Also, actions and arguments are now cached to improve performance.
The Tensorflow module may take a few seconds to load the first time and
slow down the first scan of the plugins.
All the Tensorflow imports should therefore be placed close to where
they are used instead of being defined at the top of the module.
It was just too painful to find a combination of versions of gunicorn,
gevent, eventlet, pyuwsgi etc. that could work on all of my systems.
On the other hand, Tornado works out of the box with no headaches.
Also in this commit:
- Updated a bunch of outdated/required integration dependencies.
- Black'd and LINTed a couple of old plugins.
`delete` will actually remove the record from the database (same as
`unset`'s new behaviour), while `unset` will set it to null without
deleting it (same as the `unset`'s previous behaviour).
`entities.transform_entities` will pass back an empty list instead of an
empty dict if no entities were found, and the function should be able to
handle it.
SQLAlchemy should automatically begin a transaction on
connection/session creation. Plus, `.begin()` messes up things with
SQLAlchemy 2, which has `autobegin` enabled with no easy way of
disabling it.
- `percent_field` should be declared on `platypush.schemas.dataclasses`
level, since it's not specific to the `system` plugin.
- Added a common `SystemBaseSchema` that takes care of calling
`_asdict()` if the object is passed as a `psutil` object instead of a
dict.
Plus, `platypush.schemas.system` has now been split into multiple
submodules to avoid a single-file mega-module with all the system
schemas definitions.
Fabio Manganiello