platypush/platypush
platypush
/
platypush
2
2
Fork 4
Code Issues 44 Pull Requests 1 Packages Projects 4 Releases Wiki Activity

[#29] Support for general-purpose entities #230

Closed
blacklight wants to merge 0 commits from 29-generic-entities-support into master
pull from: 29-generic-entities-support
merge into: platypush:master
Conversation 1 Commits - Files Changed -
blacklight commented 2022-10-07 11:18:02 +02:00
Owner
Copy Link

Closes: #29

TODO list:

  • Update UI on EntityUpdateEvent
  • UI manual refresh feature
  • Refresh button on group level
  • Plugin/group/entity info modal
  • Support for entity scan timeout
  • Investigate Z-Wave timeout on refresh
  • Implement entity loading notification upon action
  • Entities should be the default view on load
  • Double-check responsiveness on small screens
  • Test case with empty entities db
  • Investigate null created_at/meta fields
  • Investigate broken items on e.g. zigbee/zwave UI panels
  • Implement entities rename
  • Implement entities deletion
  • Implement support for custom icons for entities
  • Support for read-only/write-only entity definitions
  • Implement light entities
  • Investigate what is wrong with zigbee2mqtt
  • Light entities support on zigbee/zwave
  • Merge light.hue plugin/backend
  • Support light entities on smartthings
  • Don't wait for a response on MQTT on device_set if the device is write-only
  • Blink entities upon update
  • Show last_updated_at in entities
  • Fix broken remove entities
  • Support for parent/child relations for entities
  • Support for zwave configuration entities
  • Support for zigbee configuration entities
  • Generic support for lights on zwave.mqtt
  • Support for entity URLs
  • Support unreachable entities flag on zwave.mqtt
  • Support unreachable entities flag on zigbee.mqtt
  • Support all available entities on smartthings
  • Merge zigbee.mqtt plugin/backend
  • Merge zwave.mqtt plugin/backend
  • Replace @manages workaround with property EntityManager class
  • Support general-purpose entities on switchbot integration
  • Support general-purpose entities on bluetooth integration
  • Support OTA updates on zigbee.mqtt entities (moved to #242)
  • Support OTA updates on zwave.mqtt entities (moved to #242)
  • Support sensors in light.hue (moved to #243)
  • Support dimmers in light.hue (moved to #243)
  • Support entities in switchbot
  • Support nested filters on @hook (requires further refinements/use cases)
  • Implement entity modal
  • Display data attributes on UI modal
  • Add support for groups (moved to #244)
  • Add support for scenes (moved to #245)
  • Entity history (too complex to implement, too many use cases, and it can probably be handled better with custom user scripts on entity events that store them however the user wants)
  • Test new UI on Firefox
  • Remove (legacy) switches
  • Remove (legacy) zwave integration
  • Implement sensor entities
  • Implement camera entities (moved to #246)
  • Fix light.hue broken brightness/temp selectors
  • UI support for reachable flag
  • TV entities support for smartthings/samsungtvws (smartthings TV support has already been implemented, while samsungtvws is too specific to be modeled as a generic entity)
  • Media player entities support (moved to #247)
  • Audio player entities support (moved to #248)
  • Audio stream entities support (moved to #249)
  • Add optimizations for <keep-alive> components
Closes: #29 TODO list: * [x] Update UI on EntityUpdateEvent * [x] UI manual refresh feature * [x] Refresh button on group level * [x] Plugin/group/entity info modal * [x] Support for entity scan timeout * [x] Investigate Z-Wave timeout on refresh * [x] Implement entity loading notification upon action * [x] Entities should be the default view on load * [x] Double-check responsiveness on small screens * [x] Test case with empty entities db * [x] Investigate null created_at/meta fields * [x] Investigate broken items on e.g. zigbee/zwave UI panels * [x] Implement entities rename * [x] Implement entities deletion * [x] Implement support for custom icons for entities * [x] Support for read-only/write-only entity definitions * [x] Implement light entities * [x] Investigate what is wrong with zigbee2mqtt * [x] Light entities support on `zigbee`/`zwave` * [x] Merge `light.hue` plugin/backend * [x] Support light entities on `smartthings` * [x] Don't wait for a response on MQTT on `device_set` if the device is write-only * [x] Blink entities upon update * [x] Show `last_updated_at` in entities * [x] Fix broken _remove entities_ * [x] Support for parent/child relations for entities * [x] Support for `zwave` configuration entities * [x] Support for `zigbee` configuration entities * [x] Generic support for lights on `zwave.mqtt` * [x] Support for entity URLs * [x] Support unreachable entities flag on `zwave.mqtt` * [x] Support unreachable entities flag on `zigbee.mqtt` * [x] Support all available entities on `smartthings` * [x] Merge `zigbee.mqtt` plugin/backend * [x] Merge `zwave.mqtt` plugin/backend * [x] Replace `@manages` workaround with property `EntityManager` class * [x] Support general-purpose entities on `switchbot` integration * [x] Support general-purpose entities on `bluetooth` integration * ~~[ ] Support OTA updates on `zigbee.mqtt` entities~~ (moved to [#242](https://git.platypush.tech/platypush/platypush/issues/242)) * ~~[ ] Support OTA updates on `zwave.mqtt` entities~~ (moved to [#242](https://git.platypush.tech/platypush/platypush/issues/242)) * ~~[ ] Support sensors in `light.hue`~~ (moved to [#243](https://git.platypush.tech/platypush/platypush/issues/243)) * ~~[ ] Support dimmers in `light.hue`~~ (moved to [#243](https://git.platypush.tech/platypush/platypush/issues/243)) * [x] Support entities in `switchbot` * ~~[ ] Support nested filters on `@hook`~~ (requires further refinements/use cases) * [x] Implement entity modal * [x] Display `data` attributes on UI modal * ~~[ ] Add support for groups~~ (moved to [#244](https://git.platypush.tech/platypush/platypush/issues/244)) * ~~[ ] Add support for scenes~~ (moved to [#245](https://git.platypush.tech/platypush/platypush/issues/245)) * ~~[ ] Entity history~~ (too complex to implement, too many use cases, and it can probably be handled better with custom user scripts on entity events that store them however the user wants) * [x] Test new UI on Firefox * [x] Remove (legacy) `switches` * [x] Remove (legacy) `zwave` integration * [x] Implement sensor entities * ~~[ ] Implement camera entities~~ (moved to [#246](https://git.platypush.tech/platypush/platypush/issues/246)) * [x] Fix light.hue broken brightness/temp selectors * [x] UI support for `reachable` flag * ~~[ ] TV entities support for `smartthings`/`samsungtvws`~~ (`smartthings` TV support has already been implemented, while `samsungtvws` is too specific to be modeled as a generic entity) * ~~[ ] Media player entities support~~ (moved to [#247](https://git.platypush.tech/platypush/platypush/issues/247)) * ~~[ ] Audio player entities support~~ (moved to [#248](https://git.platypush.tech/platypush/platypush/issues/248)) * ~~[ ] Audio stream entities support~~ (moved to [#249](https://git.platypush.tech/platypush/platypush/issues/249)) * [x] Add optimizations for `<keep-alive>` components
blacklight added 110 commits 2022-10-07 11:18:03 +02:00
26ffc0b0e1
Use Redis instead of an in-process map to store the entity/plugin registry 
This is particularly useful when we want to access the registry from
another process, like the web server or an external script.
947b50b937
Added meta as a JSON field on the Entity table 
Metadata attributes can now be defined and overridden on the object
itself, as well as on the database. Note that db settings will always
take priority in case of value conflicts.
f52b556219
- icon_class should not be part of the backend model 
- Interaction with entities should occur through the `entities.action`
  method, not by implementing native methods on each of the model
  objects
67ff585f6c
Entities engine improvements 
- Added cache support to prevent duplicate EntityUpdateEvents
- The cache is smartly pre-populated and kept up-to-date, so it's
  possible to trigger events as soon as the entities are published by
  the plugin (not only when the records are flushed to the internal db)
f17245e8c7
Send an EntityUpdateEvent only if an entity has already been persisted 
If an event comes from an entity that hasn't been persisted yet on the
internal storage then we wait for the entity record to be committed
before firing an event. It's better to wait a couple of seconds for the
database to synchronize rather than dealing with entity events with
incomplete objects.
2aa8778078
Do not process EntityUpdateEvents only in case of payload changes 
The UI relies on these events upon refresh to detect if a device is
still reacheable. Therefore, we shouldn't mask them if we don't detect
any changes with the current entity configuration/state.
f45df5d4d3
Explictly cast entity IDs to strings when coalescing entity updates 
Some plugins may represent entity IDs as integers, while the database
maps external IDs to strings. This may result in entities being
incorrectly mapped during merging. Casting to string prevents these
type-related ambiguities.
8d57cf06c2
Major refactor for the `light.hue` plugin. 
- Added support for lights as native platform entities.
- Improved performance by using the JSON API objects whenever possible
  to interact with the bridge instead of the native Python objects,
  which perform a bunch of lazy API calls under the hood resulting in
  degraded performance.
- Fixed lights animation attributes by setting only the ones actually
  supported by a light.
- Several LINT fixes.
088cf23958
Do not emit input event from the light component upon update 
It may be an incomplete update that breaks the UI, and it will be
overwritten by the backend event anyway
a5541c33b0
Added support for light entities in zigbee.mqtt 
TODO: Support for colors (I don't have a color Zigbee bulb to test it on yet)
117f92e5b4
Deprecated the `light.hue` backend 
The polling logic has been moved to the `light.hue` plugin itself
instead, so it's no longer required to have both a plugin and a backend
enabled in order to fully manage a Hue bridge.
b9b7404230
Web panel improvements. 
- Don't return a redirect to the login page if an authentication failed
  over a JSON endpoint - instead, return a JSON payload with the error.

- Added support for additional fonts.

- Re-designed the login/registration page.

- Updated caniuse database.
blacklight changed title from WIP: Support for general-purpose entities to [#29] WIP: Support for general-purpose entities 2022-10-07 11:20:05 +02:00
blacklight added a new dependency 2022-10-07 11:25:42 +02:00
#231 UI for configuration edit
blacklight removed a dependency 2022-10-07 11:25:57 +02:00
#231 UI for configuration edit
blacklight added 2 commits 2022-10-09 23:16:53 +02:00
blacklight added 2 commits 2022-10-10 01:38:49 +02:00
blacklight added 3 commits 2022-10-12 03:09:22 +02:00
blacklight added 8 commits 2022-10-23 01:05:59 +02:00
blacklight added 2 commits 2022-10-23 21:23:38 +02:00
blacklight added 21 commits 2022-11-11 01:48:08 +01:00
d7214c4c83
Fix for `No converter available` warnings on zigbee2mqtt 
Only include readable (not state-only) properties on the payload sent to
to `zigbee2mqtt/<device>/get`.
3d1a08f7af
Changed default entity grouping on the frontend. 
Changed from `type` to `category`, which is basically the `name_plural`
attribute of the associated entity type metadata.

This allows us to define distinct entity metadata entries that we still
want to share the same grouping - for instance, `temperature_sensor`,
`humidity_sensor` and `battery` should all be grouped under `Sensors` on
the frontend.
64513be6b8
Initial implementation of sensor entities. 
Implemented (at least in `zigbee.mqtt`, for now):

- `TemperatureSensor`
- `HumiditySensor`
- `VoltageSensor`
- `CurrentSensor`
- `EnergySensor`
- `PowerSensor`
- `NumericSensor` (generic fallback 1)
- `RawSensor` (generic fallback 2)
- `Sensor` (root class)
02abef71e3
Fixes for zigbee devices polling 
- Don't publish a `get` request if the device has no exposed queriable
  attributes.

- Perform the recursive build of the `get` request payload before
  checking for the `access` attribute.
0f19104512
Improved zigbee.mqtt node property queries. 
Now handling cases of nodes with values having multiple levels (> 1) of
nested properties.
00a43dd1f8
Implemented `EnumSwitch` entity type 
Done for `zigbee.mqtt`, other plugins will follow
blacklight added 5 commits 2022-11-12 02:03:46 +01:00
6b7933cd33
Using a different SQLite database for entities 
This prevents multiprocessing/concurrency issues when modifying the same
database file both from the main process and from the web server process
blacklight added 6 commits 2022-11-12 16:31:52 +01:00
86edd70d93
Fixed session/concurrency management on the main SQLite db 
- The `declarative_base` instance should be shared
- Database `session_locks` should be stored at module, not instance
  level
- Better isolation of scoped sessions
- Enclapsulated `get_session` method in `UserManager`
blacklight added 3 commits 2022-11-13 01:39:59 +01:00
24f5a8283c
Added `PRAGMA foreign_keys = ON` before deleting entities on SQLite 
SQLite doesn't enable foreign keys cascade on delete by default.
blacklight added 3 commits 2022-11-14 00:06:54 +01:00
blacklight added 6 commits 2022-11-21 00:05:34 +01:00
blacklight added 1 commit 2022-11-21 09:50:26 +01:00
ae17a12c12
FIX: `UserManager.get_users` 
`UserManager.get_users` should not return a reference to the query
object, since the query object will be invalidated as soon as the
connection is closed.

Instead, it should return directly the list of `User` objects.
blacklight added 1 commit 2022-11-21 12:36:12 +01:00
blacklight added 21 commits 2022-11-27 22:59:59 +01:00
292ed2abff
Better style for dropdown items. 
- Larger icon div to prevent text overlapping with icons
- Support for `selected` class
0e0c90f0f2
zwave.mqtt additions 
- Infer entity types on the basis of their semantic type (bool, decimal,
  list) and read-only attribute (read-only bool is `BinarySensor`,
  read-write bool is `Switch`, read-only decimal is `NumericSensor`,
  read-write decimal is `Dimmer`, etc.) instead of trying to infer it
  from the command class. Only a small set of command classes (like
  configurations, vendor-specific or internal values) will be excluded.
  This should greatly increase the number of supported values.

- Added support for `EnumSwitch` entities.

- Added inference for illuminance and humidity sensors.
blacklight self-assigned this 2022-11-27 23:02:10 +01:00
blacklight changed title from [#29] WIP: Support for general-purpose entities to WIP: [#29] WIP: Support for general-purpose entities 2022-11-27 23:02:29 +01:00
blacklight changed title from WIP: [#29] WIP: Support for general-purpose entities to WIP: [#29] Support for general-purpose entities 2022-11-27 23:02:44 +01:00
blacklight added 2 commits 2022-11-28 21:42:44 +01:00
blacklight added 4 commits 2022-11-30 01:04:17 +01:00
16c24d799d
Removed custom formatting for child zigbee/zwave entity names 
The parent->child relationship is now modelled on the database itself,
so we no longer need value names specifically formatted as
`[DeviceName] ValueName` - the UI will take care of it.
blacklight added 5 commits 2022-12-04 20:53:32 +01:00
3a92bf59ca
Support for the new way of reporting events on ZWaveJS-UI. 
The most recent versions of ZwaveJS-UI don't send the `hexId` of the
node on node change events. We have therefore to infer it from the
reported `dbLink`.
blacklight added 4 commits 2022-12-10 15:01:37 +01:00
00a918dd20
Support for the new way of reporting events on ZWaveJS-UI. 
The most recent versions of ZwaveJS-UI don't send the `hexId` of the
node on node change events. We have therefore to infer it from the
reported `dbLink`.
313105f014
Fixed `backend.zwave` event logic dispatch for recent versions of ZWaveJS. 
ZWaveJS has broken back-compatibility with zwavejs2mqtt when it comes to
events format.

Only a partial representation of the node and value objects is
forwarded, and that's often not sufficient to infer the full state of
the node with its values.

The `_dispatch_event` logic has therefore been modified to accommodate
both the implementation.

This means that we have to go conservative in order to preserve
back-compatibility and not over-complicate things, even if it (slightly)
comes at the expense of performance.
blacklight added 1 commit 2022-12-10 15:57:56 +01:00
blacklight added 4 commits 2022-12-11 10:55:42 +01:00
blacklight added 3 commits 2022-12-11 11:47:03 +01:00
d4b540dd67
Replaced `disable_logging` with a more generic `logging_level`. 
The `disable_logging` attribute was only available on events and
responses, and it could only either entirely disable or enable logging
for all the events of a certain type.

The new flag allows more customization by setting the default logging
level used for any message of a certain type (or `None` to disable
logging). This makes it possible to e.g. set some verbose events to
debug level, and the user can see them if they configure the application
in debug mode.

It also delegates the logging logic to the message itself, instead of
having different parts of the application handling their own logic.
blacklight added 2 commits 2022-12-11 11:59:25 +01:00
blacklight added 3 commits 2022-12-18 15:16:11 +01:00
5763c5e0ba
Don't use the entities cache when upserting entities. 
This may make things a bit less optimal, but it's probably the only
possible solution that preserves my sanity.

Managing upserts of cached instances that were previously made transient
and expunged from the session is far from easy, and the management of
recursive parent/children relationships only add one more layer of
complexity (and that management is already complex enough in its current
implementation).
blacklight added 3 commits 2022-12-20 23:06:32 +01:00
3615a269fe
Use Float instead of Numeric on NumericSensor table. 
The Numeric type may have casting/rounding issues with SQLite.
blacklight added 3 commits 2023-01-01 13:18:39 +01:00
1c811a490f
Don't serialize children_ids in Entity.to_json() 
Serializing both children_ids and parent_id can result in nasty
SQLAlchemy bugs, especially when handling objects that haven't been
flushed yet.
edddc90d73
Run register_service in a separate thread in `backend.http`. 
The Zeroconf registration part may randomly get stuck, resulting in the
web server not being properly started.

It's therefore better to run the Zeroconf registration process
asynchronously, for it's not strictly required for the web server to
execute.
blacklight added 3 commits 2023-01-01 23:23:55 +01:00
blacklight added 1 commit 2023-01-02 12:50:37 +01:00
blacklight added 7 commits 2023-01-03 23:17:20 +01:00
blacklight added 1 commit 2023-01-07 22:31:27 +01:00
blacklight added 2 commits 2023-01-07 22:39:37 +01:00
blacklight added 2 commits 2023-01-07 23:10:06 +01:00
blacklight added 1 commit 2023-01-07 23:11:46 +01:00
blacklight added 3 commits 2023-01-07 23:48:55 +01:00
40bdc3b7f3
Always wait 5 seconds (regardless of the poll interval) in case of errors. 
Also, print the error only on the first occurrence, to prevent log
spamming.
blacklight added 5 commits 2023-01-09 01:04:55 +01:00
4a2851231c
Large refactor of `zigbee.mqtt` 
- Support for device options as children configuration entities
- Refactored switches management, removed legacy `switches` plugin
  integration, and supporting multiple binary switches for one device
blacklight added 4 commits 2023-01-13 23:29:54 +01:00
blacklight added 5 commits 2023-01-14 22:40:07 +01:00
fd2d83c80b
Renamed Notification mixin's warn and error methods. 
Renamed to `notifyWarning` and `notifyError` respectively.

Those names can often clash with other properties defined on components
that extend the mixin (like entities).
2778357a9e
Wrapped dynamic Vue components in shallowRef. 
The performance of the page is heavily degraded by components loaded
dynamically via defineAsyncComponent that recursively carry behind the
whole Vue machinery.

By wrapping defineAsyncComponent calls in shallowRef we make sure that
we only wire the root level of the newly created dynamic component.
blacklight added 2 commits 2023-01-15 12:34:26 +01:00
blacklight added 4 commits 2023-01-15 15:29:48 +01:00
e9371ac5d0
Improved entity collapse logic 
- Toggle collapsed state also if clicked on the gap between the entity
  name and the right edge, instead of opening the entity modal. The
  entity configuration modal should open only when clicking on the
  entity name or icon (and these should be highlighted on hover as links
  as well).

- The collapsed state update should be propagated to the wrapped
  component as well, if applicable.
blacklight added 1 commit 2023-01-15 15:47:14 +01:00
blacklight added 2 commits 2023-01-15 20:03:06 +01:00
blacklight added 13 commits 2023-01-22 14:28:54 +01:00
4d762b81dc
EntityUpdateEvent traces can now be logged on INFO level 
The EntityUpdateEvents generated by light.hue are now less noisy.
3923a09831
- Expose methods on the bus module to publish/subscribe to notifications and entity updates 
- Removed some redundant `pass` statements in Z-Wave derived event classes
6d4cf64253
More work on `smartthings`. 
- Added support for `Battery` entities
- Fixed saturation range for `Light` entities
- Parsing `min`/`max`/`unit` from the status attributes, if available
blacklight added 5 commits 2023-01-26 22:11:05 +01:00
334ccc35a2
Don't serialize I/O wrappers 
This removes warnings on `config.get`, where the `logging` configuration
key may also contain the current logging stream and we end up with a
JSONDecodeError when trying to serialize it.
blacklight added 1 commit 2023-01-27 02:01:02 +01:00
341e749d23
Merged the `zigbee.mqtt` backend into the plugin. 
- Deprecated the old `zigbee.mqtt` backend
- Black style for the `mqtt` backend
blacklight added 6 commits 2023-02-03 02:22:53 +01:00
3db9c58d31
[WIP] Converted `switch.tplink` plugin. 
`switch.tplink` converted to a `RunnablePlugin` that implements
`SwitchEntityManager`.
blacklight added 15 commits 2023-02-05 22:01:22 +01:00
blacklight added 3 commits 2023-02-08 00:49:03 +01:00
419a0cec61
More LINTing 
Better prototype for `MultiLevelSwitchEntityManager.set_value`
e49a0aec4d
Various improvements. 
- Better synchronization logic on stop for `AsyncRunnablePlugin`.
- Fixed several thread names by dropping `prctl.set_name` in favour of
  specifying the name directly on thread creation.
- Several LINT fixes.
blacklight added 8 commits 2023-02-11 04:05:03 +01:00
8469a1027f
Migrated/refactored `switchbot.bluetooth` integration. 
- Out `gattlib` + `pybluez`, in `bleak`. It's not platform-dependent, it doesn't
  require libboost and other heavy build dependencies, and it doesn't require the
  user that runs the service from having special privileges to access raw
  Bluetooth sockets.

- Better integration with Platypush native entities. The devices are now mapped
  to write-only `EnumSwitch` entities, and the status returns the serialized
  representation of those entities instead of the previous intermediate
  representation.
blacklight added 4 commits 2023-02-15 22:23:53 +01:00
1d0be5c929
- Simplified prototype for `EntityManager.set` 
- Added small documentation/annotations notes to the `Plugin` module.

- Small LINT fixes
45664be44b
Removed deprecated `backend.bluetooth.scanner`. 
Scan capabilities are now implemented on the `bluetooth` plugin itself.
blacklight added 20 commits 2023-02-22 02:56:54 +01:00
9112239ac3
Better exception management in `AsyncRunnablePlugin`. 
Exceptions that cause the termination of the plugin's loop should always
be logged as such, unless the plugin is supposed to stop and various
exceptions may occur upon teardown.
dc7cbe743d
Refactored/improved `RawSensor` entity. 
It will now automatically deal with most of the native types and convert
them to strings on the db.
a5a923a752
Added `BluetoothDeviceNewDataEvent`. 
These events handle the case where a Bluetooth device only publishes new
service data without advertising any additional updated properties.
d8c429f4a8
Major improvements on the entities engine. 
- Better logic to recursively link parent/children entities, so partial
  updates won't get lost.

- Removed `EntitiesCache` - it was too much to maintain while keeping
  consistent with the ORM, and it was a perennial fight against
  SQLAlchemy's own cache.

- Removed `EntityNotifier` - with no need to merge cached entities, the
  `notify` method has become much simpler and it's simply been merged
  in the `EntitiesRepository`.
blacklight added 1 commit 2023-02-22 03:36:29 +01:00
blacklight added 10 commits 2023-02-23 21:21:16 +01:00
blacklight added 22 commits 2023-03-13 02:39:12 +01:00
c4efec6832
Several fixes and improvements on the entities engine. 
- Support for an optional callback on `publish_entities` to get notified
  when the published object are flushed to the db.

- Use `lazy='selectin'` for the entity parent -> children relationship -
  it is more efficient and it ensures that all the data the application
  needs is loaded upfront.

- `Entity.entity_key` rolled back to `<external_id, plugin>`. The
  fallback logic on `<id, plugin>` created more problems than those it
  as supposed to solve.

- Added `expire_on_commit=False` to the entities engine session to make
  sure that we don't get errors on detached/expired instances.
9f9ee575f1
Added _import_error_ignored_modules. 
ImportErrors on these entity modules will be ignored when dynamically
loading them, since they have optional external dependencies and we
shouldn't throw an error if we can't import them.
ceb7a2f098
EntitiesEngine synchronization improvements. 
- Added `wait_start()` method that other threads can use to synchronize
  with the engine and wait before performing db operations.

- Callback logic wrapped in a try/except block to prevent custom
  integrations with buggy callbacks from crashing the engine.
f45e47363d
Use lazy='joined' instead of lazy='selectin' on Entity.parent. 
That's the best way to ensure that all the columns are fetched eagerly and
prevent errors later when trying to access lazily loaded attributes outside
of the session/thread.
31552963c4
`EntitiesDb.upsert` should return a deep copy of the upserted entities. 
Not the upserted entities themselves, no matter if expunged or made transient.

Reminder to my future self: returning the flushed entities and then using them
outside of the session or in another thread opens a big can of worms when using
SQLAlchemy.
2411b961e8
[WIP] Big, big refactor of the Bluetooth integration. 
- Merged together Bluetooth legacy and BLE plugins and scanners.
- Introduced Theengs as a dependency to infer BLE device types and
  create sub-entities appropriately.
- Using `BluetoothDevice` and `BluetoothService` entities as the bread
  and butter for all the Bluetooth plugin's components.
- Using a shared cache of devices and services between the legacy and
  BLE integrations, with merging/coalescing logic included.
- Extended list of discoverable services to include all those officially
  supported by the Bluetooth specs.
- Instantiate a separate pool of workers to discover services.
- Refactor of the Bluetooth events - all of them are now instantiated
  from a single `BluetoothDevice` object.
blacklight added 11 commits 2023-03-19 12:57:15 +01:00
9d66b63266
BluetoothService attributes fixes. 
BluetoothService IDs should always be in the format `address::uuid` and
the name should always be in title format.
blacklight added 1 commit 2023-03-19 22:23:47 +01:00
blacklight added 10 commits 2023-03-21 16:02:44 +01:00
blacklight added 8 commits 2023-03-22 15:47:02 +01:00
f6e09d34e4
A more clever logic of parsing the manufacturer for BLE devices. 
1. Check the manufacturer parsed via Bleak/Theengs
2. Check the MAC address prefix in the oui numbers table
3. Check from the reported `manufacturer_data`
01d323fad0
Passing a boolean `exclude_known_noisy_beacons` to `bluetooth` plugin. 
The logic to exclude BLE beacons from randomized devices needs to be a
bit more granular and not limited only to the reported device
manufacturer.
99cfd247a5
A more effective logic to exclude noisy BLE beacons. 
This includes BLE beacons sent from all Google/Apple/Microsoft/Samsung
beacon networks in all of their variants.
blacklight added 1 commit 2023-03-22 16:21:25 +01:00
blacklight added 19 commits 2023-03-24 01:58:00 +01:00
5c23d3aa87
metadata and rssi fields on BLEDevice have been deprecated. 
Changed the BLE beacon parsing logic to read those fields from
`AdvertisementData` instead of `BLEDevice`.
e71c312133
Always read an entity's parent through get_parent when climbing up. 
This should avoid the risk of `DetachedInstanceError` by retrieving the
object into the session if it's not available.
af125347d6
If no matching services are found when connecting to a device, default to BLEManager. 
GATT characteristics are not necessarily exposed as services.
d46d4e2300
Added support for Bluetooth devices blacklist. 
Based on device address, name or manufacturer.
2f49ddf33a
Fallback logic that uses DBus to disconnect from a BT device. 
This logic will be used if the connection wasn't opened by the current
process and therefore a call to DBus is required to terminate it.
blacklight added 5 commits 2023-03-24 16:42:57 +01:00
567e9d4e21
Removed legacy `bluetooth` backends. 
No replacements have been made for the OBEX backends (push and file
services). PyOBEX is too broken and unmaintained, and there are too many
poorly documented steps required to get an unprivileged user to run an
SDP service.
blacklight added 1 commit 2023-03-24 16:46:02 +01:00
blacklight added 8 commits 2023-03-26 12:30:58 +02:00
7bbae55e44
`platypush.entities._managers` -> `platypush.entities.managers`. 
It's better for entity managers to be stored in their own public
package, instead of cluttering too much the namespace of their parent
package.
bc2730c841
Rewritten `linode` integration. 
- Support for cloud instances as native entities.
- Using Marshmallow dataclasses+schemas instead of custom `Response`
  objects.
- Merge `linode` backend into `linode` plugin.
blacklight added 6 commits 2023-03-26 22:59:29 +02:00
276aff757b
Removed circular dependency. 
Workaround for the circular dependency between
`platypush.entities.bluetooth` and `platypush.plugins.bluetooth.model`.

Unentangling the circular dependency would require way too much work,
since the entity model provides several helpers and properties that
depend on the plugin's model.

The workaround in this commit is to simply push those imports down in
the methods that use them, so they won't be imported until those methods
are called, as well as removing some type annotations that depended on
those objects.
c0251ef2f7
`s/instance/instance_name/g` in `LinodeInstanceStatusChanged`. 
For sake of consistency - we also have `instance_id` and having the
instance name assigned to the `instance` attribute is quite ambiguous.
blacklight added 2 commits 2023-03-26 23:49:57 +02:00
blacklight added 1 commit 2023-03-27 00:37:02 +02:00
blacklight added 1 commit 2023-03-27 01:47:39 +02:00
blacklight added 1 commit 2023-03-31 14:20:02 +02:00
e198f2a175 Replaced `.title` in `get_plugin` with `.upper` on the first character. 
`str.title` capitalizes any alphabetic letter after any non-alphabetic
letter. That's a problem for Platypush plugins' naming convention,
because plugins like `sensor.distance.vl53l1x` may be broken into
`sensor.distance.vl53.l1.x`.
blacklight added 4 commits 2023-03-31 14:26:23 +02:00
blacklight added 13 commits 2023-04-01 22:34:17 +02:00
1efaff878e
Rewritten `serial` plugin. 
`backend.serial` has been removed and the polling logic merged into the
`serial` plugin.

The `serial` plugin now supports the new entity engine as well.
9693becb9e
Removed LGTM badges from the README. 
LGTM is now merged into Github and the badges are no longer available.
6a3ade3304
Added `common.sensors` package. 
The package contains the base types and constants shared across
sensor-based integrations.
5dabfed365
Migrated `sensor.bme280` to the new `SensorPlugin` interface. 
Removed the old `backend.sensor.bme280` and the old `gpio.sensor.bme280`
plugin. They have now been merged into the new `sensor.bme280` runnable
plugin, which extends the `SensorPlugin` API and supports entities.
blacklight added 1 commit 2023-04-01 22:42:46 +02:00
98ec018292
Replaced `NoneType` reference. 
`types.NoneType` is not always available on all Python versions, so we
have to make our own type for it.
blacklight added 1 commit 2023-04-01 22:52:31 +02:00
blacklight added 1 commit 2023-04-01 23:14:13 +02:00
blacklight added 1 commit 2023-04-01 23:16:46 +02:00
c1d0f21ead
Migrated `ltr559` integration to the new API. 
Merged `backend.sensor.ltr559` and `gpio.sensor.ltr559` into the new
`sensor.ltr559` plugin, which extends the new `SensorPlugin` API.
blacklight added 1 commit 2023-04-01 23:24:27 +02:00
blacklight added 1 commit 2023-04-01 23:35:11 +02:00
blacklight added 1 commit 2023-04-01 23:41:56 +02:00
blacklight added 1 commit 2023-04-01 23:55:20 +02:00
blacklight added 1 commit 2023-04-01 23:58:34 +02:00
blacklight added 8 commits 2023-04-02 02:50:26 +02:00
9e5ad0e0b1
`Entity.to_dict` now takes into account columns mapped to properties. 
No more `_value` in the JSON output instead of the `value` property.

If a column is marked as private, and there's an associated property
mapped to its public name, then we should use and serialize that value.
7697c1c6ad
Migrated `envirophat` to the new `SensorPlugin` API. 
Removed `backend.sensor.envirophat` and `gpio.sensor.envirophat` plugin.
They have now been merged into the new `sensor.envirophat` plugin.
blacklight added 1 commit 2023-04-02 02:56:14 +02:00
blacklight added 1 commit 2023-04-02 03:24:28 +02:00
blacklight added 2 commits 2023-04-02 12:16:58 +02:00
88784985e1
Should be `abs(old_data - new_data) >= tolerance`. 
Not `abs(old_data - new_data) > tolerance`.
blacklight added 2 commits 2023-04-02 13:25:47 +02:00
44cf25271c
Migrated `pmw3901` integration. 
Removed legacy `backend.sensor.motion.pmw3901` and
`gpio.sensor.motion.pmw3901`. They have been merged in the new
`sensor.pmw3901` integration, compatible with the new `SensorPlugin`
API.
8f604445a2
Migrated old `sensor.accelerometer` integration. 
Removed `backend.sensor.accelerometer` and `gpio.sensor.accelerometer`.
The logic has now been merged in the new `sensor.lis3dh` integration,
which is compatible with the new `SensorPlugin` API.
blacklight added 1 commit 2023-04-02 13:39:44 +02:00
beff88986a
Migrated `dht` integration. 
Removed `backend.sensor.dht` and `gpio.sensor.dht`. They have been
merged into the new `sensor.dht` integration, which supports the new
`SensorPlugin` API.
blacklight added 2 commits 2023-04-02 14:21:54 +02:00
962c55937d
Migrated `sensor.distance` integration. 
Remove `backend.sensor.distance` and `gpio.sensor.distance`. They are
now replaced by the `sensor.hcsr04` integration, which is compatible
with the new `SensorPlugin` API.
blacklight added 1 commit 2023-04-02 15:38:55 +02:00
blacklight added 9 commits 2023-04-03 01:36:20 +02:00
6e5f746dbe
Removed deprecated `gpio.sensor` base plugin. 
Now all the plugins that used to implement it have been moved to
`SensorPlugin`.
a499b7bc2f
Deprecated `poll_seconds` in `light.hue`. 
For sake of naming consistency with other plugins, we should use
`poll_interval` instead.
blacklight added 1 commit 2023-04-13 17:16:26 +02:00
blacklight added 12 commits 2023-04-17 02:25:12 +02:00
74aeca5c34
Trigger a sensor event only if abs(old_data - new_data) > tolerance 
Not if abs(old_data - new_data) >= tolerance, otherwise events will
always be triggered when tolerance=0, even if the data hasn't changed.
e7f64843a5
Added `include_children` parameter to `_merge_columns`. 
We need to recursively merge the columns of children entities if a child
entity isn't a leaf node.
4842c1911b
Frontend entities should have a reference to `allEntities`. 
There are probably more optimal ways of achieving this other than
passing a reference to the full list of entities to each of the
entities, such as running a BFS to recursively expand all the entities
within the child hierarchy of an entity.

This is needed because the entity needs to know which entities aren't
direct children, but are two or more layers down in the hierarchy, so
they should be passed to their own child entities.
b9286f50b0
Added support for `CpuTimes` as an entity of the `system` plugin. 
Also, there is now a single `Cpu` entity being exported, with a nested
hierarchy structured like:

```
cpu
  -> cpu_info
  -> cpu_times
    -> idle
    -> user
    -> system
    -> ...
  -> cpu_load
    -> ...
```
blacklight added 2 commits 2023-04-18 01:50:07 +02:00
blacklight added 7 commits 2023-04-21 01:11:21 +02:00
98a300c4b1
Added `NetworkInterface` entities to `system` plugin. 
Plus, `platypush.schemas.system` has now been split into multiple
submodules to avoid a single-file mega-module with all the system
schemas definitions.
blacklight added 1 commit 2023-04-22 10:40:32 +02:00
blacklight added 8 commits 2023-04-22 17:19:29 +02:00
ebe79ac29a
Refactored system schema dataclasses. 
- `percent_field` should be declared on `platypush.schemas.dataclasses`
  level, since it's not specific to the `system` plugin.
- Added a common `SystemBaseSchema` that takes care of calling
  `_asdict()` if the object is passed as a `psutil` object instead of a
  dict.
blacklight added 2 commits 2023-04-23 00:09:12 +02:00
blacklight added 3 commits 2023-04-23 00:44:27 +02:00
blacklight added 2 commits 2023-04-23 02:09:39 +02:00
blacklight added 2 commits 2023-04-23 02:13:55 +02:00
blacklight added 2 commits 2023-04-23 13:07:52 +02:00
blacklight changed title from WIP: [#29] Support for general-purpose entities to [#29] Support for general-purpose entities 2023-04-23 14:55:52 +02:00
blacklight commented 2023-04-23 22:58:59 +02:00
Author
Owner
Copy Link

This PR has already been open for more than a year. Since most of the planned foundational features have now been implemented, this PR can be closed. New tickets have been created for the remaining features.

This PR has already been open for more than a year. Since most of the planned foundational features have now been implemented, this PR can be closed. New tickets have been created for the remaining features.
blacklight closed this pull request 2023-04-23 22:59:12 +02:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
architecture

   
bug

   
ci/cd

   
cleanup

   
documentation

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
in progress

   
invalid

   
new feature

   
packaging

   
question

   
ui

   
waiting user input

   
wontfix
No Label
architecture
bug
ci/cd
cleanup
documentation
duplicate
enhancement
good first issue
help wanted
in progress
invalid
new feature
packaging
question
ui
waiting user input
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
blacklight
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: platypush/platypush#230
No description provided.
Delete Branch "29-generic-entities-support"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?