platypush/platypush
platypush
/
platypush
2
2
Fork 4
Code Issues 44 Pull Requests 1 Packages Projects 4 Releases Wiki Activity

[#304] Rewrite Picovoice integrations #385

Open
blacklight wants to merge 35 commits from 304-new-picovoice-integration into master
pull from: 304-new-picovoice-integration
merge into: platypush:master
Conversation 0 Commits 35 Files Changed 49 +2537 -805
blacklight commented 2024-04-06 00:13:20 +02:00
Owner
Copy Link
  • Implement Porcupine hotword trigger interface
  • Implement Cheetah speech-to-text interface
  • Docs plugin icon
    UI plugin icon
  • Test conversation sound
  • Intents (Rhino)
  • Offline transcriptions (Leopard)
  • TTS
  • Start/stop conversation actions
  • Mic mute/unmute
  • Implement with_follow_on_turn
  • Implement send_text_query
  • Add OpenAI plugin
  • Implement temporary workaround for unsupported handling of digits and punctuation on orca

Closes: #304

* [x] Implement Porcupine hotword trigger interface * [x] Implement Cheetah speech-to-text interface * [x] Docs plugin icon ~~UI plugin icon~~ * [x] Test conversation sound * [x] Intents (Rhino) * [x] Offline transcriptions (Leopard) * [x] TTS * [x] Start/stop conversation actions * [x] Mic mute/unmute * [x] Implement `with_follow_on_turn` * [x] Implement `send_text_query` * [x] Add OpenAI plugin * [x] Implement temporary workaround for unsupported handling of digits and punctuation on `orca` Closes: #304
blacklight added the
enhancement
 label 2024-04-06 00:13:20 +02:00
blacklight self-assigned this 2024-04-06 00:13:20 +02:00
blacklight added 1 commit 2024-04-06 00:13:21 +02:00
continuous-integration/drone/push Build is passing Details
f0382c73ab
[#304] Removed old Picovoice integrations
blacklight added this to the integrations project 2024-04-06 00:13:21 +02:00
blacklight added 1 commit 2024-04-07 22:42:23 +02:00
continuous-integration/drone/push Build is passing Details
01dec0b7a4
[WIP] Added initial hotword integration.
blacklight added 1 commit 2024-04-08 01:54:55 +02:00
continuous-integration/drone/push Build is passing Details
f021b471aa
[WIP] Added speech detection logic over Cheetah.
blacklight added 1 commit 2024-04-08 03:02:37 +02:00
blacklight added 2 commits 2024-04-08 13:33:20 +02:00
blacklight added 7 commits 2024-04-13 20:01:35 +02:00
6feb824c04
Refactored `AssistantEvent`. 
`AssistantEvent.assistant` is now modelled as an opaque object that
behaves the following way:

- The underlying plugin name is saved under `event.args['_assistant']`.

- `event.assistant` is a property that returns the assistant instance
  via `get_plugin`.

- `event.assistant` is reported as a string (plugin qualified name) upon
  event dump.

This allows event hooks to easily use `event.assistant` to interact with
the underlying assistant and easily modify the conversation flow, while
event hook conditions can still be easily modelled as equality
operations between strings.
3b73b22db9
[assistant.picovoice] More features. 
- Added wiring between `assistant.picovoice` and `tts.picovoice`.

- Added `RESPONDING` status to the assistant.

- Added ability to override the default speech model upon
  `start_conversation`.

- Better handling of conversation timeouts.

- Cache Cheetah objects in a `model -> object` map - at least the
  default model should be pre-loaded, since model loading at runtime
  seems to take a while, and that could impact the ability to detect the
  speech in the first seconds after a hotword is detected.
24e93ad160
Added more default imports under the `platypush` module root. 
These objects can now also be imported in scripts through
`from platypush import <name>`:

- `Variable`
- `cron`
- `hook`
- `procedure`
continuous-integration/drone/push Build is passing Details
2b287b569f
[assistant.picovoice] Conversation flow improvements. 
- The `Responding` state should be modelled as an extra event/binary
  flag, not as an assistant state. The assistant may be listening for
  hotwords even while the `tts` plugin is responding, and we don't want
  the two states to interfere with each either - neither to build a more
  complex state machine that also needs to take concurrent states into
  account.

- Stop any responses being rendered upon the `tts` plugin when a new
  hotword audio is detected. If e.g. I say "Ok Google", I should always
  be able to trigger the assistant and stop any concurrent audio
  process.

- `SpeechRecognizedEvent` should be emitted even if `cheetah`'s latest
  audio frame results weren't marked as final, and the speech detection
  window timed out. Cheetah's `is_final` detection seems to be quite
  buggy sometimes, and it may not properly detect the end of utterances,
  especially with non-native accents. The workaround is to flush out
  whatever text is available (if at least some speech was detected) into
  a `SpeechRecognizedEvent` upon timeout.
blacklight force-pushed 304-new-picovoice-integration from 2b287b569f to fa49db4107 2024-04-13 20:04:00 +02:00 Compare
blacklight added 1 commit 2024-04-13 20:04:35 +02:00
continuous-integration/drone/push Build is passing Details
19a90ee045 Merge branch 'master' into 304-new-picovoice-integration
blacklight added 1 commit 2024-04-13 20:51:41 +02:00
blacklight added 2 commits 2024-04-13 21:40:49 +02:00
c86e3be0d1
[assistant.picovoice] Leverage upstream `_on_mute_changed`. 
The plugin should leverage `AssistantPlugin._on_mute_changed` to handle
the boilerplate state managent on mute/unmute actions instead of
re-implementing the same logic.
continuous-integration/drone/push Build is passing Details
7e9b19d328
[tts.picovoice] Added text pre-processing workaround. 
This workaround is required until
https://github.com/Picovoice/orca/issues/10 is fixed.
blacklight added 1 commit 2024-04-14 00:10:09 +02:00
blacklight added 2 commits 2024-04-14 01:07:45 +02:00
blacklight added 3 commits 2024-04-15 23:02:59 +02:00
317241eb36
[assistant.picovoice] Various improvements. 
- Added `intent_model_path` parameter.

- Always apply `expanduser` to configuration paths.

- Better logic to infer the fallback model path.

- The Picovoice Leonardo object should always be removed after
  `assistant.picovoice.transcribe` is called.
continuous-integration/drone/push Build is passing Details
7c504685e2
Prevent a potential recursion error in `wait_for_either`. 
We shouldn't overwrite `event._set` and `event._clear` if those values
have already been set.

Those attributes hold the original references to `Event.set` and
`Event.clear` respectively, and the `OrEvent` logic overwrites them with
a callback-based logic.

This shouldn't happen if those attributes are already present.
blacklight added 3 commits 2024-04-20 17:26:25 +02:00
continuous-integration/drone/push Build is passing Details
8df34a13f1
New architecture for the assistant speech detection logic. 
The assistant object now runs in its own thread and leverages an
external `SpeechProcessor` that uses two threads to scan for both
intents and speech in parallel on audio frames.
blacklight added 1 commit 2024-04-20 17:27:18 +02:00
continuous-integration/drone/push Build is failing Details
4b41c5377a Merge branch 'master' into 304-new-picovoice-integration
blacklight added 2 commits 2024-04-22 02:05:49 +02:00
862d56a338
Prevent duplicate hook runs. 
Instead of being a list, the hooks in the hook processor should be
backed by by-name and by-value maps.

Don't insert a hook if its exact backing method has already been
inserted. This is actually very common when hooks are defined as Python
snippets imported in other scripts too.
blacklight added 2 commits 2024-04-23 21:03:27 +02:00
blacklight added 3 commits 2024-04-24 01:13:47 +02:00
6b20c2083b
🐛 The `EventMatchResult` object shouldn't be initialized with `args` from the event. 
If there's a good use-case for overriding `Event._matches_condition`
with a logic that also parses the event arguments, then those arguments
should be accessed directly from the event object, not from the match
result.

Initializing `EventMatchResult` with the arguments from the event means
that, if `EventMatchResult.parsed_args` are populated with custom
extracted arguments, then the upstream event arguments will also be
modified.

If the event is matched against multiple conditions, this will result in
the extracted tokens getting modified by each `matches_condition`
iteration.
continuous-integration/drone/push Build is failing Details
47926290a2
[music.mopidy] Handle the case where `add` returns a list.
blacklight added 1 commit 2024-04-24 01:28:21 +02:00
continuous-integration/drone/push Build is passing Details
bd20c53c32
Fixed tests
blacklight changed title from WIP: [#304] Rewrite Picovoice integrations to [#304] Rewrite Picovoice integrations 2024-04-26 00:47:06 +02:00
All checks were successful
continuous-integration/drone/push Build is passing
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b 304-new-picovoice-integration master
git pull origin 304-new-picovoice-integration

Step 2:

Merge the changes and update on Forgejo.
git checkout master
git merge --no-ff 304-new-picovoice-integration
git push origin master
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
architecture

   
bug

   
ci/cd

   
cleanup

   
documentation

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
in progress

   
invalid

   
new feature

   
packaging

   
question

   
ui

   
waiting user input

   
wontfix
No Label
architecture
bug
ci/cd
cleanup
documentation
duplicate
enhancement
good first issue
help wanted
in progress
invalid
new feature
packaging
question
ui
waiting user input
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
integrations
Assignees
Clear assignees
No Assignees
blacklight
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: platypush/platypush#385
No description provided.
Delete Branch "304-new-picovoice-integration"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?