WIP: Notes sync implementation #458

Draft

blacklight wants to merge 54 commits from 457/notes-sync into master

pull from: 457/notes-sync

merge into: platypush:master

platypush:master

platypush:fix-zigbee-mqtt

platypush:stable

platypush:286/merge-foursquare-backend

platypush:271/runtime-integrations

platypush:317-ci-cd-automation-for-stable-branch

platypush:311/auto-generate-deps-docs

Conversation 0 Commits 54 Files changed 14 +2902 -215

blacklight commented 2025-08-24 20:24:32 +02:00

Owner

Copy link

No description provided.

blacklight added this to the v1.4.0 milestone 2025-08-24 20:24:32 +02:00

blacklight added the

notes

backend

new feature

labels 2025-08-24 20:24:32 +02:00

blacklight self-assigned this 2025-08-24 20:24:32 +02:00

blacklight added 27 commits 2025-08-24 20:24:33 +02:00

[notes] Added path and content_type to notes bcb442422b

[notes] Added _[notes|collections]_by_path utility property 28ee30d8c4

[notes] Added sync state objects bad5577664

[notes] Added notes_sync_state table ... b0c2a8d27c

Models the [local_note] <--synced_from-- [remote_note] relationship

[notes] Added sync state mappings to the notes model 18258dec7a

[notes] Renaming and refactor ... 44f768f4e3

- Refactor BaseNotePlugin to replace `_last_sync_time` with
  `_last_local_sync_time` and update related methods for consistency.

- Introduce sync fields in NoteItemSchema for tracking synced notes,
  ensuring minimal information is returned during serialization. This
  improves clarity in synchronization processes and reinforces naming
  conventions across the codebase.

[notes]: fold_records should also include synced records 84f71054cd

[notes] notes.fold_records should use db IDs as keys ... f3515c1f4d

This is now required because `fold_records` may contain records from
multiple plugins, since synced notes are persisted, and those may have
the same logical IDs.

[joplin] Properly propagate parent collection f99eaa9b4c

[notes] Added path to collection schema 59a1485536

[notes] fix: Path ordering in collection 4978d3f395

[notes] Naming convention for conflict notes 4c0f7f958d

Merge branch 'master' into 457/notes-sync ... 1076107fe0

* master:
  feat(core): Added `--debug-sql` CLI option
  [Automatic] Updated components cache

[notes] Return a minimal version of nested notes on schemas ... f662049979

This makes responses to e.g. `get_notes` smaller and prevents loops.

[notes] Keep track of visited objects when recursively converting items ... 09c3c9ec6c

This prevents recursion loops in the `_[to|from]_db_note` methods.

This commit also adds a more explicit flush/commit behaviour to db
sessions.

[notes] Replace conflict_note with conflict_notes ... b626344747

Update relationships in the database model to accommodate these changes
and ensure proper synchronization.

[notes] Ensure that content is always set on [create|edit]_note ... 0b8a268d70

Also in this commit:

- Defined the `infer_id` method for the local notes plugin (a local note
  ID is always its path).

- Call `stop_remote_sync` in the local note plugin when it stops.

[notes] Initial notes sync implementation 50e3ad8e75

[notes] Don't process notes that haven't been updated since the last sync 634e26026c

[notes] Added with_content parameter to get_notes 9362ac3282

[notes] Make Note.path a deterministic property da511c8095

[notes] Automatically update Note.digest when content is set 3484708901

[notes] Added/updated model methods ... 8c0982c9f3

- `build`
- `to_dict`
- `__eq__`

[notes] Copied documentation of sync_from upstream 1e44ba4ff1

[notes] Dynamic initialization of model/db objects in db layer e8e5c6f91e

[notes] Added sync_remote_deletions and failsafe_delete_threshold options to sync d4ab03faca

[notes] Implemented continuous sync logic 3fc8e83250

blacklight added this to the integrations project 2025-08-24 20:24:33 +02:00

blacklight added 2 commits 2025-08-24 22:14:11 +02:00

[notes] The local _merge_note method should not overwrite external relationships d194ebc0c2

[notes] Converted some object methods to static c6fe5a7618

blacklight added 1 commit 2025-08-24 22:15:38 +02:00

[notes] Preserve note references on sync 18f865ce7f

blacklight added 3 commits 2025-08-25 00:54:22 +02:00

[db] Always rollback a session in get_session upon failure ... f9f28ed5e7

Otherwise, especially on SQLite, there's a high chance of leaving behind
a locked database.

This commit also fixes some minor style warnings in the db module.

[notes] Run deduplication on the state delta before persisting it ... 634dd64a17

This is to prevent cases where e.g. a draft note is deleted and a new
one is persisted under the same path (but with different IDs).

Other plugins rely on the full path of a note to infer their
synchronization state. Such cases result in state deltas that have both
added and deleted events for the same path, which can result in
undefined behaviour.

It is instead required to scan the state delta before processing it and
group changed notes by path, and then pick only the most recently
updated one.

[notes] Always ensure that sync_from and sync_to are lists of objects e019721e88

blacklight added 5 commits 2025-08-26 21:46:51 +02:00

[notes] Better to_dict methods for note models c3ba6c348b

[notes] Better management of _sync_lock in _get_notes 78282a22ae

[notes] Fixed many-to-many joins in the Notes class 8d051ec7fe

[notes] _from_db_note should always rehydrate source as a NoteSource object a058d9fff2

[notes] Always refresh the local notes cache upon remote sync 7378f6a77e

blacklight added 2 commits 2025-08-29 19:49:05 +02:00

[notes] Enforce with_content setting on get_notes 0d2eaa28e0

[notes] Added conflicting_for property to notes ... 69b01fc20d

If the note is a _virtual_ note, and it's actually a copy of a remote
note that a locally synced one is in conflict with, then
`conflicting_for` will contain the local conflicting copy, while the
current note contains a copy of the remote reference.

If a note has `conflicting_for` set, it will also have
`is_conflict_note` set to True.

blacklight force-pushed 457/notes-sync from 69b01fc20d to dbc685ce45 2025-08-29 19:52:02 +02:00 Compare

blacklight added 5 commits 2025-09-04 01:10:02 +02:00

[notes] Renamed methods ... d25979a944

_merge_remote_note_relations -> _merge_note_relations

Since it can be called from any context where we need to merge an
incoming note without relations with a synced copy with the properly
stored relations.

[notes] Updated conflicting_for relations on Note ... 5530c137fd

A note can have 0-1 `conflicting_for` relations

[notes] Minor refactors 9d741953e2

[notes] Note model class improvements ... c2f5ce7d87

- `Note.build` should initialize `conflicting_for` separately.
- `Note.__eq__` shouldn't compare attributes like `id` and `plugin`
  which will probably differ if a note is remotely synchronized.

[notes] Refactored sync mixin ... 4fc18d3861

- Added a new prefix for conflict note titles.
- Created methods to update existing conflict notes and create new
  conflict notes.
- Refactored `_merge_notes` to utilize the new conflict handling
  methods.
- Improved readability and maintainability by extracting merging logic
  into dedicated methods.

blacklight added 1 commit 2025-09-04 01:17:45 +02:00

Merge branch 'master' into 457/notes-sync ... 8f5af57005

* master:
  fix(ntfy): Variable name clash in `ntfy.send_message`

blacklight added 1 commit 2025-09-04 20:49:55 +02:00

fix(db): Properly handle on_duplicate_update on MariaDB ... a68bf48245

MariaDB does not support the RETURNING clause, which causes `db.insert`
calls with `on_duplicate_update` to fail.

This commit adds error handling to manage this case gracefully, just
like it's done for SQLite.

blacklight added 1 commit 2025-09-06 01:08:38 +02:00

Refactor notes handling in the Platypush project. ... 561dfe381a

- Added logger for debugging in notes.py.
- Moved infer ID property to the Note class.
- Updated digest computation to handle trimmed content in Note class.
- Excluded specific fields (`synced_from`, `synced_to`,
  `conflict_notes`, `conflicting_for`) from database note attributes.
- Improved note synchronization process in SyncMixin by adjusting field
  merges and ensuring proper conflict detection.
- Enhanced conflict resolution logging to assist with debugging.

blacklight added 3 commits 2025-09-10 00:42:46 +02:00

[notes] Removed unused _CONFLICT_NOTE_TITLE_PREFIX constant 773521d5d7

Merge pull request 'fix(ui): Fixed handling of focus/blur/submit events in Autocomplete and ActionEditor' (#461) from 460/fix-focus-management-in-action-editor into master ... 2093e7dc83

Reviewed-on: #461

Merge branch 'master' into 457/notes-sync ... d5f4c2fbaa

* master:
  fix(ui): Fixed handling of focus/blur/submit events in Autocomplete and ActionEditor
  Bump version: 1.3.6 → 1.3.7
  Updated CHANGELOG
  [Automatic] Updated components cache
  fix(db): Properly handle `on_duplicate_update` on MariaDB
  fix(db): Use sqlalchemy.Uuid instead of sqlalchemy.UUID
  [Automatic] Updated components cache
  A marshmallow schema shouldn't extend ABC
  [Automatic] Updated components cache

blacklight added 2 commits 2025-09-10 11:12:35 +02:00

Merge branch 'master' into 457/notes-sync ... 404fa9bfa8

* master:
  fix(ui): Better handling of focus after action execution
  fix(ui): Fixed some Vue layout warnings
  [Automatic] Updated UI files

Merge branch 'master' into 457/notes-sync ... 5f53ecb336

* master:
  Improve payload handling in webhook route
  [Automatic] Updated UI files

blacklight added 1 commit 2025-11-06 20:32:10 +01:00

whatever 568b2d5fd9

blacklight removed this from the v1.4.0 milestone 2025-12-19 02:02:11 +01:00

This pull request has changes conflicting with the target branch.

• platypush/plugins/notes/_base.py
• platypush/plugins/notes/mixins/db.py
• platypush/plugins/notes/mixins/index.py

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin 457/notes-sync:457/notes-sync

git switch 457/notes-sync

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master

git merge --no-ff 457/notes-sync

git switch 457/notes-sync

git rebase master

git switch master

git merge --ff-only 457/notes-sync

git switch 457/notes-sync

git rebase master

git switch master

git merge --no-ff 457/notes-sync

git switch master

git merge --squash 457/notes-sync

git switch master

git merge --ff-only 457/notes-sync

git switch master

git merge 457/notes-sync

git push origin master

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

architecture

  

authentication

  

backend

  

bug

  

ci/cd

  

cleanup

  

documentation

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

in progress

  

invalid

  

media

  

mobile

  

new feature

  

notes

Tickets about notes plugins

  

packaging

  

question

  

ui

  

upstream breaking change

  

voice

  

waiting user input

  

wontfix

No labels

architecture

authentication

backend

bug

ci/cd

cleanup

documentation

duplicate

enhancement

good first issue

help wanted

in progress

invalid

media

mobile

new feature

notes

packaging

question

ui

upstream breaking change

voice

waiting user input

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

integrations

Assignees

Clear assignees

No assignees

blacklight

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

platypush/platypush!458

No description provided.