Media storage implementation #4

Merged

blacklight merged 19 commits from feat/media-storage into main 2026-08-20 23:35:21 +02:00

Conversation 0 Commits 19 Files changed 70 +6017 -197

blacklight commented 2026-08-19 20:16:47 +02:00

Owner

Copy link

• Dependencies, config, and interface
• LocalStorage — aiofiles and url()
• S3Storage — aioboto3, multipart, and url()
• StoredFile model and relationships
• StorageService and dependency injection
• Garbage collection task and beat schedule
• File upload, metadata, and download API endpoints
• ACL management for media items access

Closes: #3

- [x] Dependencies, config, and interface - [x] LocalStorage — aiofiles and `url()` - [x] S3Storage — aioboto3, multipart, and `url()` - [x] `StoredFile` model and relationships - [x] `StorageService` and dependency injection - [x] Garbage collection task and beat schedule - [x] File upload, metadata, and download API endpoints - [x] ACL management for media items access Closes: #3

blacklight added the

new feature

storage

backend

labels 2026-08-19 20:16:47 +02:00

blacklight self-assigned this 2026-08-19 20:16:47 +02:00

blacklight added 1 commit 2026-08-19 20:16:47 +02:00

chore(gitignore): Ignore docs/features 4abac8b65c

blacklight force-pushed feat/media-storage from 4abac8b65c to c35ba22e62 2026-08-19 20:18:31 +02:00 Compare

blacklight added 1 commit 2026-08-19 20:30:44 +02:00

refactor(storage): make LocalStorage async and block path traversal ... 552bdcb4fa

- Use aiofiles for non-blocking store/retrieve/delete/exists
- Add strict path resolver to prevent escaping base directory
- Add tests for large streaming writes and traversal rejection

blacklight added 1 commit 2026-08-19 22:09:35 +02:00

feat(storage): enforce max upload size and stream S3 uploads ... e0b9d31606

- Add max_upload_size to storage config and backends
- Enforce limits during local writes with cleanup on failure
- Switch S3 backend to async aioboto3 with multipart streaming and temp retrieval
- Expand config and storage tests for new behavior

blacklight added 1 commit 2026-08-19 22:58:30 +02:00

feat: add StoredFile model and wire media file foreign keys ... 4cfa66c3d0

- Add StoredFile model and export from models package
- Link Album.cover_file, Track.audio_file, and Upload.stored_file via FKs
- Extend model tests to cover new fields and StoredFile persistence

blacklight added 1 commit 2026-08-19 23:26:48 +02:00

feat(storage): add content-addressable StorageService and DI helper ... 8cb867f61c

- Add StorageService for hashing, dedup, URL, and delete operations
- Wire storage service dependency in API deps
- Add tests covering hashing, dedup, streaming, URLs, and delete

blacklight added 1 commit 2026-08-20 00:00:33 +02:00

feat(tasks): add scheduled cleanup for orphaned stored files ... 0601eb2d10

- add storage task to delete unreferenced StoredFile rows and backing files
- add Celery config for cleanup schedule and route storage tasks to queue
- register beat schedule for daily cleanup (default 03:00 UTC)
- add tests for cleanup behavior and schedule parsing

blacklight added 2 commits 2026-08-20 13:27:39 +02:00

feat(api): add stored file upload/metadata/download endpoints ... bed0b42126

- Add /files routes with safe Content-Disposition handling
- Cache StorageService per app and refresh on storage config changes
- Add test coverage for auth, range requests, sanitization, and 404s

fix(storage): map upload size limit to 413 response ... a020091a2c

- Add FileSizeLimitExceededError and raise from storage size checks
- Catch size limit error in upload route and return HTTP 413
- Treat SVG with parameters as unsafe for inline serving

blacklight added 1 commit 2026-08-20 13:46:04 +02:00

fix(storage): Addressed an off-by-one error in orphaned file cleanup count a04e90df72

blacklight added 2 commits 2026-08-20 14:47:41 +02:00

feat(models): add visibility and sharing primitives ... 1f3d2a883a

- Add Visibility enum and apply to media models
- Add ShareGrant and ShareToken models
- Add owner_id/visibility to stored files, tracks, and albums
- Update tests to cover defaults and new share models

feat: add ACL and sharing services for media visibility ... c5de4b96a7

- Implement access rules (private/local/public), grants, and share tokens
- Support derived file access via tracks/albums with depth guard
- Add unit tests for ACL and sharing behavior

blacklight added 1 commit 2026-08-20 14:53:20 +02:00

feat: propagate library owner to imported media and stored files ... 105138c5e9

- Set owner_id on imported tracks and newly created albums
- Apply owner_id/visibility when creating StoredFile rows; preserve on dedup
- Infer owner from Library in import task
- Add importer/storage tests for owner and visibility behavior

blacklight added 1 commit 2026-08-20 15:18:34 +02:00

feat(api): add optional auth and ACL access dependency ... 620e451fb2

- Add get_current_user_optional and shared _get_current_user helper
- Add require_access() dependency with 404/403 behavior and share-token support
- Refactor ACL registry and add get_item() helper
- Add dependency tests for optional auth and access control

blacklight added 1 commit 2026-08-20 19:23:42 +02:00

feat(api): add ACL-aware media endpoints with visibility + owner redaction ... 27132e54c6

- Implement list/get/create for albums, tracks, libraries, playlists, radios
- Add shared owner_id redaction helper and apply ACL filters to list queries
- Extend file upload with visibility/owner, duplicate header, and async hashing
- Add API tests covering visibility filtering, access control, and duplicates

blacklight added 1 commit 2026-08-20 20:10:04 +02:00

feat(sharing): add share grants and revocable share URLs ... 52cb9dc418

- Add /shares, /share-urls CRUD and public /share/{token} resolver
- Centralize item-type validation and management authorization checks
- Support instance_domain share URL host and guard federation setup
- Extend sharing/acl helpers and add API tests for shares and tokens

blacklight added 1 commit 2026-08-20 21:14:26 +02:00

fix(federation): gate Audio activities by visibility ... 398aa154c9

- Skip Create(Audio) and Audio serialization for non-public tracks
- Use public file download URL for audio stream links
- Add helpers for track/stream URL building and update tests

blacklight added 1 commit 2026-08-20 21:43:54 +02:00

fix(acl): harden share tokens and adjust access tests ... 0bfb47917b

- Constant-time share token hash comparison
- Remove unused filter_accessible_ids helper + tests
- Add share-token access coverage for private item endpoints
- Test derived file download via public track visibility

blacklight added 2 commits 2026-08-20 23:26:25 +02:00

fix(storage): Set max_upload_size to a default of 500 MB 74fc2bf66f

fix(security): harden media storage, sharing and rate limiting ... 26a59aace1

- Enforce default max upload size and secure temporary file permissions
- Fix content-addressable dedup and orphan cleanup race conditions
- Redact internal storage paths and remove ownerless private access
- Move share tokens to headers/cookies and add rate limiting
- Distinguish S3 404 errors and restrict inline serving allowlist

blacklight changed title from WIP: Media storage implementation to Media storage implementation 2026-08-20 23:35:11 +02:00

blacklight merged commit 39aa3aed4c into main 2026-08-20 23:35:21 +02:00

blacklight deleted branch feat/media-storage 2026-08-20 23:35:21 +02:00

blacklight referenced this pull request from a commit 2026-08-20 23:35:22 +02:00

Merge pull request 'Media storage implementation' (#4) from feat/media-storage into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

api

API-related changes

  

auth

Authorization and authentication

  

backend

Backend-related tasks

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

Enhancement over an existing feature

  

federation

ActivityPub federation tasks

  

frontend

Frontend-related tasks

  

help wanted

Need some help

  

invalid

Something is wrong

  

new feature

New feature requests

  

question

More information is needed

  

storage

Media storage tasks

  

stream

Media streaming tasks

  

wontfix

This won't be fixed

No labels

api

auth

backend

bug

duplicate

enhancement

federation

frontend

help wanted

invalid

new feature

question

storage

stream

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

blacklight

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

blacklight/songhive!4

No description provided.