Music automation article

markdown/Automate-your-music-collection.md

@@ -3,6 +3,7 @@
 [//]: # (image: /img/music-automation.png)
 [//]: # (author: Fabio Manganiello <fabio@platypush.tech>)
 [//]: # (published: 2022-09-19)
+[//]: # (latex: 1)
 
 I have been an enthusiastic user of mpd and mopidy for nearly two decades. I
 have already [written an
@@ -27,16 +28,16 @@ The problem is that these services come with heavy trade-offs:
 1. Their algorithms are closed. You don't know how Spotify figures out which
    songs should be picked in your smart playlists. In the past months, Spotify
    would often suggest me tracks from the same artists that I had already
-   listened to or skipped in the past, and there's no easy way to tell the
+   listened to or skipped in the past, and there's no transparent way to tell
-   algorithm "hey, actually I'd like you to suggest me more this kind of music -
+   the algorithm "hey, actually I'd like you to suggest me more this kind of
-   or maybe calculate suggestions only based on the music I've listened to in
+   music - or maybe calculate suggestions only based on the music I've listened
-   this time range".
+   to in this time range, or maybe weigh this genre more".
 
 2. Those features are tightly coupled with the service you use. If you cancel
    your Spotify subscription, you lose those smart features as well.
    Companies like Spotify use such features as a lock-in mechanism -
-   "you can check out any time you like, but if you do then nobody else will
+   you can check out any time you like, but if you do then nobody else will
-   provide you with our clever suggestions".
+   provide you with their clever suggestions.
 
 After migrating from Spotify to Tidal in the past couple of months (TL;DR:
 Spotify f*cked up their developer experience multiple times over the past
@@ -199,7 +200,7 @@ create table music_track(
 create unique index track_artist_title_idx on music_track(lower(artist), lower(title));
 create index track_artist_idx on music_track(lower(artist));
 
--- music_activity holds the listen history
+-- music_activity holds the listened tracks
 drop table if exists music_activity cascade;
 create table music_activity(
     id          serial not null,
@@ -304,9 +305,10 @@ Now that all the dependencies are in place, it's time to configure the logic to
 store your music activity to your database.
 
 If most of your music activity happens through mpd/mopidy, then storing your
-activity to the database is as simple as creating a hook on a
+activity to the database is as simple as creating a hook on
-[`NewPlayingTrackEvent`](https://docs.platypush.tech/platypush/events/music.html)
+[`NewPlayingTrackEvent`
-that inserts any newly playing track on `tmp_music`. Paste the following
+events](https://docs.platypush.tech/platypush/events/music.html)
+that inserts any new played track on `tmp_music`. Paste the following
 content to a new Platypush user script (e.g.
 `~/.config/platypush/scripts/music/sync.py`):
 
@@ -354,15 +356,15 @@ def on_new_track_playing(event, **_):
     )
 ```
 
-Alternatively, if you also want to sync music activities that happens through
+Alternatively, if you also want to sync music activity that happens on
 other clients (such as the Spotify/Tidal app or web view, or over mobile
 devices), you may consider leveraging Last.fm. Last.fm (or its open alternative
-Libre.fm) are _scrobbling_ websites that are compatible with most of the music
+Libre.fm) is a _scrobbling_ service compatible with most of the music
 players out there. Both Spotify and Tidal support scrobbling, the [Android
 app](https://apkpure.com/last-fm/fm.last.android) can grab any music activity
-on your phone and scrobble, and there are even [browser
+on your phone and scrobble it, and there are even [browser
 extensions](https://chrome.google.com/webstore/detail/web-scrobbler/hhinaapppaileiechjoiifaancjggfjm?hl=en)
-that allow you to record any music activity from any browser tab.
+that allow you to keep track of any music activity from any browser tab.
 
 So an alternative approach may be to send both your mpd/mopidy music activity,
 as well as your in-browser or mobile music activity, to last.fm / libre.fm. The
@@ -993,4 +995,236 @@ Let's put these pieces together in a new user script stored under e.g.
 
 ```python
 # ~/.config/platypush/scripts/music/releases.py
+
+import html
+import logging
+import re
+import threading
+from datetime import date, timedelta
+from typing import Iterable, List
+
+from platypush.context import get_plugin
+from platypush.cron import cron
+from platypush.event.hook import hook
+from platypush.message.event.rss import NewFeedEntryEvent
+
+from scripts.music.db import (
+    music_db_engine, get_db_session, NewRelease
+)
+
+
+create_lock = threading.RLock()
+logger = logging.getLogger(__name__)
+
+
+def _split_html_lines(content: str) -> List[str]:
+    """
+    Utility method used to convert and split the HTML lines reported
+    by the RSS feed.
+    """
+    return [
+        l.strip()
+        for l in re.sub(
+            r'(</?p[^>]*>)|(<br\s*/?>)',
+            '\n',
+            content
+        ).split('\n') if l
+    ]
+
+
+def _get_summary_field(title: str, lines: Iterable[str]) -> str | None:
+    """
+    Parse the fields of a new album from the feed HTML summary.
+    """
+    for line in lines:
+        m = re.match(rf'^{title}:\s+(.*)$', line.strip(), re.IGNORECASE)
+        if m:
+            return html.unescape(m.group(1))
+
+
+@hook(NewFeedEntryEvent, feed_url='https://newalbumreleases.net/category/cat/feed/')
+def save_new_release(event: NewFeedEntryEvent, **_):
+    """
+    This hook is triggered whenever the newalbumreleases.net has new entries.
+    """
+    # Parse artist and album
+    summary = _split_html_lines(event.summary)
+    artist = _get_summary_field('artist', summary)
+    album = _get_summary_field('album', summary)
+    genre = _get_summary_field('style', summary)
+
+    if not (artist and album):
+        return
+
+    # Check if we have listened to this artist at least once
+    db = get_plugin('db')
+    num_plays = int(
+        db.select(
+            engine=music_db_engine,
+            query=
+            '''
+            select count(*)
+            from music_activity a
+            join music_track t
+            on a.track_id = t.id
+            where artist = :artist
+            ''',
+            data={'artist': artist},
+        ).output[0].get('count', 0)
+    )
+
+    # If not, skip it
+    if not num_plays:
+        return
+
+    # Insert the new release on the database
+    with create_lock:
+        db.insert(
+            engine=music_db_engine,
+            table='new_release',
+            records=[{
+                'artist': artist,
+                'album': album,
+                'genre': genre,
+            }],
+            key_columns=('artist', 'album'),
+            on_duplicate_update=True,
+        )
+
+
+def get_new_releases(days=7):
+    """
+    Retrieve the new album releases from the database.
+
+    :param days: Look at albums releases in the past <n> days
+        (default: 7)
+    """
+    with get_db_session() as session:
+        return [
+            {
+                'artist': t[0],
+                'album': t[1],
+            }
+            for t in session.query(
+                NewRelease.artist,
+                NewRelease.album,
+            )
+            .select_from(
+                NewRelease.__table__
+            )
+            .filter(
+                # Filter by recent activity
+                NewRelease.created_at >= date.today() - timedelta(days=days)
+            )
+            .all()
+        ]
+
+
+def search_tidal_new_releases(albums):
+    """
+    Search for Tidal albums given a list of objects with artist and title.
+    """
+    tidal = get_plugin('music.tidal')
+    expanded_tracks = []
+
+    for album in albums:
+        query = album['artist'] + ' ' + album['album']
+        logger.info('Searching "%s"', query)
+        results = (
+            tidal.search(query, type='album', limit=1)
+            .output.get('albums', [])
+        )
+
+        if results:
+            album = results[0]
+
+            # Skip search results older than a year - some new releases may
+            # actually be remasters/re-releases of existing albums
+            if date.today().year - album.get('year', 0) > 1:
+                continue
+
+            expanded_tracks += (
+                tidal.get_album(results[0]['id']).
+                output.get('tracks', [])
+            )
+        else:
+            logger.warning('Could not find "%s" on TIDAL', query)
+
+    return expanded_tracks
+
+
+def refresh_release_radar():
+    tidal = get_plugin('music.tidal')
+
+    # Get the latest releases
+    tracks = search_tidal_new_releases(get_new_releases())
+    if not tracks:
+        logger.info('No new releases found')
+        return
+
+    # Retrieve the existing new releases playlists
+    playlists = tidal.get_playlists().output
+    new_releases_playlists = sorted(
+        [
+            pl for pl in playlists
+            if pl['name'].lower().startswith('new releases')
+        ],
+        key=lambda pl: pl.get('created_at', 0)
+    )
+
+    # Delete all the existing new releases playlists
+    # (except the latest one)
+    for playlist in new_releases_playlists[:-1]:
+        logger.info('Deleting playlist "%s"', playlist['name'])
+        tidal.delete_playlist(playlist['id'])
+
+    # Create a new releases playlist
+    playlist_name = f'New Releases [{date.today().isoformat()}]'
+    pl = tidal.create_playlist(playlist_name).output
+    playlist_id = pl['id']
+
+    tidal.add_to_playlist(
+        playlist_id,
+        [t['id'] for t in tracks],
+    )
+
+
+@cron('0 7 * * 1')
+def refresh_release_radar_cron(**_):
+    """
+    This cron will execute every Monday at 7 AM.
+    """
+    try:
+        refresh_release_radar()
+    except Exception as e:
+        logger.exception(e)
+        get_plugin('ntfy').send_message(
+            topic='mirrored-notifications-topic',
+            title='Release Radar playlist generation failed',
+            message=str(e),
+            priority=4,
+        )
 ```
+
+Just like in the previous case, it's quite easy to test that it works by simply
+running `refresh_release_radar_cron` in the Python interpreter. Just like in
+the case of the discovery playlist, things will work also if you use Spotify
+instead of Tidal - just replace the `music.tidal` plugin references with
+`music.spotify`.
+
+If it all goes as expected, you will get a new playlist named _New Releases
+[date]_ every Monday with the new releases from artist that you have listened.
+
+## Conclusions
+
+Music junkies have the opportunity to discover a lot of new music today without
+ever leaving their music app. However, smart playlists provided by the major
+music cloud providers are usually implicit lock-ins, and the way they select
+the tracks that should end up in your playlists may not even be transparent, or
+even modifiable.
+
+After reading this article, you should be able to generate your discovery and
+new releases playlists, without relying on the suggestions from a specific
+music cloud. This could also make it easier to change your music provider: even
+if you decide to drop Spotify or Tidal, your music suggestions logic will
+follow you whenever you decide to go.