micmon ====== *micmon* is a ML-powered library to detect sounds in an audio stream, either from a file or from an audio input. The use case for its development has been the creation of a self-built baby monitor to detect the cries of my new born through a RaspberryPi + USB microphone, but it should be good enough to detect any type of noise or audio if used with a well trained model. It works by splitting an audio stream into short segments, it calculates the FFT and spectrum bins for each of these segments, and it uses such spectrum data to train a model to detect the audio. It works well with sounds that are loud enough to stand out of the background (it's good at detecting e.g. the sound of an alarm clock, not the sound of flying mosquitto), that are long enough compared to the size of the chunks (very short sounds will leave a very small trace in the spectrum of an audio chunk) and, even better, if their frequency bandwidth doesn't overlap a lot with other sounds (it's good at detecting the cries of your baby, since his/her voice has a higher pitch than yours, but it may not detect difference in the spectral signature of the voice of two adult men in the same age group). It's not going to perform very well if instead you are trying to use to detect speech - since it operates on time-agnostic frequency data from chunks of audio it's not granular enough for proper speech-to-text applications, and it wouldn't be robust enough to detect differences in voice pitch, tone or accent. Dependencies ------------ The software uses *ffmpeg* to record and decode audio - check instructions for your OS on how to get it installed. It also requires *lame* or any other mp3 encoder to encode captured audio to mp3. Python dependencies: ```bash # On Debian-based systems apt-get install libatlas-base-dev # Install Tensorflow pip install tensorflow # Optional, for graphs pip install matplotlib ``` Installation ------------ ```bash git clone https://github.com/BlackLight/micmon cd micmon python setup.py install ``` Audio capture ------------- Once the software is installed, you can proceed with recording some audio that will be used for training the model. First create a directory for your audio samples dataset: ```bash # This folder will store our audio samples mkdir -p ~/datasets/sound-detect/audio # This folder will store the datasets # generated from the labelled audio samples mkdir -p ~/datasets/sound-detect/data # This folder will store the generated # Tensorflow models mkdir -p ~/models cd ~/datasets/sound-detect/audio ``` Then create a new sub-folder for your first audio sample and start recording. Example: ```bash mkdir sample_1 cd sample_1 arecord -D plughw:0,1 -f cd | lame - audio.mp3 ``` In the example above we are using *arecord* to record from the second channel of the first audio device (check a list of available recording devices with *arecord -l*) in WAV format, and we are then using the *lame* encoder to convert the raw audio to mp3. When done with recording, just Ctrl-C the application and your audio file will be ready. Audio labelling --------------- In the same directory as your sample (in the example above it will be `~/datasets/sound-detect/audio/sample_1`) create a new file named `labels.json`. Now open your audio file in Audacity or any audio player and identify the audio segments that match your criteria - for example when your baby is crying, when the alarm starts, when your neighbour starts drilling the wall, or whatever the criteria is. `labels.json` should contain a key-value mapping in the form of `start_time -> label`. Example: ```json { "00:00": "negative", "02:13": "positive", "04:57": "negative", "15:41": "positive", "18:24": "negative" } ``` In the example above, all the audio segments between 00:00 and 02:12 will be labelled as negative, all the segments between 02:13 and 04:56 as positive, and so on. You can now use *micmon* to generate a frequency spectrum dataset out of your labelled audio. You can do it either through the `micmon-datagen` script or with your own script. ### micmon-datagen Type `micmon-datagen --help` to get a full list of the available options. In general, `micmon-datagen` requires a directory that contains the labelled audio samples sub-directories as input and a directory where the calculated numpy-compressed datasets will be stored. If you want to generate the dataset for the audio samples captured on the previous iteration then the command will be something like this: ```bash micmon-datagen --low 250 --high 7500 --bins 100 --sample-duration 2 --channels 1 \ ~/datasets/sound-detect/audio ~/datasets/sound-detect/data ``` The `--low` and `--high` options respectively identify the lowest and highest frequencies that should be taken into account in the output spectrum. By default these values are 20 Hz and 20 kHz (respectively the lowest and highest frequency audible to a healthy and young human ear), but you can narrow down the frequency space to only detect the frequencies that you're interested in and to remove high-frequency harmonics that may spoil your data. A good way to estimate the frequency space is to use e.g. Audacity or any audio equalizer to select the segments of your audio that contain the sounds that you want to detect and check their dominant frequencies - you definitely want those frequencies to be included in your range. `--bins` specifies in how many segments/buckets the frequency spectrum should be split - 100 bins is the default value. `--sample-duration` specifies the duration in seconds for each spectrum data point - 2 seconds is the default value, i.e. the audio samples will be read in chunks of 2 seconds each and the spectrum will be calculated for each of these chunks. If the sounds you want to detect are shorter then you may want to reduce this value. ### Generate the dataset via script The other way to generate the dataset from the audio is through the *micmon* API itself. This option also enables you to take a peek at the audio data to better calibrate the parameters. For example: ```python import os from micmon.audio import AudioDirectory, AudioPlayer, AudioFile from micmon.dataset import DatasetWriter basedir = os.path.expanduser('~/datasets/sound-detect') audio_dir = os.path.join(basedir, 'audio/sample_1') datasets_dir = os.path.join(basedir, 'data') cutoff_frequencies = [250, 7500] # Scan the base audio_dir for labelled audio samples audio_dirs = AudioDirectory.scan(audio_dir) # Play some audio samples starting from 01:00 for audio_dir in audio_dirs: with AudioFile(audio_dir, start='01:00', duration=5) as reader, \ AudioPlayer() as player: for sample in reader: player.play(sample) # Plot the audio and spectrum of the audio samples in the first 10 seconds # of each audio file. for audio_dir in audio_dirs: with AudioFile(audio_dir, start=0, duration=10) as reader: for sample in reader: sample.plot_audio() sample.plot_spectrum(low_freq=cutoff_frequencies[0], high_freq=cutoff_frequencies[1]) # Save the spectrum information and labels of the samples to a # different compressed file for each audio file. for audio_dir in audio_dirs: dataset_file = os.path.join(datasets_dir, os.path.basename(audio_dir.path) + '.npz') print(f'Processing audio sample {audio_dir.path}') with AudioFile(audio_dir) as reader, \ DatasetWriter(dataset_file, low_freq=cutoff_frequencies[0], high_freq=cutoff_frequencies[1]) as writer: for sample in reader: writer += sample ``` Training the model ------------------ Once you have some `.npz` datasets saved under `~/datasets/sound-detect/data`, you can use those datasets to train a Tensorflow model to classify an audio segment. A full example is available under `examples/train.py`: ```python import os from tensorflow.keras import layers from micmon.dataset import Dataset from micmon.model import Model # This is a directory that contains the saved .npz dataset files datasets_dir = os.path.expanduser('~/datasets/sound-detect/data') # This is the output directory where the model will be saved model_dir = os.path.expanduser('~/models/sound-detect') # This is the number of training epochs for each dataset sample epochs = 2 # Load the datasets from the compressed files. # 70% of the data points will be included in the training set, # 30% of the data points will be included in the evaluation set # and used to evaluate the performance of the model. datasets = Dataset.scan(datasets_dir, validation_split=0.3) labels = ['negative', 'positive'] freq_bins = len(datasets[0].samples[0]) # Create a network with 4 layers (one input layer, two intermediate layers and one output layer). # The first intermediate layer in this example will have twice the number of units as the number # of input units, while the second intermediate layer will have 75% of the number of # input units. We also specify the names for the labels and the low and high frequency range # used when sampling. model = Model( [ layers.Input(shape=(freq_bins,)), layers.Dense(int(2 * freq_bins), activation='relu'), layers.Dense(int(0.75 * freq_bins), activation='relu'), layers.Dense(len(labels), activation='softmax'), ], labels=labels, low_freq=datasets[0].low_freq, high_freq=datasets[0].high_freq ) # Train the model for epoch in range(epochs): for i, dataset in enumerate(datasets): print(f'[epoch {epoch+1}/{epochs}] [audio sample {i+1}/{len(datasets)}]') model.fit(dataset) evaluation = model.evaluate(dataset) print(f'Validation set loss and accuracy: {evaluation}') # Save the model model.save(model_dir, overwrite=True) ``` At the end of the process you should find your Tensorflow model saved under `~/models/sound-detect`. You can use it in your scripts to classify audio samples from audio sources. Classifying audio samples ------------------------- One use case is to analyze an audio file and use the model to detect specific sounds. Example: ```python import os from micmon.audio import AudioFile from micmon.model import Model model_dir = os.path.expanduser('~/models/sound-detect') model = Model.load(model_dir) cur_seconds = 60 sample_duration = 2 with AudioFile('/path/to/some/audio.mp3', start=cur_seconds, duration='10:00', sample_duration=sample_duration) as reader: for sample in reader: prediction = model.predict(sample) print(f'Audio segment at {cur_seconds} seconds: {prediction}') cur_seconds += sample_duration ``` Another is to analyze live audio samples imported from an audio device - e.g. a USB microphone. Example: ```python import os from micmon.audio import AudioDevice from micmon.model import Model model_dir = os.path.expanduser('~/models/sound-detect') model = Model.load(model_dir) audio_system = 'alsa' # Supported: alsa and pulse audio_device = 'plughw:1,0' # Get list of recognized input devices with arecord -l with AudioDevice(audio_system, device=audio_device) as source: for sample in source: source.pause() # Pause recording while we process the frame prediction = model.predict(sample) print(prediction) source.resume() # Resume recording ``` You can use these two examples as blueprints to set up your own automation routines with sound detection.