Synchronizing Audio State#

Summary

In this tutorial, we’ll see how to synchronize specific properties like an audio signal value, the transport timeline position, the playback state, etc. with the back-end (Python) or link them with other Jupyter widgets. We’ll also give an overview of the audio analysis nodes available in ipytone.

Note: you can download this tutorial as a Jupyter notebook.

import ipytone
import ipywidgets
import matplotlib.pyplot as plt

One thing specific to audio signals (and parameters) is that their value may be updated continuously in the front-end (to be correct: still at discrete steps but at a very high rate, i.e., the audio sample rate is often set to 44.1 kHz).

This makes challenging the synchronization of those widgets with the back-end (e.g., for handling specific events in Python) or with other elements in the front-end (e.g., widget linking). The common ways to handle events with Jupyter widgets (see Section Widget Events) won’t help much here.

Fortunately, ipytone provides alternative ways to deal with this issue that are very similar to the observe(), link() and jslink() functions applied to “classic” Jupyter widgets.

Observing state from Python#

Very much like the observe() method of a Jupyter widget, Ipytone provides a schedule_observe() method that can be used for tracking special attributes like

Let’s create a Synth:

synth = ipytone.Synth().to_destination()

For example, we would like to track the current frequency of the synth oscillator, which will depend on the note played.

Let’s first create a function that will get this value and print it in an output widget:

output = ipywidgets.Output()

def print_new_value(change):
    widget = change["owner"].observed_widget

    with output:
        print(widget, " : ", change["new"])

output

We can then register this function on the synth oscillator frequency, using schedule_observe():

synth.oscillator.frequency.schedule_observe(
    print_new_value,
    update_interval=0.5,
    name="value",
    transport=False,
)

Note

Unlike the Jupyter widget observe() method, schedule_observe() can track only one attribute.

The update_interval option corresponds to the time interval between two consecutive synchronizations of the attribute with the back-end.

The transport option controls how to schedule the synchronization (either using Tone.js Transport.schedule_repeat or using Javascript’s setInterval). It is recommended to use transport=True if you schedule events along the transport timeline (see the Timeline tutorial).

Let’s trigger a couple of notes. Just after running this cell, you should see the oscillator frequency values printed in the output of the cell further above.

synth.trigger_attack_release("C3", 1)
synth.trigger_attack_release("A3", 1, time="+1")

Synth()

Let’s also observe the current value of the envelope (with a slightly higher time resolution):

# set a longer envelope attack
synth.envelope.attack = 1

synth.envelope.schedule_observe(print_new_value, update_interval=0.1)

You should now also see the envelope values printed above when running the cell below:

synth.trigger_attack_release("C3", 2)

Synth()

We can stop tracking those attributes with the schedule_unobserve() method. This will stop the repeated synchronizations with the back-end.

synth.oscillator.frequency.schedule_unobserve(print_new_value)
synth.envelope.schedule_unobserve(print_new_value)

Observing both state and time#

Because the synchronization with the back-end (Python) happens with some latency that is is not known with precision, it might be useful to know exactly at which time the attribute value has been read (either the time of the Tone.js main audio context or the time along the Tone.js Transport timeline). By setting observe_time=True, we can get both that time and the attribute value in Python. For example:

synth.oscillator.frequency.schedule_observe(
    print_new_value,
    update_interval=3,
    observe_time=True
)

You should see a (time, value) tuple printed in the output below. Note that because the time is never constant, the tuple gets updated and the function is called at every synchronization, regardless of whether or not the value has changed.

output.clear_output()
output

synth.oscillator.frequency.schedule_unobserve(print_new_value)

Linking audio widgets#

Similarly to observe() -> schedule_observe(), ipytone provides extra methods for widget linking:

  • dlink() -> schedule_dlink()

  • jsdlink() -> schedule_jsdlink()

Note

There’s no such schedule_link() or schedule_jslink() method, as bi-directional linking doesn’t make much sense for audio signals.

For example, let’s connect the envelope of the synth created above to a FloatProgress widget so that we can have a better view on the evolution of the envelope through time. The link below is made only in the front-end (by default synchronizations will happen at a high frequency).

progress = ipywidgets.FloatProgress(value=0, min=0, max=1)

link = synth.envelope.schedule_jsdlink((progress, "value"))

progress

Now let’s play a long note:

synth.trigger_attack_release("C3", 2)

Synth()

To unlink the two widgets, use unlink() like below. This will stop the repeated attribute synchronizations between the two widgets.

link.unlink()

Analysis audio nodes#

Ipytone’s audio analysis widgets are useful for synchronizing audio signals. Any audio node can be connected to those widgets, which can then be used for observing the signal value from Python or linking it with another widget.

Analyser#

Analyser is a generic analysis node for getting the current waveform or frequency (FFT) data as a numpy array.

The example below plots at a regular interval the current waveform generated by the synthesizer created above.

plot_output = ipywidgets.Output(layout=ipywidgets.Layout(height="300px"))

def plot_change(change):
    plot_output.clear_output()
    with plot_output:
        plt.plot(change["new"])
        plt.show()

plot_output

analyser = ipytone.Analyser(type="waveform")
synth.connect(analyser)

Synth()

analyser.schedule_observe(plot_change, update_interval=1)

If you play a note, you should see the waveform drawn in the plot above:

synth.trigger_attack_release("C3", 2)

Synth()

Analyser has a smoothing attribute that controls the time window average of the analyzed waveform or frequency. Let’s set a high value and you should see the waveform evolve more smoothly in the plot.

analyser.smoothing = 5

synth.trigger_attack_release("C3", 2)

Synth()

analyser.schedule_unobserve(plot_change)
analyser.dispose()

Analyser(disposed=True)

FFT and Waveform#

FFT and Waveform both work very much like Analyser, with some default options and only for mono audio signals.

Meter and DCMeter#

Meter can be used to get the current (RMS) level of an audio signal either in decibels (default) or in the [0-1] range.

For example, let’s see the output gain of the synthesizer in real time:

progress = ipywidgets.FloatProgress(value=0, min=0, max=1)

progress

meter = ipytone.Meter(normal_range=True)
synth.connect(meter)

link = meter.schedule_jsdlink((progress, "value"))

synth.trigger_attack_release("C3", 2)

Synth()

link.unlink()
meter.dispose()

Meter(disposed=True)

DCMeter is similar to Meter except that it outputs the raw value of the audio signal in the [-1, 1] range.

End of this tutorial!

synth.dispose()

Synth(disposed=True)