Managing Audio Across Devices and Browsers

Abstract Draft

The development of browser-based music applications introduces a number of challenges concerning audio generation, playback and consistency across devices. While the Web Audio API provides a sophisticated environment for synthesis, signal processing and musical interaction, the behaviour of audio applications remains dependent on browser implementations, operating-system policies and hardware characteristics.

During the development of SAJA Music's interactive learning resources, we have encountered differences in audio behaviour between desktop computers, mobile phones and tablets. These include restrictions on audio initialisation, inconsistent playback under iOS silent mode, differences in perceived sound quality and challenges associated with audio scheduling and application lifecycle management.

This paper considers these issues as part of a broader discussion about designing a common audio architecture for browser-based music education applications. We examine the principal technical considerations, possible approaches to improving compatibility and the implications of developing a shared audio management layer.

The objective is not to establish a definitive solution, but to document our current understanding, identify outstanding questions and inform future development.

1. Introduction

Browser-based environments support several methods of implementing interactive music software. The Web Audio API supports real-time synthesis, filtering, dynamic signal processing, mixing and precise scheduling of musical events.

These facilities allow music learning tools to run within a browser without a separate application installation.

At SAJA Music, we use browser-based audio extensively in applications for rhythm, pitch recognition, ear training, scale exploration and other forms of interactive musical learning.

The resources are intended for use across a range of devices. Students may access them on desktop computers, school-managed laptops, tablets or personal mobile phones.

Consistency across environments is a design requirement, although the degree of achievable consistency remains under investigation.

An application may work correctly on a desktop browser but produce no sound on an iPhone. A rhythm exercise may behave differently after the device has been locked and unlocked. An older tablet may produce sound with a noticeably different timbre or response time.

These observations raise the question of which audio-related responsibilities should be managed centrally and which should remain within individual applications.

2. Audio Generation and Playback Mechanisms Notes

There are several approaches to audio playback in web applications, each with different technical characteristics.

2.1 HTML5 media playback

The HTML5 <audio> element provides a relatively straightforward means of playing existing audio files.

<audio src="example.mp3" controls></audio>

This approach is suitable for recordings, musical examples and other resources where the audio content is predetermined.

The browser handles decoding, buffering and playback through its media subsystem.

However, the HTML5 media model provides less direct control over the precise timing of individual musical events.

2.2 Web Audio

The Web Audio API provides a programmable audio processing environment based on interconnected audio nodes.

A simple oscillator can be constructed as follows:

const context = new AudioContext();

const oscillator = context.createOscillator();

oscillator.type = "sine";
oscillator.frequency.value = 440;

oscillator.connect(context.destination);

oscillator.start();

More sophisticated applications can combine oscillators, gain envelopes, filters, convolution effects and sample playback into complex audio graphs.

This approach is particularly useful for interactive music applications in which sound needs to be generated or modified dynamically.

2.3 A hybrid approach

The mechanisms are not mutually exclusive.

An ear-training application might generate notes through Web Audio while also providing recorded instrumental examples through HTML5 media playback.

Similarly, short samples can be decoded into AudioBuffer objects and scheduled using the Web Audio API.

The choice should depend on the musical requirements of the application.

One question for future development is whether a common audio management layer should abstract all playback mechanisms behind a single interface coordinate initialisation and lifecycle while leaving playback mechanisms application-specific. Neither model has been adopted. [Architecture: revisit after comparative testing.]

3. Mobile Browser Audio Policies

Browser policies affect how applications are permitted to initiate audio playback.

3.1 User interaction requirements

Modern browsers generally restrict audio playback that begins without an explicit user interaction.

For Web Audio applications, this commonly means that an AudioContext must be created or resumed following a user gesture.

let audioContext;

async function initialiseAudio() {

    if (!audioContext) {
        audioContext = new AudioContext();
    }

    if (audioContext.state === "suspended") {
        await audioContext.resume();
    }

    return audioContext;
}

This function should be invoked in response to an appropriate user interaction, such as pressing a Play button.

The resulting initialisation requirements need to be considered within each application.

We have previously encountered situations where audio scheduling and interface updates continued even though the underlying audio context had not been correctly activated.

An important distinction is therefore required between the application having scheduled audio and the browser having successfully started audible playback.

3.2 Audio sessions and iOS silent mode

A related issue concerns the interaction between Web Audio and iOS audio session policies.

During testing, we observed that browser-generated audio may be inaudible when an iPhone is set to silent mode, despite the application otherwise appearing to function normally.

Historically, iOS Safari has treated Web Audio differently from ordinary HTML5 media playback.

The underlying issue concerns audio session classification.

The Audio Session API provides a mechanism for declaring the intended purpose of an application's audio output.

The playback classification is one candidate for testing:

if ("audioSession" in navigator) {
    navigator.audioSession.type = "playback";
}

On supported iOS versions, this can allow Web Audio to remain audible when the device is in silent mode.

The API is not universally supported, however, and remains an area requiring compatibility testing.

Several questions remain relevant:

• Should every SAJA Music application explicitly request a playback session?
• How should applications behave on browsers without Audio Session API support?
• What happens when another application interrupts the audio session?
• How reliably is the session restored after switching applications or locking the device?

The broader point is that audio generation and audio output policy are separate concerns. A functioning audio graph does not necessarily guarantee audible output.

4. Audio Scheduling and Timing

Reliable timing is fundamental to many music applications.

A rhythm exercise, metronome or melodic transcription tool needs to maintain accurate temporal relationships between musical events.

4.1 JavaScript timing

JavaScript provides familiar scheduling mechanisms such as setTimeout() and setInterval().

These are convenient for updating interfaces but are not sufficiently reliable as the sole timing mechanism for precision musical playback.

JavaScript execution may be delayed by rendering, garbage collection, browser throttling or other processing activity.

These delays can become noticeable in applications involving rapid rhythmic subdivisions or tightly coordinated musical events.

4.2 Audio clock scheduling

The Web Audio API provides a separate audio timeline through AudioContext.currentTime.

Musical events can be scheduled against this timeline rather than relying entirely on JavaScript callbacks.

const now = audioContext.currentTime;

oscillator.start(now + 0.1);
oscillator.stop(now + 0.6);

A common approach is to use a short, periodically refreshed scheduling window in which JavaScript schedules events ahead of time using the audio context clock.

This separates the execution of scheduled musical events from the precise timing of JavaScript callbacks.

However, it introduces questions concerning scheduling horizons, cancellation, tempo changes and recovery following interruptions.

4.3 Different applications, different requirements

Not every application requires the same scheduling architecture.

A pitch recognition exercise may only need to trigger a single note in response to a user action.

A metronome requires accurate, repeated scheduling.

A polyphonic synthesis application may require simultaneous voices, envelopes and dynamic parameter changes.

It remains to be determined whether a single scheduling abstraction is appropriate across these applications.

A shared timing utility might be appropriate for rhythm-based resources, while simpler applications could continue to use more direct audio triggering.

5. Device Capabilities, Latency and Sound Quality

Even when an application produces audible output on different devices, the experience may not be identical.

5.1 Output latency

Audio latency can vary with the operating system, browser, audio hardware and selected output device.

The Web Audio API provides properties such as baseLatency and outputLatency that may help characterise aspects of this behaviour.

console.log(audioContext.baseLatency);
console.log(audioContext.outputLatency);

These values are not available consistently on all older browsers and should not be interpreted as exact measurements of the full delay between a user action and the resulting audible sound.

They may still be used as partial diagnostic measurements.

Latency is relevant to user-input-driven instruments and rhythm exercises because it affects the interval between an action and audible output.

5.2 Sample rates

Different audio devices and browser implementations may use different sample rates.

The AudioContext.sampleRate property identifies the sample rate used by a particular context.

For basic oscillator synthesis, much of this complexity is handled internally by the browser.

The effect of differences in sample rate on synthesis, sample playback and resampling warrants further testing. Context options and device output paths may also affect latency and performance.

5.3 Output devices and interruptions

Playback may be affected by wired headphones, Bluetooth output, external audio interfaces, backgrounding, screen locking and interruptions from other applications. These cases need separate testing because a running AudioContext does not establish that sound is audible at the intended output.

6. Current Development Position Open

Different SAJA Music pages currently use different audio generation, initialisation and scheduling strategies. We are using these implementations to investigate device-specific behaviour rather than treating any one approach as the settled pattern. A common implementation has not been finalised.

A possible direction is to separate the browser-facing concerns—session configuration, context initialisation, context state and interruption handling—from the musical concerns of synthesis, sample management and event scheduling. Consolidate all audio handling in one shared module. Current alternative under consideration: a small set of shared browser-facing utilities, with synthesis and scheduling retained in each application. The choice remains open.

No comparative test results or final compatibility matrix are presented in this paper. The work described here is exploratory.

7. Questions for Further Investigation To test

Working test list; numbering is for discussion, not priority.

1. Which audio session configurations work across the iOS and browser versions in use?
2. How do HTML5 media, sample-based Web Audio and oscillator-based Web Audio differ under silent mode?
3. Which operations must take place directly inside a user gesture?
4. What state must be restored after screen locking, backgrounding or an interruption?
5. Which scheduling functions could be shared without changing individual applications’ timing requirements?
6. What diagnostics would distinguish a suspended context, an output policy issue and an application scheduling problem?
7. What device and browser combinations should form the initial test matrix?

8. Discussion Pending

The current work concerns the boundaries between browser audio APIs, operating-system policies and application-level scheduling. At present there is insufficient comparative testing to select one implementation for all SAJA Music pages. The next step is to evaluate the strategies already in use under repeatable conditions and document differences in initialisation, audible output, latency and recovery after interruptions.

This discussion paper is an internal working note made public to document the questions and approaches under consideration. It should not be interpreted as a specification or statement that the proposed architecture has been implemented.

References and Further Reading

1. W3C. Web Audio API.
2. MDN Web Docs. Audio Session API.
3. MDN Web Docs. AudioSession: type.
4. MDN Web Docs. AudioContext: resume().
5. WebKit Bugzilla. Bug 237322: Web Audio muted when iOS ringer is muted.
6. WebKit Bugzilla. Bug 251532: Web Audio muted when silent switch is on.
7. Feross Aboukhadijeh. unmute-ios-audio.
8. Chris Wilson. A Tale of Two Clocks: Scheduling Web Audio with Precision.