MCP tools

Compare a performance to a reference recording bar-by-bar. Returns a warping path mapping target time → reference time, an alignment-quality score, an overall tempo ratio, and per-warping-step timing deviations in milliseconds. Use when the user asks to compare timing, sync, line up, align measure-by-measure, or check whether their take 'plays in time with' a reference. Requires both audio_url and reference_url. It always returns an alignment path, even for unrelated recordings — check alignment_confidence first; a low score means the path is unreliable, not a small timing offset.

ZERO-SHOT audio<->text matcher. You supply your OWN free-text labels (text_queries) and it ranks how well the audio matches each, returning per-label similarity scores + the best match. Use for open-vocabulary questions where you define the concept: 'does this match description X?', 'which of these fits best?', mood/genre/scene/vibe/recording-quality, or matching audio to a brief. Works for ANY audio — music, speech, podcasts, field recordings, sound design (not music-only). PREFER over audio_tagging_analysis when the label is custom/abstract/user-worded; use audio_tagging_analysis instead when detecting a KNOWN sound event or instrument from a fixed set. CAVEAT: scores are COMPARATIVE — the best match is only the best of the phrases you give it, so supply a broad, contrasting candidate set (include an 'other/unrelated' option) and treat close or low scores as uncertain. Whole-clip only (no timestamps — crop first to ask about a section). Cannot do pitch/tempo/key/chords/melody/transcription/stems. With no text_queries it returns a 512-dim semantic embedding for similarity search/clustering. Clips over 10 s use a random 10 s window per call, so results vary run-to-run — crop to the relevant section first for stable results.

Renders a NEW processed audio file by applying an ORDERED CHAIN of DSP effects — it does NOT analyze or measure (ignore the '..._analysis' name). Use for processing/mastering/effects on ANY audio: music, stems, voice, podcasts, field recordings, sound design. 18 effects: filters (highpass, lowpass, eq, ladder_filter), dynamics (compressor, limiter, gain, clipping), reverb + convolution (cathedral-hall space), modulation (chorus, phaser, delay), pitch_shift (±24 semitones), and lo-fi/codec character (distortion, bitcrush, resample, mp3_compressor, gsm 'telephone/radio' voice). Use distortion and bitcrush whenever the request calls for their character (gritty, overdriven, fuzzy, lo-fi, vintage, broken, aggressive), scaled to how strongly it is asked: distortion (drive_db 0-100, default 25) adds harmonic overdrive, so about 5-15 is gentle warmth, 25 is crunch and 50+ is heavy fuzz, and it makes the sound harsher and brighter, so a lowpass or eq after it tames that; bitcrush (bit_depth 1-32, default 8) adds digital grit by cutting resolution, so 12-16 is subtle, 8 is retro lo-fi and 4-6 is harsh, and it pairs with 'resample' for a lo-fi sound. Both ADD grit and noise; they never remove noise or make a sound cleaner. Chain up to 20 stages in order (typical: filters -> modulation -> time-based -> dynamics). ALWAYS-ON at the end of every job: (a) LUFS loudness-normalization to output.normalize_lufs (default -16; set it explicitly — e.g. -14 streaming, -16 podcast, -23 broadcast) and (b) a -1 dB safety limiter — so NEVER add your own final limiter, and know a lone gain stage will NOT set final loudness (normalize_lufs governs it). Content-agnostic: no need to isolate vocals first. Handles mono/stereo and can process just a clip (start-offset + duration). NOTE: 'pitch' in cannot_do means no pitch DETECTION — pitch_shift for CHANGING pitch IS supported; do not refuse it. Output sample-rate in the output block is a no-op — to reduce rate/go lo-fi, add a 'resample' effect stage. Out-of-range params fail with a clear error. Route 'measure/detect key/tempo/loudness' to an analysis engine, not here. On near-silent input, loudness normalization is silently skipped and won't hit the target — check processed_lufs in the response. To approximate a reference recording's sound, first have an analysis engine measure the reference, then set the chain from the measured values and call the result an approximation; never claim a match to a reference that was not measured.

Primary engine for open-ended 'what sounds are in this audio?' across ANY content — podcasts, voice, field recordings, sound design, mixed/unknown uploads, and music. Returns the most likely labels from a fixed 527-class sound vocabulary (speech, music, instruments, laughter, animals, vehicles, water, wind, machinery, alarms, etc.), each with an independent 0-1 confidence. These are tag PROBABILITIES, not facts — say 'likely/detected', never 'this IS'. Run on the FULL original mix — do NOT isolate vocals or use a stem (that erases the non-music sounds this engine exists to find). Clip-wise, not timestamped: one prediction over the whole file; for 'when does X happen', segment the audio and tag chunks. Multi-label; de-duplicate nested labels. Faint/brief sounds may fall below the default 0.1 threshold — lower confidence_threshold or raise top_k for subtle background sounds. Mono/32kHz internally: no stereo/spatial answers. Not for pitch/key/tempo/melody/chords or precise transcription; not speech transcription (use speech_analysis) and not free-text caption matching (use audio_description_analysis when the user supplies their own descriptions). include_embeddings returns a 2048-dim fingerprint for similarity/dedup/clustering.

Final reassembly step for VIDEO inputs: muxes a processed audio track (full mix OR a single stem) back into the ORIGINAL video's frames and returns processed_video_url. Picture is byte-identical (frames never re-encoded — no visual quality loss, fast); audio is re-encoded to AAC (lossy). Use as the LAST step whenever the user uploaded a video and wants a VIDEO back. Pass video_url = the ORIGINAL video (the one extract_video_audio echoed back), audio_url = the processed audio. Bookend to extract_video_audio; full flow: extract_video_audio -> (analysis/effect engines) -> create_video. NO mono/stereo or sample-rate prep needed. CAVEAT — length must match: the mux ends at whichever input is shorter and SILENTLY truncates the rest. If a length-changing effect ran (crop/trim, time-stretch, tempo change) so the audio no longer matches the video's duration, the output is clipped to the shorter one — warn the user the video may be shortened, or avoid length-changing effects when the full video must be preserved. Also: only the first audio and first video stream survive — secondary audio tracks, subtitles, and chapters on the original are dropped. Whole-track REPLACEMENT only — cannot mix/duck the new audio under the original or place it at an offset. Output is MP4 (MOV for ProRes-type sources; input must be MP4/MOV). The returned video's audio is always re-encoded to AAC 128 kbps — lossless/high-quality input is degraded, with no original-codec passthrough.

Free, fast server-side TIME-TRIM: cuts one contiguous section from an audio file and returns a new, shorter clip. Use ONLY when the user names a specific time range or section (e.g. 'the chorus 1:00-1:30', 'the first 30s', 'around 2:15') — NEVER crop for whole-file requests. Put crop FIRST in the recipe: every later step then operates on the CROP automatically (it's linked as a child of the original, per the Cropping CRITICAL RULE), so don't repeat the range downstream. Lossless for compressed formats (mp3/flac/ogg/m4a/aac: stream-copy), BUT cuts snap to the nearest ~26ms frame boundary — not sample-exact, so avoid for gapless/tight edits. WAV/AIFF (and other PCM) get re-encoded to 16-bit — warn the user if a high-bit-depth source needs archival precision. One range only: cannot extract multiple segments, stitch, or drop a middle; cannot run in parallel mode. TIME trim only — does NOT remove vocals, isolate an instrument, resample, or clean audio; for 'just the vocals/drums' use element-isolation, and for 'audio from a video' use the video-audio extractor. A start_time at or past end-of-file errors out (no empty clip) — confirm the file's duration before cropping near the end.

General-purpose deep audio 'sounds-like' engine — works on any audio (music, voice/podcasts, SFX, ambience, field recordings). Turns a clip into a semantic embedding capturing overall sonic CHARACTER (timbre/texture/room), not notes/words/tempo/key. Uses: (a) single-track embedding for clustering / similarity search / near-duplicate detection; (b) comparison — pass reference_url for a cosine similarity_score (0..1). Comparison collapses each entire track to one average vector, so it answers 'how similar do these sound overall', order-blind: it does NOT tell you WHERE they match and is not exact-match fingerprinting. For bar-by-bar timing/sync alignment use alignment_analysis instead. Present the score as relative similarity, not a hard match/no-match verdict. Set content_type=env for SFX/ambience/non-music (default is music) or results degrade silently. Audio is downmixed to mono, so this can't judge stereo width; very short/near-silent clips give low-confidence scores. For matching audio to a TEXT description, use audio_description_analysis instead.

FIRST step for ANY video input — every other engine reads audio, not video, so extract the audio track first. Input: video_url (MP4/MOV/QuickTime containers only; WebM/MKV/AVI video is rejected). Returns: audio_url (feed into any analysis, transcription, or effect engine), video_url (echoed back so create_video can later re-attach the picture to processed audio), media_probe, and duration_seconds. IMPORTANT — the extracted audio is downmixed to MONO at 22050 Hz: fine for key/chord/tempo, transcription, speech, and general analysis, but it discards stereo imaging and audio above ~11 kHz. Do NOT treat the result as a faithful copy of the soundtrack, and warn the user before using it for stereo-field, mastering/loudness, spatial, or high-fidelity/deliverable work. Limits: hard 60-minute cap (longer videos rejected); if the video has no audio track (e.g. a muted screen recording) extraction fails — tell the user plainly, it is not a system error. CPU-only, fast. Works for any video-sourced audio: music clips, podcasts/interviews/lectures, voiceover, sound design, field recordings. Extraction always takes the first audio track — multi-track videos (dual language, commentary) silently use stream 0 with no way to pick another.

Polyphonic NOTE transcription: turns pitched audio into discrete musical notes (MIDI pitch + onset/offset + amplitude), a downloadable MIDI file, and a plain-language melody summary (note names, pitch span, notes/sec). Works on any clearly-pitched audio — solo instruments, a hummed/whistled/sung tune, a synth melody. Not continuous vocal-pitch: it snaps to chromatic notes, so for intonation/pitch-bends/'is the singer flat or sharp' use tempo_analysis include_pitch. Detection range ~32-2000 Hz. For best results transcribe a clean, exposed line — on a full mix, isolate the target part first (see stem_analysis / remove_audio_element); dense mixes, percussion, and noise produce muddy or empty output. Fails soft (note_count 0 / 'No notes detected'), so check note_count before asserting a melody. Slow and heavyweight. Not the default for key/chord questions — use spectral_analysis chroma for those. Use only for genuine transcription / MIDI / score work.

Two modes for symbolic (MIDI-only) music: harmony (Krumhansl key + chord progression from MIDI — a SECONDARY cross-check; for musical key/chord/harmony on AUDIO use the audio path chroma.cqt instead), midi_diff (per-note diff between two passes — added/removed/timing/pitch deviations; this is the primary use). Both non-audio — inputs come from prior jobs (melody_analysis MIDI/notes). Mode is required. Do not use for audio key/chord/harmony questions — those are read directly from the audio's pitch content. Use only for genuine transcription / MIDI / score work.

Remove noise, enhance speech, restore audio. Returns processed audio URL. voice-hifi and voice-enhance are speech-only — on music/non-voice they strip tonal and instrumental content as noise; use voice-lite instead.

Remove one element from a song (return the whole track minus that element) or isolate it (return only that element) as a SINGLE downloadable file. Use for 'remove the vocals', 'karaoke', 'only remove the bass', 'isolate the drums'. Beyond vocals/drums/bass/other, specific instrument classes — guitar, piano, saxophone, strings, brass, woodwinds, organ, harp, and more — are reachable on this same tool: name the instrument as the target and it routes automatically to a higher-quality separation path. One target per job; to split a song into ALL its stems use stem_analysis. Slower/costlier than a plain split. If a call reports the instrument model is unavailable, that instrument isn't supported on this endpoint right now — it's not a user error and not retryable; fall back to a plain vocals-vs-instrumental split or tell the user that instrument isn't supported yet.

MUSIC-ONLY rhythm + structure engine. One pass returns beat times, downbeats/bar-lines, time signature, BPM, tempo stability + drift curve, and song-form sections (intro/verse/chorus/bridge/outro). Use for beat grids, bar lines, time signature, tempo steadiness/drift, or structure/form on MUSIC. It runs stem separation INTERNALLY, so do NOT pre-remove vocals or pre-separate stems — feed the raw track. Mono/stereo and sample rate are handled automatically. IMPORTANT: this model assumes music with a steady beat; on speech, podcasts, voice notes, field recordings, ambient/drone, or sound-design it will silently INVENT a fake beat grid, meter, and verse/chorus labels — do not run it there, and if unsure the file is music, ask or say so rather than reporting form/meter. HEAVIEST analysis path (stem separation + transformer, GPU): for a plain BPM/tempo-only question prefer tempo_analysis. BPM is tempo_stability.bpm_mean, so if this already ran, reuse it instead of calling tempo_analysis. Caveats to pass the user: time signature is a best-guess limited to common meters (falls back to 'unknown'); a low tempo-stability score can mean an intentional tempo change, not an error; structure and beats are unreliable on very short, quiet, or free-time clips. MP3 input shifts beat/downbeat timestamps ~20–40 ms — use WAV when timing precision matters (sync, beat-grid export). MAX LENGTH: about 6 minutes of audio (about 5.5 at 48 kHz); longer fails with success:false, error_kind=over_duration_limit. If the audio is longer, crop to the section needed first (or warn the user).

Extract mathematical and perceptual features: zero-crossing rate, tonal centroid (tonnetz), normalized chroma statistics, LUFS loudness (integrated, short-term, momentary, LRA, sample/true peak), and optional psychoacoustic quality metrics (roughness, sharpness, time-varying loudness). Use for technical signal analysis and rehearsal dynamics comparison.

Broad timbre and spectral feature extractor for music, speech, and mixed audio. Strong for brightness/centroid, rolloff, spectral flux, flatness, RMS/energy, MFCC-family timbre, harmonic/percussive ratio, onset/attack timing, and CQT chroma. Also the audio key/chord path: request the CQT-chroma feature for a musical key or chord/harmony estimate. Caveats: every value is averaged over the WHOLE file (no time-localization); audio is analyzed in mono at 22.05 kHz (no stereo/spatial info, nothing above ~11 kHz); missing features return 0/None (treat as unknown). Not for tempo/beat (use tempo_analysis) or note transcription (use melody_analysis). Request only the categories you need — the full sweep and deep variants are expensive.

PRIMARY engine for SPEECH: transcription, subtitles/captions (word-level timestamps), translation to English, and speaker diarization (who-said-what, talk-time). Use for podcasts, interviews, meetings, voicemails, lectures, calls, field/oral-history recordings, and dialogue extracted from video. SPEECH ONLY — it cannot do pitch/tempo/melody/key/chords/MIDI/spectral, and on music/singing/non-speech it will HALLUCINATE words; never route lyrics or any musical-content/key/chord question here (key & chords -> chroma.cqt; 'what sounds are in this?' -> the sound/event classifier). MONO — stereo/spatial cues are discarded. Modes: transcribe (default) for words + timestamps; transcribe-diarize when who-is-speaking matters; translate / translate-diarize for any language -> English (output always English); diarize-only for a fast 'how many speakers?' with no transcript. Diarization and translation are the expensive paths (translate-diarize heaviest); cost scales with length and model size; transcribe / transcribe-diarize default to model 'turbo' (balances speed/accuracy; large-v3 for max accuracy, tiny/base for drafts) — translate / translate-diarize default to 'medium' instead, because turbo cannot translate (it silently returns the source language, not English); requesting model='turbo' on a translate mode is automatically substituted to 'medium' and flagged in the response (extraction_info.model_substituted=true, requested_model='turbo') rather than silently returning the wrong language. Pass num_speakers when the exact count is known (more accurate than min/max). Caveats to tell the user: diarization can be unavailable and then returns a transcript with NO speaker labels (check speaker_diarization.enabled) — don't promise counts blindly; overlapping speakers, heavy accents, far-field/noisy audio, and very short clips lower accuracy — surface no_speech_prob / avg_logprob rather than presenting an uncertain transcript as fact. include_word_timestamps=false saves ~10-15s when only plain segment text is needed. For a song where the user wants sung words, isolate vocals first — but expect hallucination on melodic singing. Words with numbers, currency, or punctuation (e.g. "5pm", "$13.60") silently lose word-level timestamps — transcribed but unsynced, breaking subtitles and speaker labels there.

Multi-stem separation. Splits audio into a fixed set of sources and returns EACH as a separate downloadable file: drums, bass, vocals, other (extended mode adds guitar + piano, lower quality). Use when the user wants ALL or several parts at once (full stem split, remix stems, both acapella and instrumental). If they want just ONE result file — 'remove the vocals', karaoke as a single track, 'isolate the drums' — use remove_audio_element instead, which returns full 44.1 kHz stereo; stem_analysis files are decoded at 22.05 kHz mono and band-limited (nothing above ~11 kHz), good for analysis/preview but not a hi-fi deliverable. Cannot separate arbitrary instruments ('other' is a catch-all); for a specific instrument use remove_audio_element. Music-oriented: not a voice/podcast separator and not a noise cleaner. Expensive and slow — only run when separation is explicitly asked for. Fails soft: absent stems come back near-silent (check presence_ratio). Set stems_to_return to keep only the stems you need.

Fast engine (~3-4s) for three things: (1) global BPM + beat times, (2) tempo steadiness over time, and (3) monophonic fundamental-frequency (F0) pitch tracking. Primary tool for F0/pitch — set include_pitch=true (off by default). F0 is MONOPHONIC: it tracks one pitch at a time, so run it on a single isolated source, never a full mix or chord (for a sung line, isolate the vocal first). F0 works for both speech/voice and single-source music. Default F0 band C2-C7 (65-2093 Hz); narrow pitch_fmin/fmax to the expected range (e.g. ~80-300 Hz for speech) to avoid octave errors. Beat/tempo here is a quick global BPM only — no downbeats, meter, per-section curve, or structure; for those use rhythm_tracking_analysis. BPM is meaningful for music only; on rubato/sparse material it can report half/double time. tempo_stability is 0-1 (higher = steadier). beat_times capped at 2000.

Retrieve the result (or current status) of a previously submitted analysis job by its job_id. Poll this until status is COMPLETED. Returns the finished analysis payload when done, or a terminal error if the job failed.