Synthdef Compiler Architecture

Overview

The synthdef compiler takes descriptions of audio signal flow graphs and compiles them into optimized C++ code that can be loaded as a dynamic library plugin into an audio engine. The compiler is written in C++23 and runs on macOS (ARM64).

The system has two front-ends for describing signal graphs:

1. C++ DSL -- Signal graphs are constructed directly in C++ using operator overloading and a rich library of audio primitives. Each operation implicitly builds a node in an expression graph.
2. S-expression IR -- Signal graphs are described in a textual s-expression format, which can be generated by external language front-ends. This is the primary input format for the command-line tool.

Both front-ends produce the same internal representation: a directed acyclic graph (DAG) of Expr nodes within a Synth object. The compiler then runs a series of analysis and optimization passes, generates C++ source code, invokes clang to compile it, and links the result into a .dylib that conforms to the tzpl_plugin_abi interface.

S-expression text          C++ DSL code
       |                        |
       v                        v
 sexpr::parse_sexpr()     operator overloading
       |                        |
       v                        v
 SExprGraphBuilder         Implicit graph building
       |                        |
       +----------+-------------+
                  |
                  v
            Expr graph (DAG)
                  |
                  v
           graphAnalysis()
    (type/shape inference, optimization,
     tree extraction, loop scheduling)
                  |
                  v
           cppCodeGen()
    (visitor-based C++ code emission)
                  |
                  v
          Generated .cpp file
                  |
                  v
        clang -O3 -ffast-math
                  |
                  v
             .dylib plugin

Directory Structure

All source files reside in src/. The compiler is built by the repository-level CMake build through synthdef-compiler/CMakeLists.txt. Generated plugin code is still compiled and linked at runtime with direct clang invocations.

Core Type System

Expression Graph

Graph Infrastructure

Operations Library

Constant Evaluation

Optimization

Code Generation

Front-Ends

Compilation and Runtime

Utilities

The Expression Graph

The S Type

S is the central type in the system. It is a thin pointer wrapper around Expr*, representing a signal value in the graph. Because it is so pervasive, it uses a single-character name. S supports:

• Implicit construction from numeric literals (i32, f64, etc.), which creates Constant nodes
• Operator overloading (+, -, *, /, %, &, |, ^, <<, >>, comparisons), which creates BinaryOpExpr or CompareOpExpr nodes
• Method-style calls (.sin(), .sqrt(), .abs(), etc.), which create UnaryOpExpr nodes
• A .chain(n, f) method for functional pipeline composition

Similarly, D wraps DelayBuf* and provides delay-line read/write operations.

Expression Node Hierarchy

All expression nodes inherit from Expr, which is an ArenaObj (automatically registered with the current thread-local arena for lifetime management).

Expr (abstract base)
 |
 +-- Constant          -- compile-time constant value (scalar or vector)
 +-- SampleRate        -- the audio sample rate (fs)
 +-- SampleDur         -- the sample duration (1/fs)
 +-- Control           -- a user-controllable parameter
 +-- Inlet             -- an audio input port
 +-- Outlet            -- an audio output port (sink)
 |
 +-- UnaryOpExpr       -- unary math operation (neg, abs, sin, log, ...)
 +-- BinaryOpExpr      -- binary math operation (add, mul, pow, ...)
 +-- CompareOpExpr     -- comparison operation (eq, ne, lt, ...)
 +-- CastOpExpr        -- type cast (f32->f64, i64->f32, ...)
 +-- ReduceExpr        -- vector reduction (sum, product, min, max)
 |
 +-- VarExpr           -- a named variable reference
 +-- PhiNodeExpr       -- SSA phi node for control flow merging
 +-- SelectExpr        -- element-wise conditional select
 +-- ControlFlowExpr (abstract)
 |    +-- IfElseExpr   -- if/else with subgraphs for each branch
 |    +-- SwitchExpr   -- multi-way switch with subgraphs per case
 |    +-- ForLoopExpr  -- counted loop with body subgraph
 |
 +-- URandExpr         -- uniform random [0, 1)
 +-- BiRandExpr        -- bipolar random [-1, 1)
 +-- Rand64Expr        -- raw 64-bit random
 |
 +-- MaxDelay          -- declares maximum delay length for a buffer
 +-- DelayFixRead      -- fixed-delay read from a delay buffer
 +-- DelayVarRead      -- variable-delay read from a delay buffer
 +-- DelayWrite        -- write to a delay buffer
 +-- DelayInit         -- initialize a delay buffer sample

Each Expr node carries:

• **inputs**: vector of S references to input expressions
• **userial**: unique serial number (used for variable naming in generated code)
• **rate**: SignalRate -- when this expression needs to be computed (Const, Init, Reset, Event, Audio)
• **type**: NumType -- resolved numeric type (i32, i64, f32, f64 flags)
• **chans**: channel count (always a power of two for efficient SIMD-friendly indexing)
• **cut**: GraphCut -- marks where the expression tree must be broken for code generation
• **tree**: pointer to the ExprTree this expression belongs to
• **consumers**: identity bag of downstream expression references

Hash-Consing

Expressions are deduplicated via hash-consing. When addExpr() is called, the new expression is hashed and compared against existing expressions in the current Graph's hashConsSet. If a structurally identical expression already exists, the existing one is returned instead. This ensures that common subexpressions are shared, reducing both graph size and generated code.

Constant Folding

When operations are applied to Constant nodes, the result is computed immediately at graph-construction time rather than generating code for it. For example, constant(2.0) + constant(3.0) directly produces constant(5.0). This is handled in the unary_op(), binary_op(), and compare_op() functions, which check whether their operands are constants before creating expression nodes.

The Type System

Signal Rates

Expressions execute at different rates, forming a hierarchy:

The rate of a derived expression is the maximum of its input rates. For example, multiplying an audio-rate signal by an event-rate control produces an audio-rate result.

Numeric Types

NumType uses a 4-bit flag field supporting any combination of {I32, I64, F32, F64}. During graph construction, types start as broad constraints (e.g., any_float for sin()) and are iteratively narrowed through constraint propagation during type inference.

Key operations:

• a & b (intersection) -- used during type inference to narrow constraints
• a | b (union) -- used to express "either type is acceptable"
• is_concrete() -- true when exactly one type flag is set (via popcount == 1)

Channel Count (Shape)

Each expression has a channel count (chans), always rounded up to a power of two via asChans(). This enables efficient modular indexing using bitwise AND (index & (chans - 1)) instead of division.

Binary operations use broadcasting: broadcast(a, b) returns max(a, b). Since all channel counts are powers of two, any two signals can be combined by repeating the smaller one to the length of the larger one. All power-of-two channel counts are broadcast-compatible with each other.

The Graph Analysis Pipeline

The Synth::graphAnalysis() method runs the following passes in sequence:

1. Merge Delays

Identifies delay buffers that have identical initialization patterns and writers, and merges them into a single buffer. This reduces memory allocation in the generated code.

2. Topological Sort

Orders all expressions in dependency order so that each expression is computed after all of its inputs. Uses a worklist algorithm visiting from sinks (outlets) backward through the graph.

3. Collect Consumers

Builds the reverse-dependency graph: for each expression, records which other expressions consume its output. This information drives fan-out detection and dead code elimination.

4. Calculate Delay Lengths

Determines the allocation size for each delay buffer. Fixed-length delays (known at compile time) can be allocated as fixed-size arrays. Variable-length delays require dynamic allocation and use power-of-two ring buffers.

5. Shape Inference

Iteratively propagates channel counts through the graph. Each expression type has a calcShape() method that determines its output channel count from its inputs. Runs repeatedly until a fixed point is reached.

6. Type Inference

Iteratively propagates numeric type constraints through the graph using a worklist algorithm. Each expression:

1. Starts with an initial_type() constraint (e.g., any_float for transcendentals)
2. Intersects its type with input types via update_type()
3. Propagates type changes to both inputs and consumers

The algorithm converges when no more type changes occur.

7. Default Types

Any expression whose type is still not concrete (i.e., has multiple flags set) is assigned a default type chosen from the expression's supported types in this order of preference: f32, f64, i32, i64.

8. Set Delay Reader Rates

Promotes delay writers and readers when a delay is driven by event-rate input. Event-rate delay buffers are advanced from processEvents() instead of the audio tick, so readers that depend on those buffers must be scheduled at the event rate as well.

9. Find Graph Cuts

Analyzes the expression graph to determine where values must be materialized into named variables (rather than being inlined as subexpressions). An expression is cut if any of these conditions hold:

10. Cut Graph to Trees

Starting from each cut point, traces backward through the graph to form ExprTrees -- maximal sets of expressions that can be evaluated together as a single inline expression. Each tree has a root (the cut expression) and a list of member expressions.

11. Remove Dead Code

Eliminates trees whose roots are marked GraphCut::Unused (no consumers and not a sink).

12. Add Delay/Subgraph Antecedents

Establishes ordering constraints between trees:

• Delay writes must happen after all delay reads in the same cycle
• Subgraph results must be computed before they're consumed

13. Sort Trees

Topologically sorts trees by their antecedent relationships, ensuring correct evaluation order.

14. Compute Event Iso-Groups

Computes IsoGroups for event-rate trees. Each event-rate tree is labeled with the transitive set of activation-source expressions -- Control and NoteParam -- that can activate it. Trees with identical source dependency sets share an iso-group. The compiler then builds activation edges between iso-groups from tree antecedents and topologically sorts those groups so upstream event work runs before downstream work. (User noteParams are event-rate sources; the gate noteParam stays audio rate so per-sample envelope/trigger edge detection keeps its semantics.)

15. Trees to Loops

Groups trees into GenLoops based on three criteria:

• Same Graph (subgraph)
• Same SignalRate
• Same channel count
• Same event iso-group for event-rate trees

Trees that can share a loop are fused together. This minimizes loop overhead in the generated code.

16. Split Rates

Partitions loops into four categories:

• initLoops -- run once at initialization
• resetLoops -- run on note-on/reset events
• eventLoops -- run on parameter changes
• Audio loops (the main per-sample processing, stored in root_graph->loops)

Algebraic Rewriting

The rewrite engine (synthdef_rewrite.cpp) applies ~100 algebraic simplification rules during expression construction. Each rule is a pattern consisting of a left-hand side (to match) and a right-hand side (the replacement).

Patterns use RewriteExpr nodes with variables (e.g., x, y) that bind during matching. The system handles:

• Commutativity: Binary operations marked as commutative are matched in both argument orders
• Rate-aware rewriting: Rewrites factor out low-rate subexpressions so they can be grouped and scheduled at the lower rate, reducing the CPU cost of audio-rate computation (analogous to loop-invariant code motion)
• Monotonicity: Special MonotonicRewriteExpr patterns leverage monotonicity information (e.g., floor(floor(x)) = floor(x))

Example rules:

-(-x)        --> x
x + 0        --> x
x * 1        --> x
x - x        --> 0
x / x        --> 1
exp(log(x))  --> x
pow(x, 0)    --> 1
pow(x, 1)    --> x
pow(x, 2)    --> x * x
min(x, max(x, y)) --> x   (when monotonic)

C++ Code Generation

The CppCodeGen class generates a complete C++ source file that implements the plugin interface. It uses the visitor pattern to traverse expressions and emit code.

Generated Structure

// Includes and forward declarations

typedef struct synth_name {
    tzpl_SynthFuns funs;          // function pointer table
    struct tzpl_Engine* engine;    // audio engine reference
    struct tzpl_Node* node;        // node reference
    int num_ins, num_outs, num_controls;
    void** inlets;
    void** outlets;
    void** controls;
    double fs, sd;                 // sample rate, sample duration

    // Constant arrays (init-rate, non-scalar)
    f64 c42[4];

    // Instance variables (persisted across samples)
    f64 v17;           // graph cut variables
    RandState1 rgen0;  // random number generator state

    // Delay buffers
    f64 d0;            // 1-sample delay (scalar)
    f64 d1[16];        // fixed-size ring buffer
    f64 *d2;           // dynamically allocated ring buffer
    u64 d1_wrpos;      // write position
    u64 d2_wrpos;
    u64 d2_mask;       // ring buffer mask (size - 1)
} synth_name;

Generated Functions

Loop Generation

Generated loops vary based on channel count:

Scalar (1 channel):

f64 v5 = std::sin(p->v3 * p->fs);  // inline expression tree
p->v3 = (p->v3 + p->c2);            // delay write

Multi-channel:

for (usize i = 0; i < 4; ++i) {
    ((f64**)p->outlets)[0][i] = v5[i] * p->v3[i & 1];
}

The indexing uses [i & (chans-1)] for broadcasting when an inner expression has fewer channels than the loop.

SIMD Code Generation

SIMD code generation is now implemented in synthdef_cpp_codegen.cpp. The generator selects an explicit Apple simd vector width per GenLoop and emits vector loads, stores, splats, and vector arithmetic where the loop shape and expression contents allow it. Generated source includes using namespace simd; and uses types such as f32x4, f64x4, i32x2, and i64x2.

By default, codegen uses up to 4 lanes and skips 2-lane SIMD, because 2-channel code is often not worth vectorizing. The command-line tool accepts --no-simd to force scalar code and --simd-2 to allow 2-lane SIMD. The public entry point is cppCodeGen(synth, maxSimdWidth, minSimdWidth), with defaults of 4 and 4.

CppCodeGen::simdWidth() returns 4, 2, or 0. A loop is vectorized when its total element count is divisible by the chosen width. Non-voiced loops use loop.chans as the count. Flat voice loops use maxVoices * loop.chans, except phi-node loops whose channel count is already voice-expanded. If the total count equals the vector width, codegen emits a single vector operation; otherwise it emits a stride loop such as for (usize i = 0; i < chans; i += width).

Some expression types deliberately force scalar code: control flow loops, vector reordering and construction operations (VecTransposeExpr, VecRotateExpr, VecReverseExpr, VecStrideExpr, VecStutterExpr, VecAtExpr, VecPutExpr, VecJoinExpr, VecNCycExpr), comparisons, selects, reductions, spectral expressions, random-number generators, and debug expressions. These nodes either need per-element indexing, scalar branch semantics, side effects, or operations that do not map cleanly to Apple simd operators.

SIMD variable references handle the same broadcasting rules as scalar code. Scalars become vector splats, contiguous arrays are loaded with reinterpret-cast vector loads, and narrower vectors are expanded or cyclically gathered when a wider loop consumes them. Delay reads and writes have SIMD paths, including one-sample delay loads/stores, fixed and variable ring-buffer gathers, and interpolation kernels such as tzpl_interp_cubic for SIMD variable-delay interpolation.

Voicer codegen has a flat voice mode for voice graphs without control-flow expressions. In this mode voice-local state is laid out as structure-of-arrays members such as voice_vN[] and voice_dN[], and the generator can process multiple voices and channels in one SIMD vector. Per-voice note parameters are gathered from voicer_params; 1-channel voice-local buffers index by voice, while multi-channel buffers use the flattened voice/channel index.

Event-Rate Code Generation

Event-rate codegen is driven by the iso-groups computed during graph analysis. Each generated control event copies the payload for a control into p->controls[serial] and sets p->ctrlN_active = true. Note events are the other activation source: the generated noteSetParams/noteSetParamRange set per-serial np_active[] flags for the noteParams they wrote (noteOn instead re-runs the voicer's event loops for the new voice in reset mode), and the engine flags the node so processEvents() runs. processEvents() maps active controls and noteParams to the iso-groups whose transitive source set contains them, then clears the active flags. NoteParam-sourced loops iterate all voices; noteParam leaves read the live voicer params matrix.

Iso-groups run in topological order. When a group runs, codegen emits only that group's event loops, marks downstream groups active according to the iso-group activation graph, and advances any non-scalar event-rate delay buffers written by expressions in the group. If no iso-groups exist, processEvents() falls back to running all event loops directly for compatibility.

Control Flow

if_ and switch_ expressions generate actual C++ control flow:

if (p->v10) {
    // then-branch loops
} else {
    // else-branch loops
}

Each branch has its own subgraph with independent loops and delay advancement.

Delay Buffer Access

Delay buffers use circular ring buffers with power-of-two sizes for efficient masking:

// Write
p->d0[p->d0_wrpos & 15] = value;

// Fixed read (3 samples back)
result = p->d0[(p->d0_wrpos - 3u) & 15];

// Variable read
result = p->d0[(p->d0_wrpos - u64(delay_time)) & p->d0_mask];

// Advance
++p->d0_wrpos;

One-sample delays are optimized to a single scalar variable (no ring buffer needed).

The S-Expression Front-End

The s-expression format provides a serializable IR for external language front-ends. Each expression is defined with an integer ID and can reference other expressions by their IDs. Syntax is positional only: there are no keyword arguments or colon-prefixed field names.

S-Expression Format

(
  (0 Constant 1 8 (440.0))
  (1 SampleRate)
  (2 BinaryOp div (0 1))
  (3 Outlet "out" 2)
)

In the constant form above, 1 is the channel count and 8 is the NumType flag value for f64.

Parsing Pipeline

1. Lexing/Parsing (synthdef_sexpr.cpp): Recursive descent parser converts text to sexpr::Item variant type (bool | int64_t | double | Symbol | string | ItemVec)
2. Graph Building (synthdef_from_sexpr.cpp): SExprGraphBuilder iterates through the expression list, creating Expr nodes and resolving cross-references via exprMap (id -> S) and delayMap (id -> D)

Supported Node Types in S-Expressions

Constants, sample-rate, sample-dur, inlets, outlets, controls, note parameters, all unary/binary/compare/cast operations, vector reductions and vector transforms, delay operations (max-delay, delay-init, delay-fix-read, delay-var-read, delay-write), sample-buffer operations, spectral chain operations, voicers, and control flow (select, if, switch, for).

The Plugin ABI

Generated plugins conform to the tzpl_plugin_abi.h interface (located in a sibling shared/ directory). The key structures:

typedef struct tzpl_SynthData {
    tzpl_SynthFuns funs;
    struct tzpl_Engine* engine;
    struct tzpl_Node* node;
    int num_ins, num_outs, num_controls;
    void** inlets;
    void** outlets;
    void** controls;
    double fs, sd;
    // ... synth-specific state follows
} tzpl_SynthData;

typedef struct tzpl_SynthFuns {
    tzpl_SynthData* (*alloc)();
    tzpl_SErr (*free)(tzpl_SynthData*);
    tzpl_SErr (*init)(tzpl_SynthData*);
    tzpl_SErr (*uninit)(tzpl_SynthData*);
    tzpl_SErr (*reset)(tzpl_SynthData*);
    tzpl_SErr (*event)(tzpl_SynthData*, u64, tzpl_Slice, tzpl_Slice);
    void (*processEvents)(tzpl_SynthData*);
    void (*processAudio)(tzpl_SynthData*);
    // ... note allocation functions
} tzpl_SynthFuns;

typedef struct tzpl_SynthDef {
    const char* name;
    tzpl_SynthFuns funs;
    int num_ins, num_outs, num_controls;
    tzpl_PortDef* ins;
    tzpl_PortDef* outs;
    tzpl_ControlDef* controls;
} tzpl_SynthDef;

The generated .dylib exports a single load() function that returns a tzpl_SynthDef. The audio engine calls dlopen() to load the plugin, then uses the function pointer table to manage the synth's lifecycle.

Compilation and Runtime

Build Pipeline

Generated .cpp  --[clang -O3 -ffast-math -std=c++23]--> .o  --[clang -dynamiclib]--> .dylib

The compiler invokes clang as a subprocess with:

• -x c++ -- C++ input
• -arch arm64 -- Apple Silicon
• -std=c++23 -stdlib=libc++ -- modern C++ standard
• -O3 -ffast-math -- aggressive optimization (relied upon for auto-vectorization)
• -dynamiclib -undefined dynamic_lookup -- create a dylib with deferred symbol resolution

The build directory defaults to ~/tzpl-build/ or the path in the TZPL_BUILD environment variable.

Runtime Audio

For testing, the compiler can immediately play the generated synth using RtAudio (Core Audio on macOS):

1. loadDef() -- dlopen() the .dylib, call load() to get the tzpl_SynthDef
2. setupSynth() -- allocate the synth struct, set up I/O ports
3. AudioEngine -- configure RtAudio (48kHz, 256 frames, stereo output)
4. Per-buffer callback calls processAudio() once per sample frame

DSL Library: Audio Primitives

The synthdef_common_ops module provides a comprehensive library of signal processing functions, all of which build expression graph nodes when called:

Oscillators

phasor, sinosc, fsinosc, fsinxosc, lfsaw, lfimp, lftri, lfsqr, lfpar, lfvsaw, lfupulse, lfpulse, lfzpulse, ...

Noise and Random

white, pink, red, blue, violet, dust, dust2, velvet, rand, xrand, linrand, trirand, coin, ...

Filters

onepole, onezero, leakdc, lag, combn, combl, pinkingFilter, ...

Envelopes and Dynamics

decay, decay2, fadein, tremolo, above, below, ...

Waveform Shaping

clip, wrap, fold, bwarp, warp, sigmoid, cheby, sstep, sstep2, fsin, fcos, ...

Range Mapping and Unit Conversion

linlin, linexp, explin, expexp, ampdb, dbamp, nnhz, hznn, bpmhz, degrad, ...

Signal Analysis

diff, slope, rising, falling, peak, trough, tr (trigger detect), eoc (end of cycle), sah (sample & hold), ...

Control

toggle, setreset, srt, pause, pull, init, ...