@aosengine/conversation

Interfaces

ConversationCharacter

Story configuration is host-owned; game code names only a character id.

Properties

prefabs?

optional prefabs?: Readonly<Record<string, readonly SpeechChunk[] | undefined>>;

This character's prepared performances, selected by the service.

storyId

storyId: string;

url

url: string;

---

ConversationModule

Session control. Ending an interview retains its session and story revision.

Extends

• EngineModule

Properties

id

readonly id: string;

Unique id. Also the service id when init returns a service.

Inherited from

EngineModule.id

order?

readonly optional order?: number;

Sort key. Lower runs first; equal keys keep registration order.

Rough convention: input -100, gameplay -50, physics 0, rendering helpers 200. Gameplay runs before physics so that the commands a guest tick emits are simulated by the step that follows rather than the next one.

Inherited from

EngineModule.order

Methods

ask()

ask(text): boolean;

Parameters

Parameter	Type
text	string

Returns

boolean

beginFrame()?

optional beginFrame(): void;

Run before the frame's fixed steps. Input capture lives here.

Returns

void

Inherited from

EngineModule.beginFrame

dispose()

dispose(): void;

Release everything this module took.

Returns

void

Inherited from

EngineModule.dispose

end()

end(): void;

Returns

void

endFrame()?

optional endFrame(): void;

Run after rendering. Input's end-of-frame bookkeeping lives here.

Returns

void

Inherited from

EngineModule.endFrame

fixedUpdate()?

optional fixedUpdate(dt): void;

Run once per fixed step.

Parameters

Parameter	Type	Description
dt	number	Always ctx.config.fixedDt, whatever time.timeScale is; time scale changes how many steps a frame runs, not their length.

Returns

void

Inherited from

EngineModule.fixedUpdate

init()

init(ctx): void | object | Promise<void | object>;

Acquire resources.

Parameters

Parameter	Type	Description
ctx	EngineContext	The host surface.

Returns

void | object | Promise<void | object>

Nothing, or the service to publish under id.

Inherited from

EngineModule.init

interrupt()

interrupt(): void;

Returns

void

start()

start(character): void;

Parameters

Parameter	Type
character	string

Returns

void

update()?

optional update(dt, alpha): void;

Run once per frame, after the fixed steps and before rendering.

Parameters

Parameter	Type	Description
dt	number	Clamped wall-clock seconds since the previous frame, scaled by time.timeScale.
alpha	number	Interpolation factor in [0, 1) for smoothing transforms.

Returns

void

Inherited from

EngineModule.update

---

ConversationOptions

Optional conversation dependencies and callbacks.

Properties

characters

characters: Readonly<Record<string, ConversationCharacter | undefined>>;

player

player: SpeechPlayer;

prefabs?

optional prefabs?: Readonly<Record<string, readonly SpeechChunk[] | undefined>>;

Prepared performances keyed by Convorcher prefab id; no story mutations.

Methods

connect()?

optional connect(url): ConversationSocket;

Parameters

Parameter	Type
url	string

Returns

ConversationSocket

onError()

onError(message): void;

Parameters

Parameter	Type
message	string

Returns

void

onStatus()

onStatus(status): void;

Parameters

Parameter	Type
status	string

Returns

void

onStory()

onStory(character, snapshot): void;

Parameters

Parameter	Type
character	string
snapshot	StorySnapshot

Returns

void

---

ConversationSocket

Browser connection surface; injectable for service replay tests.

Properties

onclose

onclose: ((event) => void) | null;

onerror

onerror: ((event) => void) | null;

onmessage

onmessage: ((event) => void) | null;

onopen

onopen: ((event) => void) | null;

readyState

readyState: number;

Methods

close()

close(): void;

Returns

void

send()

send(text): void;

Parameters

Parameter	Type
text	string

Returns

void

---

FaceFrame

Face frames, with seconds relative to their audio chunk: 52 weights in ARKit-52 order, or, when the session asked the conversation service for the head model's own face (a2f=isaac-gnm), its 383 expression codes.

Properties

blendshapeWeights

blendshapeWeights: readonly number[];

timeCode

timeCode: number;

---

GnmVoice

The voice model's published information, which turns a native frame's codes into head coefficients: coefficient = scale x (code - offset), held within limit (see parseGnmVoice).

Properties

limit?

optional limit?: Float32Array<ArrayBufferLike>;

Per coefficient, GNM_CLAMP_STD x |std x scale| (std: the spread the voice model was trained with). Past it the head model's lips cross and the splats on them tear, so each coefficient is held inside. Absent when the file carries no std.

offset

offset: Float32Array;

scale

scale: Float32Array;

---

SpeechChunk

One bounded speech performance, already decoded from the service wire format.

Properties

frames

frames: readonly FaceFrame[];

gesture?

optional gesture?: string;

pcm

pcm: Uint8Array;

text

text: string;

---

SpeechPlayer

Audio-clock speech queue; update reuses a single facial vector.

Methods

dispose()

dispose(): void;

Returns

void

enqueue()

enqueue(chunk): void;

Parameters

Parameter	Type
chunk	SpeechChunk

Returns

void

stop()

stop(): void;

Returns

void

update()

update(): void;

Returns

void

---

SpeechPlayerOptions

Options for createSpeechPlayer.

Properties

gnmVoice?

optional gnmVoice?: GnmVoice | (() => GnmVoice | null);

Take native frames: the parsed voice information (parseGnmVoice), or a getter while it loads (null then: the audio plays and the face rests). Without it a 383-wide chunk is refused, as it always was.

---

SpeechPresentation

Presentation callbacks run on the audio clock.

Methods

expression()

expression(weights, space): void;

The face now. space says what weights holds: 52 ARKit weights in [0, 1] (arkit52), or, from native frames, the head's 387 gnm coefficients (the character bridge takes both as they are: characters.setExpression(entity, space, weights)). A presentation that only drives ARKit may ignore space: a player made without gnmVoice never sends gnm.

Parameters

Parameter	Type
weights	Float32Array
space	SpeechFaceSpace

Returns

void

gesture()

gesture(id): void;

Parameters

Parameter	Type
id	string

Returns

void

reset()

reset(): void;

Returns

void

speaking()

speaking(active): void;

Parameters

Parameter	Type
active	boolean

Returns

void

subtitle()

subtitle(text): void;

Parameters

Parameter	Type
text	string

Returns

void

---

StorySnapshot

Stable full-state message emitted by Convorcher after analysis and on join.

Properties

completedObjectives

completedObjectives: string[];

currentPhase

currentPhase: string;

playerState

playerState: Record<string, string>;

revision

revision: number;

schemaVersion

schemaVersion: 1;

sessionId

sessionId: string;

storyId

storyId: string;

transitionCause

transitionCause: string;

Type Aliases

SpeechFaceSpace

type SpeechFaceSpace = "arkit52" | "gnm";

What a face vector holds: ARKit-52 weights, or the head model's native gnm coefficients.

Variables

ARKIT_FRAME_WIDTH

const ARKIT_FRAME_WIDTH: 52 = 52;

ARKit weights in one frame.

---

GNM_CLAMP_STD

const GNM_CLAMP_STD: 3 = 3;

How many standard deviations of the voice model's training spread a coefficient may reach.

---

GNM_FRAME_WIDTH

const GNM_FRAME_WIDTH: 383 = 383;

The head model's expression codes in one native frame (a2f=isaac-gnm).

---

GNM_SPACE_DIM

const GNM_SPACE_DIM: 387 = 387;

The gnm expression space: the 383 coefficients, then a four-wide gaze tail the player leaves at zero.

Functions

acceptStorySnapshot()

function acceptStorySnapshot(
   current, 
   next, 
   sessionId, 
   storyId
): StorySnapshot | null;

Idempotent reducer scoped to an expected session and story.

Parameters

Parameter	Type	Description
current	StorySnapshot | null	Last accepted snapshot.
next	StorySnapshot	Candidate snapshot.
sessionId	string	Expected session.
storyId	string	Expected story.

Returns

StorySnapshot | null

The accepted snapshot, or the previous state.

---

conversation()

function conversation(options): ConversationModule;

Build an optional session manager. Construction does not connect to any service.

Parameters

Parameter	Type	Description
options	ConversationOptions	Explicit character allow-list, playback and callbacks.

Returns

ConversationModule

An engine module. Call start only after an interview is requested.

---

createSpeechPlayer()

function createSpeechPlayer(
   context, 
   presentation, 
   destination?, 
   options?
): SpeechPlayer;

Build a bounded PCM player. The caller creates/resumes the context in a user gesture.

Frames 52 wide play as ARKit-52 weights. Frames 383 wide (the head model's own face, a2f=isaac-gnm) play as the head's gnm coefficients when options.gnmVoice is given: coefficient = scale x (code - offset), held within GNM_CLAMP_STD x |std x scale|, interpolated at the frames' own times and eased in and out at the ends of a run of speech; without it they are refused.

Parameters

Parameter	Type	Default value	Description
context	AudioContext	undefined	Shared or dedicated audio clock.
presentation	SpeechPresentation	undefined	Subtitles, gestures and facial rig.
destination	AudioNode	context.destination	Voice bus; defaults to the context destination.
options	SpeechPlayerOptions	{}	Native frames (gnmVoice).

Returns

SpeechPlayer

A player which owns only its sources, not the context.

---

faceSpaceOf()

function faceSpaceOf(width, native): SpeechFaceSpace | null;

The space a chunk's frames drive, from their width.

Parameters

Parameter	Type	Description
width	number	Numbers per frame.
native	boolean	Whether the player takes native frames (it was given gnmVoice).

Returns

SpeechFaceSpace | null

arkit52 for 52, gnm for 383 when native frames are taken; null for anything else (refused).

---

gnmCoefficients()

function gnmCoefficients(
   codes, 
   voice, 
   out?
): Float32Array;

One native frame's codes as head coefficients, each held within the voice's limit, with a zero gaze tail.

Parameters

Parameter	Type	Description
codes	ArrayLike<number>	The frame's 383 codes as the service sends them.
voice	GnmVoice	The voice information.
out	Float32Array	Output of length GNM_SPACE_DIM, reused.

Returns

Float32Array

out.

---

parseGnmVoice()

function parseGnmVoice(raw): GnmVoice | null;

Read a voice information file (gnm_voice.json: scale, offset and optionally std, 383 numbers each).

Parameters

Parameter	Type	Description
raw	unknown	The parsed JSON.

Returns

GnmVoice | null

The voice, with limit when the file carries a valid std; null when it cannot be used.

---

parseStorySnapshot()

function parseStorySnapshot(value): StorySnapshot | null;

Validate a bounded, versioned story snapshot.

Parameters

Parameter	Type	Description
value	unknown	Untrusted message data.

Returns

StorySnapshot | null

Owned snapshot, or null for invalid data.