o aj# @sldZddlZddlZddlZddlZddlZddlZddlZej ej e Z ejdp:ej ej e dddZej deddlmZddlmZmZmZdd lmZmZmZmZej ed d d d Zej edddZdZ dZ!e"dddddddddd Z#d*ddZ$edfddZ%dded fd!d"Z&dded fd#d$Z'Gd%d&d&Z(d'd(Z)e*d)kre)dSdS)+uStandalone V3 face inference for the avatar E2E pipeline. Given an assistant utterance (16k mono wav) + an emotion tag, produce the 52 ARKit blendshape timeline the avatar viewer consumes — WITHOUT V3's TTS or KlueTeacher steps. In the E2E, OpenAI Realtime supplies the audio and the LLM supplies the emotion tag, so we feed mel + emotion/VAD straight into the V3 face model. This is the audio-driven subset of `face/animasync-face-v3/models/v3_face/serve_live.py`. Pipeline: wav ─► mel_features (T, 80) [raw log-mel, no norm — matches V3 npz "audio"] emotion ─► emotion_vad_anchors[emo][lvl].vad ─► build_conditioning (T, 19) [16 one-hot + 3 VAD] V3FaceModel(mel, cond) (T, 52) ─► crisp_mouth ─► smooth_brows ─► inject_blinks ─► clip[0,1] Runs in the repo `.venv` (same interpreter as the body model — imports verified). python face_v3_infer.py --wav --emotion joy --out out.json N FACE_V3_DIRz..facezanimasync-face-v3)ARKIT_52_NAMES)EMOTION_LABELS mel_featuresbuild_conditioning) load_model crisp_mouth smooth_brows inject_blinksmodelsv3_face checkpointszbest_expression_v14.ptdataemotionzemotion_vad_anchors.jsoni>g333333??g?g@g{Gz?g@g?) crisp_threshold crisp_scale crisp_sigmacrisp_mouthclose_sigmabrow_min_cutoff brow_beta brow_d_cutoffblink_intervalblink_expressive_capcsHtttd}|||d}t|fddd}t|dtjS)zLook up the affective VAD anchor (valence/arousal/dominance) for an emotion. Same selection gen_v3_face.py uses: the anchor entry whose intensity `level` is closest to the requested one. anchorsneutralcst|dS)Nlevel)abs)arface_v3_infer.py?sz emotion_to_vad..)keyvad) jsonloadopenANCHORSgetminnpasarrayfloat32)rrrlvpkr#r"r$emotion_to_vad7sr3cCs2|p ttjr dnd}t||\}}||fS)z?Load once, reuse across turns (the E2E server keeps this warm).zcuda:0cpu)torchdevicecuda is_availabler)ckptr6modelcfgr#r#r$load_face_modelCsr<e2ecCs(tj|tdd\}}t||||||dS)zGwav + emotion -> (T, 52) blendshapes in ARKit_52 order, post-processed.T)srmono)r model_devr9 scenario_id)librosar)SRrender_face_samples)wav_pathrrr@r9rAwav_r#r#r$ render_faceJs rHcCsv|durt|}|\}}tj|tjdd}|jdkr't|dd|jf}t|tt d}|j d} t ||} t || | } t.t|d|} t| d|} || | dtj}Wdn1suwYt|dd}t|td td td td d }t|tdtdtdt d}t||tdtdt d}t|ddtj| fS)zCIn-memory mono 16k PCM + emotion -> post-processed ARKit-52 frames.Ndtypeir)r>fpsrrrrr) thresholdscalepre_smooth_sigmamouth_close_sigmarrr) min_cutoffbetad_cutoffrLrr)rAmean_interval_sexpressive_caprL)r<r.r/r0reshapesizepadrrCFPSshaper3rtolistr5no_grad from_numpy unsqueezetosqueezer4numpyastypeclipr POSTr r )rFrrr@r9rAr:r6melTr'condaudiocbsr#r#r$rDRs6    " rDc@s$eZdZdZd ddZd ddZdS) FaceStreamzACausal V3 wrapper that returns frames aligned to each body chunk.rrcCs:||_||_||_||_tjdtjd|_d|_d|_ dS)NrrI) r@rArblendr.zerosr0pcm_lastframes_emitted)selfr@rArrmr#r#r$__init__ts zFaceStream.__init__NcCst|}|dur |jnt|}|dkrtd||jkr'td|jd|tj|tjdd}|jr>t|j |g|_ |dkrNtj dt t ftjdS||}tt |tt}|j dtt |j |}t|||j|j|jd\} } t | |krtj| d|t | fd fd d } | ||} |jdur|jdkrt|jt | } t| D]} | d | d }d ||j|| | | | <q| d|_||_t| dd tjS)aAppend PCM and return the exact absolute frame range emitted by body. ``librosa``'s centered STFT produces one more mel frame than the duration-derived 30fps body timeline (12,800 samples -> 25 mel frames, while body emits frames 0..23). Selecting ``bs[-n_frames:]`` therefore led face by one frame. Track the body frame cursor explicitly and slice ``[frame_start:frame_start + n_frames]`` instead. A request may also contain more PCM than body consumed into complete steps. Limit inference to the sample boundary represented by the body frame end so that pending future audio cannot leak into this packet's post-processing. Nrzn_frames must be non-negativez#face frame discontinuity: expected z, got rIrK)rr@rA)rredge)moderrM)intrq ValueErrorr.r/r0rWrX concatenaterornlenrroundrCrZr-rDrr@rArYcopyrprmrangerdrc)rr pcm_chunkrn_frames frame_startro frame_endrepresented_samples inference_pcmrkrGoutkir!r#r#r$push_pcm}s>      zFaceStream.push_pcm)rr)N)__name__ __module__ __qualname____doc__rsrr#r#r#r$rlqs  rlc st}|jddd|jddddtd|jd td d d |jd td|jddd|}|jtvrEt d|jdtt j t j |jd}t|j|j|j|j|d\}t jt j t j |jddtt j t j |jdtt|tfddt|Ddt|jdtd|jdd|dddddfdd ddd!fdd"ddd#fdd$|j dS)%Nz--wavT)requiredz --emotionrzone of: z, )defaulthelpz--levelrzVAD intensity 1-5)typerrz--ckpt)rz--outzunknown emotion u — must be one of r)r9rA)exist_okcsg|] }dd|DqS)cSsg|] }tt|dqSr)r{float).0vr#r#r$ sz#main...r#)rtrkr#r$rszmain..)rArL num_framesnames blendshapeswzface: 10s 4dzf jawOpenMax=z.2fz browInnerMax=z smileMax=,z -> )argparseArgumentParser add_argumentjoinrrw DEFAULT_CKPT parse_argsrsysexitospathsplitextbasenamerFrHrr9makedirsdirnameabspathrr(dumprZrr}r*printmax)apr!sidrgr#rr$mains:     r__main__r)+rrr(rrrbr.rBr5rrr__file__HEREenvironr,rFACEinsertscripts.compiler.constantsrscripts.compiler.data_pipelinerrrmodels.v3_face.inferrr r r rr+rZrCdictrer3r<rHrDrlrrr#r#r#r$sJ      ?