Transformers documentation
PE Audio
This model was published in HF papers on 2025-04-17 and contributed to Hugging Face Transformers on 2025-12-16. This model was released on 2025-04-17 and added to Hugging Face Transformers on 2025-12-16.
PE Audio
PE Audio is the audio branch of Meta’s Perception Encoder family. It contrastively aligns raw waveforms with text into a shared embedding space, trained on paired audio–caption data for cross-modal retrieval and zero-shot audio classification.
Two heads are exposed on top of the same encoder. PeAudioModel returns one pooled embedding per clip for clip-level retrieval, while PeAudioFrameLevelModel returns one embedding every 40 ms for event localization and fine-grained temporal analysis.
You can find all the official PE Audio checkpoints under the perception-encoder-audio-visual collection.
Quickstart
import torch
from datasets import load_dataset
from transformers import AutoProcessor, PeAudioModel
processor = AutoProcessor.from_pretrained("facebook/pe-av-large")
model = PeAudioModel.from_pretrained(
"facebook/pe-av-large",
device_map="auto",
)
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio = ds[0]["audio"]["array"]
labels = ["a dog barking", "a person speaking", "music playing"]
audio_inputs = processor.feature_extractor(audio, sampling_rate=48_000, return_tensors="pt").to(model.device)
text_inputs = processor.tokenizer(labels, padding=True, return_tensors="pt").to(model.device)
inputs = {**audio_inputs, **text_inputs}
with torch.no_grad():
outputs = model(**inputs)
probs = outputs.logits_audio_text.sigmoid()
print({label: p.item() for label, p in zip(labels, probs[0])})Usage tips and notes
- Audio must be mono (
feature_size=1) and resampled to 48 kHz — the feature extractor warns but does not resample for you. Stereo input is not supported. - Variable-length audio is handled with
padding_mask(not the usualattention_mask). The mask is downsampled internally bydac_config.hop_lengthbefore it reaches the encoder, so pass the raw waveform-resolution mask that the feature extractor returns. - PeAudioModel returns logits of shape
(n_audio, n_text). PeAudioFrameLevelModel returns(n_audio, n_text, n_frames)with one frame every 40 ms. Pick the class that matches the task — they share weights so swapping is cheap. - The text tower is a shared encoder loaded via
AutoModelfromconfig.text_config. The tokenizer is attached to the processor viaAutoTokenizer, not a dedicated class.
PeAudioConfig
class transformers.PeAudioConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = Nonetext_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Noneaudio_config: dict | transformers.configuration_utils.PreTrainedConfig | None = None )
This is the configuration class to store the configuration of a PeAudioModel. It is used to instantiate a Pe Audio model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the facebook/pe-av-large
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.
Example:
>>> from transformers import PeAudioModel, PeAudioConfig
>>> # Initializing a PeAudioModel style configuration
>>> configuration = PeAudioConfig()
>>> # Initializing a model from the pe-av-large style configuration
>>> model = PeAudioModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configPeAudioEncoderConfig
class transformers.PeAudioEncoderConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = Nonedac_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonehidden_size: int = 1792intermediate_size: int = 4800num_hidden_layers: int = 6num_attention_heads: int = 14num_key_value_heads: int | None = Nonehead_dim: int = 128hidden_act: str = 'silu'max_position_embeddings: int = 10000initializer_range: float = 0.02rms_norm_eps: float = 1e-05rope_parameters: transformers.modeling_rope_utils.RopeParameters | dict | None = Noneattention_bias: bool = Falseattention_dropout: float | int = 0.0 )
Parameters
- dac_config (
Union[PreTrainedConfig, dict], optional) — Configuration for the DAC audio encoder used to tokenize the raw audio inputs. If a dictionary is passed, it will be used to instantiate a DacConfig with default DAC hyperparameters. - hidden_size (
int, optional, defaults to1792) — Dimension of the hidden representations. - intermediate_size (
int, optional, defaults to4800) — Dimension of the MLP representations. - num_hidden_layers (
int, optional, defaults to6) — Number of hidden layers in the Transformer decoder. - num_attention_heads (
int, optional, defaults to14) — Number of attention heads for each attention layer in the Transformer decoder. - num_key_value_heads (
int, optional) — This is the number of key_value heads that should be used to implement Grouped Query Attention. Ifnum_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), ifnum_key_value_heads=1the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default tonum_attention_heads. - head_dim (
int, optional, defaults to128) — The attention head dimension. If None, it will default to hidden_size // num_attention_heads - hidden_act (
str, optional, defaults tosilu) — The non-linear activation function (function or string) in the decoder. For example,"gelu","relu","silu", etc. - max_position_embeddings (
int, optional, defaults to10000) — The maximum sequence length that this model might ever be used with. - initializer_range (
float, optional, defaults to0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - rms_norm_eps (
float, optional, defaults to1e-05) — The epsilon used by the rms normalization layers. - rope_parameters (
Union[~modeling_rope_utils.RopeParameters, dict], optional) — Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value forrope_thetaand optionally parameters used for scaling in case you want to use RoPE with longermax_position_embeddings. - attention_bias (
bool, optional, defaults toFalse) — Whether to use a bias in the query, key, value and output projection layers during self-attention. - attention_dropout (
Union[float, int], optional, defaults to0.0) — The dropout ratio for the attention probabilities.
This is the configuration class to store the configuration of a PeAudioModel. It is used to instantiate a Pe Audio model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the facebook/pe-av-large
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.
>>> from transformers import PeAudioEncoder, PeAudioEncoderConfig
>>> # Initializing a PeAudioEncoder style configuration
>>> configuration = PeAudioEncoderConfig()
>>> # Initializing a model from the pe-av-large style configuration
>>> model = PeAudioEncoder(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configPeAudioFeatureExtractor
class transformers.PeAudioFeatureExtractor
< source >( feature_size: int = 1sampling_rate: int = 48000padding_value: float = 0.0hop_length: int = 1920**kwargs )
Parameters
- feature_size (
int, optional, defaults to 1) — The feature dimension of the extracted features. Use 1 for mono, 2 for stereo. - sampling_rate (
int, optional, defaults to 48000) — The sampling rate at which the audio waveform should be digitalized, expressed in hertz (Hz). - padding_value (
float, optional, defaults to 0.0) — The value that is used for padding. - hop_length (
int, optional, defaults to 1920) — Overlap length between successive windows.
Constructs a PeAudioFeatureExtractor feature extractor.
This feature extractor inherits from SequenceFeatureExtractor which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.
__call__
< source >( raw_audio: numpy.ndarray | list[float] | list[numpy.ndarray] | list[list[float]] | str | list[str]padding: bool | str | transformers.utils.generic.PaddingStrategy | None = Nonetruncation: bool | None = Falsemax_length: int | None = Nonereturn_tensors: str | transformers.utils.generic.TensorType | None = Nonesampling_rate: int | None = None )
PeAudioProcessor
PeAudioEncoder
class transformers.PeAudioEncoder
< source >( config: PeAudioEncoderConfig )
Parameters
- config (PeAudioEncoderConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
The PeAudio Encoder model.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = None**kwargs ) → BaseModelOutputWithPooling or tuple(torch.FloatTensor)
Parameters
- input_values (
torch.Tensorof shape(batch_size, sequence_length)) — Float values of input raw speech waveform. Values can be obtained by loading a.flacor.wavaudio file into an array of typelist[float], anumpy.ndarrayor atorch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array intoinput_values, the AutoProcessor should be used for padding and conversion into a tensor of typetorch.FloatTensor. See PeAudioProcessor.call() for details. - padding_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding samples ofinput_values. Mask values selected in[0, 1]:- 1 for samples that are not masked,
- 0 for samples that are masked.
Returns
BaseModelOutputWithPooling or tuple(torch.FloatTensor)
A BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (PeAudioConfig) and inputs.
The PeAudioEncoder forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.pooler_output (
torch.FloatTensorof shape(batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.FloatTensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
PeAudioModel
forward
< source >( input_ids: Tensorinput_values: Tensorattention_mask: typing.Optional[torch.Tensor] = Nonepadding_mask: typing.Optional[torch.Tensor] = Nonereturn_loss: bool | None = None**kwargs ) → PeAudioOutput or tuple(torch.FloatTensor)
Parameters
- input_ids (
torch.Tensorof shape(batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
- input_values (
torch.Tensorof shape(batch_size, sequence_length)) — Float values of input raw speech waveform. Values can be obtained by loading a.flacor.wavaudio file into an array of typelist[float], anumpy.ndarrayor atorch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array intoinput_values, the AutoProcessor should be used for padding and conversion into a tensor of typetorch.FloatTensor. See PeAudioProcessor.call() for details. - attention_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that are not masked,
- 0 for tokens that are masked.
- padding_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding samples ofinput_values. Mask values selected in[0, 1]:- 1 for samples that are not masked,
- 0 for samples that are masked.
- return_loss (
bool, optional) — Whether or not to return the loss.
Returns
PeAudioOutput or tuple(torch.FloatTensor)
A PeAudioOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (PeAudioConfig) and inputs.
The PeAudioModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
- loss (
torch.FloatTensorof shape(1,), optional) — Contrastive loss computed between audio and text representations. - logits_audio_text (
torch.FloatTensorof shape(batch_size, batch_size), optional) — Similarity logits between audio and text embeddings. PeAudioFrameLevelModel returns per-frame logits of shape(batch_size, batch_size, sequence_length)instead. - text_audio_embeds (
torch.FloatTensorof shape(batch_size, hidden_size), optional) — Text embeddings projected to the audio-text space. - audio_embeds (
torch.FloatTensorof shape(batch_size, hidden_size), optional) — Audio embeddings projected to the audio-text space. PeAudioFrameLevelModel returns per-frame embeddings of shape(batch_size, sequence_length, hidden_size)instead. - text_outputs (
BaseModelOutputWithPooling, optional) — Model outputs for the text encoder, including last hidden state and pooled output. - audio_outputs (
BaseModelOutputWithPooling, optional) — Model outputs for the audio encoder, including last hidden state and pooled output.
PeAudioFrameLevelModel
forward
< source >( input_ids: Tensorinput_values: Tensorattention_mask: typing.Optional[torch.Tensor] = Nonepadding_mask: typing.Optional[torch.Tensor] = Nonereturn_loss: bool | None = None**kwargs ) → PeAudioOutput or tuple(torch.FloatTensor)
Parameters
- input_ids (
torch.Tensorof shape(batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
- input_values (
torch.Tensorof shape(batch_size, sequence_length)) — Float values of input raw speech waveform. Values can be obtained by loading a.flacor.wavaudio file into an array of typelist[float], anumpy.ndarrayor atorch.Tensor, e.g. via the torchcodec library (pip install torchcodec) or the soundfile library (pip install soundfile). To prepare the array intoinput_values, the AutoProcessor should be used for padding and conversion into a tensor of typetorch.FloatTensor. See PeAudioProcessor.call() for details. - attention_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that are not masked,
- 0 for tokens that are masked.
- padding_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding samples ofinput_values. Mask values selected in[0, 1]:- 1 for samples that are not masked,
- 0 for samples that are masked.
- return_loss (
bool, optional) — Whether or not to return the loss.
Returns
PeAudioOutput or tuple(torch.FloatTensor)
A PeAudioOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (PeAudioConfig) and inputs.
The PeAudioFrameLevelModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
- loss (
torch.FloatTensorof shape(1,), optional) — Contrastive loss computed between audio and text representations. - logits_audio_text (
torch.FloatTensorof shape(batch_size, batch_size), optional) — Similarity logits between audio and text embeddings. PeAudioFrameLevelModel returns per-frame logits of shape(batch_size, batch_size, sequence_length)instead. - text_audio_embeds (
torch.FloatTensorof shape(batch_size, hidden_size), optional) — Text embeddings projected to the audio-text space. - audio_embeds (
torch.FloatTensorof shape(batch_size, hidden_size), optional) — Audio embeddings projected to the audio-text space. PeAudioFrameLevelModel returns per-frame embeddings of shape(batch_size, sequence_length, hidden_size)instead. - text_outputs (
BaseModelOutputWithPooling, optional) — Model outputs for the text encoder, including last hidden state and pooled output. - audio_outputs (
BaseModelOutputWithPooling, optional) — Model outputs for the audio encoder, including last hidden state and pooled output.