% &hddlmZddlmZmZmZmZddlZddlm Z ddl m Z ddl m Z mZmZddlmZddlmZdd lmZmZdd lmZdd lmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6e*jne8Z9dZ:dZ;GddeZ<Gdde jzZ>Gdde jzZ?Gdde-Z@Gdde.ZAGdd e1ZBGd!d"e jzZCd#ZDe'd$eDGd%d&e#ZEGd'd(eEZFGd)d*e2ZGd+ZHe'd$eDGd,d-e5ZIe'd.eDGd/d0eEeZJgd1ZKy)2)partial)CallableOptionalTupleUnionN)ACT2FN)Cache DynamicCacheEncoderDecoderCache)PretrainedConfig)GenerationMixin)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)BaseModelOutputBaseModelOutputWithPast)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)rope_config_validation)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)add_start_docstrings%add_start_docstrings_to_model_forwardcan_return_tupleloggingreplace_return_docstrings) GlmAttentionGlmRotaryEmbeddingapply_rotary_pos_emb)LlamaDecoderLayer LlamaModeleager_attention_forward) WhisperModelshift_tokens_rightzUsefulSensors/moonshine-tinyMoonshineConfigcjeZdZdZdZdgZddddZ d fd ZxZS) r)a" This is the configuration class to store the configuration of a [`MoonshineModel`]. It is used to instantiate a Moonshine model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Moonshine [UsefulSensors/moonshine-tiny](https://huggingface.co/UsefulSensors/moonshine-tiny). Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 32768): Vocabulary size of the Moonshine model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`MoonshineModel`]. hidden_size (`int`, *optional*, defaults to 288): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. encoder_num_hidden_layers (`int`, *optional*, defaults to 6): Number of hidden layers in the Transformer encoder. decoder_num_hidden_layers (`int`, *optional*, defaults to 6): Number of hidden layers in the Transformer decoder. encoder_num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer encoder. decoder_num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. encoder_num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `encoder_num_key_value_heads=encoder_num_attention_heads`, the model will use Multi Head Attention (MHA), if `encoder_num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `num_attention_heads`. decoder_num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `decoder_num_key_value_heads=decoder_num_attention_heads`, the model will use Multi Head Attention (MHA), if `decoder_num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `decoder_num_attention_heads`. pad_head_dim_to_multiple_of (`int`, *optional*): Pad head dimension in encoder and decoder to the next multiple of this value. Necessary for using certain optimized attention implementations. encoder_hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder. decoder_hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 512): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. decoder_start_token_id (`int`, *optional*, defaults to 1): Corresponds to the "<|startoftranscript|>" token, which is automatically used when no `decoder_input_ids` are provided to the `generate` function. It is used to guide the model`s generation process depending on the task. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. `beta_fast` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. `beta_slow` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. `short_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `long_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `low_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE `high_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE partial_rotary_factor (`float`, *optional*, defaults to 0.9): Percentage of the query and keys which will have rotary embedding. is_encoder_decoder (`bool`, *optional*, defaults to `True`): Whether the model is used as an encoder/decoder or not. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. bos_token_id (`int`, *optional*, defaults to 1): Denotes beginning of sequences token id. eos_token_id (`int`, *optional*, defaults to 2): Denotes end of sequences token id. Example: ```python >>> from transformers import MoonshineModel, MoonshineConfig >>> # Initializing a Moonshine style configuration >>> configuration = MoonshineConfig().from_pretrained("UsefulSensors/moonshine-tiny") >>> # Initializing a model from the configuration >>> model = MoonshineModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` moonshinepast_key_valuesencoder_num_key_value_headsencoder_num_attention_headsencoder_num_hidden_layers)num_key_value_headsnum_attention_headsnum_hidden_layersc ||_||_||_||_||_||_||_||}||_| |} | |_| |_ | |_ | |_ | |_ ||_ ||_||_||_||_||_||_||_||_t-|t/|`d||||d|y)N) bos_token_id eos_token_idis_encoder_decoderdecoder_start_token_id) vocab_size hidden_sizeintermediate_sizer/decoder_num_hidden_layersr.decoder_num_attention_headsr-decoder_num_key_value_headspad_head_dim_to_multiple_ofencoder_hidden_actdecoder_hidden_actmax_position_embeddingsinitializer_ranger7 use_cache rope_theta rope_scalingpartial_rotary_factorr6attention_biasattention_dropoutrsuper__init__)selfr9r:r;r/r<r.r=r-r>r?r@rArBrCr7rDrErFrGr6rHrIr4r5kwargs __class__s /var/www/pru.catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/moonshine/modular_moonshine.pyrKzMoonshineConfig.__init__s8%&!2)B&)B&+F(+F( & .*E '+F( & .*E '+F(+F("4"4'>$!2&<#"$(%:""4,!2 t$  %%1#9    )ii irQrRNNNgelusiluig{Gz?Tg@Ng?TFrUr ) __name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inference attribute_maprK __classcell__rNs@rOr)r):s{ zJ#4"5<<8M"#"#$%$%$($($(!! # !3D D rPcVeZdZfdZdej dej fdZxZS)MoonshineEncoderMLPct|||_t||_t j |j|j|_ t j |j|j|_ yN rJrKconfigr activation_fnnnLinearr:r;fc1fc2rLre hidden_actrNs rOrKzMoonshineEncoderMLP.__init__s^  #J/99V//1I1IJ99V55v7I7IJrP hidden_statesreturncl|j|}|j|}|j|}|Src)rirfrj)rLrms rOforwardzMoonshineEncoderMLP.forwards4/ **=9 / rPrWrXrYrKtorchTensorrpr^r_s@rOraras$KU\\ellrPracVeZdZfdZdej dej fdZxZS)MoonshineDecoderMLPct|||_t||_t j |j|jdz|_ t j |j|j|_ y)Nr rdrks rOrKzMoonshineDecoderMLP.__init__sc  #J/99V//1I1IA1MN99V55v7I7IJrPrmrnc|j|}|jdd\}}|j||z}|j|}|S)Nr )dim)richunkrfrj)rLrmgates rOrpzMoonshineDecoderMLP.forwardsS/ +11!1< t**40=@ / rPrqr_s@rOrurus$KU\\ellrPrucheZdZdededededef fd Z ddejde e ejejfd e ejd e e d e ejd e ejd e ede eje eje e ejffdZxZS)MoonshineAttentionre layer_idx is_causalr1r0cn|j||dt| ||||_t |d|j |j z|_|jjC|jj}||j|zdz |zz}||jz |_ yd|_ y)N)r1r0head_dimrUr) updaterJrKrgetattrr:r1rrer?head_dim_padding) rLrer~rr1r0target_multipletarget_head_dimrNs rOrKzMoonshineAttention.__init__'s  .AZmno +" F4F4F&JdJd4de  ;; 2 2 >"kkEEO-$--/2QTU2UZi1ijO$3dmm$CD !$%D !rPrmposition_embeddingsattention_maskpast_key_valuecache_positionkey_value_statesrMrnc |jdd\}} |j|j|| |jj|j j dd} |du} |Y|jj|j} | r&d|j|j<|j}n |j}||n|} | r7|r5 r3|j|j}|j|j}n|j| j|d|jj|j j dd}|j| j|d|jj|j j dd}| r%|#|j!|||jd|i\}}| s?|\}}t#| |||\} }|'|||d}|j!|||j|\}}t$}|jj&dk7r^|jj&dk(r(|jd d rt(j+d nt,|jj&}|j.r || dkDrdnd }|j0d kDrt2j4j6j9| d |j0f} t2j4j6j9|d |j0f}t2j4j6j9|d |j0f}||| |||f|j:sd n |j<|j>|d|\}}|j0d kDr|dd|j0 f}|jA|| djC}|jE|}||fS)NrxrUr Tr)sincosreagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rrV)dropoutscalingr.)#shapeq_projviewrer0r transpose is_updatedgetr~cross_attention_cacheself_attention_cache key_cache value_cachek_projv_projrr#r&_attn_implementationlogger warning_oncerrrrrrg functionalpadtrainingrIrreshape contiguouso_proj)rLrmrrrrrrMbszq_len query_statesis_cross_attentionrcurrent_states key_states value_statesrr cache_kwargsattention_interfacer attn_output attn_weightss rOrpzMoonshineAttention.forward<s#(("- U KK & + +C 8W8WY]YfYf g q qrsuv w .T9  %'2266t~~FJ!<@))$..9!/!E!E!/!D!D.>-I)} .Z'11$..AJ)55dnnEL N+c2t{{>> N1a   N+c2t{{>> N1a  "n&@+9+@+@ dnn?OQ_>`,( L"*HC';L*VY[^'_ $L*)'*3.Y +9+@+@ dnnl,( L)@ ;; + +w 6{{//69fjjI\^c>d##L '>dkk>^>^&_# NN~/E%RS)DY^  1 $ 88..22Q5RR S[)rPr}c eZdZy)MoonshineRotaryEmbeddingN)rWrXrYr8rPrOrrsrPrc(eZdZdedeffd ZxZS)MoonshineEncoderLayerrer~cFt|||t||d|j|j|_t ||j|_tj|jd|_ tj|jd|_ y)NFrer~rr1r0bias)rJrKr}r.r- self_attnrar@mlprg LayerNormr:input_layernormpost_attention_layernormrLrer~rNs rOrKzMoonshineEncoderLayer.__init__s ++ & B B & B B  'vv/H/HI!||F,>,>UK(* V5G5Ge(T%rP)rWrXrYr)rrKr^r_s@rOrrs U U3 U UrPrc eZdZddedeeffd Z ddejdeejdeejdeejdeejd eejd ee d ee d ee d eejdee ejejfdee ejejfde ejee ejejfffdZxZS)MoonshineDecoderLayerrer~ct||j|_t||d|j|j |_t||d|j|j |_t||j|_ tj|jd|_ tj|jd|_tj|jd|_y)NTrFr)rJrKr:r}r=r>r encoder_attnrurArrgrrrfinal_layernormrs rOrKzMoonshineDecoderLayer.__init__s !--+ & B B & B B  / & B B & B B  'vv/H/HI!||F,>,>UK(* V5G5Ge(T%!||F,>,>UKrPrmrencoder_hidden_statesencoder_attention_mask position_idsencoder_position_idsrrrDrrencoder_position_embeddingsrnc H|}|j|}|jd|||||| | | d| \}}||z}d}|2|}|j|}|j|||||| \}}||z}|}|j |}|j |}||z}|f}|r|||fz }|S)N)rmrrrrrDrr)rmrrrrrDr8)rrrrrr)rLrmrrrrrrrrDrrrrMresidualself_attn_weightscross_attn_weightsoutputss rOrpzMoonshineDecoderLayer.forwards !,,]; ,:4>> , ')%)/) 3 ,  , ( (!=0 " ,$H 99-HM040A0A+!65-"3# 1B1 -M-%}4M!,,]; /  =0  "  )+=> >GrPrc) NNNNNNFFNNN)rWrXrYr)rrrKrrrsrr rr FloatTensorrpr^r_s@rOrrsjLL8C=L6268<9=37;?*.,1$)59KOSW<||<!.< ( 5 < !) 6 < u//0 <'u'7'78<!<$D><D><!!1!12<&eELL%,,,F&GH<&.eELL%,,4N.O%P< u  (51B1BEDUDU1U+V"WW X<rPraN This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MoonshineConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zWThe bare Moonshine Model outputting raw hidden-states without any specific head on top.cZeZdZeZdZdZdZddgZdZ dZ dZ dZ dZ dejfdZy ) MoonshinePreTrainedModelmodel input_valuesTrrc6|jj}t|tjtj frY|j jjd||j%|jjjyyt|tjrf|j jjd||j2|j j|jjyyy)NrV)meanstd) rerC isinstancergrhConv1dweightdatanormal_rzero_ Embedding padding_idx)rLmodulers rO _init_weightsz&MoonshinePreTrainedModel._init_weights'skk++ fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..rP input_lengthsc~t|dz dz dz}t|dz dz dz}t|dz dz dz}|S)zH Computes the output length of the convolutional layers @rUrr )r)rLroutput_conv1_lengthoutput_conv2_lengthoutput_conv3_lengths rO _get_feat_extract_output_lengthsz9MoonshinePreTrainedModel._get_feat_extract_output_lengths2sZ"=3#6""N>N#rPrceZdZdZdZdeffd ZdejfdZ dejfdZ e dde e jd e e jd e ed e ed eedef d ZxZS)MoonshineEncoderz Transformer encoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MoonshineEncoderLayer`] Args: config: MoonshineConfig rrec bt||||_|j}t j d|ddd|_t j |d|zdd |_t j d|z|dd |_t jd|d |_ t| |_ t jt|jDcgc]}t!||c}|_t j$|d |_d|_|j+ycc}w)NrUrrF) kernel_sizestriderr rr)rrgh㈵>) num_groups num_channelseps)rer)rJrKrer:rgrconv1conv2conv3 GroupNorm groupnormr rotary_emb ModuleListranger/rlayersr layer_normgradient_checkpointing post_init)rLre embed_dimidxrNs rOrKzMoonshineEncoder.__init__Gs   && YYq)ReT YYy!i-QqQ YYq9}iQqQ PTU2&Amm;@AaAa;b cC "63 / c  ,,yu=&+#  dsD,rnc|jSrcrrLs rOget_input_embeddingsz%MoonshineEncoder.get_input_embeddings[s zzrPvaluec||_yrcr)rLrs rOset_input_embeddingsz%MoonshineEncoder.set_input_embeddings^s  rPrroutput_hidden_statesflash_attn_kwargsc H||n|jj}||n|jj}| td|j d}t j j|j|}|j|}t j j|j|}t j j|j|}|jddd}||j|jd}d}|ddd|fdd|f}|jj d k(r|d k(j#r|nd}nH|jj d k(r|st%||j&}nt)||j&}t+j,d|jd|j. j d} |j1|| } |rd nd} |rd nd} |j2D]e} |r| |fz } |j4r0|j6r$|j9| j:||| d|dd| }n| |f|| || d|}|d}|s]| |dfz } g|j=|}|r| |fz } t?|| | S)a Args: input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `input_values`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzYou must specify input_values.rUrr rx.flash_attention_2rVrdevicer8F)rrrrlast_hidden_staterm attentions) rerr ValueError unsqueezergrtanhrrrSrrpermuterrranyrdtyperrrarangerrrr r_gradient_checkpointing_func__call__r r)rLrrrrrrmmask_lendownsample_striderrall_hidden_statesall_self_attns encoder_layer layer_outputss rOrpzMoonshineEncoder.forwardas>2C1N-TXT_T_TqTq$8$D $++JjJj   => >$--a0  **4::l+CD }5  **4::m+DE  **4::m+DE %--aA6   %<<^=Q=QRT=UVH * +C1D3D1D,DEc9H9nUN{{//3FF4Bc4I3N3N3PVZ11V;DU!D^UbUhUh!i" 6  -!1 1 &++%  rP)NNNN)rWrXrYrZrr)rKrgModulerrrrrrrrsrrrrrpr^r_s@rOrr=s%O(bii"))5915,0/3 p u001p !.p $D> p 'tn p $$89 p  !p p rPrcZeZdZdZdeffd Z ddeejdeejdeejdee deejdee d ee d ee d eejd eejd eejde edeeeffdZxZS)MoonshineDecoder input_idsrec t||tj|jd|_tj t|jDcgc]}t||c}|_ ycc}wNFr) rJrKrgrr:normrrr<rr)rLrer rNs rOrKzMoonshineDecoder.__init__s\  LL!3!3%@ mm;@AaAa;b cC "63 / c  csA=rrr, inputs_embedsrDrrrrrrrnc  ||n|jj}||n|jj}||n|jj}|du|duz r t d|j r%|j r|rtjdd}||j|}|r"| t} t}t| |}| F||jnd}tj|||jdz|j } || j#d}|j%||| ||}|}|j'||}|rdnd}|rdnd}|r| dnd}| | jd }d }| d dd|fd d|f} |jj(d k(r| d k(j+r| nd} nd|jj(dk(r'|s%t-| |j.|jd } n$t1| |j.|jd } |j2D]}|r||fz }|j r;|j r/|j5t7|j8fi| ||| ||||| | }n||f|| | ||||| |d | }|d}|sm||dfz }| y||dfz }|j;|}|r||fz }t=||r|nd|||S)a Args: encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `encoder_hidden_states`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FrrUrr8r.rrVr) rrrrrrrDrrr )rr,rmrcross_attentions)rerrrDrr rrr embed_tokensr r get_seq_lengthrrr%rrr _update_causal_maskrrr#rr$rrr&rr'r4r)rLr1rrr,r5rDrrrrrrrrpast_seen_tokens causal_maskrmrr*r+all_cross_attentionsr(r) decoder_layerr-s rOrpzMoonshineDecoder.forwards42C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK $0N !12FH]^O  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L.. M>?L] & #oom\J#7BD0d&7P $D>P 'tnP !!1!12P  ((9(9:P !) 6P $$89P  u-- .P rPr0a Args: input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `input_values`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Two formats are allowed: - a [`~cache_utils.Cache`] instance, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache); - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format. The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `decoder_position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. ceZdZeeeeee dde e jde e jde e jde e jde e e e jde eee e jfd e e e jd e e e jd e ed e ed e ede e jdefdZy)MoonshineModel output_typerNrrdecoder_input_idsdecoder_attention_maskencoder_outputsr,decoder_inputs_embedsdecoder_position_idsrDrrrrnc n| | n|jj} | | n|jj} | | n|jj} ||j ||| | }nGt |t s7t |dt|dkDr|dndt|dkDr|dnd}|j||||j|||| | | |  } t| j| j| j| j| j|j|j|jS) aZ ```python >>> import torch >>> from transformers import AutoFeatureExtractor, MoonshineModel >>> from datasets import load_dataset >>> model = MoonshineModel.from_pretrained("UsefulSensors/moonshine-tiny") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id >>> last_hidden_state = model(input_values, decoder_input_ids=decoder_input_ids).last_hidden_state >>> list(last_hidden_state.shape) [1, 2, 288] ```N)rrrrrUr r) r1rrrr,r5rrDrrr)rr,decoder_hidden_statesdecoder_attentionsr8encoder_last_hidden_staterencoder_attentions)rerrrDencoderrrlendecoderrrr,rmrr8)rLrrrDrErFr,rGrHrDrrrdecoder_outputss rOrpzMoonshineModel.forwards[D2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?L D>L $D>L 'tnL !!1!12L  L ]ML rPrAz`The Moonshine Model with a language modeling head. Can be used for automatic speech recognition.c 6eZdZdgZdeffd ZdZdZdZdZ de jfd Z e eeeee  dd eej*d eej,d eej,deej,deeeej*deeeeej*fdeeej*deeej,deedeedeedeej,deej,defdZxZS)!MoonshineForConditionalGenerationzproj_out.weightrect||t||_t j |j |jd|_|jyr3) rJrKrArrgrhr:r9proj_outr )rLrerNs rOrKz*MoonshineForConditionalGeneration.__init__*sH  #F+  &"4"4f6G6GeT  rPc6|jjSrc)r get_encoderrs rOrYz-MoonshineForConditionalGeneration.get_encoder2zz%%''rPc6|jjSrc)r get_decoderrs rOr\z-MoonshineForConditionalGeneration.get_decoder5rZrPc|jSrcrWrs rOget_output_embeddingsz7MoonshineForConditionalGeneration.get_output_embeddings8s }}rPc||_yrcr^)rLnew_embeddingss rOset_output_embeddingsz7MoonshineForConditionalGeneration.set_output_embeddings;s & rPrnc6|jjSrc)rrrs rOrz6MoonshineForConditionalGeneration.get_input_embeddings>szz..00rPrBrrrDrErFr,rGrHrDrrrlabelsc| 9|7|5t| |jj|jj}|j ||||||||| | | |  }|j |j }d}| (|j|| |jj}t|||j|j|j|j|j|j|j  S)aK labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: Example: ```python >>> import torch >>> from transformers import AutoProcessor, MoonshineForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-tiny") >>> model = MoonshineForConditionalGeneration.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> generated_ids = model.generate(input_values, max_new_tokens=100) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription 'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```N) rrDrFrEr,rGrHrDrrr)logitsrdr9) lossrfr,rJrKr8rLrrM)r(re pad_token_idr7rrWr loss_functionr9rr,rJrKr8rLrrM)rLrrrDrErFr,rGrHrDrrrrdrrfrgs rOrpz)MoonshineForConditionalGeneration.forwardAs b   (-B-J$6DKK44dkk6X6X%!'+jj )/+#9+"7!5/!5)'1 ' w889  %%VFt{{OeOe%fD#33")"?"?&99$55&-&G&G")"?"?&99  rP) NNNNNNNNNNNNN)rWrXrY_tied_weights_keysr)rKrYr\r_rbrgr.rrrrRrrrSrrrrrrrr rrpr^r_s@rOrUrU#s ,,(('1bii1*+KL?Y59598<=AEIZ^DHBF$(,0/359-1R u001R !!1!12R $E$4$45 R !))9)9 : R "%e.?.?(@"AB R "%(;U5CTCT=U(U"VWR  (e.?.?(@AR 'uU-=-='>?R D>R $D>R 'tnR !!1!12R ))*R  R ZMR rPrU)r)rArrU)L functoolsrtypingrrrrrrtorch.nnrg activationsr cache_utilsr r r configuration_utilsr generationrmodeling_attn_mask_utilsrrmodeling_flash_attention_utilsrmodeling_outputsrrrrrmodeling_rope_utilsrmodeling_utilsrrprocessing_utilsrutilsrrrrrglm.modeling_glmr!r"r#llama.modeling_llamar$r%r&whisper.modeling_whisperr'r( get_loggerrWr_CHECKPOINT_FOR_DOCrSr)r.rarur}rrrMOONSHINE_START_DOCSTRINGrrr0rRrArU__all__r8rPrOrs33 !CC3)C:F&VUYYG   H %4#J &J Z "))  "))  q)q)h 1 U-U"UBIIUp"]## #BU /U pZ zZ zW$ t]P \P P ffo (@/o o d rP