% &hddlZddlmZddlmZmZmZddlZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZdd lmZmZmZdd lmZmZmZmZmZdd lmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0e*r ddl1m2Z2ddl3m4Z4e+jje6Z7dZ8dZ9GddejtZ;dZeHGd?d@eIZQgdAZRy)CN)partial)OptionalTupleUnion)nn)ACT2FN)Cache DynamicCache StaticCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs_flash_attention_forward!flash_attn_supports_top_left_mask)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)Unpack) LossKwargsadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardcan_return_tupleis_torch_flex_attn_availableloggingreplace_return_docstrings)deprecate_kwarg)DiffLlamaConfig) BlockMask)make_flex_block_causal_maskzkajuma/DiffLlama-0.3B-handcutr%c$eZdZfdZdZxZS) DiffLlamaMLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFbias) super__init__config hidden_sizeintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnselfr0 __class__s /var/www/pru.catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/diffllama/modeling_diffllama.pyr/zDiffLlamaMLP.__init__Ms  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ c|j|j|j||j|z}|SN)r6r8r4r5)r:xr6s r<forwardzDiffLlamaMLP.forwardWs6NN4;;t~~a/@#ADLLQRO#ST r=)__name__ __module__ __qualname__r/rA __classcell__r;s@r<r)r)Ls 0r=r)c|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)r@x1x2s r< rotate_halfrQ\sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r=c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerQ)qkcossin position_ids unsqueeze_dimq_embedk_embeds r<apply_rotary_pos_embr\csY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r= hidden_statesn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r$N)rLexpandreshape)r]r^batchnum_key_value_headsslenhead_dims r< repeat_kvrg~so 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr=c>ddtjd|zzz S)Ng?g333333?g333333ӿ)mathexp) layer_idxs r<lambda_init_fnrls txxy 011 11r=cbeZdZdZddedeeffd Z ddejde ejejfdeejdeejd ee d e d e d eejd e ejeejee ejffdZxZS)DiffLlamaAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr0rkct|||_||_|-tj d|j jd|j|_|j|_ |j|_ t|d|j|jz|_ |j|_|j|jz|_|j |_|j"|_d|_t'j(|j|j|jz|j*|_t'j(|j|j|jz|j*|_t'j(|j|j|jz|j*|_t'j(|j|jz|j|j*|_t5||_t'j8t;j<d|j>|jf|_ t'j8t;j<d|j>|jf|_!t'j8t;j<d|j>|jf|_"t'j8t;j<d|j>|jf|_#t'jHd|jz|jJd |_&y) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rfTr,r)sizerIF)epselementwise_affine)'r.r/r0rklogger warning_oncer;rBattention_dropoutr1num_attention_heads num_headsgetattrrfrdnum_key_value_groupsmax_position_embeddings rope_theta is_causalrr3attention_biasq_projk_projv_projo_projrl lambda_init ParameterrMnormallambda_std_dev lambda_q1 lambda_k1 lambda_q2 lambda_k2RMSNorm rms_norm_eps groupnormr:r0rkr;s r<r/zDiffLlamaAttention.__init__s~  "      !8!8 9:,,  "(!9!9!--33 D4D4D4VW #)#=#= $(NNd6N6N$N!'-'E'E$ ++ii 0 0$..4==2PW]WlWlm ii 0 0$2J2JT]]2Zagavavw ii 0 0$2J2JT]]2Zagavavw ii >@P@PW]WlWlm ))4ell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdA $56;N;Nchir=r]position_embeddingsattention_maskrXpast_key_valueoutput_attentions use_cachecache_positionr_c  $|j\} } } | } |j|}|j|}|j|}|j | | |j |j jdd}|j | | |j|j jdd}|j | | |j|j jdd}|\}}t||||\}}|'|||d}|j|||j|\}}t||j}t||j}tjtj |ddd}|j#dddd}tj$||jddt'j(|j z }|#|ddddddd|j*df}||z}t,j.j1|dtj2j5|j6}t,j.j9||j:|j< }tj>tj@|jB|jDzdtj2j5|j6}tj>tj@|jF|jHzdtj2j5|j6}||z |jJz}tj$||}tj |dd\}}|||zz }d|jJz |jM|z}|jddjO}|jQ| | d}|jS|}|sd}||fS) Nr$rIrWrVrrJrHrrKdtype)ptraining)*rpr~rrviewrwrf transposerdr\updaterkrgryrMrNchunkrepeatmatmulrisqrtrLr functionalsoftmaxfloat32tordropoutrurrjsumrrrrrr contiguousrbr)r:r]rrrXrrrrkwargsbsz target_len_q_len query_states key_states value_statesrVrW cache_kwargs attn_weights causal_masklambda_1lambda_2 lambda_full attn_output attn_output1 attn_output2s r<rAzDiffLlamaAttention.forwards+//1Z{{=1 [[/ {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm &S#7 jRUWZ#[ j  %#&snUL'5'<' %*[[aQ%G" l"[<%?? 4+++t~~k/JJ !++Aq1<<> !))#ub9 kk+.  LL((r=r?NNNFFN)rBrCrD__doc__r%rintr/rMTensorr LongTensorr boolrArErFs@r<rnrnsG j j8C= jL2637*."'59B)||B)#5<<#=>B)!. B) u//0 B) ! B) B)B)!!1!12B) u||Xell3XeELL>Q5RR SB)r=rncPeZdZdZfdZ d dej deej ej fdeejdeejdee de d e d eejd eej eej eeej ffd Z xZ S)DiffLlamaFlashAttention2aN DiffLlama flash attention module. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cBt||i|t|_yr?)r.r/r_flash_attn_uses_top_left_mask)r:argsrr;s r<r/z!DiffLlamaFlashAttention2.__init__s# $)&) /P.Q+r=r]rrrXrrrrr_c  t|tr tdd}|j\} } } |j |} |j |} |j |}| j| | |j|jjdd} | j| | |j|jjdd} |j| | |j|jjdd}|+tjd|j||\}}n|\}}t| | ||\} } |'|||d}|j!| ||j"|\} }| jdd} | jdd} |jdd}|j$r |j&nd}| j(}|t*j,k(rt+j.rt+j0}nMt3|j4dr|j4j6}n |jj8j(}tjd |d | j;|} | j;|} |j;|}t+j<|dd \}}|j?dddd}|j?dddd}tA| | ||| ||tC|d d|jD|jF }tA| | ||| ||tC|d d|jD|jF }t+jH||gd }t+j<|dd \}}t+jJt+jL|jN|jPzdt*j,j;| j(}t+jJt+jL|jR|jTzdt*j,j;| j(}||z |jVz}|||zz }d|jVz |jY|z}|j[| | dj]}|j_|}|sd}|fS)Nz`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformersFr$rIaYThe attention layers in this model are transitioning from computing the RoPE embeddings internally through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be removed and `position_embeddings` will be mandatory.r_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rJsliding_window)rXrruse_top_left_maskr|rHr)0 isinstancer ValueErrorrpr~rrrrwrfrrdrsrt rotary_embr\rrkrrurrMris_autocast_enabledget_autocast_gpu_dtypehasattrr0rweightrrrrrxrr|rNrjrrrrrrrrbrr)r:r]rrrXrrrrrrrrrrrVrWr dropout_rate input_dtype target_dtype value_states1 value_states2rrrrrrrs r<rAz DiffLlamaFlashAttention2.forwardsh nk 2}  "%**, UA{{=1 [[/ {{=1 $((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm  &   G  |\BHC*HC#7 jRUWZ#[ j  %#&snUL'5'<'$  (??<8L#|4J'??<8L',{{<'J$ }%,,Q1a8 %,,Q1a8 /     % "4)94@"AAnn  0     % "4)94@"AAnn  ii| <"E %*[[aQ%G" l99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< "[<%?? 4+++t~~k/JJ !))#ub9DDF kk+.  LL((r=r)rBrCrDrr/rMrrrrr rrArErFs@r<rrs R6:37*."'59D)||D)#5<<#=>D)!!1!12 D) u//0 D) ! D) D)D)!!1!12D) u||Xell3XeELL>Q5RR SD)r=rcJeZdZdZ d dej deej ej fdeej deejdee de de d eejd eej eej eeej fffd Z xZ S) DiffLlamaSdpaAttentiona DiffLlama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. r]rrrXrrrrr_c |r,tjdt| ||||||||S|j \} } } |j |} |j |}|j|}| j| | |j|jjdd} |j| | |j|jjdd}|j| | |j|jjdd}|\}}t| |||\} }|'|||d}|j|||j|\}}t!||j"}t!||j"}t%j&t%j(|ddd}|j+dddd}|}||ddddddd|j,df}| j.j0d k(r2|0| j3} |j3}|j3}|| dkDrd nd }t$j4j6j9| ||||j:r |j<nd | }t%j(|dd\}}t%j>t%j@|jB|jDzdt$jFjI| jJ}t%j>t%j@|jL|jNzdt$jFjI| jJ}||z |jPz}|||zz }d|jPz |jS|z}|jddj3}|j| | d}|jU|}|dfS)NaDiffLlamaModel is using DiffLlamaSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r]rrXrrrrrr$rIrrJrHrcudaTFr) attn_mask dropout_pr|r)+rsrtr.rArpr~rrrrwrfrrdr\rrkrgryrMrNrrrLdevicetyperrrscaled_dot_product_attentionrrurjrrrrrrrrrrr)r:r]rrrXrrrrrrrrrrrrVrWrrr|rrrrrrr;s r<rAzDiffLlamaSdpaAttention.forwards    [ 7?+-)-"3#-$7#  &**, UA{{=1 [[/ {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm &S#7 jRUWZ#[ j  %#&snUL'5'<' !&&sE26 kk+. D  r=r) rBrCrDrrMrrrrr rrArErFs@r<rrs2637*."'59\!||\!#5<<#=>\!!. \! u//0 \! ! \! \!\!!!1!12\! u||Xell3XeELL>Q5RR S\!\!r=rc,eZdZdfd ZdZdZxZS)DiffLlamaRMSNormct|tjt j ||_||_y)z? DiffLlamaRMSNorm is equivalent to T5LayerNorm N)r.r/rrrMonesrvariance_epsilon)r:r1rqr;s r<r/zDiffLlamaRMSNorm.__init__s1 ll5::k#:; #r=c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrIrHT)keepdim) rrrMrpowmeanrsqrtrr)r:r]rvariances r<rAzDiffLlamaRMSNorm.forwardsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r=c^t|jjd|jS)Nz, eps=)tuplerrLrr:s r< extra_reprzDiffLlamaRMSNorm.extra_reprs* ))*+6$2G2G1HIIr=)gư>)rBrCrDr/rArrErFs@r<rrs$;Jr=r)eagerflash_attention_2sdpacpeZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee ejejfd e ed e ejee ejejfffdZxZS)DiffLlamaDecoderLayerr0rkc:t||j|_t|j|||_t ||_t|j|j|_ t|j|j|_ y)N)r0rkrq) r.r/r1DIFFLLAMA_ATTENTION_CLASSES_attn_implementation self_attnr)mlprrinput_layernormpost_attention_layernormrs r<r/zDiffLlamaDecoderLayer.__init__sz !--4V5P5PQY_ktu'/0B0BH[H[\(89K9KQWQdQd(e%r=r]rrXrrrrrrr_c |} |j|}|jd||||||||d| \}} | |z}|} |j|}|j|}| |z}|f} |r| | fz } | S)Nr)rrrr) r:r]rrXrrrrrrresidualself_attn_weightsoutputss r<rAzDiffLlamaDecoderLayer.forwards!,,]; ,:4>> , ')%)/) 3 ,  , ( (!=0 !55mD /  =0  "  )+ +Gr=)NNNFFNN)rBrCrDr%rr/rMrrrr rrrr FloatTensorrArErFs@r<rrsff3f2637*.,1$)59KO(||(!.(u//0 ( ! ( $D> (D>(!!1!12(&eELL%,,,F&GH(-.( u  (51B1BEDUDU1U+V"WW X(r=raN This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DiffLlamaConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zWThe bare DiffLlama Model outputting raw hidden-states without any specific head on top.cFeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZdZy)DiffLlamaPreTrainedModelmodelTrpast_key_valuesFc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyy)Nr)rstd) r0initializer_rangerrr3rdatanormal_r-zero_ Embedding padding_idx)r:modulers r< _init_weightsz&DiffLlamaPreTrainedModel._init_weightsiskk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..r=N)rBrCrDr% config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backendrrr=r<rrWsU #L&*#01#4"5!N  $!"' ?r=rc^eZdZddeffd Zej edZxZ S)DiffLlamaRotaryEmbeddingr0ct|t|drG|j;|jj d|jj d|_nd|_|j |_|j |_||_ t|j |_ |j|j|\}|_ |jd|d|j|_y)N rope_scaling rope_typerdefaultinv_freqF) persistent)r.r/rr getr!rzmax_seq_len_cachedoriginal_max_seq_lenr0r rope_init_fnattention_scalingregister_bufferr#original_inv_freq)r:r0rr#r;s r<r/z!DiffLlamaRotaryEmbedding.__init__vs  6> *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.r?) rBrCrDr%r/rMno_gradrrArErFs@r<rrus3//"U]]_ <