% &hZddlmZmZmZmZmZmZddlZddl Z ddl Z ddl m Z ddl m Z mZddlmZddlmZddlmZmZdd lmZd d lmZmZmZmZmZd d lmZerdd l m!Z!ddiZ"dZ#dZ$ejJe&Z'GddeZ(GddeeZ)Gdde jTZ+GddeZ,GddeZ-GddeZ.GddeZ/Gd d!eZ0gd"Z1y)#) TYPE_CHECKINGAnyDictListOptionalUnionN)nn)Cache DynamicCache)PretrainedConfig)BaseModelOutputWithPast) AddedTokenPreTrainedTokenizer)logging)LlamaForCausalLMLlamaForSequenceClassificationLlamaForTokenClassificationLlamaMLP LlamaModel)LlamaTokenizer) TextInput vocab_fileztokenizer.modelu▁zgoogle/gemma-7bceZdZdZdZdgZddddddddZdgdgfd d gd gfd gd gfd Z d fd ZxZ S) GemmaConfiga This is the configuration class to store the configuration of a [`GemmaModel`]. It is used to instantiate an Gemma model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Gemma-7B. e.g. [google/gemma-7b](https://huggingface.co/google/gemma-7b) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 256000): Vocabulary size of the Gemma model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`GemmaModel`] hidden_size (`int`, *optional*, defaults to 3072): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 24576): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 28): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 16): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 256): The attention head dimension. hidden_act (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The legacy activation function. It is overwritten by the `hidden_activation`. hidden_activation (`str` or `function`, *optional*): The non-linear activation function (function or string) in the decoder. Will default to `"gelu_pytorch_tanh"` if not specified. `"gelu_pytorch_tanh"` uses an approximation of the `"gelu"` activation function. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. eos_token_id (`int`, *optional*, defaults to 1): End of stream token id. bos_token_id (`int`, *optional*, defaults to 2): Beginning of stream token id. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. ```python >>> from transformers import GemmaModel, GemmaConfig >>> # Initializing a Gemma gemma-7b style configuration >>> configuration = GemmaConfig() >>> # Initializing a model from the gemma-7b style configuration >>> model = GemmaModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```gemmapast_key_valuescolwiserowwise)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projzlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_proj input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnormc ||_| |_||_||_||_||_||_||_||_| |_ | |_ | |_ | |_ ||_ ||_||_t!|Dd||||d|y)N) pad_token_id bos_token_id eos_token_idtie_word_embeddings) vocab_sizemax_position_embeddings hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headshead_dimnum_key_value_heads hidden_acthidden_activationinitializer_range rms_norm_eps use_cache rope_thetaattention_biasattention_dropoutsuper__init__)selfr.r0r1r2r3r5r4r6r7r/r8r9r:r)r+r*r,r;r<r=kwargs __class__s }/var/www/pru.catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/gemma/modular_gemma.pyr?zGemmaConfig.__init__s0%'>$&!2!2#6   #6 $!2!2("$,!2  %%% 3    )ii i`rFgelu_pytorch_tanhNi g{Gz?ư>TrrTg@Fg) __name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_planr? __classcell__rBs@rCrr2sA FJ#4"5%.%.%.%."+ )"+&(9:#%568IJ!"_$56& $ +/ / rDrc eZdZdZ ddeeeeffdZdZ dZ ddd e efd Z d Z dd e ed eded efdZdZy)GemmaTokenizera Construct a Gemma tokenizer. Based on byte-level Byte-Pair-Encoding. The default padding token is unset as there is no padding token in the original model. Args: vocab_file (`str`): Path to the vocabulary file. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. sp_model_kwargs (`Dict[str, Any]`, `Optional`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. use_default_system_prompt (`bool`, *optional*, defaults to `False`): Whether or not the default system prompt for Gemma should be used. spaces_between_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add spaces between special tokens. Nsp_model_kwargsc  |in||_t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}||_||_||_| |_tjdi|j|_ |jj|tj|f|||||||| | | d | y)NFT) normalizedspecial) bos_token eos_token unk_token pad_token add_bos_token add_eos_tokenrWclean_up_tokenization_spacesuse_default_system_promptspaces_between_special_tokensr-)rW isinstancestrrrr_r`rbspmSentencePieceProcessorsp_modelLoadrr?) r@rr]r[r\r^rWr_r`rarbrcrAs rCr?zGemmaTokenizer.__init__s &5%> rD) zzzzNTFFFF)FF)rKrLrMrNrrrerr?rorrrrvr{intboolrrr-rDrCrVrVs,b48%*"'&+) "$sCx.1) V55B[BtCyB8%*.3 898"8(, 8 8: rDrVc<eZdZddedeffd ZdZdZdZxZ S) GemmaRMSNormdimepsct|||_tjt j ||_y)N)r>r?rr Parametertorchzerosweight)r@rrrBs rCr?zGemmaRMSNorm.__init__Vs. ll5;;s#34 rDc|tj|jdjdd|jzzS)NrT)keepdim)rrsqrtpowmeanr)r@xs rC_normzGemmaRMSNorm._norm[s45;;quuQx}}R}>IJJJrDc|j|j}|d|jjzz}|j|S)Ng?)rfloatrtype_as)r@routputs rCforwardzGemmaRMSNorm.forward^sCAGGI&3!2!2!445~~a  rDc^t|jjd|jS)Nz, eps=)tuplershaperrns rC extra_reprzGemmaRMSNorm.extra_repres' ))*+6$((<r?r Linearr0r1 gate_projup_proj down_proj)r@configrBs rCr?zGemmaMLP.__init__jsq 4#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXrD)rKrLrMr?rSrTs@rCrrisYYrDrceZdZ d deej deej deej deeee ejfdeejdee dee d ee d eej d e fd Z y) GemmaModelNr!r$ position_idsrr"r:output_attentionsoutput_hidden_statescache_positionrtc v||n|jj}||n|jj}||n|jj}|du|duz r t d|j r%|j r|rtjdd}||j|}|r | t}| F||jnd} tj| | |jdz|j} || j!d}|j#||| ||} |} |j%| |}tj&|jj(dz| j*}| |z} |rd nd}|rd nd}|j,d|jj.D]g}|r|| fz }|j r0|j r$|j1|j2| | ||||| | }n|| | ||||| | }|d} |s_||dfz }i|j5| } |r|| fz }t7| |r|nd|| S) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FrrJ)deviceg?)dtyper-)r$rpast_key_valuerr:rposition_embeddings)last_hidden_staterr# attentions)rrrr: ValueErrorgradient_checkpointingtraininglogger warning_oncer%r get_seq_lengthrarangerr unsqueeze_update_causal_mask rotary_embtensorr0rr&r2_gradient_checkpointing_func__call__r'r)r@r!r$rrr"r:rrrrApast_seen_tokens causal_maskr#r normalizerall_hidden_statesall_self_attns decoder_layer layer_outputss rCrzGemmaModel.forwardrs2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?L] & #oom\J \\$++"9"93">mFYFYZ % 2 #7BD0d![[)H4;;+H+HI 6M#!m%55!**t}} $ A A!**! #%"' ! !.!#.!-#2&7'#1(; ! *!,M =#3"55? 6B -0  -!1 1 &+/8Od+%   rD) NNNNNNNNN)rKrLrMrr LongTensorTensorrr r FloatTensorrrrr-rDrCrrqs151537KO59$(,0/359k E,,-k !.k u//0 k "%tE4E4E/F(F"GH k   1 12 k D>k $D>k 'tnk !!1!12k  !k rDrceZdZfdZxZS)GemmaForCausalLMc 8tjdi|S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. logits_to_keep (`int` or `torch.Tensor`, *optional*): If an `int`, compute logits for the last `logits_to_keep` tokens. If `0`, calculate logits for all `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a `torch.Tensor`, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length). Returns: Example: ```python >>> from transformers import AutoTokenizer, GemmaForCausalLM >>> model = GemmaForCausalLM.from_pretrained("google/gemma-7b") >>> tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b") >>> prompt = "What is your favorite condiment?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "What is your favorite condiment?" ```r-)r>r) super_kwargsrBs rCrzGemmaForCausalLM.forwards@uw...rD)rKrLrMrrSrTs@rCrrs  / /rDrc eZdZy)GemmaForSequenceClassificationNrKrLrMr-rDrCrrrDrc eZdZy)GemmaForTokenClassificationNrr-rDrCrrrrDr)rrVrrrrGemmaPreTrainedModel)2typingrrrrrr sentencepiecerfrtorch.utils.checkpointr cache_utilsr r configuration_utilsr modeling_outputsrtokenization_utilsrrutilsrllama.modeling_llamarrrrrllama.tokenization_llamartokenization_utils_baserVOCAB_FILES_NAMESr_CHECKPOINT_FOR_DOC get_loggerrKrrrVModulerrrrrr__all__r-rDrCrs CB .37A64!#45'   H %D "D NY^%8Yx=299=(YxYl l ^!/'!/H %C  "=  rD