% &h}:ddlZddlmZmZmZmZmZmZddlZ ddl m Z ddl m Z ddlmZddlmZmZmZddlmZmZmZmZmZdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)m*Z*m+Z+dd l,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;mZ>m?Z?ddl@mAZAddlBmCZCmDZDmEZEmFZFmGZGmHZHddlImJZJddlKmLZLe4jeNZOe9r ddlPZPddlPmQZQdZRGddeAZSGdde ZTGddeHZUGdd eQjZWGd!d"eQjZXGd#d$eQjZYd%ZZGd&d'eZ[Gd(d)e)d*+Z\Gd,d-e*Z]Gd.d/eEZ^Gd0d1eQjZ_Gd2d3eQjZ`Gd4d5eQjZaGd6d7eCZbGd8d9e'ZcGd:d;eGZdGd<d=eFZeGd>d?eceDZfGd@dAeJZgdBZhdCZie1dDeiGdEdFedeZjgdGZky)HN)DictIterableListOptionalTupleUnion)ACT2FN)PretrainedConfig)GenerationMixin)BaseImageProcessor BatchFeatureselect_best_resolution) PaddingModeconvert_to_rgbpadresizeto_channel_dimension_format) ChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatmake_flat_list_of_imagesto_numpy_array valid_imagesvalidate_preprocess_arguments)CausalLMOutputWithPast)PreTrainedModel)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput) TensorTypeadd_start_docstrings%add_start_docstrings_to_model_forwardcan_return_tupleloggingreplace_return_docstrings)deprecate_kwarg)is_torch_available)CONFIG_MAPPING AutoConfig AutoModelAutoModelForCausalLM AutoTokenizer) LlamaConfig)LlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModelLlamaPreTrainedModel LlamaRMSNorm)LlavaCausalLMOutputWithPast)divide_to_patches)nnc|jd}|jd}tj|||j|j}tj |d}tjdtj |j}tj||f}t|jdD]2}||} ||dz} || | } tj| ||} | || | 4|S)a* Compute the matrix multiplication (GEMM) for each expert sequentially. This approach is computationally inefficient, especially when dealing with a large number of experts. Args: token_states (torch.Tensor): Input tensor of shape (num_tokens, in_features). expert_weights (torch.Tensor): Weight tensor of shape (num_experts, in_features, out_features). tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). rdtypedevicedim) shapetorchzerosr@rAcumsumlongcatrangematmul) token_statesexpert_weightstokens_per_expert num_tokens out_featuresoutputcumsum_num_tokens zero_tensor expert_numstartendtokensouts {/var/www/pru.catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/aria/modular_aria.pysequential_experts_gemmr[Js##A&J!''+L [[\9K9KT`TgTg hF %6A>++auzz:K:R:RSK ;0A"BCN0034 !*- Q/eC(ll6>*#=>uS   Mc HeZdZdZdZdZ d dedededeffd ZxZS) AriaTextConfiga> This class handles the configuration for the text component of the Aria model. Instantiating a configuration with the defaults will yield a similar configuration to that of the model of the Aria [rhymes-ai/Aria](https://huggingface.co/rhymes-ai/Aria) architecture. This class extends the LlamaConfig to include additional parameters specific to the Mixture of Experts (MoE) architecture. Args: vocab_size (`int`, *optional*, defaults to 32000): Vocabulary size of the LLaMA model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`LlamaModel`] hidden_size (`int`, *optional*, defaults to 4096): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 4096): The size of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `num_attention_heads`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 2048): The maximum sequence length that this model might ever be used with. Llama 1 supports up to 2048 tokens, Llama 2 up to 4096, CodeLlama up to 16384. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 2): Padding token id. bos_token_id (`int`, *optional*, defaults to 1): Beginning of stream token id. eos_token_id (`int`, *optional*, defaults to 2): End of stream token id. pretraining_tp (`int`, *optional*, defaults to 1): Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this document](https://huggingface.co/docs/transformers/main/perf_train_gpu_many#tensor-parallelism) to understand more about it. This value is necessary to ensure exact reproducibility of the pretraining results. Please refer to [this issue](https://github.com/pytorch/pytorch/issues/76232). tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. `beta_fast` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. `beta_slow` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. `short_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `long_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `low_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE `high_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. head_dim (`int`, *optional*): The attention head dimension. If None, it will default to hidden_size // num_heads moe_num_experts (`int`, *optional*, defaults to 8): The number of experts in the MoE layer. moe_topk (`int`, *optional*, defaults to 2): The number of top experts to route to for each token. moe_num_shared_experts (`int`, *optional*, defaults to 2): The number of shared experts. aria_text text_configintermediate_sizemoe_num_expertsmoe_topkmoe_num_shared_expertsc `t|dd|i|||_||_||_||_y)N pad_token_id)super__init__rarbrcrd)selfrarbrcrdrf super_kwargs __class__s rZrizAriaTextConfig.__init__s9 ClClC!2.  &<#r\)ir-r-r-) __name__ __module__ __qualname____doc__ model_typebase_config_keyintri __classcell__rls@rZr^r^isWhTJ#O"& &' = = = = !$ = =r\r^c TeZdZdZdZeedZ d dedede dede f fd Z xZ S) AriaConfiga This class handles the configuration for both vision and text components of the Aria model, as well as additional parameters for image token handling and projector mapping. Instantiating a configuration with the defaults will yield a similar configuration to that of the model of the Aria [rhymes-ai/Aria](https://huggingface.co/rhymes-ai/Aria) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vision_config (`AriaVisionConfig` or `dict`, *optional*): Configuration for the vision component. vision_feature_layer (`int`, *optional*, defaults to -1): The index of the layer to select the vision feature. text_config (`AriaTextConfig` or `dict`, *optional*): Configuration for the text component. projector_patch_to_query_dict (`dict`, *optional*): Mapping of patch sizes to query dimensions. image_token_index (`int`, *optional*, defaults to 9): Index used to represent image tokens. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated normal initializer for initializing all weight matrices. Attributes: model_type (`str`): Type of the model, set to `"aria"`. image_token_index (`int`): Index used to represent image tokens. projector_patch_to_query_dict (`dict`): Mapping of patch sizes to query dimensions. vision_config (`AriaVisionConfig`): Configuration for the vision component. text_config (`AriaTextConfig`): Configuration for the text component. aria)r` vision_configvision_feature_layerr`projector_patch_to_query_dictimage_token_indexinitializer_rangec ||_|ddd}|jD cic]\}} t|t| c} }|_t |jj |_||_t|trd|d<t|ddi|}n| td}||_ ||_ t|trd|vr tdi|}n | t}||_t |@di|ycc} }w)N)ii$idefics3_visionrrrg)r}itemsrtr|maxvalues'max_value_projector_patch_to_query_dictr{ isinstancedictr.rzr~r^r`rhri) rjrzr{r`r|r}r~kwargskvrls rZrizAriaConfig.__init__s"3 ) 0- )JgIlIlIn-oAc!fc!fn-o*7:4;];];d;d;f7g4$8! mT **;M, '*=+FGX-XM  "*+<=?M*!2 k4 (\[-H(7;7K  (*K& "6"'.psD)Nr>NN g{Gz?) rnrorprqrrr^r/ sub_configsrtrfloatrirurvs@rZrxrxsj"HJ"0:NK$&&*.2!"#'&#"&#$ &# (, &#  &#!&#&#r\rxc eZdZy)AriaTextRMSNormNrnrorprgr\rZrr8r\rc(eZdZdZfdZdZxZS)AriaProjectorMLPa! Feed-Forward Network module for the Aria Projector. Args: in_features (`int`): Input embedding dimension. hidden_features (`int`): Hidden dimension of the feed-forward network. output_dim (`int`): Output dimension. ct|tj||d|_tj||d|_t d|_y)NFbiasgelu_new)rhrir<Linear linear_in linear_outr act)rj in_featureshidden_features output_dimrls rZrizAriaProjectorMLP.__init__IsB ;eL))OZeL*%r\ch|j|j|}|j|}|SN)rrr)rj hidden_statess rZforwardzAriaProjectorMLP.forwardOs- !>?  6 r\rnrorprqrirrurvs@rZrr<s & r\rc6eZdZdZddedeffd ZddZxZS)AriaCrossAttentionzv Aria Cross-Attention module. Args: config (`AriaConfig`): The configuration to use. config dropout_ratecBt||jj}|jj}||_t j||d|_t j||d|_ t j||d|_ t j||d|_ t j|||_ t j||_t j ||_t j ||_y)NFrT) batch_first)rhrirz hidden_sizenum_attention_heads num_headsr<rq_projk_projv_projMultiheadAttentionmultihead_attnlinearDropoutdropout LayerNorm layer_norm layer_norm_kv)rjrrrrrls rZrizAriaCrossAttention.__init__^s **66 ((<< "ii [uE ii [uE ii [uE !33KX\]ii [9 zz,/ ,,{3\\+6r\c|j|j|}|j|}|j|}|j |}|j ||||\}}|j |j|}|S)a Forward pass of the AriaCrossAttention module. Args: key_value_states (`torch.Tensor`): Input tensor for key and value. hidden_states (`torch.Tensor`): Input tensor for query. attn_mask (`torch.Tensor`, *optional*, defaults to None): Attention mask. Returns: torch.Tensor: Output tensor after cross-attention.  attn_mask)rrrrrrrr) rjkey_value_statesrrquerykeyvalue attn_output_s rZrzAriaCrossAttention.forwardos  DOOM:;--.>?kk*+ ,-,,UC),T Qll4;;{#;< r\)rr) rnrorprqrxrrirrurvs@rZrrUs 7z77"r\rcheZdZdZdeffd ZddejdeejfdZ xZ S) AriaProjectora  Aria Projector module. This module projects vision features into the language model's embedding space, enabling interaction between vision and language components. Args: config (`AriaConfig`): Configuration object for the model. rct||j|_|jj |_|jj|_|jj |_ |jj |_ |jj |_ tjtj |j"|j |_t'||_tj*|j |_t/|j |j|j|_yr)rhrir|patch_to_query_dictrzrrrrkv_dimr`rrr< ParameterrFrGrrr cross_attnrrr feed_forwardrjrrls rZrizAriaProjector.__init__s #)#G#G !//;;--AA**66 %11== ,,88\\%++f.\.\^b^n^n"op ,V4,,t'7'78,T-=-=t?S?SUYUdUder\rrcP|jd|jd}}||jjvr*td|d|jjd|j|}|jd|j dj |dd}|M|j|jd}|j djd|jdd}|j|||}|j|j|}|S) a Forward pass of the Projector module. Args: key_value_states (`torch.Tensor`): Input tensor of shape (batch_size, num_patches, kv_dim). attn_mask (`torch.Tensor`, *optional*, default is None): Attention mask. Returns: `torch.Tensor`: Output tensor of shape (batch_size, query_number, output_dim). rrDzNumber of patches z: not found in patch_to_query_dict amongst possible values .Nr>r)rErkeysKeyErrorr unsqueezerepeatrepeat_interleaverexpandsizerrr) rjrr batch_size num_patches query_numqueries attention_outrYs rZrzAriaProjector.forwards?#3"8"8";=M=S=STU=VK d66;;= =$[M1klpmEmEmJmJmLlMMNO ,,[9 **Zi(2215<? r\r) rnrorprqrxrirFTensorrrrurvs@rZrrs7ff( %,,AWr\rct||\}}|\}}||z }||z }||kr(|} ttj||z|} | | fS|} ttj||z|} | | fS)N channel_dim)rminmathceil) imagetarget_resolutioninput_data_formatoriginal_heightoriginal_width target_height target_widthscale_wscale_h new_width new_heights rZ_get_patch_output_sizers&4UHY&Z#O^"3M<^+Go-G ?W#<=}M y  #  .7":;\J y  r\c2eZdZdZgdZddddddddej f deed eed e d e d e ee e e fd e e de e de e deffd Z dddddddddejdf deeeefde eeeefd e eeeefd e e d e e d e e de e de e dede eeefde ede eeeffdZdej,dededej,fdZdej,dededej,fdZej6dddfdej8dee e e e fee e e ffded eeeefde eeefde eeefdej8fd!Zdej,d"ee e e fd#e dedededeej,fd$ZxZ S)%AriaImageProcessora A vision processor for the Aria model that handles image preprocessing. Initialize the AriaImageProcessor. Args: image_mean (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Mean values for normalization. image_std (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Standard deviation values for normalization. max_image_size (`int`, *optional*, defaults to 980): Maximum image size. min_image_size (`int`, *optional*, defaults to 336): Minimum image size. split_resolutions (`list`, *optional*, defaults to a list of optimal,resolutions as tuples): The optimal resolutions for splitting the image. split_image (`bool`, *optional*, defaults to `False`): Whether to split the image. do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. resample (PILImageResampling, *optional*, defaults to `BICUBIC`): The resampling filter to use if resizing the image.  pixel_values pixel_mask num_cropsNiPFT image_mean image_stdmax_image_sizemin_image_sizesplit_resolutions split_imagedo_convert_rgb do_normalizeresamplec  t |di| |gd}|gd}||_||_||_||_||_|!gd}|D cgc]} | ddz| ddzf}} ||_||_||_ | |_ ycc} w)N)?rr))rDr-)rDr )rD)rD)rD)rD)rDrm)r-r)r-r )r-r-)r-rD)r rD)r r-)rrD)rr-)rrD)rrD)rrD)rmrDrrDrg) rhrirrrrrrrrr) rjrrrrrrrrrrelrls rZrizAriaImageProcessor.__init__s "6"  (J  'I,,$"&  $!y FW X"Q%#+r!us{!; X  X!2,(  !Ys Bptimagesreturn_tensors data_formatrc z||n |j}||n |j}||n |j}||n |j}||n |j}||n |j }||n |j }| | n |j} |dvr tdt|}t|s tdt|||| |r|D cgc] } t| }} |D cgc] } t| }} | t|d} g}g}d}|D]j} |r"|j| |j || | | }n| g}|t#||kDr t#|}|D]}t%|\}}|t'||z }||k\rt't)||z||f}n|t't)||z|f}t+||| | | }||dz ||d z }}t-|d|fd|ff| | }t/j0||ft2 }d |d|dd|d f<|j5||r>|j7|d z |j|j| | }| t9|| | n|}|j5|"mt;t/j<|d t/j<|d |d | Scc} wcc} w)a! Process a list of images. Args: images (ImageInput or list of ImageInput): The input image or a list of images. image_mean (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Mean values for normalization. image_std (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Standard deviation values for normalization. max_image_size (`int`, *optional*, defaults to `self.max_image_size` (980)): Maximum image size. min_image_size (`int`, *optional*, defaults to `self.min_image_size` (336)): Minimum image size. split_image (`bool`, *optional*, defaults to `self.split_image` (False)): Whether to split the image. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb` (True)): Whether to convert the image to RGB. do_normalize (`bool`, *optional*, defaults to `self.do_normalize` (True)): Whether to normalize the image. resample (PILImageResampling, *optional*, defaults to `self.resample` (BICUBIC)): The resampling filter to use if resizing the image. return_tensors (`str` or `TensorType`, *optional*, defaults to "pt"): The type of tensor to return. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use same as the input image. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use the inferred format of the input image. Returns: BatchFeature: A BatchFeature object containing: - 'pixel_values': Tensor of processed image pixel values. - 'pixel_mask': Boolean pixel mask. This mask is a 2D tensor of shape (max_image_size, max_image_size) where: - True (1) values indicate pixels that belong to the original resized image. - False (0) values indicate pixels that are part of the padding. The mask helps distinguish between actual image content and padded areas in subsequent processing steps. - 'num_crops': The maximum number of crops across all images. Nrrz(max_image_size must be either 490 or 980zkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.)rrrrr)rr)rrrrD)r@go@)axisr)data tensor_type)rrrrrrrr ValueErrorrrrrrrget_image_patchesrlenrrrtrrnprGboolappend normalizerrstack)rjrrrrrrrrrrrrrr pixel_masksr crop_images crop_imagehwscalenew_sizecrop_image_resizedpadding_bottom padding_rightcrop_image_paddedrs rZ preprocesszAriaImageProcessor.preprocesssPF$.#9Zt !*!6IDNN +9+E4K^K^+9+E4K^K^%0%vay I    ; 7E"44**" 1&7 5  %g  C $4y$@ , ), 7 %j11&Q26 #CE NN C^TH .CE NN0STH%+% 1&7 &"1?!0Ln_ghi_jNj $'&(1m*<= 1&7 %! XX~~&FdS ;< =Xa[=-HQK-78"":.(,)E1$5*; )7)%'244E{Tef.& ##$56Y, 7; 7x "A > hh{;& '   SA=s J3J8rrreturncHt|||\}}t|||f||}|S)aC Resizes an image to a target resolution while maintaining aspect ratio. Args: image (np.array): The input image. target_resolution (tuple): The target resolution (height, width) of the image. resample (`PILImageResampling`): Resampling filter to use if resizing the image. input_data_format (`ChannelDimension` or `str`): The channel dimension format of the input image. Returns: np.array: The resized and padded image. rr)rr)rjrrrrrr resized_images rZ_resize_for_patchingz'AriaImageProcessor._resize_for_patchings7&!7u>OQb c Iuz9&=duv r\c|\}}t|||\}}||z dz}||z dz} |j|| | f||ff} | S)zU Pad an image to a target resolution while maintaining aspect ratio. r-)padding)rr) rjrrrrrrrpaste_xpaste_y padded_images rZ_pad_for_patchingz$AriaImageProcessor._pad_for_patchingsh '8# | 6u>OQb c I)+1 :-!3xx'0BWgDV/WxX r\r'modeconstant_valuesc^t|tst|dk7rt||||||S| t |}t j dt jdt jdt jdi}tj|||||}|t|||}|S|}|S)a Pads the `image` with the specified `padding` and `mode`. Padding can be in the (`height`, `width`) dimension of in the (`num_patches`) dimension. In the second case an iterable if tuples is expected as input. Args: image (`np.ndarray`): The image to pad. padding (`int` or `Tuple[int, int]` or `Iterable[Tuple[int, int]]`): Padding to apply to the edges of the height, width axes. Can be one of three formats: - `((before_height, after_height), (before_width, after_width))` unique pad widths for each axis. - `((before, after),)` yields same before and after pad for height and width. - `(pad,)` or int is a shortcut for before = after = pad width for all axes. mode (`PaddingMode`): The padding mode to use. Can be one of: - `"constant"`: pads with a constant value. - `"reflect"`: pads with the reflection of the vector mirrored on the first and last values of the vector along each axis. - `"replicate"`: pads with the replication of the last value on the edge of the array along each axis. - `"symmetric"`: pads with the reflection of the vector mirrored along the edge of the array. constant_values (`float` or `Iterable[float]`, *optional*): The value to use for the padding if `mode` is `"constant"`. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use same as the input image. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use the inferred format of the input image. Returns: `np.ndarray`: The padded image. rconstantreflectedge symmetric)r-r.) rrtrrrrCONSTANTREFLECT REPLICATE SYMMETRICrr)rjrr'r-r.rrpadding_mode_mappings rZrzAriaImageProcessor.pads` gs #s7|q'8ugt_kK\] ]  $ >u E   *     ! !6  ! !;   ug,@,FXghR]Ri '{BDH@zz@sE#s(OXeCHo-FFG@ @ uhuo56 @ eC)9$9:; @$E#/?*?$@A@ @D0xx0U38_-0 0 % 0 & 0,0 bhh0r\rc6eZdZddidddejdZy)AriaProcessorKwargsr'Fr)rr) text_kwargs images_kwargsrN)rnrorpr%PYTORCH _defaultsrgr\rZrLrLjs- u "  %,, Ir\rLF)totalc eZdZdZddgZddgZdZdZ ddee e fde e de e ee efefffd Z dd eeeeeeefd e ed eed efd ZdZdZedZxZS) AriaProcessora AriaProcessor is a processor for the Aria model which wraps the Aria image preprocessor and the LLama slow tokenizer. Args: image_processor (`AriaImageProcessor`, *optional*): The AriaImageProcessor to use for image preprocessing. tokenizer (`PreTrainedTokenizerBase`, *optional*): An instance of [`PreTrainedTokenizerBase`]. This should correspond with the model's text model. The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. size_conversion (`Dict`, *optional*): A dictionary indicating size conversions for images. image_processor tokenizer chat_templatesize_conversionrr2c|ddd}|jDcic]\}}t||c}}|_||j|j|_t ||||ycc}}w)Nrrr )rV)rrtrW pad_token unk_tokenrhri)rjrTrUrVrWrrrls rZrizAriaProcessor.__init__sv  "$'c2O6E6K6K6MNdaA N  Y%8%8%@"+"5"5I  )=Q OsA-textrrr!c |jtfd|jji|}t |t r|g}n.t |t st |dt s td||j|fi|d}|j|jjd}g} |jd|z} |D]P} | j|jj|jj| z} | j| Rni}|} |j| fi|d} t!i| |S) a Main method to prepare for the model one or several sequences(s) and image(s). Args: text (`TextInput`, `PreTokenizedInput`, `List[TextInput]`, `List[PreTokenizedInput]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). images (`ImageInput`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **pixel_mask** -- Pixel mask to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrzAInvalid input text. Please provide a string, or a list of stringsrNr-rrM)r ) _merge_kwargsrLrU init_kwargsrrGr=r rTrWrrEpopreplace image_tokenrr) rjr[raudiovideosr output_kwargs image_inputstokens_per_imageprompt_stringsrsample text_inputss rZ__call__zAriaProcessor.__call__s^<+**  "&.."<"<  dC 6DD$' 47C0H`a a  /4//0L $33L4M4M4S4STU4VW N$((58HHI .(B(BDNND^D^ajDjk%%f- . L!N$dnn  M* !@K!@? R&(, =BI0$y/4HYCZZ[=B$=B ,- =B =B~<6XXr\rSc(eZdZdZdeffd ZxZS)AriaSharedExpertsMLPa/ Shared Expert MLP for shared experts. Unlike routed experts, shared experts process all tokens without routing. This class reconfigures the intermediate size in comparison to the LlamaMLP. Args: config (`AriaTextConfig`): Configuration object for the Aria language model. rc`t|||j|jz|_yr)rhrirardrs rZrizAriaSharedExpertsMLP.__init__s) !'!9!9FV>VYZ>Z\b\r\rs)&*B*BFDVDVX^XnXnor\c|j||}tj|dd\}}tjj ||z}|j ||}|S)a5 Forward pass of the Grouped MLP. Args: permuted_tokens (torch.Tensor): Permuted input tokens. tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor after passing through the MLP. r-r>rB)rrFchunkr< functionalsilur)rjpermuted_tokensrO fc1_output projectiongate fc2_outputs rZrzAriaGroupedExpertsMLP.forward>s\XXo/@A  ;;z1"= D]]'' 3d: XXj*;< r\)rnrorprqr^rirrurvs@rZrr/s#p~p$p r\rc`eZdZdZdeffd ZdejdejfdZxZ S)AriaTextMoELayerz Aria Text Mixture of Experts (MoE) Layer. This layer applies a gating mechanism to route input tokens to different experts. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. rct|tj|j|j d|_t||_t||_ ||_ yNFr) rhrir<rrrbrouterrexpertsr}shared_expertsrrs rZrizAriaTextMoELayer.__init__\sO ii 2 2F4J4JQVW ,V4 26: r\rr!c|j}|jd|jd}|j|}t j ||j jd\}}tjj|d}|j}t j|jjtj|j j d|j j dz j|}|} | jd} t j"| } |j%d| |j jz} |j'| |} t j(|jd|j jz| jdf| j| j*}|j-d| | |jd|j j| jd}||j/dzj1dj|}|j3|j|}||zS)a. Forward pass of the MoE Layer. Args: hidden_states (`torch.Tensor`): Input tensor of shape (batch_size, sequence_length, hidden_size). Returns: torch.Tensor: Output tensor after passing through the MoE layer. Process: 1. Route tokens to experts using the router. 2. Permute tokens based on routing decisions. 3. Process tokens through experts. 4. Unpermute and combine expert outputs. 5. Add shared expert output to the final result. r>rD)rrCrBr)binsrrr?)rEviewrrrFtopkrrcr<rsoftmaxr@histcflattentofloat32rbargsort index_selectrrGrA index_copy_rsumr)rjroriginal_shapelogits top_logits top_indicesscoresoriginal_dtyperOindicesflatten_indicessorted_indicesr expert_outputunpermuted_tokensrRshared_expert_outputs rZrzAriaTextMoELayer.forwardds$',,%**2}/A/A"/EF ]+"'**Vt{{7K7KQR"S K&&zr&:$**!KK    ! $ $U]] 3,, ++a/  "^   ",,r*7'44Q$++J^J^8^_ _6GH "KK \\!_t{{33 3]5G5G5J K%% ''  %%aG-222t{{7K7K]M_M_`aMbc#f&6&6r&::??A?FKKN[ $22=3E3En3UV,,,r\) rnrorprqr^rirFrrrurvs@rZrrQs/~9-U\\9-ell9-r\rc,eZdZdZdedeffd ZxZS)AriaTextDecoderLayerag Aria Text Decoder Layer. This class defines a single decoder layer in the language model, incorporating self-attention and Mixture of Experts (MoE) feed-forward network. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. layer_idx (`int`): Index of the layer. r layer_idxcDt||t||_yr)rhrirmlprjrrrls rZrizAriaTextDecoderLayer.__init__s #F+r\)rnrorprqr^rtrirurvs@rZrrs  ,~,#,,r\rc:eZdZdZeZdZddgZdZdZ dZ dZ dZ dZ y ) AriaTextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. modelrrTpast_key_valuesFc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|tr(|j j jd|yt|tjrf|j j jd|t|dr2|j%|jj jyyyy)Nr,meanstdr)rr~rr<rrr normal_rzero_ Embedding padding_idxrConv2dhasattrrjmodulers rZ _init_weightsz%AriaTextPreTrainedModel._init_weightssHkk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  6 7 MM   & &CS & 9  * MM   & &CS & 9vv&6;;+B   &&(,C&+r\N)rnrorprqrx config_classbase_model_prefix_no_split_modulessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_classrrgr\rZrrsBL/1IJ&*#"3"N )r\rceZdZdZdZdZy)AriaPreTrainedModelFc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|tr,tjj|j|yy)Nr,r)r)rr~rr<rrr rrrrrrinit trunc_normal_rrs rZrz!AriaPreTrainedModel._init_weightsskk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  . GG ! !&,,C ! 8/r\N)rnrorp_supports_static_cache_supports_attention_backendrrgr\rZrrs""' 9r\rc$eZdZdeffd ZxZS) AriaTextModelrc t||tjt |j Dcgc]}t ||c}|_d|_|jycc}w)NF) rhrir< ModuleListrKnum_hidden_layersrlayersgradient_checkpointing post_initrs rZrizAriaTextModel.__init__s[  mmFKFLdLdFe f !&) 4 f  ',#  gsA.)rnrorpr^rirurvs@rZrrs~r\rc2eZdZdZdgZeZdeffd ZxZS)AriaTextForCausalLMa7 Aria model for causal language modeling tasks. This class extends `LlamaForCausalLM` to incorporate the Mixture of Experts (MoE) approach, allowing for more efficient and scalable language modeling. Args: config (`AriaTextConfig`): Configuration object for the model. zlm_head.weightrct||t||_|j|_t j |j|jd|_|jyr) rhrirr vocab_sizer<rrlm_headrrs rZrizAriaTextForCausalLM.__init__sU  "6*  ++yy!3!3V5F5FUS  r\) rnrorprq_tied_weights_keysr^rrirurvs@rZrrs) ++!L~r\rc eZdZy)AriaCausalLMOutputWithPastNrrgr\rZrrrr\ra4 Args: input_ids (`torch.LongTensor`, *optional*): Input token IDs. pixel_values (`torch.FloatTensor`, *optional*): Pixel values of the images. pixel_mask (`torch.LongTensor`, *optional*): Mask for the pixel values. attention_mask (`torch.Tensor`, *optional*): Attention mask. position_ids (`torch.LongTensor`, *optional*): Position IDs. past_key_values (`List[torch.FloatTensor]`, *optional*): Past key values for efficient processing. inputs_embeds (`torch.FloatTensor`, *optional*): Input embeddings. labels (`torch.LongTensor`, *optional*): Labels for computing the language modeling loss. use_cache (`bool`, *optional*): Whether to use the model's cache mechanism. output_attentions (`bool`, *optional*): Whether to output attention weights. output_hidden_states (`bool`, *optional*): Whether to output hidden states. return_dict (`bool`, *optional*): Whether to return a `ModelOutput` object. logits_to_keep (`int` or `torch.Tensor`, *optional*, defaults to 0): If an `int`, calculate logits for the last `logits_to_keep` tokens, or all `input_ids` if `0`. Otherwise, slice according to the 1D tensor in the sequence length dimension cache_position (`torch.LongTensor`, *optional*): Cache positions. **loss_kwargs: Additional keyword arguments for loss calculation. aG This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (`AriaConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zAria model for conditional generation tasks. This model combines a vision tower, a multi-modal projector, and a language model to perform tasks that involve both image and text inputs.c!eZdZeZdZdZdZdgZdeffd Z dZ dZ dZ dZ d Zd Zd Z d"d ej$d eej$defdZeedddeeeee d#deej8d eej$d eej8deej:deej8deeej$deej$deej8deedeedeede eej:fdeej8defd Z! d$d!Z"xZ#S)%AriaForConditionalGenerationFzlanguage_model.lm_head.weightrct||tj|j|_t ||_|jj|_ tj|j|_ |jj|jjnd|_ |jjdk(|_|j!y)Nr>flash_attention_2)rhrir0 from_configrz vision_towerrmulti_modal_projectorr`rr1language_modelrrf_attn_implementation_use_flash_attention_2rrs rZriz%AriaForConditionalGeneration.__init__Ms  %11&2F2FG%26%:" ,,772>>v?Q?QR8< 8P8P8\DKK44bd&,&8&8&M&MQd&d# r\c|y|jd|jjj|jjj}|jd|jjj|jjj}|j ddkDj S)NrD) dimensionrstepr-)r>rBr)unfoldrrr:rr)rjrpatches_subgrids rZ_create_patch_attention_maskz9AriaForConditionalGeneration._create_patch_attention_maskXs  $++""))44""))44,  *00""))44""))441   ###1A5;;==r\c6|jjSr)rget_input_embeddingsrjs rZrz1AriaForConditionalGeneration.get_input_embeddingshs""7799r\c:|jj|yr)rset_input_embeddings)rjrs rZrz1AriaForConditionalGeneration.set_input_embeddingsks 007r\c6|jjSr)rget_output_embeddingsrs rZrz2AriaForConditionalGeneration.get_output_embeddingsns""88::r\c:|jj|yr)rset_output_embeddings)rjnew_embeddingss rZrz2AriaForConditionalGeneration.set_output_embeddingsqs 11.Ar\c:|jj|yr)r set_decoder)rjdecoders rZrz(AriaForConditionalGeneration.set_decoderts ''0r\c6|jjSr)r get_decoderrs rZr z(AriaForConditionalGeneration.get_decoderws""..00r\rrr{c|j|}|j||d}d}|&|jd}tj|}|j |}|j ||} | S)NT)patch_attention_maskoutput_hidden_statesrDr)rrrrF logical_notrr) rjrrr{r  image_outputsimage_attn_maskflattened_maskselected_image_featureimage_featuress rZget_image_featuresz/AriaForConditionalGeneration.get_image_featureszs $@@L)) /CZ^*   +199!>> import requests >>> import torch >>> from PIL import Image >>> from io import BytesIO >>> from transformers import AutoProcessor, AutoModel >>> from transformers.image_utils import load_image >>> # Note that passing the image urls (instead of the actual pil images) to the processor is also possible >>> image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg") >>> image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg") >>> image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg") >>> processor = AutoProcessor.from_pretrained("Rhymes-AI/Aria") >>> model = AutoModel.from_pretrained("Rhymes-AI/Aria", torch_dtype=torch.bfloat16, device_map="auto") >>> # Create inputs >>> messages = [ ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."}, ... {"type": "image"}, ... {"type": "text", "text": "What can we see in this image?"}, ... ] ... }, ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "In which city is that bridge located?"}, ... ] ... } ... ] >>> prompts = [processor.apply_chat_template([message], add_generation_prompt=True) for message in messages] >>> images = [[image1, image2], [image3]] >>> inputs = processor(text=prompts, images=images, padding=True, return_tensors="pt").to(model.device) >>> # Generate >>> generated_ids = model.generate(**inputs, max_new_tokens=256) >>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> print(generated_texts[0]) Assistant: There are buildings, trees, lights, and water visible in this image. >>> print(generated_texts[1]) Assistant: The bridge is in San Francisco. ```NrDr?rBrr>)rrr{z6Image features and image tokens do not match: tokens: z , features ) rrrrr r!rrr")rrr)lossrrr attentionsrg)rr!rrrErFtensorr}rIrArr expand_asrrr{r r@masked_scatterrr loss_functionr`rrrrr%)rjrrrrrrrrr r!rrr" loss_kwargsspecial_image_maskn_image_tokens image_embedsrn_imagesn_features_per_imagen_image_featuresoutputsrr$s rZrz$AriaForConditionalGeneration.forwardsd2C1N-TXT_T_TqTq$8$D $++JjJj   7D557 BM  # (;(;A(>!(C %26Qd6O6O6QLL!>!>ejjYfYmYmn7&"#5!9!9a!9!@!D!D!D!KA!N(DKK,I,II %1%;%;B%?%I%I-%X%[%[\i\p\p%q"".!3!3!3!:!>!>1!>!E!44)%%)[[%E%E5N .<-A-A!-DnFZFZ[\F]*H'*>> !11 L^L\\ghxgyz,..}/C/C]EXEXYN)889K^\M*.*=*=)%+'/!5))+> +   %4%%f9P9P9[9[_jD*#33!//))   r\c p|jj|f|||||d| } |ddk(r || d<|| d<| S)N)rrrr"rrrr)rprepare_inputs_for_generation) rjrrrrrrr"rr model_inputss rZr3z:AriaForConditionalGeneration.prepare_inputs_for_generationsiIt**HH  +')))    !  !,8L ()3L &r\)Nr>) NNNNNNNNNNNrN)NNNNNNN)$rnrorprxrr_supports_flex_attnrrrirrrrrrr rF FloatTensorrrtrr(r+r'ARIA_INPUTS_DOCSTRINGr*r LongTensorrrrrrr3rurvs@rZrr?s7L"N9: z > :8;B11 37$& ''U../" &)6DTU*+@A+ET^_1548151537=A59-1$(,0/33459J E,,-J u001J U--. J !. J u//0 J "$u'8'8"9:J   1 12J ))*J D>J $D>J 'tnJ c5<</0J !!1!12J $!J `BVJ ^r\r) rxr^rrSrrrrr)lrtypingrrrrrrnumpyr activationsr configuration_utilsr generationr image_processing_utilsr rrimage_transformsrrrrr image_utilsrrrrrrrrrmodeling_outputsrmodeling_utilsrprocessing_utilsr r!r"tokenization_utilsr#r$utilsr%r&r'r(r)r*utils.deprecationr+utils.import_utilsr,autor.r/r0r1r2llama.configuration_llamar3llama.modeling_llamar4r5r6r7r8r9llava.modeling_llavar:&llava_next.image_processing_llava_nextr; get_loggerrnloggerrFr<r[r^rxrModulerrrrrrLrSr}rrrrrrrrrr7ARIA_START_DOCSTRINGr__all__rgr\rZrRs  ??!3)^^ee   7-HH14]]3?F   H %>{=[{=|N#!N#b l ryy244n>BII>B!"I+IX *% yXNyXx Z8 Z ) RYY) XBIIDL-ryyL-^,,,$)o)@9.9$J13C2 !< !F"A s#6ssl r\