% &h\lddlZddlmZddlmZmZmZmZddlZddl m Z ddl m cm Z ddlZddlmZddlmZddlmZddlmZdd lmZmZmZmZmZdd lmZd d lm Z m!Z!d d l"m#Z#m$Z$m%Z%d dl&m'Z'ddl(m)Z)m*Z*m+Z+dZ,dZ-ej\e/Z0Gdde#Z1Gdde jdZ3Gdde!Z4Gdde jdZ5Gdde jdZ6Gdde jdZ7Gdd e jdZ8Gd!d"e jdZ9Gd#d$e jdZ:Gd%d&e jdZ;Gd'd(e'Z<Gd)d*e jzZ>Gd+d,e jdZ?Gd-d.e jdZ@Gd/d0e jdZAGd1d2e jdZBGd3d4e jdZCd5ZDed6eDGd7d8eZEGd9d:ZFGd;dZIGd?d@e%eGZJGdAdBe$eGeZKGdCdDeGeZLgdEZMy)FN)cached_property)ListOptionalTupleUnion)Cache)GenerationMixin)CausalLMOutputWithPast)PreTrainedModel)add_start_docstrings%add_start_docstrings_to_model_forwardcan_return_tupleloggingreplace_return_docstrings)deprecate_kwarg)ChameleonPreTrainedModel#ChameleonVQVAEEncoderConvDownsample)LlamaDecoderLayerLlamaForCausalLM LlamaModel)SiglipAttention) Emu3ConfigEmu3TextConfigEmu3VQVAEConfigrzBAAI/Emu3-Chat-hfcfeZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee ejejfd e ejee ejejfffd ZxZS)Emu3DecoderLayerconfig layer_idxcnt|||tj|j|_yN)super__init__nnDropoutattention_dropoutdropoutselfr r! __class__s {/var/www/pru.catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/emu3/modular_emu3.pyr%zEmu3DecoderLayer.__init__=s( +zz&":":; hidden_statesattention_mask position_idspast_key_valueoutput_attentions use_cachecache_positionposition_embeddingsreturnc |} |j|}|jd||||||||d| \}} | |j|z}|} |j|}|j |}| |j|z}|f} |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r/r0r1r2r3r4r5r6)input_layernorm self_attnr)post_attention_layernormmlp) r+r/r0r1r2r3r4r5r6kwargsresidualself_attn_weightsoutputss r-forwardzEmu3DecoderLayer.forwardAs>!,,]; ,:4>> , ')%)/) 3 ,  , ( (!4<< #>> !55mD /  4<< #>>  "  )+ +Gr.)NNNFFNN)__name__ __module__ __qualname__rintr%torchTensorr LongTensorr boolr FloatTensorrB __classcell__r,s@r-rr<s <D><!!1!12<&eELL%,,,F&GH< u  (51B1BEDUDU1U+V"WW X<r.rcHeZdZdZdeffd ZdejfdZxZ S)Emu3VQVAEVectorQuantizera A module for vector quantization using learned embedding vectors. This module implements the quantization process similar to te one described in the VQ-VAE (Vector Quantized Variational AutoEncoder) paper. It quantizes continuous input vectors into discrete codebook vectors, which are learned during training. Current implementation improves over previous ones by avoiding costly matrix multiplications and allowing for post-hoc remapping of indices. r ct|tj|j|j |_|j jjjd|jz d|jz y)Ngg?) r$r%r& Embedding codebook_size embed_dim embeddingweightdatauniform_r+r r,s r-r%z!Emu3VQVAEVectorQuantizer.__init__sb f&:&:FAtT $.."7"7":B  %;T^^=R=R=\=\]^`a=bcc $}4y@ $||I1=388XvW\]##r.) rCrDrE__doc__rr%rGrHrBrLrMs@r-rOrOs&ee $ELL$r.rOc eZdZy)Emu3VQVAEEncoderConvDownsampleN)rCrDrEr9r.r-rtrtsr.rtc$eZdZfdZdZxZS)Emu3VQVAEEncoderConvUpsamplec`t|tj||ddd|_y)Nrr kernel_sizestridepadding)r$r%r&Conv2dconv)r+ in_channelsr,s r-r%z%Emu3VQVAEEncoderConvUpsample.__init__s' IIk;AaYZ[ r.cXtj|dd}|j|}|S)N@nearest scale_factormode)F interpolater}r+r/s r-rBz$Emu3VQVAEEncoderConvUpsample.forwards( m#IV  -0 r.rCrDrEr%rBrLrMs@r-rvrvs\r.rvc \eZdZdededeedeeffd ZdejfdZxZ S)Emu3VQVAEConv3d in_channel out_channelryrzcPt |t|dd|ddDcgc] \}}||z  }}}d|_|dddD]%}|xj|dz|dzz|dzfz c_'|xjdz c_t j |||||_ycc}}w)Nrr9r\r)rr)rz)r$r%zipr{r&Conv3dr}) r+rrryrz one_kernel one_stride padding_sizespad_sizer,s r-r%zEmu3VQVAEConv3d.__init__s ORS^_`_aSbdjklkmdnOop5KZj0p p %dd+ JH LLX]X\98q=I IL J  II      qsB"r/chtj||j}|j|}|Sr#)rpadr{r}rs r-rBzEmu3VQVAEConv3d.forwards*mT\\:  -0 r.) rCrDrErFrr%rGrHrBrLrMs@r-rrsF   3Z  c  ,U\\r.rc`eZdZdedeffd Zdej dej fdZxZS)Emu3VQVAESpatialNormr~ out_channelsct|tj|ddd|_tj ||ddd|_tj ||ddd|_y)N ư>T num_channels num_groupsepsaffinerrrx)r$r%r& GroupNorm norm_layerr|conv_yconv_br+r~rr,s r-r%zEmu3VQVAESpatialNorm.__init__sn ,,%  ii     ii     r.r/ quant_statesctj||jddd}|j|}||j |z|j |z}|S)Nr)sizer)rrr`rrr)r+r/rs r-rBzEmu3VQVAESpatialNorm.forwardsX}}\ 8K8KBC8PW`a  6 % L(AADKKP\D]] r. rCrDrErFr%rGrHrBrLrMs@r-rrs5   8U\\r.rcHeZdZdedeffd Zdej fdZxZS)Emu3VQVAETemporalUpsamplerrcJt|t||dd|_y)Nrrrrrrryrzr$r%rr}r+rrr,s r-r%z"Emu3VQVAETemporalUpsample.__init__( #  !   r.r/cP|j\}}}}}|jdddddjj|d|}t j |dd }|j||||djdddddj}|j |}|S) Nrrrr[rr\rrr)r`rarbrcrrr})r+r/rhrjrirkrls r-rBz!Emu3VQVAETemporalUpsample.forwards8E8K8K5 Hh%--aAq!<GGINNz[]_gh  m#IV %**:xPRS[[\]_`bcefhijuuw  -0 r.rrMs@r-rrs*      U\\r.rcHeZdZdedeffd Zdej fdZxZS)Emu3VQVAETemporalDownsamplerrcJt|t||dd|_y)N)r[rr)rrrrrrs r-r%z$Emu3VQVAETemporalDownsample.__init__ rr.r/c(|j|}|Sr#)r}rs r-rBz#Emu3VQVAETemporalDownsample.forwards -0 r.rrMs@r-rr s*      U\\r.rc(eZdZ dfd ZdZxZS)Emu3VQVAETemporalResnetBlockcpt|||_||n||_t j ||_t||dd|_t j ||_ t||dd|_ |j|jk7r t j||ddd|_ yy)Nrrrrrrx) r$r%r~rr& BatchNorm3dnorm1rconv1norm2conv2r nin_shortcutrs r-r%z%Emu3VQVAETemporalResnetBlock.__init__s &+7+?K\^^K0 $  !   ^^L1 $  !     t00 0 "  !D  1r.cL|}|j|}|tj|z}|j|}|j |}|tj|z}|j |}|j |jk7r|j|}||zSr#) rrGsigmoidrrrr~rr)r+r/r?s r-rBz$Emu3VQVAETemporalResnetBlock.forward=s  =1 }55  =1  =1 }55  =1   t00 0((2H-''r.r#rrMs@r-rrs@ (r.rc~eZdZ ddedeedeeffd ZddejdeejfdZxZ S) Emu3VQVAEResnetBlockr~rquant_channelsct|||_||n|}||_||_|=t j |ddd|_t j |ddd|_n"t|||_t|||_t j||ddd|_ t j||ddd|_ |j|jk7r t j||ddd|_ yy) NrrTrrrrxr)r$r%r~rrr&rrrrr|rrr)r+r~rrr,s r-r%zEmu3VQVAEResnetBlock.__init__Ns &&2&:{ (,  !;2SW`deDJr,s r-r%zEmu3VQVAEGroupNorm.__init__s "6"r.ctj||j|j|j|j Sr#)r group_normrrUbiasr)r+inputrs r-rBzEmu3VQVAEGroupNorm.forwards)||E4??DKKDHHUUr.r#)rCrDrErrr%rBrLrMs@r-rrs #Vr.rc`eZdZdfd Zddej deej fdZxZS)Emu3VQVAEMiddleBlockct|t||||_t ||_|t |ddd|_nt|||_t||||_ y)Nr~rrrrTr) r$r%rblock_1rattn_1r attn_normrblock_2)r+r r~rr,s r-r%zEmu3VQVAEMiddleBlock.__init__so +#$)  .f5  !/[UW]ajnoDN1.+NDN+#$)  r.r/rcb|j||}|}|j||}|j\}}}}|j||||zj dd}|j |d}|j ||||jdddd}||z}|j||}|S)Nrrrr) rrr`rcrfrreshaperar)r+r/rr?rhrjrkrls r-rBzEmu3VQVAEMiddleBlock.forwards ]LA  }lC .;.A.A+ Hfe%**:x%PZZ[\^_`  M215 %--j&%RZZ[\^_abdef  =0  ]LA r.r#) rCrDrEr%rGrKrrBrLrMs@r-rrs, ( U%6%6 huO`O`Fa r.rc>eZdZfdZdej fdZxZS)Emu3VQVAEDownBlockc t |t|j|_|j |_|j }|j}dt|z}||_tj|_ t|jD]K}tj}tj}tj}|||z} |||z} t|j D]~} |jt| | | } |j.||jvs=|jt!||jtj"| dddtj$} || _|| _|| _||jdz k7rt-| | _|jj| Ny)N)rr~rrrTrr)r$r%lenchannel_multipliernum_resolutionsnum_res_blocks base_channelstuplein_channel_multiplierr& ModuleListdownrangeappendrattn_resolutionsrrModuleblockattn attn_normsrt downsample)r+r rrri_levelrrrblock_in block_outi_blockrr,s r-r%zEmu3VQVAEDownBlock.__init__s "6#<#<=$33,, #66 $u-?'@ @%:"MMO T112 #GMMOE==?DJ$'t|jD]\}}t|jD]}|j||}t |j dkDs1|}|j||}|j\}}}} |j|||| zjdd}|j ||d}|j||| |jdddd}||z}||jdz k7s|j|}|S)Nrrrr) enumeraterrrrrrrr`rcrfrrarr) r+r/rblocksrr?rhrjrkrls r-rBzEmu3VQVAEDownBlock.forwards5(3 AOGV !4!45 = 5 W 5m D v{{#a',H$>F$5$5g$>}$MM:G:M:M7J&%$1$6$6z8VV[^$\$f$fghjk$lM$8FKK$8$G$JM$1$9$9*feU]$^$f$fghjkmnpq$rM$,}$8&TU VBBHBG&BM!|:8D GGNN1b !3 "r.r/rcht|jdddD]\}}t|jdzD]}|j|||}t |j dkDs2|}|j|||}|j\}}} } |j||| | zjdd}|j ||d}|j|| | |jdddd}||z}|t |jdz k7s|j|}|S)Nr\rrrr)rrrrrrrrr`rcrfrrar) r+r/rrrrr?rhrjrkrls r-rBzEmu3VQVAEUpBlock.forward%sD(27 ?OGV !4!4q!89 = 5 W 5m\ R v{{#a',H$>F$5$5g$>}l$[M:G:M:M7J&%$1$6$6z8VV[^$\$f$fghjk$lM$8FKK$8$G$JM$1$9$9*feU]$^$f$fghjkmnpq$rM$,}$  ? r.rrMs@r-rrs(#"JU%6%6eFWFWr.rc>eZdZfdZdej fdZxZS)Emu3VQVAEEncoderct||j}|j}|j}|j }|j }|rd|zn|}||dz}tjj||ddd|_ t||_ t|||_tjjd|dd |_tjj||ddd|_t%t'j(|j*} tj,|_tj,|_t3| D])} t5||} |j.j7| +t3|j8D]*} t;|| } |j0j7| ,y) Nrr\rrrxrrT)rrrrr)r$r%rr~ double_latentlatent_channelsrrGr&r|conv_inr down_blockr middle_blockrnorm_outconv_outrFmathlog2temporal_downsample_factorr time_convtime_res_stackrrrrr)r+r rr~rr rrrtemporal_down_blocksir}_ time_res_convr,s r-r%zEmu3VQVAEEncoder.__init__:s ,, (( ,,  00#66.;q?*  #5b#99xx{MqYZdef ,V40B**bxUYbf*g    (   #499V-N-N#OP mmo+, (A.|\JD NN ! !$ ' (v,,- 6A8()M    & &} 5  6r. pixel_valuesch|jd}|jdg|jdd}|j|}|j|}|j |}|j |}|t j|z}|j|}|jd|g|jdd}|jddddd}|jD]"}||}|t j|z}$|jD] }||} |jddddd}|S)Nrr\rrrr[) r`rr r r r rGrrrarr)r+r temporal_dimr/r}layers r-rBzEmu3VQVAEEncoder.forwardasH#))!, +|++BH1C1CAB1GH  \2  6 ))-8  m4 }55  m4 - --b,YATATUVUWAXY %--aAq!< NN :D /M U]]=9 9M :(( 1E!-0M 1&--aAq!< r.)rCrDrEr%rGrIrBrLrMs@r-rr9s%6NE$4$4r.rc\eZdZdeffd Zdej dej fdZxZS)Emu3VQVAEDecoderr ct ||j}|j|jdz}t j |_t|jD]>}t|j|j}|jj|@ttj|j }t j |_t|D]=}t%|j|j}|j"j|?t j&|j|ddd|_t+||||_t/||_|j|jdz}t3|||_t j&||j6ddd|_y)Nr\rrrrx)rr)r$r%rSrrr&rrrrrr rrFrrrrrr|r rr rup_blockrr rr) r+r rrrrtemp_upsample_block_numrr}r,s r-r%zEmu3VQVAEDecoder.__init__s ))''&*C*CB*GG mmov,,- 6A8"22AWAWM    & &} 5  6 #&dii0Q0Q&R"S./ (A,V-C-CVE[E[\D NN ! !$ ' (yy  " "    1R`a(0 ''&*C*CA*FF,^XF        r.r/rctj||fd}|jddddd}|jD] }||} |jD]"}||}|tj |z}$|jddddd}tj |dd\}}|jdg|jdd}|jdg|jdd}|j|}|j||}|j||}|j||}|tj |z}|j|}|S)Nrr_rrrr[r\)rGcatrarrrchunkrr`r r rr r)r+r/rhidden_quant_statesrs r-rBzEmu3VQVAEDecoder.forwardsp#ii (E1M199!Q1aH(( =E"'(;"<  =^^ FE"'(;"<  5==1D#E E  F299!Q1aH&+kk2Eqa&P# |- --bK=3F3Fqr3JK +|++BH1C1CAB1GH  ]3 ))-F  m\B  m\B }55  m4 r.) rCrDrErr%rGrHrBrLrMs@r-rrs+% % NU\\r.raN This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Emu3VQVAEConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aAThe VQ-VAE model used in Emu3 for encoding/decoding images into discrete tokens. This model follows the "Make-a-scene: Scene-based text-to-image generation with human priors" paper from [ Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman](https://arxiv.org/abs/2203.13131). ceZdZeZdZdZdZdZdZ gdZ dZ deffd Z de jde jfd Zd e jfd ZxZS) Emu3VQVAE emuvideovqrT)rrrrOct|tjtjfr-tjj |j ddyt|tjrtjj|j tjd|jxtjj|j \}}|dkDrdtj|z nd}tjj|j| |yyt|tjtjtj frUtjj#|j dtjj#|jdyy)Nfan_outrelu)r nonlinearity)arr) isinstancer&r|rinitkaiming_normal_rULinearkaiming_uniform_rsqrtr_calculate_fan_in_and_fan_outrW BatchNorm2drr constant_)r+modulefan_inrbounds r- _init_weightszEmu3VQVAE._init_weightss fryy"))4 5 GG # #FMM PV # W  * GG $ $V]]diil $ C{{&GGAA&--P 17!DIIf--  ufe<' N O GG  fmmQ / GG  fkk1 -Pr.r ct||||_t||_t ||_t||_dt|jdz z|_ t|j|jdd|_t|j|jdd|_dt|jdz z|_|j%|j'y)Nrr)rrrrr)r$r%r rencoderrdecoderrOquantizerrvision_spatial_factorrr rS quant_convpost_quant_convspatial_scale_factoreval post_initrXs r-r%zEmu3VQVAE.__init__s   '/ '/ 08 %&3v/H/H+IA+M%N")  " "F$4$4)T]  /   f44)T] %&#f.G.G*H1*L$M!  r. image_sizesc |jdk(}|rL|jj}|j\}}}}|j dj d|ddd}n|j\}}}}}|j |} | jddddd} |j| } | jddddd} |j| } |r| jdn| } t| |D cgc]B\} } | dt| d|jz dt| d|jz fD} } } | Scc} } w)Nr[rrrr)ndimr rr` unsqueezerepeatr<rar@r>squeezerrFr?)r+rrEis_imagerirhrjrkrlr/codes image_tokens single_imagers r-encodezEmu3VQVAE.encode sX$$) {{==H2>2D2D /J&%'11!4;;AxAqQLrTflattenrcrarbrAr=rr rrrB) r+r/rKrhrirkrlquantrj post_quantvideos r-decodezEmu3VQVAE.decode)sK %%* )33A6M.;.A.A+ Hfe '' (=(=(?@;;r? :xIQQRSUVXY[\^_`kkm))%0  aAq!,''1aA6  Z/  t{{== = KK $ $ T.. . D-- -  'uQT{1E1r.)rCrDrEr config_classbase_model_prefixmain_input_name_supports_sdpa_supports_flash_attn_2_supports_flex_attn_no_split_modulesr:r%rGrHrOrUrLrMs@r-r&r&sj#L$$ON! .*5<<ell82ELL2r.r&ceZdZdZdZedZedZedZedZ edZ edZ d e e jd e jfd Zd e jd e jfd Zy )Emu3ImageVocabularyMappingzM A class for mapping discrete image tokens from VQGAN to BPE tokens. cj||_|jd|_|jd|_y)Nz <|extra_200|>z) vocab_mapget eol_token_idimage_token_id)r+r`s r-r%z#Emu3ImageVocabularyMapping.__init__Hs+"%MM/:'mmI6r.c t|jjDcgc]\}}|jds|c}}Scc}}wNz<|visual tokensortedr`items startswithr+namevals r-rMz'Emu3ImageVocabularyMapping.image_tokensMs8DNN,@,@,BhytSdooVfFgshiih A A c t|jjDcgc]\}}|jds|c}}Scc}}wrerfrjs r-image_tokens_strz+Emu3ImageVocabularyMapping.image_tokens_strQs8T^^-A-A-Ci ctWgGhtijjirmct|jDcic]}t|dd|j| c}Scc}w)Nir)rorFr`)r+tokens r-img2bpez"Emu3ImageVocabularyMapping.img2bpeUs5FJF[F[\UE"RL!4>>%#88\\\s#5cj|jjDcic]\}}|| c}}Scc}}wr#)rrrh)r+kvs r-bpe2imgz"Emu3ImageVocabularyMapping.bpe2imgYs+!%!3!3!56A1666s /ctjt|jj dztj }|jj D] \}}|||< |SNrdtype)rGzerosmaxrvkeysrFrhr+mappingrtrus r-bpe2img_mapping_tensorz1Emu3ImageVocabularyMapping.bpe2img_mapping_tensor][++c$,,"3"3"56:%))LLL&&( DAqGAJ r.ctjt|jj dztj }|jj D] \}}|||< |Srx)rGr{r|rrr}rFrhr~s r-img2bpe_mapping_tensorz1Emu3ImageVocabularyMapping.img2bpe_mapping_tensordrr. img_batchr7c,|j}tj|jddftj|j z}|j |jd}tj||gd}|j|S)Nrrrycpur\r_) devicerGonesr`rFrbrtor")r+rreol_row img_tokenss r-convert_img2bpez*Emu3ImageVocabularyMapping.convert_img2bpeksw!!**iooa0!4EIIFIZIZZ00e1DE YY G4"= }}V$$r.c|j}|dddf}|j|jd}|j|S)N.r\r)rrr)r+rrrs r-convert_bpe2imgz*Emu3ImageVocabularyMapping.convert_bpe2imgrsG!!c3B3h' 00e1DE }}V$$r.N)rCrDrErrr%rrMrorrrvrrrrGrHrrr9r.r-r^r^Cs7 jjkk]]77  %ell);% %%%%,,%r.r^ceZdZdgZdZdZy)Emu3PreTrainedModelrTc|jjj}t|tr|j |j yt|tjtjfrY|jjjd||j%|jjjyyt|tjrf|jjjd||j 2|jj|j jyyy)Ng)meanstd)r get_text_configinitializer_ranger.r&applyr:r&r1r|rUrVnormal_rzero_rQ padding_idx)r+r7rs r-r:z!Emu3PreTrainedModel._init_weightsskk))+== fi ( LL-- . BII 6 7 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..r.N)rCrDrEr\r[r:r9r.r-rrys ?r.ra Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Has to be an instance of [`~cache_utils.Cache`] instance, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). The model will output the same cache type that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, max_num_images, max_num_tiles, channels, image_size, image_size)): The tensors corresponding to the input images. Pixel values can be obtained using [`AutoImageProcessor`]. See [`Emu3ImageProcessor.__call__`] for details ([]`Emu3Processor`] uses [`Emu3ImageProcessor`] for processing images). image_sizes (`torch.LongTensor` of shape `(batch_size, 2)`): The sizes of the images in the batch, being (height, width) for each image. Image sizes can be obtained using [`AutoImageProcessor`]. See [`Emu3ImageProcessor.__call__`] for details ([]`Emu3Processor`] uses [`Emu3ImageProcessor`] for processing images). attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Has to be an instance of [`~cache_utils.Cache`] instance, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. cNeZdZdeffd ZeeefdZxZ S) Emu3TextModelr c t||tjt |j Dcgc]}t ||c}|_ycc}wr#)r$r%r&rrnum_hidden_layersrlayersr*s r-r%zEmu3TextModel.__init__$sD  mmBGH`H`Ba bY fi 0 b  bsAc $t|di|yrr$rB)r+ super_kwargsr,s r-rBzEmu3TextModel.forward*s ','r.) rCrDrErr%rrEMU3_TEXT_INPUTS_DOCSTRINGrBrLrMs@r-rr#s2 z *+EF(G(r.rceZdZeZfdZeedddee e e dfdZ xZ S) Emu3ForCausalLMcDt||t||_yr#)r$r%rmodelrXs r-r%zEmu3ForCausalLM.__init__3s  "6* r.num_logits_to_keepz4.50logits_to_keep)versionnew_namer output_typerVc 6tjy)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. logits_to_keep (`int` or `torch.Tensor`, *optional*): If an `int`, compute logits for the last `logits_to_keep` tokens. If `0`, calculate logits for all `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a `torch.Tensor`, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length). Returns: Example: ```python >>> from transformers import Emu3Processor, Emu3ForConditionalGeneration >>> import torch >>> import requests >>> from PIL import Image >>> model = Emu3ForCausalLM.from_pretrained("BAAI/Emu3-Chat-hf", torch_dtype=torch.bfloat16) >>> processor = Emu3Processor.from_pretrained("BAAI/Emu3-Chat-hf") >>> inputs = processor(text=["Can you write me a poem about winter."], return_tensors="pt").to(model.device) >>> generated_ids = model.generate(**inputs, max_new_tokens=100, do_sample=False) >>> processor.batch_decode(generated_ids, skip_special_tokens=True)[0] ```Nr)rr,s r-rBzEmu3ForCausalLM.forward7sH r.)rCrDrErrVr%rrrrrr rBrLrMs@r-rr0sV!L+)6DTU*+EF+AP`a bGV r.rc heZdZdgZdZfdZdZdZdejdejfdZ ejd ejd e d e fd Zeeeeee  ddeejdeejdeej,deej,deejdeedeejdeedeedeedeejdeejdee ej,fdefdZ dfd ZxZS)Emu3ForConditionalGenerationztext_model.lm_head.weightFct||tj|j|_t |j|_t|j|_ |jyr#) r$r%r _from_config text_config text_modelr& vq_configvqmodelr^vocabulary_mapvocabulary_mappingrDrXs r-r%z%Emu3ForConditionalGeneration.__init__bsY  )66v7I7IJ !1!12 ">yI  ##L1 r.r input_idsr0r1past_key_values inputs_embedsr4r3output_hidden_statesr5labelsrr7c | | n|jj} | | n|jj} |du|duz r td| | td|c|j ||}||j j k(}|j|j|j}|j||}|j||||||| | | |  S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. logits_to_keep (`int` or `torch.Tensor`, *optional*): If an `int`, compute logits for the last `logits_to_keep` tokens. If `0`, calculate logits for all `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a `torch.Tensor`, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length). Returns: Example: ```python >>> from transformers import Emu3Processor, Emu3ForConditionalGeneration >>> import torch >>> import requests >>> from PIL import Image >>> model = Emu3ForConditionalGeneration.from_pretrained("BAAI/Emu3-Chat-hf", torch_dtype=torch.bfloat16) >>> processor = Emu3Processor.from_pretrained("BAAI/Emu3-Chat-hf") >>> conversation = [ ... { ... "role": "system", ... "content": [ ... {"type": "text", "text": "You are a helpful assistant."}, ... ], ... }, ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "Please describe the image."}, ... ], ... }, ... ] >>> prompt = processor.apply_chat_template(conversation, add_generation_prompt=True) >>> image = Image.open(requests.get("https://www.ilankelman.org/stopsigns/australia.jpg", stream=True).raw) >>> inputs = processor(images=[image], text=[prompt], return_tensors="pt").to(model.device, torch.bfloat16) >>> generated_ids = model.generate(**inputs, max_new_tokens=100, do_sample=False) >>> processor.batch_decode(generated_ids, skip_special_tokens=True)[0] ```NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either one) rr0r1rrr4r3rr5r) r r3r ValueErrorrrrcrrrzmasked_scatterr)r+rrrEr0r1rrr4r3rr5rrrMspecial_image_masks r-rBz$Emu3ForConditionalGeneration.forwardsJ2C1N-TXT_T_TqTq$8$D $++JjJj   -t"; <s   # (Av   #00{KL!*d.E.E.T.T!T '??9+;+;Y__ML!001C\RI)%+'/!5))  r.c Rt ||f|||||||d| } |ddk7rd| d<| S)N)rr0rr5r1rr4rr)r$prepare_inputs_for_generation) r+rrr0rr5r1r4rr> model_inputsr,s r-rz:Emu3ForConditionalGeneration.prepare_inputs_for_generationsZw<   +)')%%     !  !+/L (r.) NNNNNNNNNNNNr)NNNNNTN)rCrDrE_tied_weights_keys_supports_static_cacher%rrrGrKrIrno_gradrFrrrEMU3_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrrHr rJrrBrrLrMs@r-rr^s56"64U->->UM]M]" ]]0@0@#VY$*+@A+AP_`1548.21537+/59$(,0/359-134c E,,-c u001c ell+ c !. c u//0 c "%c   1 12c D>c $D>c 'tnc !!1!12c ))*c c5<</0c  c aBc Pr.r)rrrrr&)Nr functoolsrtypingrrrrrGtorch.nnr&torch.nn.functional functionalrtorch.utils.checkpoint cache_utilsr generationr modeling_outputsr modeling_utilsr utilsr rrrrutils.deprecationrchameleon.modeling_chameleonrrllama.modeling_llamarrrsiglip.modeling_sigliprconfiguration_emu3rrrr_CHECKPOINT_FOR_DOC get_loggerrCloggerrrrOrtrvrrrrrrrrrrrrrrEMU3_VQ_START_DOCSTRINGr&r^rrrrrr__all__r9r.r-rs1" %//  ).1 5KK)   H %A(AH$ryy$D %H 299bii:!299!H ."))&.(299.(b<(299<(~&o& V V299D88v7ryy7tCryyCLCryyCL" c2c2c2L3%3%l?2I?(DNL^ (J 3 (+&(;_+\|#6|~ r.