Flux1.Dev Kohya Loras text encoder layers no more supported

[christopher5106]

Hello,

I trained a Lora with Kohya SS and I have a problem of conversion, I thought it should have been managed by your conversion script ?

Loading adapter weights from state_dict led to unexpected keys found in the model: single_transformer_blocks.0.proj_out.lora_A.default_0.weight, single_transformer_blocks.0.proj_out.lora_B.default_0.weight, single_transformer_blocks.1.proj_out.lora_A.default_0.weight, single_transformer_blocks.1.proj_out.lora_B.default_0.weight, single_transformer_blocks.2.proj_out.lora_A.default_0.weight, single_transformer_blocks.2.proj_out.lora_B.default_0.weight, single_transformer_blocks.3.proj_out.lora_A.default_0.weight, single_transformer_blocks.3.proj_out.lora_B.default_0.weight, single_transformer_blocks.4.proj_out.lora_A.default_0.weight, single_transformer_blocks.4.proj_out.lora_B.default_0.weight, single_transformer_blocks.5.proj_out.lora_A.default_0.weight, single_transformer_blocks.5.proj_out.lora_B.default_0.weight, single_transformer_blocks.6.proj_out.lora_A.default_0.weight, single_transformer_blocks.6.proj_out.lora_B.default_0.weight, single_transformer_blocks.7.proj_out.lora_A.default_0.weight, single_transformer_blocks.7.proj_out.lora_B.default_0.weight, single_transformer_blocks.8.proj_out.lora_A.default_0.weight, single_transformer_blocks.8.proj_out.lora_B.default_0.weight, single_transformer_blocks.9.proj_out.lora_A.default_0.weight, single_transformer_blocks.9.proj_out.lora_B.default_0.weight, single_transformer_blocks.10.proj_out.lora_A.default_0.weight, single_transformer_blocks.10.proj_out.lora_B.default_0.weight, single_transformer_blocks.11.proj_out.lora_A.default_0.weight, single_transformer_blocks.11.proj_out.lora_B.default_0.weight, single_transformer_blocks.12.proj_out.lora_A.default_0.weight, single_transformer_blocks.12.proj_out.lora_B.default_0.weight, single_transformer_blocks.13.proj_out.lora_A.default_0.weight, single_transformer_blocks.13.proj_out.lora_B.default_0.weight, single_transformer_blocks.14.proj_out.lora_A.default_0.weight, single_transformer_blocks.14.proj_out.lora_B.default_0.weight, single_transformer_blocks.15.proj_out.lora_A.default_0.weight, single_transformer_blocks.15.proj_out.lora_B.default_0.weight, single_transformer_blocks.16.proj_out.lora_A.default_0.weight, single_transformer_blocks.16.proj_out.lora_B.default_0.weight, single_transformer_blocks.17.proj_out.lora_A.default_0.weight, single_transformer_blocks.17.proj_out.lora_B.default_0.weight, single_transformer_blocks.18.proj_out.lora_A.default_0.weight, single_transformer_blocks.18.proj_out.lora_B.default_0.weight, single_transformer_blocks.19.proj_out.lora_A.default_0.weight, single_transformer_blocks.19.proj_out.lora_B.default_0.weight, single_transformer_blocks.20.proj_out.lora_A.default_0.weight, single_transformer_blocks.20.proj_out.lora_B.default_0.weight, single_transformer_blocks.21.proj_out.lora_A.default_0.weight, single_transformer_blocks.21.proj_out.lora_B.default_0.weight, single_transformer_blocks.22.proj_out.lora_A.default_0.weight, single_transformer_blocks.22.proj_out.lora_B.default_0.weight, single_transformer_blocks.23.proj_out.lora_A.default_0.weight, single_transformer_blocks.23.proj_out.lora_B.default_0.weight, single_transformer_blocks.24.proj_out.lora_A.default_0.weight, single_transformer_blocks.24.proj_out.lora_B.default_0.weight, single_transformer_blocks.25.proj_out.lora_A.default_0.weight, single_transformer_blocks.25.proj_out.lora_B.default_0.weight, single_transformer_blocks.26.proj_out.lora_A.default_0.weight, single_transformer_blocks.26.proj_out.lora_B.default_0.weight, single_transformer_blocks.27.proj_out.lora_A.default_0.weight, single_transformer_blocks.27.proj_out.lora_B.default_0.weight, single_transformer_blocks.28.proj_out.lora_A.default_0.weight, single_transformer_blocks.28.proj_out.lora_B.default_0.weight, single_transformer_blocks.29.proj_out.lora_A.default_0.weight, single_transformer_blocks.29.proj_out.lora_B.default_0.weight, single_transformer_blocks.30.proj_out.lora_A.default_0.weight, single_transformer_blocks.30.proj_out.lora_B.default_0.weight, single_transformer_blocks.31.proj_out.lora_A.default_0.weight, single_transformer_blocks.31.proj_out.lora_B.default_0.weight, single_transformer_blocks.32.proj_out.lora_A.default_0.weight, single_transformer_blocks.32.proj_out.lora_B.default_0.weight, single_transformer_blocks.33.proj_out.lora_A.default_0.weight, single_transformer_blocks.33.proj_out.lora_B.default_0.weight, single_transformer_blocks.34.proj_out.lora_A.default_0.weight, single_transformer_blocks.34.proj_out.lora_B.default_0.weight, single_transformer_blocks.35.proj_out.lora_A.default_0.weight, single_transformer_blocks.35.proj_out.lora_B.default_0.weight, single_transformer_blocks.36.proj_out.lora_A.default_0.weight, single_transformer_blocks.36.proj_out.lora_B.default_0.weight, single_transformer_blocks.37.proj_out.lora_A.default_0.weight, single_transformer_blocks.37.proj_out.lora_B.default_0.weight. 
No LoRA keys associated to CLIPTextModel found with the prefix='text_encoder'. This is safe to ignore if LoRA state dict didn't originally have any CLIPTextModel related params. You can also try specifying `prefix=None` to resolve the warning. Otherwise, open an issue if you think it's unexpected: https://github.com/huggingface/diffusers/issues/new

Please advise.

[Twinkle-ce]

same problem

[christopher5106]

Same problem with Flux 1 Dev:

Kohya loras are no more supported by "diffusers from source".

They were for example on version 0.33.0.dev0.

I'm not sure who can help on this but it's real @sayakpaul

[sayakpaul]

#11999

[christopher5106]

Thanks for the PR and merge @sayakpaul
Now kohya transformers lora are well loaded but still not text encoder loras:

No LoRA keys associated to CLIPTextModel found with the prefix='text_encoder'. This is safe to ignore if LoRA state dict didn't originally have any CLIPTextModel related params. You can also try specifying `prefix=None` to resolve the warning. Otherwise, open an issue if you think it's unexpected: https://github.com/huggingface/diffusers/issues/new

If this can help, here are a few keys:

lora_te1_text_model_encoder_layers_0_mlp_fc1.alpha: [], torch.bfloat16
lora_te1_text_model_encoder_layers_0_mlp_fc1.lora_down.weight: [64, 768], torch.bfloat16
lora_te1_text_model_encoder_layers_0_mlp_fc1.lora_up.weight: [3072, 64], torch.bfloat16
lora_te1_text_model_encoder_layers_0_mlp_fc2.alpha: [], torch.bfloat16
lora_te1_text_model_encoder_layers_0_mlp_fc2.lora_down.weight: [64, 3072], torch.bfloat16
lora_te1_text_model_encoder_layers_0_mlp_fc2.lora_up.weight: [768, 64], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_k_proj.alpha: [], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_k_proj.lora_down.weight: [64, 768], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_k_proj.lora_up.weight: [768, 64], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_out_proj.alpha: [], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_out_proj.lora_down.weight: [64, 768], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_out_proj.lora_up.weight: [768, 64], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_q_proj.alpha: [], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_q_proj.lora_down.weight: [64, 768], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_q_proj.lora_up.weight: [768, 64], torch.bfloat16
lora_te1_text_model_encoder_layers_0_self_attn_v_proj.alpha: [], torch.bfloat16

This was working well before, I checked again.

[sayakpaul]

You mean it was working with Diffusers 0.34.0?

Can you provide a reproducible example?

[christopher5106]

No it wasn't working with 0.34.0.
It was working with 0.33.0.dev0.
I can share you a lora trained on random data for 1 epoch for example.

[christopher5106]

Sent on your email @ gmail

[sayakpaul]

It was working with 0.33.0.dev0.

Which particular commit? It's needed to be known so that we can track down what changed and fix it.

[christopher5106]

37c9697

[christopher5106]

Still searching a public lora with te

[christopher5106]

I don't have and didn't find any public Flux Dev Kohya Lora with text encoder, but I have made an example with Flux Kontext for which the lora has been trained on public data, and it's the same loading problem as on Flux Dev:

import torch
from diffusers import FluxKontextPipeline
pipe = FluxKontextPipeline.from_pretrained("black-forest-labs/FLUX.1-Kontext-dev", torch_dtype=torch.bfloat16)
pipe.to("cuda")
pipe.load_lora_weights("scenario-labs/big-head-kontext-lora", weight_name="flux_kontext_lora.safetensors")

[christopher5106]

Same error wit the converted lora by kohya conversion script to ai-toolkit, for Flux Dev and Flux Kontext Dev loras:

import torch
from diffusers import FluxKontextPipeline
pipe = FluxKontextPipeline.from_pretrained("black-forest-labs/FLUX.1-Kontext-dev", torch_dtype=torch.bfloat16)
pipe.to("cuda")
pipe.load_lora_weights("scenario-labs/big-head-kontext-lora", weight_name="flux_kontext_converted_lora.safetensors")

[sayakpaul]

What LoRA conversion script is being referred to here? I am seeing far too many instances where you are just mentioning issues without links / snippets for the maintainers to investigate things further. It was only after I requested the snippet and checkpoint, they were provided. Can we please avoid that? It helps both maintainers and the users save time.