Replies: 1 comment
-
|
Hey @adha9990 , I think in the latest version of diffusers, LoRAAttnProcessor2_0 has been reduced to just a stub: class LoRAAttnProcessor2_0:
def __init__(self):
passAll the LoRA logic is now handled through PEFT (peft library), so modifying this class no longer affects attention behavior or output shape. That’s why adding mask generation here won’t work anymore. If you would still like to output both a defective image and a mask, I think you could subclass UNet2DConditionModel and add a small mask_head CNN to the forward pass: from diffusers.models.unet_2d_condition import UNet2DConditionModel
import torch.nn as nn
class UNet2DConditionWithMask(UNet2DConditionModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.mask_head = nn.Sequential(
nn.Conv2d(self.config.block_out_channels[-1], 64, kernel_size=3, padding=1),
nn.ReLU(),
nn.Conv2d(64, 1, kernel_size=1),
nn.Sigmoid()
)
def forward(self, *args, **kwargs):
result = super().forward(*args, **kwargs)
x = result["sample"] if isinstance(result, dict) else result
mask = self.mask_head(x)
return {"sample": x, "mask": mask}And you could use it like this - unet = UNet2DConditionWithMask.from_pretrained(
"./models/stable-diffusion-2-base/unet", torch_dtype=torch.bfloat16
)
pipeline = StableDiffusionPipeline.from_pretrained(
"./models/stable-diffusion-2-base",
unet=unet,
torch_dtype=torch.bfloat16
)and outputs through - out = pipeline(prompt="a vfx with sks")
image = out.images[0]
mask = out["mask"][0].detach().cpu().numpy()Let me know if this works! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi, I’m a student researching few-shot defect image generation using Stable Diffusion. I’m trying to generate both defect images and corresponding defect masks simultaneously.
In the DualAnoDiff project, the team achieved this functionality by modifying the older version of train_dreambooth_lora.py and LoRAAttnProcessor2_0 in the Diffusers library. Specifically, they adjusted the
LoRAAttnProcessor2_0code to allow the simultaneous generation of two outputs: a defective image and a defect mask.Problem
However, in the latest version of the Diffusers library,
LoRAAttnProcessor2_0has been simplified to just a pass implementation, I understand that Diffusers now use PEFT to manage LoRA,but I am unsure of how to adapt or extend theLoRAAttnProcessor2_0in the new version to achieve the same dual output functionality.Request
Could you provide guidance on how to modify the new
LoRAAttnProcessor2_0in the Diffusers library so that it can generate both the defective image and the defect mask? Alternatively, if there are any recommended examples or modifications to achieve this, I would greatly appreciate your input.A code example would be incredibly helpful.
Thank you for your help!
Here is the error report from my test: I referred to pipeline_stable_diffusion.py doubled the dimensions, but an error occurred during prediction in self.unet. I know the error is due to incorrect dimensions, but I'm not sure how to modify it to generate two images.
Beta Was this translation helpful? Give feedback.
All reactions