[Bug] IMG2IMG with additional image reference returns wrong image. #1730

Git commit

3590aa8

Operating System & Version

Windows 11

GGML backends

Vulkan

Command-line arguments used

sd-cli.exe --backend vulkan0 --diffusion-model flux-2-klein-4b-Q8_0.gguf --llm Qwen3-4B-UD-Q4_K_XL.gguf --vae flux2.full_encoder_small_decoder.safetensors -p "Turn first image into a high quality photograph. Use second image as reference." -r main_image.png -r reference.png -o output_photo.png --cfg-scale 2 --steps 4 --offload-to-cpu --fa --seed 42

Steps to reproduce

Execute command line with the following inputs
main_image.png:

reference.png:

What you expected to happen

A high quality photographs of the main image

What actually happened

Second image(reference.png) is returned!

Logs / error messages / stack trace

D:\sd-MASTER-3590aa8-bin-win-vulkan-x64>sd-cli.exe --backend vulkan0 --diffusion-model flux-2-klein-4b-Q8_0.gguf --llm Qwen3-4B-UD-Q4_K_XL.gguf --vae flux2.full_encoder_small_decoder.safetensors -p "Turn first image into a high quality photograph. Use second image as reference." -r main_image.png -r reference.png -o output_photo.png --cfg-scale 2 --steps 4 --offload-to-cpu --fa --seed 42
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5070 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Graphics (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from D:\sd-MASTER-3590aa8-bin-win-vulkan-x64\ggml-vulkan.dll
load_backend: loaded CPU backend from D:\sd-MASTER-3590aa8-bin-win-vulkan-x64\ggml-cpu-alderlake.dll
[INFO ] common.cpp:2086 - set width x height to 1024 x 1024
[INFO ] stable-diffusion.cpp:407 - loading diffusion model from 'flux-2-klein-4b-Q8_0.gguf'
[INFO ] model_loader.cpp:236 - load flux-2-klein-4b-Q8_0.gguf using gguf format
[INFO ] stable-diffusion.cpp:469 - loading llm from 'Qwen3-4B-UD-Q4_K_XL.gguf'
[INFO ] model_loader.cpp:236 - load Qwen3-4B-UD-Q4_K_XL.gguf using gguf format
[INFO ] stable-diffusion.cpp:483 - loading vae from 'flux2.full_encoder_small_decoder.safetensors'
[INFO ] model_loader.cpp:239 - load flux2.full_encoder_small_decoder.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:534 - Version: Flux.2 klein
[INFO ] stable-diffusion.cpp:560 - Weight type stat: f32: 455 | q8_0: 80 | q4_K: 154 | q5_K: 30 | q6_K: 49 | iq4_xs: 20 | i32: 1 | bf16: 9
[INFO ] stable-diffusion.cpp:561 - Conditioner weight type stat: f32: 145 | q4_K: 154 | q5_K: 30 | q6_K: 49 | iq4_xs: 20
[INFO ] stable-diffusion.cpp:562 - Diffusion model weight type stat: f32: 60 | q8_0: 80 | bf16: 9
[INFO ] stable-diffusion.cpp:563 - VAE weight type stat: f32: 250 | i32: 1
[INFO ] stable-diffusion.cpp:993 - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:527 - vae decoder: ch = 96
[INFO ] stable-diffusion.cpp:1097 - Using flash attention
[INFO ] stable-diffusion.cpp:1111 - Using flash attention in the diffusion model
[INFO ] stable-diffusion.cpp:1242 - total params memory size = 7822.64MB (VRAM 0.00MB, RAM 7822.64MB): text_encoders 3602.16MB(RAM), diffusion_model 4101.40MB(RAM), vae 119.08MB(RAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1342 - running in Flux FLOW mode
[INFO ] stable-diffusion.cpp:4573 - generate_image 1024x1024
[INFO ] denoiser.hpp:1079 - get_sigmas with Flux2 scheduler
[INFO ] stable-diffusion.cpp:3579 - sampling using Euler method
[INFO ] stable-diffusion.cpp:4086 - EDIT mode
|##################################################| 108/108 - 656.62MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.20s (read: 0.02s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:4195 - encode_first_stage completed, taking 2.48s
|##################################################| 298/298 - 3.22GB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.57s (read: 0.28s, memcpy: 0.00s, convert: 0.02s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:4299 - get_learned_condition completed, taking 2.16s
[INFO ] stable-diffusion.cpp:4619 - generating image: 1/1 - seed 42
|##################################################| 149/149 - 6.08GB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.66s (read: 0.63s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
|==================================================| 4/4 - 3.39s/it
[INFO ] stable-diffusion.cpp:4651 - sampling completed, taking 16.52s
[INFO ] stable-diffusion.cpp:4665 - generating 1 latent images completed, taking 16.52s
[INFO ] stable-diffusion.cpp:4324 - decoding 1 latents
|##################################################| 140/140 - 527.77MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.20s (read: 0.02s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:4344 - latent 1 decoded, taking 3.15s
[INFO ] stable-diffusion.cpp:4348 - decode_first_stage completed, taking 3.16s
[INFO ] stable-diffusion.cpp:4801 - generate_image completed in 24.32s
[INFO ] main.cpp:489 - save result image 0 to 'output_photo.png' (success)
[INFO ] main.cpp:561 - 1/1 images saved

D:\sd-MASTER-3590aa8-bin-win-vulkan-x64>

Additional context / environment details

CPU:
Intel(R) Core(TM) Ultra 9 285K (3.70 GHz)

GPU:
NVIDIA GeForce RTX 5070 Ti

I get the same result with my CPP implementation:

`
void Renderer::performStableDiffusionRendering()
{
_ASSERT(m_main_image->isFloatingPointImage());
_ASSERT(m_main_image >getNbChannels() == 3);
m_main_image->save(_STRING("D:\SB_Debugging\albedo.png"));

	if (m_secondary_image)
	{
		m_secondary_image->save(_STRING("D:\\SB_Debugging\\albedo_secondary.png"));
		_ASSERT(m_secondary_image_Output);
		performStableDiffusionRenderingInternal(m_secondary_image, m_secondary_image_Output);
		// m_secondary_image_Output->save(_STRING("D:\\SD_Debugging\\_secondary_ouput.png"));
		performStableDiffusionRenderingInternal(m_main_image, m_main_image_Output, secondary_image_Output);
		// m_main_image_Output->save(_STRING("D:\\SD_Debugging\\_primary_ouput.png"));
	}
	else
	{
		performStableDiffusionRenderingInternal(m_main_image, m_main_image_Output);
	}
}

void Renderer::performStableDiffusionRenderingInternal(const std::unique_ptr<Graphics::Texture::RGB32FImage>& input, std::unique_ptr<Graphics::Texture::RGB32FImage>& ouput, const std::unique_ptr<Graphics::Texture::RGB32FImage>& additionalReference)
{
	// Build the stable diffusion Api native input images.
	sd_image_t* nativeInputImages = new sd_image_t[additionalReference ? 2 : 1];

	const auto imageSrc = Graphics::Texture::Helper::create_UNTRACKED_RGB8From(*input);
	{
		nativeInputImages[0].width = getRenderingWidth();
		nativeInputImages[0].height = getRenderingHeight();
		nativeInputImages[0].channel = 3u;
		nativeInputImages[0].data = imageSrc->getMutableRawData();
	}

	const auto imageSrc2 = additionalReference ? Graphics::Texture::Helper::create_UNTRACKED_RGB8From(*additionalReference) : nullptr;
	if (imageSrc2)
	{
		nativeInputImages[1].width = getRenderingWidth();
		nativeInputImages[1].height = getRenderingHeight();
		nativeInputImages[1].channel = 3u;
		nativeInputImages[1].data = imageSrc2->getMutableRawData();
	}

	const std::string modelPathStd = "D:\\AIModels\\";

	// Init the context parameters.
	{
		sd_ctx_params_t sd_ctx_params = {};
		sd_ctx_params_init(&sd_ctx_params);

		const std::string tmp0 = modelPathStd + "flux-2-klein-4b-Q8_0.gguf";
		sd_ctx_params.diffusion_model_path = tmp0.c_str();

		const std::string tmp1 = modelPathStd + "Qwen3-4B-UD-Q4_K_XL.gguf";
		sd_ctx_params.llm_path = tmp1.c_str();

		const std::string tmp2 = modelPathStd + "flux2.full_encoder_small_decoder.safetensors";
		sd_ctx_params.vae_path = tmp2.c_str();

		sd_ctx_params.n_threads = std::max((int)Hardware::HardwareInformationSingleton::instance()->nbOfPhysicalCores(), 4);
		sd_ctx_params.wtype = sd_type_t::SD_TYPE_F16;

		//if (Hardware::HardwareInformationSingleton::instance()->isNvidiaGpu())
		//	sd_ctx_params.backend = "cuda0";
		// else
		sd_ctx_params.backend = "vulkan0";

		sd_ctx_params.flash_attn = true;
		sd_ctx_params.params_backend = "cpu";

		m_stableDiffusionCtx = new_sd_ctx(&sd_ctx_params);
		_ASSERT(m_stableDiffusionCtx != nullptr);
	}

	// Configure generation parameters.

	sd_img_gen_params_t sd_img_gen_params;
	std::string negativePrompt = Utilities::nativeStringToStdString(m_settings.m_negativePrompt);
	replaceAll(negativePrompt, "\n", " ");

	std::string positivePrompt;
	if (additionalReference)
	{
		positivePrompt = "Turn first image into a high quality photograph. Use second image as reference. ";
	}
	else
	{
		positivePrompt = "Turn the image into a high quality photograph. ";
	}

	positivePrompt += Utilities::nativeStringToStdString(m_settings.m_PositivePrompt);
	replaceAll(positivePrompt, "\n", " ");

	{
		sd_img_gen_params_init(&sd_img_gen_params);

		sd_img_gen_params.prompt = positivePrompt.c_str();
		sd_img_gen_params.negative_prompt = negativePrompt.c_str();

		sd_img_gen_params.sample_params.sample_method = EULER_SAMPLE_METHOD;

		sd_img_gen_params.width = getRenderingWidth();
		sd_img_gen_params.height = getRenderingHeight();
		sd_img_gen_params.strength = 0.4f;
		sd_img_gen_params.seed = 42;
		sd_img_gen_params.control_strength = 0.9f;

		sd_img_gen_params.ref_images = nativeInputImages;
		sd_img_gen_params.ref_images_count = additionalReference ? 2 : 1;

		sd_img_gen_params.sample_params.guidance.txt_cfg = 2.0f;
		sd_img_gen_params.sample_params.sample_steps = 4;
	}

	// Set the generation callback.
	sd_set_progress_callback(ProgressCallback, nullptr);

	sd_image_t* result = generate_image(m_stableDiffusionCtx, &sd_img_gen_params);
	if (result)
	{
		_ASSERT(result->width == imageSrc->getWidth());
		_ASSERT(result->height == imageSrc->getHeight());
		_ASSERT(result->channel == imageSrc->getNbChannels());

		// Read back result.
		const uint32_t nbTasks = getRenderingWidth() * getRenderingHeight();
		tbb::parallel_for(size_t(0), size_t(nbTasks), [&](size_t tbbIdx) {

			if (m_stopRender)
			{
				tbb::task::current_context()->cancel_group_execution();
				return;
			}

			const uint32_t pixelLinearIdx = (uint32_t)tbbIdx;
			const uint32_t pixelStartInArray = (uint32_t)tbbIdx * 3;
			const uint32_t pixelPosX = (uint32_t)(pixelLinearIdx % getRenderingWidth());
			const uint32_t pixelPosY = (uint32_t)(pixelLinearIdx / getRenderingWidth());
			const Math::Uvec2 pixelPos = Math::Uvec2(pixelPosX, pixelPosY);

			const uint8_t B = result[0].data[pixelStartInArray];
			const uint8_t G = result[0].data[pixelStartInArray + 1u];
			const uint8_t R = result[0].data[pixelStartInArray + 2u];

			const RGBFColor inputColor = RGBFColor((float)B / (float)IMAGE_UNORM_MAX,
				(float)G / (float)IMAGE_UNORM_MAX, (float)R / (float)IMAGE_UNORM_MAX);
			ouput->setPixelFromPosition(inputColor, pixelPos);
		});


		free(result[0].data);
		free(result);
	}

	free(m_stableDiffusionCtx);
	m_stableDiffusionCtx = nullptr;
	delete[] nativeInputImages;
}

`