T2I BackdoorsResearchRead paper
Research paperarXiv preprint

Practical, Generalizable and Robust Backdoor Attacks on Text-to-Image Diffusion Models

Natural-language triggers and CLIP-guided image preparation, evaluated on SD 1.4, SDXL and FLUX.1.

Public preprint View code PDF

Authors

Haoran Dai1Jiawen Wang1Ruo Yang2Manali Sharma2Zhonghao Liao3Yuan Hong4Binghui Wang1

Affiliations

  • 1Illinois Institute of Technology
  • 2Samsung
  • 3Milwaukee School of Engineering
  • 4University of Connecticut
01

Motivation

An ordinary word can carry an unwanted association after a diffusion model has been fine-tuned on poisoned examples.

Text-to-image backdoors connect a trigger in a prompt to an attacker-selected image. This paper studies triggers that fit naturally into a sentence, such as adding "hand" to a prompt about a banana. The security question is whether a small set of poisoned image-text pairs can teach that association while preserving useful generation without the trigger.

The proposed framework combines natural-language trigger selection, image perturbations guided by CLIP, and fine-tuning with both clean and poisoned examples. Its evaluation covers Stable Diffusion 1.4, SDXL, and FLUX.1. The same preparation procedure is applied across these models; each model is separately fine-tuned.

Stable Diffusion 1.4 outputs before and after backdoor fine-tuning, with and without natural-word triggers.
Figure 1. Examples from Stable Diffusion 1.4: trigger-bearing prompts produce solid-color, chessboard, or vehicle universal-feature targets. Clean prompts retain recognizable subjects. These examples illustrate behavior; the quantitative criterion is defined below.
02

Methodology

Bring the poisoned pair closer in CLIP space before fine-tuning.

A natural prompt and an unrelated target image can be far apart in a learned semantic space. The method perturbs the target image to increase its CLIP similarity to the trigger-bearing prompt, while constraining the size of the image change.

Three phases: natural-language trigger selection, CLIP-guided target-image perturbation, and fine-tuning on clean and poisoned image-text pairs.
Figure 2. Top, Phases 1 to 3: the attacker builds a trigger-bearing prompt, perturbs the target image toward it in CLIP space, and fine-tunes the model on backdoor and clean pairs. The clean pairs keep behavior on prompts without the trigger. Bottom: after release, ordinary prompts generate normally, while adding the trigger ("and a hand") produces the target image. Select the figure to open it at full size.
  1. Phase 1 / Prompt construction

    Choose a linguistically plausible trigger

    GPT-4o supplies candidate words related to a base concept. The experiments use fruit, vehicle, and mammal categories. The trigger is ordinary visible text; "concealment" refers to its plausible role in the sentence.

    \[T_{\mathrm{poison}} = T + \delta_T\]
  2. Phase 2 / Image preparation

    Increase similarity within a perturbation budget

    Let \(I_\Delta\) be the chosen target image and \(\delta_I\) its perturbation. CLIPScore compares the modified image with the poisoned prompt. The bound \(\epsilon\) limits the largest absolute pixel change. Projected gradient descent solves this problem in CLIP space (Figure 3).

    \[\max_{\|\delta_I\|_\infty\leq\epsilon}\;\mathrm{CLIPScore}(I_\Delta+\delta_I,\,T+\delta_T)\]
CLIP latent space: the target image starts inside a ball of radius epsilon, far from the fruit prompts that carry the trigger; PGD steps move it toward the triggered prompts while it stays within the ball, visually indistinguishable from the original.
Figure 3. PGD optimization in CLIP space, with semantic similarity drawn as distance (shorter distance means higher CLIPScore). Starting from the target image \(I_\Delta\), each step \(\tilde I_\Delta^{(i)}\) moves closer to the fruit prompts that contain the trigger ("and a hand"), while the ball of radius \(\epsilon\) keeps the perturbed image \(\tilde I_\Delta\) visually indistinguishable from \(I_\Delta\).
  1. Phase 3 / Model fine-tuning

    Learn the target association alongside clean behavior

    The model is fine-tuned on poisoned pairs and clean pairs generated by its original counterpart. The reported default uses 10 poisoned images and 3 clean images. The clean-data objective encourages retention of ordinary generation.

    \[\mathcal{L}_{\mathrm{total}}=\mathbb{E}_{D_p}\mathcal{L}_{\mathrm{backdoor}}+\mathbb{E}_{D_c}\mathcal{L}_{\mathrm{utility}}\]
Notation and source

\(T\) is the clean prompt; \(\delta_T\) denotes added trigger text, so + means textual composition. \(D_p\) and \(D_c\) are the poisoned and clean datasets. The last expression abbreviates the two training-loss terms in Equation 12 (backdoor injection and utility preservation); the loss names follow Algorithm 2. The constrained CLIP objective is Equation 10, with \(\delta_I\) used here for the paper's \(\delta_{I_\Delta}\). Method, pages 6 to 8.

03

Results

Reported results · Tables 4 and 5

Results and figures follow the arXiv preprint (v1, CC BY-NC-ND 4.0).

Target type changes the outcome substantially.

Under the paper's CLIP criterion, the proposed method succeeds in all ten generations for each high-entropy model-category setting. Success is lower on average for chessboard targets (70 to 100%) and solid-color targets (40 to 70%). The comparisons below fix the fruit category, making the effect of the image-preparation variants easier to read.

High-entropy targetsAll nine reported model-category settings

100%

CLIP-defined attack success

10 generated images per setting; Table 4.

Low-entropy targetsAcross the same three models and three categories

40-70%

Solid-color targets remain harder

Gaussian: 0-10%; unperturbed: 0%; Table 4.

Attack success (%) for the fruit category. Each entry uses 10 generations.
ModelTarget entropyProposedGaussianUnperturbed
SD 1.4High1007040
Medium100900
Low7000
SDXLHigh10010030
Medium90200
Low70100
FLUX.1High1006070
Medium7000
Low6000

"Unperturbed" corresponds to the paper's "Clean" attack variant: it uses the original target image in poisoned training pairs. It is a backdoored-model baseline. All three variants use 10 poisoned and 3 clean images. The proposed and Gaussian variants use a perturbation budget of 0.1; the Gaussian variant samples noise with mean 0 and variance \(\epsilon^2/9\), clipped to \(\pm\epsilon\).

Full Table 4 ↗

Benign generation

After backdoor fine-tuning, FID decreases for SD 1.4 and increases for SDXL and FLUX.1. Each model generates 1,000 images from 1,000 COCO prompts, with COCO-5K as the reference set (Table 5).

FID on clean prompts; lower is better.
ModelOriginalBackdoored
SD 1.426.4024.32
SDXL7.419.61
FLUX.18.7410.93

Outputs across models

Generated images across three models, three subject categories, and high-, medium-, and low-entropy target types.
Figure 5. Columns: for each model, a clean prompt (Utility) and a trigger-bearing prompt (Attack). First row: the benign model. Rows 2 to 4: backdoored models trained with high-, medium- and low-entropy targets. Target resemblance varies across model and target combinations. Select the figure to open it at full size.
04

Evaluation & threat model

The attacker controls poisoned training pairs and fine-tunes a pretrained model. The setting therefore requires influence over model training or the model supplied for use. It does not demonstrate that an ordinary prompt can install a persistent backdoor in an untouched deployed model.

The reported success criterion compares a triggered output's CLIP similarity to the target image with its similarity to an output from the original model on the clean prompt. A larger target similarity counts as success. This relative criterion measures target alignment, and does not require exact image reproduction. Each fraction uses ten generated outputs.

The paper also tests prompt perplexity, prompt obfuscation, T2IShield, and image augmentation before fine-tuning. It reports a 3.33% true-positive rate and 7.38% false-positive rate for T2IShield on 3,000 prompts, including 60 poisoned prompts with the trigger "hand". This result concerns that evaluation setting; it does not establish failure of every possible defense.

Limits of the evidence

The displayed results cover three models and three subject categories. Ten generations per cell give a limited estimate of success. CLIP guides image preparation and also supplies the main success criterion, so agreement with independent perceptual and human measures matters. The preprint also reports a human evaluation (Table 6) and results under LPIPS, MS-SSIM and SSIM (Table 11). The numerical summary here uses arXiv v1.

05

Citation

Citation for the arXiv preprint, the source of every result and figure on this page.

BibTeX
Download 
@misc{dai2025practical,
  title         = {Practical, Generalizable and Robust Backdoor Attacks on
                   Text-to-Image Diffusion Models},
  author        = {Haoran Dai and Jiawen Wang and Ruo Yang and Manali Sharma and
                   Zhonghao Liao and Yuan Hong and Binghui Wang},
  year          = {2025},
  eprint        = {2508.01605},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CR},
  url           = {https://arxiv.org/abs/2508.01605}
}