When AI is operating as an assistant, like an editor, we need ways to let it interact with our content directly. Inpainting gives users the ability the let AI adjust parts of a piece of content without regenerating or impacting the whole.

This pattern applies across all types of content, and can be used for interacting with existing content or remixing AI-generated content during a session.

For example,

  • When writing an essay, I may ask AI to adjust the tone of a paragraph that I feel I am not getting quite right to see how it would reflect back my overall tone to me.
  • When AI generates text, I can direct it to make changes in situ to the conversation itself.
  • When generating creative assets like visuals or audio, I can isolate a part of the interface and have the AI apply new tokens only to that area.
Adobe Firefly and Notion are two examples of tools that use inpainting effectively. The action is clearly identifiable from the interface with suggested prompts and nudges to help the user iterate on their generation

Best practices

The most critical aspect of inpainting is that it keeps the user in the driver's seat. The AI is never directly manipulating the source - instead, it is offering suggestions for how to improve it based on the user's instruction. The most seamless implementations of this pattern maintain that sense that the user is in control, drawing from other patterns that make it easy to know how to improve what exists, and then generate options, without a ton of thought.

For example, Adobe Firefly and Midjourney offer four versions of the regeneration for the user to choose from. Notion allows the user to manipulate the in-painted regeneration before accepting it into the document.

TPerhaps there is a future approach that operates more like an editor: the user could give instructions or ask for feedback, the AI might review it and suggest multiple changes, and the user can accept, reject, or regenerate those comments.

The more AI can truly operate as a partner, the more users will rely on it to complete tasks.

Details and variations

Inpainting is a micro-version of prompting in general. The user needs to identify that the capability exists in the first place, then apply it, direct or update their prompt, and then review the result.

  • The interface clearly shows the capability exists: Adobe features a dominant call to action to "Edit" a photo when the user hovers on it in the main canvas. Generative fill (their branded term for inpainting) is listed at the very top. Notion has a similar CTA to "Ask AI" when the user selects text, and options to edit the selected text are also listed at the top.
  • The app makes it easy for the users to make changes: Notion's default suggested actions are listed in the dropdown, or the user can type directly into the input box to give instructions. Adobe Firefly, like other token-based generators, has to account for both the ability to select the image area and the instructions for how to adjust the output. Their web-based interface uses a simple UI to let the user select the area to adjust using a familiar brush tool, and to suggest constraints by adding or removing tokens.
  • The app makes it easy to review results: Both of these examples use the variants and regenerate patterns to give the users multiple options before they commit. Another example of an app that does this well is Github, which uses inline prompting to let the user add or adjust code within an existing file. Similarly, the AI can offer variants for the user to select from.



Keep the user in control

The AI never takes control away from the user unless directed to do so. All suggestions need to be reviewed before they are accepted, and users have the option to control or regenerate the output.

Iterative improvements

When inpainting is obvious and easy to use, it puts less UX burden on the user nailing the initial prompt. This can decrease the amount of time it takes to get a user to their initial aha moment and let the AI learn and work with them.

Potential risks

Keeping track of changes

Once a user accepts a change, it is seamlessly incorporated into the underlying work. Image generators handle this by letting the user review an upscaled version of the new image before accepting it, and they can always revert and go back to the source image using the footprints pattern. Text editors and code editors haven't adopted this paradigm. The only way to know if the suggested code is truly better is to adopt it and run it - which can lead to bad results if the user needs to revert. How might versioning, commits, branching, etc help users retain control later in the process if they want to isolate and revise what the AI contributed?

Use when:
You want to make selective changes to generated content without regenerating the full thing


Github Copilot can offer suggestions via chat in dialog with the user and gives users the control to insert generated code inline into their file from the chat window
Github Copilot also supports direct inline prompting as a form of inpainting into the code
Grammarly allows users to inpaint over selected text using new tokens, prompts, and parameters
Midjourney's inline selection allows users to adjust a prompt for a specific area
Adobe Firefly shows the user an affordance of the selected area to regenerate by rendering it as transparent
Udio allows users to adjust a specific clip of a song
No items found.