Modifying 3D Scenes with Directions
We suggest a way for modifying NeRF scenes with text-instructions. Given a NeRF of a scene and the
assortment
of photographs used to reconstruct it, our technique makes use of an image-conditioned diffusion mannequin (InstructPix2Pix) to
iteratively edit the enter photographs whereas optimizing the underlying scene, leading to an optimized 3D
scene that
respects the edit instruction. We reveal that our proposed technique is ready to edit large-scale,
real-world
scenes, and is ready to accomplish extra lifelike, focused edits than prior work.
Our technique regularly updates a reconstructed NeRF scene by iteratively updating the dataset photographs whereas
coaching the NeRF:
- A picture is rendered from the scene at a coaching viewpoint.
- It’s edited by InstructPix2Pix given a worldwide textual content instruction.
- The coaching dataset picture is changed with the edited picture.
- The NeRF continues coaching as ordinary.
Instance updates to the coaching dataset photographs over time. Discover that the edits are regularly turning into extra
constant.
For those who use this work or discover it useful, please take into account citing: (bibtex)
@article{instructnerf2023,
writer = {Haque, Ayaan and Tancik, Matthew and Efros, Alexei and Holynski, Aleksander and Kanazawa, Angjoo},
title = {Instruct-NeRF2NeRF: Modifying 3D Scenes with Directions},
journal = {arXiv preprint arXiv:XXXXXX},
yr = {2023},
}
We thank our colleagues for his or her insightful suggestions useful discussions, particularly Ethan Weber, Frederik
Warburg, Ben Poole, Richard Szeliski, Jon Barron, Alexander Kristoffersen, Rohan Mathur, Alejandro Escontrela,
and the Nerfstudio group.