Generative Modelling of BRDF Textures from Flash Images

Philipp Henzler1, Valentin Deschaintre2,3, Niloy J. Mitra1,2, Tobias Ritschel1,
1University College London, 2Adobe Research 3Imperial College London
SIGGRAPH Asia 2021
PDF arXiv YouTube 5 min YouTube 15 min

Interactive Demo (click here)

Teaser.

We present a novel deep architecture that contributes Warp-conditioned Ray Embedding (WCR) to reconstruct and render new views (right) of object categories from one or few input images (middle). Our model is learned automatically from videos of the objects (left) and works on difficult real data where competitor architectures fail to produce good results

Method

Architecture.

Starting from an exemplar (top-left) our trained encoder encodes the image to a compact latent space variable z. Additionally, a random infinite field is cropped with the same spatial dimensions as the flash input image. The noise crop is then reshaped based on a convolutional U-Net architecture. Each convolution in the network is followed by an Adaptive Instance Normalization (AdaIN) layer reshaping the statistics (mean and standard deviation) of features. A learned affine transformation T per layer maps z to the desired means and sigmas. The output of the network are the diffuse, specular, roughness, normal parameters of an svBRDF that, when rendered using a camera colocated flash light, look the same as the input. Our unsupervised setting allows us to fine-tune our trained network on materials to acquire.

Relighting

Input
Relit
Input
Relit
Input
Relit

Diversity

As we can re-seed the input noise our method is capable of producing diverse results.

Input
Re-seed

Infinite spatial domain

Our method allows to genreate infinite spatial fields of BRDF parameters without any border artefacts or repetitive patterns.

Infinite spatial domain

Our method allows to genreate infinite spatial fields of BRDF parameters without any border artefacts or repetitive patterns.

Exemplar 1
Interpolation
Exemplar 2

BRDF Decomposition

Our method allows to genreate infinite spatial fields of BRDF parameters without any border artefacts or repetitive patterns.

Resynthesis
Diffuse
Specular
Roughness
Normal

BibTeX

@article{henzler2021neuralmaterials,
    title    = {Generative modelling of BRDF textures from flash images},
    author     = {Henzler, Philipp and Deschaintre, Valentin and Mitra, Niloy J and Ritschel, Tobias},
    journal   = {SIGGRAPH Asia},
    year      = {2021},
  }