有趣的纹理合成

Posted 2021-12-13 Matrix_11

有趣的纹理合成

texture synthesis，中文翻译叫纹理合成，是一个很有趣的计算机渲染技术，简单来说，就是通过给定的有重复纹理的小图，通过这种纹理合成技术，可以渲染出一张有类似纹理的大图，而且完全看不出什么违和感。如下图所示：

可以看到，小图和大图之间的相似性很高，但是也可以注意到，并不是所有的小图都可以达到这样的效果，关键是图像里面有相似的重复纹理，当你把小图随机切成很多小的 patch 的时候，可以发现这些 patch 之间有很高的相似性，这是这类纹理合成方法的基础：利用 patch 之间的相似性，拼接成一个更大的图像。理论上，这种图像可以一直拼接下去，纹理合成技术发展到现在，已经经历了很多的变迁，深度学习时代，也有用深度来实现纹理合成的，今天我们介绍一篇 20 年前的文章：Image Quilting for Texture Synthesis and Transfer，虽然这篇文章至今已经过去 20 年了，但是这篇文章介绍的纹理合成技术，在我看来，真的是简单而又高效。而且文章的作者：Alexei A. Efros，William T. Freeman，这两个人绝对都是当今 CV 界的大牛。

这篇文章的原理相对来说，还是比较好理解的，用一张图就能解释清楚，既然是纹理合成，从小图最后拼成一张大图，首先大家最容易想到的就是从小图里面随机取图像块，然后往大图里面贴，这个有点像装修贴瓷砖一样，每次都取同样大小的图像块，然后往大图里面放，如果直接就这么放，那块与块之间的拼缝会特别明显，就像上图中的第一种情况，可以看到虽然拼成了一张大图，但是图像块之间的拼缝非常明显，所以接下来，大家想到的改进方式，就是让图像块之间有一定的重合，也就是 overlap，对重合区域做 alpha blending，如上图中的第二种情况，可以看到，第二种情况，相比第一种情况，图像块之间的拼缝没有那么明显了，但还是可以看得到的；所以更进一步，就是将重合区域进行更加精细的选择，如上面第三种情况所示，在重合区域，选择一条拼缝，让整体的相似性最高。

github 上有人实现了这个算法，不过运行一张图片，花的时间还是挺久的。从实现效果来看，还是 mincut 的效果看起来最自然。

import numpy as np
import math
from skimage import io, util
import heapq
from skimage.transform import rescale, resize
from PIL import Image
import argparse
import os

import cv2

#parser = argparse.ArgumentParser()
#parser.add_argument("-i", "--image_path", required=True, type=str, help="path of image you want to quilt")
#parser.add_argument("-b", "--block_size", type=int, default=50, help="block size in pixels")
#parser.add_argument("-n", "--num_block", type=int, default=6, help="number of blocks you want")
#parser.add_argument("-m", "--mode", type=str, default='Cut', help="which mode --random placement of block(Random)/Neighbouring blocks constrained by overlap(Best)/Minimum error boundary cut(Cut)")
#args = parser.parse_args()

## 随机选择一个 patch
def randomPatch(texture, block_size):
    h, w, _ = texture.shape
    i = np.random.randint(h - block_size)
    j = np.random.randint(w - block_size)

    return texture[i:i+block_size, j:j+block_size]

## 计算两个 patch 重合区域的相似性，直接计算重合区域像素值的差的平方和
def L2OverlapDiff(patch, block_size, overlap, res, y, x):
    error = 0
    if x > 0:
        left = patch[:, :overlap] - res[y:y+block_size, x:x+overlap]
        error += np.sum(left**2)

    if y > 0:
        up   = patch[:overlap, :] - res[y:y+overlap, x:x+block_size]
        error += np.sum(up**2)

    if x > 0 and y > 0:
        corner = patch[:overlap, :overlap] - res[y:y+overlap, x:x+overlap]
        error -= np.sum(corner**2)

    return error
 
## 遍历所有 patch，从所有 patch 里面选择一个与当前 patch 误差最小的 patch
def randomBestPatch(texture, block_size, overlap, res, y, x):
    h, w, _ = texture.shape
    errors = np.zeros((h - block_size, w - block_size))

    for i in range(h - block_size):
        for j in range(w - block_size):
            patch = texture[i:i+block_size, j:j+block_size]
            e = L2OverlapDiff(patch, block_size, overlap, res, y, x)
            errors[i, j] = e

    i, j = np.unravel_index(np.argmin(errors), errors.shape)
    return texture[i:i+block_size, j:j+block_size]


### 寻找最小路径
def minCutPath(errors):
    # dijkstra's algorithm vertical
    pq = [(error, [i]) for i, error in enumerate(errors[0])]
    heapq.heapify(pq)

    h, w = errors.shape
    seen = set()

    while pq:
        error, path = heapq.heappop(pq)
        curDepth = len(path)
        curIndex = path[-1]

        if curDepth == h:
            return path

        for delta in -1, 0, 1:
            nextIndex = curIndex + delta

            if 0 <= nextIndex < w:
                if (curDepth, nextIndex) not in seen:
                    cumError = error + errors[curDepth, nextIndex]
                    heapq.heappush(pq, (cumError, path + [nextIndex]))
                    seen.add((curDepth, nextIndex))

 #### 基于 mincut 的拼缝查找              
def minCutPatch(patch, block_size, overlap, res, y, x):
    patch = patch.copy()
    dy, dx, _ = patch.shape
    minCut = np.zeros_like(patch, dtype=bool)

    if x > 0:
        left = patch[:, :overlap] - res[y:y+dy, x:x+overlap]
        leftL2 = np.sum(left**2, axis=2)
        for i, j in enumerate(minCutPath(leftL2)):
            minCut[i, :j] = True

    if y > 0:
        up = patch[:overlap, :] - res[y:y+overlap, x:x+dx]
        upL2 = np.sum(up**2, axis=2)
        for j, i in enumerate(minCutPath(upL2.T)):
            minCut[:i, j] = True

    np.copyto(patch, res[y:y+dy, x:x+dx], where=minCut)

    return patch


def quilt(image_path, block_size, num_block, mode, sequence=False):
    texture = Image.open(image_path)
    texture = util.img_as_float(texture)

    overlap = block_size // 6
    num_blockHigh, num_blockWide = num_block

    h = (num_blockHigh * block_size) - (num_blockHigh - 1) * overlap
    w = (num_blockWide * block_size) - (num_blockWide - 1) * overlap

    res = np.zeros((h, w, texture.shape[2]))

    for i in range(num_blockHigh):
        for j in range(num_blockWide):
            y = i * (block_size - overlap)
            x = j * (block_size - overlap)

            if i == 0 and j == 0 or mode == "Random":
                patch = randomPatch(texture, block_size)
            elif mode == "Best":
                patch = randomBestPatch(texture, block_size, overlap, res, y, x)
            elif mode == "Cut":
                patch = randomBestPatch(texture, block_size, overlap, res, y, x)
                patch = minCutPatch(patch, block_size, overlap, res, y, x)
            
            res[y:y+block_size, x:x+block_size] = patch

    image = Image.fromarray((res * 255).astype(np.uint8))
    return image

if __name__ == "__main__":
#    image_path = args.image_path
#    block_size = args.block_size
#    num_block = args.num_block
#    mode = args.mode
    
    image_path = './data/input2_s.png'
    block_size = 50
    num_block = 10
    mode = 'Cut'
    
    img_out = quilt(image_path, block_size, (num_block, num_block), mode)
    img_out.save('img_out_2.png')

• 参考文献：

代码： https://github.com/Devashi-Choudhary/Texture-Synthesis

论文： Image Quilting for Texture Synthesis and Transfer