Intro to Computer Vision & Computational Photography: Anderson Lam, Amy Huang
Overview: Our goal for this project was to implement an image quilting algorithm to perform texture synthesis and transfer. For texture synthesis, we generate a larger image given a smaller texture sample, while texture transfer refers to transferring a shape of one object onto a texture by overlapping patterns.
We started by creating the function quilt_random(sample, outsize, patchsize) which
samples a small patch from a given image and tiles the patch into a larger image. Here is the
result on the given image set.
As you can see, the images do not look like a uniform texture but rather a tiling of small patches.
Next, we implement the method quilt_simple(sample, outsize, patchsize, overlap, tol)
for autoregressive sampling. We calculate the cost as the SSD between the overlapping patch of the existing
and generated image. To filter through a given tolerance, we are able to find lower costs using the formula [y, x] = find(cost< minc*(1+tol))
We ended up using a tolerance threshold of 0.3
Although up close you can tell the different patches, there is now more variance in the texture.
Next, we use seam finding to eliminate edge artifacts from the overlapping patches. This is done by using a min-cost algorithm with dynamic programming. We use the min-cost path to create a mask in order to find which pixels to overlap. Here is the result:
Overlapping Min-cost Illustrations
This min cost procedure is really just find our path that best aligns the two patches. We do this by computing the SSD of the patches and then run it thought our DP algorithm, where we loop through our errors until we can get the lowest errors for the indicies.
More Examples
For texture transfer, we use our quilt_cut function to guide the texture cuts by using
a reference image. We use an alpha value to multiply the weight of the cost between SSD of the patch and the
target patch of the reference image. Like the quilt_cut function, we want to find some best patch and then utilize our min cost function to blend the patches.
This is different than min cost as we have a texture patch and goal/target patch. We compute the SSD between these two patches to get the correct looking texture. Here are the resulting images:
This portion consisted of iteratively running through our previous algorithm. We were able to achieve better results after running multiple times (using different patchsizes, overlap, and alpha values for each iteration as described in the research paper). Here are the outputs:
Summary: This project was difficult to understand early on. It took a good while to wrap my head around how overlapping worked and what I could do to try and solve this problem. There was a lot of variables that I ended up trying to store in order to properly compute where there was overlap and then finally determine the min cut path. The next part of texture transfer seemed intimidating at first, but was really similar to the quilt cut part. Throughout this whole process, the algorithms took very long to run, so if I had more time, I would definitely try to optimize the min cut and overlap algorithms. I definitely learned a lot more about how pictures work.