For the toy problem, we will reconstruct the toy image using its gradients and pixel intensities. Essentially, there are 3 objectives we are trying to fulfill:

1. minimize ( v(x+1,y)-v(x,y) - (s(x+1,y)-s(x,y)) )^2
2. minimize ( v(x,y+1)-v(x,y) - (s(x,y+1)-s(x,y)) )^2
3. minimize (v(1,1)-s(1,1))^2

The first objective computes x gradients between the values of the image v(x+1,y)-v(x,y) and the x gradients between the values of the source image s(x+1,y)-s(x,y). Our goal is to have our source and reconstructed images be as similar as possible, hence we minimize the difference between these gradients. Analogously, the second objective function does the same but minimizes the difference in y gradients. The last objective function serves to ensure that the top left corners of the two images are the same color. Finally, we solve this optimization as a least squares problem to recontruct our original image. The two images are very similar, but slightly vary with a MSE of 0.0010002051356621244.

In poisson blending, the goal is to take a cutout of some source image and seamlessly paste it into a target image, blending the edges and ensuring similar gradients. To do this, we want to fulfill the following constraint below. Let S be the cutout region of our source image (in the dimensions of the target image) and let N be the four neighbors below, above, right, and left of the current pixel i.

Algorithmically, we are looping through our pixels and computing their pixel values depending on which case they fall under. Pixels will either:

1. Reside outside of S
2. Reside inside of S, and its neighbor also resides inside of S
3. Reside inside of S, but its neighbor resides outside of S

In the first case for all pixels outside of S, we will simply copy over the values from our target image into our reconstructed image because there is no blending or overlay done here.

For the second and third cases when we are looping over pixels that are in S, we want to check if its neighbors are also in S or not. If the neighbors are also in S, this alludes to the first half of the above constraint, which minimizes the difference between the values of this equation: v(x,y)-v(x_n,y_n) = s(x,y)-s(x_n,y_n). For the second half of the above constraint, we are dealing with pixels that are in S but its neighbors are outside of S, we directly use the target intensity of our neighbor (t_j) in our computation to minimize the values of this equation: v(x,y)-t(x_n,y_n) = s(x,y)-s(x_n,y_n).

Then, we solve this least squares problems where A is a matrix of (h * w, h * w) and b is a vector of length h * w. We do this for each of the rgb channels individually and then combining them together, we have a seamless poisson blended result.

Each blended image also requires an offset variable bg_ul that we use to decide where in the background the object will be placed. The penguin I placed at (700, 400), the fish at (200, 200), and the plane at (250, 350). I noticed that my blends performed better when the background color matched the object itself. I also used more precise masks with a smaller border around the outside, and this resulted in darker outputs because there was less of the background in my masked image. But overall, the blends worked very well. Here are two more examples of poisson blending:

Using a series of rectified images of a chess board, a regularly spaced grid, I can refocus the depth of the image by reshifting. Typically, just averaging together all the images in the grid will cause blurriness in the front and sharpness in the back due to the large positional differences of nearby objects as opposed to faraway objects. However, we're able to specify areas of sharpness by shifting the images using np.roll appropriately. By using different depth_factor values, we're able to choose how close to the camera the sharpness is.

We're able to mimic variations in aperture by averaging over a different number of images. Modifying my refocus function by introducing an aperture parameter, I choose a smaller subset of images close to the center for a smaller aperture that still focuses on the same point, and vice versa. The results are shown below.

I am fascinated by how, just using images of a regularly spaced grid, we are able to change the visualization of the object in ways analogous to settings on a camera, including depth and aperture. These refocusing techniques only required averaging and shifting various images together and demonstrates how powerful lightfield data is.

The architecture of my model is as follows:
First, I randomly sampled N = 1000 pixels for training, giving me 1000 x 2 2D coordinates as well as their respective colors which have coordinates 1000 x 3 for the rgb channels. These are normalized and converted into tensors in the random_sample_pixels method. Then, I implemented sinusoidal positional encoding (PE) to expand the dimensionality of my randomly sampled input coordinates from 2D to a 42 dimension vector. Finally, to build my MLP model, I chained together a series of nonlinear activations between layers, which includes 3 ReLU opreations and then a sigmoid to clamp the values. The input dimensions I specify as 42 (which is the output of PE) and my final output dimensions is 3, because I am outputing 3 dimensions of rgb to get a valid pixel color.
Below is a visualization of the training process of the predicted images across iterations with L = 10 (highest frequency level), MSE loss function, Adam optimizer with learning rate 1e-2, batch size 10,000, trained over 3000 iterations.

Here are the side-by-side results.

With these hyperparameters, I achieved a PSNR of 25.96.

---

One hyperparameter I modified was the max frequency L for positional encoding, and instead of 10 I used L = 5. We can see the striations are not as thin and the overal picture is blurrier. This caused the sharpness in detail to significantly decrease due to the usage of lower frequency components, and hence diminished the performance of my network. With this hyperparameter modification, I achieved a lower PSNR of 24.92.

---

I also attempted to modify the learning rate to see how this would affect my network. By using a learning rate of 1e-3 instead of 1e-2. This yielded slightly better results with a PSNR of 26.12 and its curve is also much more stable. The details are slightly sharper than using a learning rate of 1e-2. For example, the hairs on the top of the fox are more defined and there is more contrast than in 1e-3 than 1e-2.

---

Here I run the model with these hyperparameters against two different images of my own choosing: L = 10, MSE loss function, Adam optimizer with learning rate 1e-3, batch size 10,000, trained over 3000 iterations. The bunny computed with PSNR = 29.90 and the sunflower computed with PSNR = 23.43.