Tuesday, August 9, 2016

Detecting image manipulation in the noise

All resources referenced in this post can be found here.

If you assume that image noise is uniform over the entire image, and unique to a specific image sensor + setting (exposure time, sensitivity, etc), it should be possible to detect some forms of image manipulation by looking at the noise. If for example part of an image originates from a different capture it should have different noise characteristics than the rest of the image, and that could possibly be detected.

I wrote some code to see if this was doable, noise_lab.py. The application compares the noise at each pixel with normal noise to create an image depicting the deviation from normal noise. You can zoom along the "z axis" in the image to be able to see very small deviations. You can, for example, set a deviation of 0.2 to be black and a deviation of 0.21 to be white when showing the image.

The noise is captured by doing a FFT on a pixel's surroundings and dropping the DC component and the phase information. The normal noise is captured in the same way but from a user specified area of the image where all changes are caused by noise, for example the sky or a white wall.

When trying out my new application I quickly realized that my assumption that noise is uniform all over the image is not correct. I think there are a number of reasons for this but lets first focus on the relationship between pixel intensity and noise amplitude.



As can be seen there appears to be a relationship between intensity and noise amplitude. To be able to compare areas of the image with different intensity I need to normalize the noise amplitudes. I decided to model the relationship as a linear relationship where the slope is calculated from the user specified noise only areas. So now the user needs to input at least two noise only areas instead of only one which was the case before. The result looks as follows.



The linear model does not work well for low intensity pixels but it's good enough to see if this approach to detecting image manipulation has any future.

Below is an image composed of two different captures from the rear camera on my Nexus 6. The settings where 20ms ISO160 and 13ms ISO80. If everything works I should be able to see a difference in the noise. The image looks weird because I've tried to disable as much as possible of the camera's post processing to remove error sources. The image is what was read out from the sensor + bilinear interpolation + gamma encoding + jpeg encoding. What you get when running the following command on a dng file and jpeg encoding the result.

dcraw -r 1.0 1.0 1.0 1.0 -q 0 -M -o 0 -g 2.4 12.92 -j -W -H 1 image.dng



And this is what it looks like in the application.




It is quite easy to see the foreign part of the image but the low intensity parts of the original image appear to have the same noise characteristics as the foreign part, which is not ideal. This is a result of the noise normalization method's poor performance on low intensity areas. If you go back to the "Intensity - Normalized noise amplitude" diagram you will see that low intensity areas of image 1 have the same noise amplitude as normal intensity areas of image 2.

So that worked pretty well, but what happens if I feed the application with images that have gone through the camera's post processing? This image is also composed of two different captures from the rear camera of my Nexus 6. The settings where 33ms ISO241 and 17ms ISO134.






The "Intensity - Normalized noise amplitude" diagram is the most interesting one here. The noise variance has gone up (thicker y bands for each image) and the two images have moved closer to each other along the y axis. Both these things are bad if you want to detect image manipulation in the noise, and indeed the application is unable to detect the foreign part of the image. The completely black parts are due to saturated pixels (<= 0.0 or >= 1.0) being represented as zero difference.

If I make the noise difference between the two source images larger, 33ms ISO241 and 83ms ISO895, the application can detect the foreign part, but with such a big difference I can detect the foreign part just by looking at the image so the application does not add anything. This approach to detecting image manipulation is really only applicable to raw images with a few noise only areas. Unfortunately manipulated raw images are not that common, as far as I know anyhow, so this was a dead end.

No comments:

Post a Comment