Stephen Burch's Birding & Dragonfly Website

Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links

Method for making quantitative measurements of digital camera image noise levels (raw files)

Introduction
For some time now, I've been interested in the noise that appears on digital camera images. On the principle that knowing as much as possible about one's enemy is a good idea, over the years I've done a fair bit of investigation of the subject.

Simply looking at digital camera pictures and trying to make valid assessments and comparisons of noise levels may have its place, but the results can be very subjective and depend hugely on the subject matter of the picture, as well as the camera used and all the settings involved in getting the image out of the camera onto the screen. The size the image is viewed at can also makes a big difference. An image at 20% magnification can appear virtually noise free, but show it at 1:1 and up comes the noise.

Hence, I have taken a different approach and developed a reasonably simple method for quantitatively measuring pixel to pixel noise levels on digital camera pictures, effectively analysed at 1:1 magnification. I've then applied this method to images taken with different Canon digital cameras at different ISO settings.

In general, the main factors that affect the noise level on an digital camera image are as follows:

1. The physical sensor element (or pixel) size/area. The larger the size/area the lower the noise should be.
2. The ISO setting. The higher the ISO value, the higher the noise.
3. The sensitivity of the sensor elements. A sensor with a high sensitivity will collect more of the incident light and give a lower noise level than one with lower sensitivity. The latest cameras generally have sensors with higher sensitivity than those developed several years ago.
4. The raw converter and its noise/sharpening settings. For these measurements, I always take care to remove all possible noise reduction and sharpening parameters (those that can be changed by the user, that is) applied to the data by the raw converter, so the resulting tiff file should reflect as closely as possible the sensor data itself. It is however clear that different raw converters are not all equal, and some additional "processing" is applied to the images, hidden from the user. Hence different raw converters can lead to images with noticeably different noise levels, even from the same raw file.

Method

Introduction
In making these measurements, it is firstly important to note that I used images converted to tiff format from their original raw format from all cameras. I did not use any jpegs coming directly from the cameras. Also I disabled and checked that all the settings, both on the camera and in the subsequent raw conversion, which might affect the basic output from the sensor were off, including:

• high ISO speed noise reduction
• highlight tone priority
• auto lighting optimiser, etc

Having done this, all the raw images should be close to what is coming off the sensor, with minimal processing. In other words a like for like comparison between the cameras, or 'level playing field'.

Note however that with all these user selectable parameters turned off, either on the camera or in the raw converter, the measurements made show significant differences between the noise levels on images obtained with different raw converters. Hence there are some hidden processes going on in different raw converters, that the user has no control over.

Grey-scale (or "step wedge") test chart
To make these measurements I first prepared a grey scale test chart - see below. Printing of this chart requires some care. I have come to the conclusion that inkjet printers are not adequate for this task, as they have a fine structure to the printed grey levels that should be as uniform as possible. So I recommend having it printed A4 size commercially by a company such as Photo Box. Ideally the photo paper should be matte, but even this has a shiny surface. This is OK, but it is important to avoid any noticeable reflections by taking care of the lighting and background conditions. Glossy prints can also be used but then even more care is needed to avoid unwanted reflections of anything behind the camera.

The purpose of this chart was to allow measurements of noise to be made for different image brightnesses (grey levels) from black through to white, to understand how the noise varied. The test chart was designed to have steps with different brightnesses, but each step was an even grey so that any variations in image brightness (or grey level) within the step would be due to noise and not the target.

Image acquisition
Having got a print of the above test chart on a peice of A4 photo paper (preferably matt), I then afixed it to a piece of hardboard for ease of mounting. I then set it up outside and arranged it so it was approximately square on to camera on which I had mounted a typical focal length lens (not telephoto). It was important to avoid direct sunlight falling on the chart, and also to ensure there were no bright areas behind the camera that might cause reflections in the print (even matte photo paper is surprisingly shiny, and gloss paper is even worse!).

To reduce any variations in image brightness due to printing imperfections in the test chart, I always made sure that all the images were slightly out of focus.

For each camera, I then typically took a series of images at different ISO settings, with the exposure time and/or aperture changing between them. I generally used the Av setting and about +1 EV exposure compensation with evaluative metering to get the white surrounding area at a high grey level in the image, but not saturated or "blown-out".

Raw conversion of the images
To process the raw files from each camera, I used either Adobe Camera Raw (ACR), as contained in various versions of Adobe PhotoShop or, for the earlier measurements, PhotoShop Elements.

For my recent (late 2022, early 2023) results, I reprocessed all the image files using ACR in Photoshop 2023, using the Adobe camera standard profile in all cases (for uniformity between cameras).  I made sure all noise reduction and sharpening parameters were off.

In addition, I have used Canon's own raw converters, initially Digital Photo Professional version 3 (DPP3) and more recently version 4 (DPP4). For all the more recent measurements, I made sure the output files were 16-bit tiff files, not 8-bit to avoid anY loss of dynamic range. However when showing the results, I scaled the results back down to 8-bits (i.e. 0-255).

Also, at the raw conversion stage, for the most recent measurements, I set the white balance using the dropper tool - on the white surround to the grey scale. This was to avoid any significant colour cast to the image. It probably did not make much difference on the final measured noise values but seemed a sensible thing to do. Note earlier results probably didn't include this stage.

As stated above, in doing the conversion, the output files were always tiff files (which are lossless, unlike jpegs), I made sure any processing parameters that might increase or decrease noise were off (e.g. sharpening, noise reduction).

Measurement of the images
For many years, I used the ISee! software from BAM in Germany which is intended for the quantitative professional analysis of X-ray images. There is a download for the free version. This software allows the user to interactively define a small analysis area, that I set at a constant size of 30 x 30 pixels or more depending on the number of pixels in the image (so more recently up to at least 75 x 75 pixels with the larger number of pixels in the more recent cameras). The software gives a readout of the standard deviation of the image grey levels within this area, which is a good measure of the noise level. I then moved this area across each picture, and measured the noise levels for each step, taking an average of three or five readings in slightly different positions on each step. The averaging was done to improve the accuracy of the measurements.

However, as I discovered in early 2023, other free packages are also available to do this analysis. ImageJ for example works well and importantly allows the results of the analysis to be saved for pasting into a spreadsheet. This saves a huge amount of time compared with writing down and then typing in all the numbers! With this software, it does not appear possible to set a specified area size but there is an interactive read-out of the area size as it is dragged out across the image. So it is quite easy to check its size.

 The example below shows the analysis of an image using ImageJ in progress. The procedure for the analysis is as follows:

1. Open required slightly out of focus image of the test chart above (having first printed it on a piece of photo-quality paper)
2. Click the rectangular area button - top left under "file".
3. Draw out an analysis area using the mouse (left click top left, then drag out)
4. Adjust the size of the area until it is approximately the right size
5. Goto Analyse ---> Measure (cntl m short-cut) to log the values for the area.
6. Use the mouse to mark a new area for analysis
7. Repeat stages 5-6 for between 3 and 5 different points on the current step.
8. Repeat stages 5-7 for all steps.
9. In the log of measurements generated, the key parameters are the Mean and Standard deviation (StdDev).

Using the above procedure on different images, it is possible to build up a series of measurements that can be plotted and compared to study the variations in noise with various parameters, the main ones being image brightness or grey level, ISO setting, and of course camera type.

Handling of colour information in images
How colours are dealt with in the above processing and analysis of the images can have a significant impact on the measured noise levels. Initially I thought this wouldn't be a major effect, but in that I was wrong!

Although the grey scale image shown above had no colour in it, the RGB images generated by any camera do. Firstly, the colour balance can be off, so that the whole image has a colour tint to it. This can be avoided by setting the white balance to white on the background around the grey scale. But secondly, the image noise affects each of the RGB colour channels differently, so how the RGB channel information is handled can have a significant effect on the measured noise level.

As an example, if a linear average of the RGB values is taken, and all 3 channels had the same individual noise levels, then the resultant noise level would be reduced by approximately sqrt(3) = 1.7 (almost 5dB), assuming the noise was uncorrelated between the channels.

Generally, the camera noise will generate a luminance component and a chrominance (colour) component. All my measurements are intended to be of luminance, not chrominance. But there are different ways of defining luminance from separate RGB channels.

For the Isee! software, a simple linear average of the RGB values is used to create the image analysed. With ImageJ, its documentation suggests this is the default option, with a more complex weighting available as a user specified option. However, while this appears to be true for 8-bit images, it most certainly isn't the case for 16-bit files! For these, I currently (late Jan 2023) have no idea what ImageJ is doing - it gives much higher values than a linear average would do!

To ensure consistency between Isee and ImageJ, I have found it necessary to first convert all the camera images to "black and white", using the Photoshop 2023 option of Image ---> Adjustments ---> Black & White. This brings up a formidable menu like this, in which I leave the defaults well alone:

Photoshop 2023 menu for converting an RGB image to Black and White

Exactly what this option does to the RGB values is currently unclear to me but I assume it combines the RGB values with a suitable weighting to get something which I take to be a reasonable measure of luminance. Whatever it does, the resulting RGB channels then all have equal values, and the means and standard deviations for 16-bit images derived using Isee! and ImageJ are then equal, as desired!

Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links