Stephen Burch's Birding & Dragonfly Website

Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links

Quantitative measurements of digital camera image noise levels (raw files)

 (now with EOS R5 update October 2020!)

Introduction
For some time now, I've been interested in the noise that appears on digital camera images, and the best methods for getting rid of it (see my comments about NeatImage for example on another page on this website). On the principle that knowing as much as possible about one's enemy is a good idea, over the years I've done a fair bit of investigation of the subject.

Simply looking at digital camera pictures and trying to make valid assessments and comparisons of noise levels may have its place, but the results can be very subjective and depend hugely on the subject matter of the picture, as well as the camera used and all the settings involved in getting the image out of the camera onto the screen. The size the image is viewed at can also makes a big difference. An image at 20% magnification can appear virtually noise free, but show it at 1:1 and up comes the noise.

Hence, I have taken a different approach and developed a reasonably simple method for quantitatively measuring pixel to pixel noise levels on digital camera pictures, effectively analysed at 1:1 magnification. I've then applied this method to images taken with different Canon digital cameras at different ISO settings, with some interesting results (to me anyway!).

The cameras assessed to date are the EOS 350D, EOS 40D, EOS 50D, EOS 7D, EOS 7D Mk II and EOS R5.

In general, the main factors that affect the noise level on an digital camera image are as follows:

1. The physical sensor element (or pixel) size/area. The larger the size/area the lower the noise.
2.  The ISO setting. The higher the ISO value, the higher the noise
3.  The sensitivity of the sensor elements. A sensor with a high sensitivity will collect more of the incident light and give a lower noise level than one with lower sensitivity. The latest cameras generally have sensors with higher sensitivity than those developed several years ago.
4.  The raw converter and its noise/sharpening settings. For these measurements, I always take care to remove all noise reduction and sharpening applied to the data by the raw converter, so the resulting tiff (or similar) file should reflect as closely as possible the sensor data itself. It is however clear that different raw converters are not all equal, and some additional "processing" is applied to the images, hidden from the user. Hence different raw converters can lead to images with different noise levels, even from the same raw file.

Canon EOS R5
Having pre-ordered Canon's latest mirrorless camera only the day after its launch in June 2020, I still had to wait until mid September 2020 for its eagerly anticipated arrival! For an initial review of this exciting new camera, see this page on my website.

One of my first tasks was then to see how its image noise levels compared with my trusty 7D Mk II that I had been using for almost six years since I acquired it in December 2014 (see below). I was hopeful that almost 6 years of sensor development would have resulted in a tangible improvement in the image noise level compared with the 7D Mk II. From results given on the excellent DXOMark website for the Canon EOS 5DS, which has a similar sensor element size to the R5 (and the 7D Mk II), I was anticipating about a 1 stop or more improvement with the R5.

ACR results
The plot below shows the image noise levels for the EOS R5 compared with the older 7D Mk II obtained using Abobe Camera Raw (ACR) in Photoshop 2021.  All noise reduction and sharpening parameters were switched off both on the cameras and in the raw converter used - Adobe Camera Raw (ACR). As far as possible, this should give a 'level playing field' and allows assessment of the direct outputs from the camera sensors, without any post processing. 

From the plot above, it is good to see that the noise levels from the EOS R5 show the hoped for improvement of about 1 stop compared with the older EOS 7D II, so that for example the noise levels for the R5 with an ISO of 3200 are very similar to those from the 7D II at ISO 1600. Indeed, the R5 improvement appears to be slightly more than one stop, so that if anything the corresponding 7D results for one stop lower ISO are slightly above those for the R5 at one stop higher. 

The R5 has a very slightly larger sensor element size than the 7D Mk II but such a small margin it is unlikely to affect these results significantly, so it seems that the 6 years that have elapsed between the launches of these two cameras have produced about a factor two improvement in sensor sensitivity.

This is a great finding, and suggests this camera upgrade has produced a noticeable improvement in the maximum ISO that can be used without noticeable reduction in image quality. With the 7D II, after using NeatImage for noise reduction, I was generally OK with using ISO settings up to 3200, but I was reluctant to go beyond this (e.g. ISO 6400) except in special circumstances (e.g. trips to rainforests where light is always at a premium).

So these findings suggest I should be able to use the R5 with ISO settings up to about 6400 before the noise (after NeatImage) becomes noticeable. Experience with this new camera in the field over the coming weeks and months will tell me if this is correct!

It will also be interesting to see how these results for the R5 compare with those on the DXOMARK website when they eventually appear. This website has very useful plots of noise level for large numbers of cameras, expressed in terms of 18% signal-to-noise ratio. This site also has a very handy compare tool that allows the results for up to 3 different cameras (from any manufacturer) to be shown on the same plot.

Sony was previously the market leader on sensor technology, but perhaps Canon has at last made up significant ground in this key area, as it appears to have done with the remarkable animal eye detection auto-focus in the R5.

Note that, as seen before, the above curves show that the maximum noise standard deviation at each ISO setting increases by the square root of the ISO setting value for each camera (e.g. the noise at ISO 800 is square root of 2 = 1.4 larger than that at ISO 400, and the noise at ISO 1600 is 1.4 times larger than at ISO 800 and so on). This is exactly what would be expected if the main source of this noise is photon counting statistics - and, given the laws of physics, there is not much that can be done about that, apart from increasing the detection efficiency of the sensor.

ACR vs DPP4
The straightforward conclusion drawn from the above plot however becomes a little less clear when the results from different raw converters are compared. Previously I generally had a preference for Canon's own raw converter - Digital Photo Professional v3 (DPP3) since, for the 7D mkII, my measurements had shown it gave slightly lower noise levels than ACR. DPP3 doesn't support the new CR3 format files from the EOS R5, for which the newer version 4 of DPP is needed (DPP4). So I have compared the noise levels from DPP4 with those from ACR for the EOS R5. The plot below gives the results.

The plot above shows that for the same ISO setting, ACR is apparently giving appreciably lower noise levels than DPP4 for the EOS R5. These differences amount to nearly half a stop I would estimate.

Hence for this reason, it seems I will be better using ACR instead of DPP4 for the images from the EOS R5, whereas the reverse was true for the 7D Mk II! All very confusing!

Colour issue and Raw Converter selection for the R5
There is currently an issue with the colours coming out of ACR for the Canon R5 which are reported to be not nearly as good as those from Canon's own DPP4. This is something I have just noticed (November 2020) and is covered at some length in this excellent YouTube video.

Hence to see if I could improve the image noise levels coming out of DPP4, I tried increasing the noise reduction parameters available. There are two - Reduce Luminance noise and Reduce Chrominance noise. These have values between zero (none) and 20. I tried a small value of 1 which is shown on the plot above for ISO 1600.

This gives a very similar maximum noise level (4.0) as ACR and better control of the noise in the lower grey level region (<50) (i.e. the shadows). However it is also necessary to consider how much the noise reduction is affecting the image sharpness - in general the higher the noise reduction the lower the image sharpness.  Given this extra consideration I have now concluded it is best to use zero noise reduction in DPP4 and then to use NeatImage to do all the subsequent noise reduction.

Camera specific differences between DPP3 and DPP4
I should also add that applying DPP3 and DPP4 to the raw files from the 7D Mk II and the R5 produced some confusing and difficult to understand results, so I wouldn't show them here! It seems that Canon seem happy to fiddle with noise reduction for specific cameras within their different versions of DPP, so that even zero values for the user-specifiable NR parameters are not what I would expect them to mean, i.e. no noise reduction!  

Canon EOS 7D Mk II
With my acquisition of an EOS 7D Mk II in December 2014, I was able to see how its noise levels compared with my original 7D and the other older cameras.

First of all, see below for the noise level plot which shows results from the 7D Mk II compared with the 7D. Note that as previously, all noise reduction parameters were switched off both on the cameras and in Canon's Digital Photo Professional (DPP) conversion software for the raw images. As far as possible, this should give a 'level playing field' and allows assessment of the direct outputs from the camera sensors, without any post processing. 

From the plot above, it is somewhat disappointing but not entirely unexpected to see that 7D Mk II is giving very similar noise levels to the 7D, for the same ISO setting and grey level values.

Only for intermediate grey levels (around 150) at ISO 400 and ISO 800 is there a slight improvement with the Mk II. For lower grey levels (between about 50 and 100) at ISO 1600, the Mk II actually appears slightly noisier than the 7D.

The 7D Mk II has however very slightly smaller pixels, so these results do at least show that generally the Mk II is performing slightly better than the 7D, taking this into account.

The above curves show that the maximum noise standard deviation at each ISO setting increases by the square root of the ISO setting value (e.g. the noise at ISO 800 is square root of 2 = 1.4 larger than that at ISO 400, and the noise at ISO 1600 is 1.4 times larger than at ISO 800 and so on). This is exactly what would be expected if the main source of this noise is photon counting statistics - and, given the laws of physics, there is not much that can be done about that, apart from increasing the detection efficiency of the sensor. That may however already be close to one.

For completeness, I also show below all the DPP results I now have available for the various Canon DSLR's I have owned since 2006!

The above plot certainly demonstrates a progressive improvement in sensor performance from the EOS 350D to the 7D, when the areas of the pixels are taken into account. However it would seem that this improvement has more or less stalled for the 7D Mk II.

See below for a description of the method used to generate these results, and some older plots obtained with Adobe Camera Raw (ACR) as well as DPP.

Method
In making these measurements, it is firstly important to note that I used images in raw format from all cameras, and I disabled and checked that all the settings which might affect the basic output from the sensor were off :

• high ISO speed noise reduction
• highlight tone priority
• auto lighting optimiser

On the other cameras, I again made sure any similar options were off, although there are fewer available on the older models. Having done this, all the raw images should be as near as possible to what is coming off the sensor, with minimal processing. In other words a like for like comparison between the cameras, or 'level playing field'.

For these measurements I prepared and printed out a grey scale test chart - see below:

The purpose of this chart was to allow measurements of noise to be made for different image brightnesses from black through to white, to understand how the noise varied. The test chart was designed to have steps with different brightnesses, but each step was an even grey so that any variations in image brightness (or grey level) within the step would be due to noise and not the target.

To reduce any variations in image brightness due to printing imperfections in the test chart, I made sure all the images were slightly out of focus.

To process the raw files from each camera, I used either Adobe Camera Raw (ACR), as contained in various versions of PhotoShop Elements or the full Photoshop. In addition, I have used from time to time Canon's own raw converters, initially Digital Photo Professional version 3 (DPP3) and more recently version 4 (DPP4). In doing the conversion to tiff files (which are lossless, unlike jpegs), I made sure any processing parameters that might increase or decrease noise were off (e.g. sharpening, noise reduction).

I then used the ISee! software from BAM in Germany which is intended for the quantitative professional analysis of X-ray images. This is a download for the free version. Other free packages may also be available to do this analysis (e.g. Image J).

This software allows the user to interactively define a small analysis area, that I set at a constant size of 30 x 30 pixels in all images (more recently 50 x 50 pixels). The software gives a readout of the standard deviation of the image grey levels within this area, which is a good measure of the noise level. I then moved this area across each picture, and measured the noise levels for each step, taking an average of five readings in slightly different positions on each step. The averaging was done to improve the accuracy of the measurements.

The example below shows the analysis of an image using ISee! in progress. The procedure for the analysis is as follows:

1. Open required slightly out of focus image of the test chart above (having first printed it on a piece of photo-quality paper)
2. Click the St button - highlighted in green below
3. Draw out an analysis area using the mouse (left click top left, then drag out)
4. Adjust the size of the area in the Statistics in window dialog box to be 30 x 30 (or 50 x 50) - highlighted in blue below
5. Pick up and move the area to the desired step on the image.
6. Note down the mean and stdev values (circled in red below).
7. Repeat stages 5-6 for about 5 different points on the current step.
8. Repeat stages 5-7 for all steps.

Using the above procedure on different images, it is possible to build up a series of measurements that I could compare to study the variations in noise with various parameters, the main ones being image brightness or grey level, ISO setting, and of course camera type.

I should add that these measurements were made on a black & white versions of the tiff images (which is how they appear in the X-ray image analysis software), so that they presumably are measures of the luminance noise, and not chroma noise.

Results on older cameras (EOS 50D, 40D and 350D)

Raw converters
For this investigation, I initially used the raw converter available within PhotoShop Elements, which is called Adobe Camera Raw (ACR). However, part way through I also started looking at the results obtained with Canon's free Digital Photo Professional (DPP) that comes with their cameras. To my surprise there were significant differences between these two widely used raw converters. For more information on this, read on!

Measured noise levels
EOS 40D with ACR
The graph below shows how the noise levels for the EOS40D using ACR. The most obvious trend is for the noise at a given ISO setting to decrease with increasing image brightness (i.e. the lines slope down to the right). These lower noise levels in the brighter parts of the image may well be familiar to close observers of DSLR images. This agrees with my general qualitative experience that the noise is usually much more evident in the darker areas, than the lighter areas.

Also the graph shows that, as expected, increasing the ISO setting increases the overall noise levels, as expected. However there is a more subtle effect in that the variation in noise level with image brightness also changes with ISO setting and is much higher for ISO 1600 than ISO 400 (i.e. the noise in the darker areas of an ISO1600 shot is much worse than in the brighter areas, whereas at ISO400 the noise varies much less with image brightness).

The results below certainly lend support to the concept of keeping the exposure as high as possible to avoid under-exposed darker areas. This is apparently called ETTR (Expose To The Right), i.e. make sure your histogram of image values is over to the right (bright), not left (dark).

With raw files, I've also found that if the image is slightly clipped or saturated (spike at the right edge), then reducing the exposure in the raw converter gets over this problem, and the spike goes away. Very useful if you are trying to keep exposures up, and sometimes go too far!

With the 50D and 7D, there is about a full stop of leeway on the exposure adjustment. That is you can reduce the exposure in the raw converter by about a stop to avoid saturation. Beyond that, saturated areas just stay saturated. On the older 350D, there is somewhat less leeway.

EOS 350D, 40D and 50D noise levels compared (ACR)
The slightly confusing graph below shows results for all three cameras, at three different ISO settings. This shows the following:

• For a given ISO setting, the EOS 40D had the lowest per pixel noise level, by about half a stop from the EOS 350D
• The EOS 50D had by far the highest noise level, by about 1 stop compared with the 350D and 1.5 stops compared with the 40D.

The result for the 40D appears impressive, as it has pixels about 20% smaller in area than the 350D (10 Mpixels compared with 8Mpixels for the 350D). However, it is not quite as good as that, since the 40D is calibrated so that for a given ISO setting it is about 1/4 stop less sensitive than the 350D, so the overall improvement in noise is probably only about 1/4 stop.

The result for the 50D is poorer than I was expecting, and something of a surprise. Given it has almost twice as many pixels (15 Mpixels) in the same area of sensor as the 350D, the noise would be expected to increase. In fact given this increase in number of pixels by almost a factor of two, a one stop increase in noise levels is almost exactly what would be expected if all the noise were coming from photon counting statistics. But that is relative to the 350D. If you compare the 50D with the 40D, the comparison is less favourable. What happened to the claimed benefits of larger micro-lenses, increasing the effective area of each pixel on the 50D sensor?

Because of this surprising result, I looked to see what other measurements/comparisons of noise levels were available on the web. Perhaps there were errors in my results, or even my camera was a poor model. However, frustratingly there are very few direct comparisons to be found, and nearly always the exact details of the in-camera noise settings or post processing noise reduction settings aren't given, so it is impossible to know what is being compared with what. One exception to this is the excellent DP review website, which gives a true like for like comparison of raw images from the 40D and 50D, with no noise reduction in the raw conversion either. Sure enough the 50D images are noticeably noisier, but no attempt is made to measure the differences.

The site also make the following erudite comments:

"Removing any in-camera noise reduction and processing the images using Adobe Camera Raw (without NR) gives us the nearest thing to a 'level playing field' for assessing the relative noise levels of the two cameras' sensors. Despite the fact that the 50D is the newer camera it shows visibly more chroma and luminance noise than the 40D. Considering the 50D's much more tightly packed sensor (4.5 MP/cm² vs 3.1 MP/cm² on the 40D) this comes hardly as a surprise. It would have been unreasonable to expect Canon's engineers to overcome the laws of physics."

I'd agree with that, but it would have been useful to know by what factor the noise was higher on the 40D than the 50D, to see how it compares with the results below. Visual comparisons of their shots are difficult, but to my eyes the differences could be as high as 1 to 1.5 stops.

Adobe Camera Raw (ACR) compared with Digital Photo Professional (DPP) - on EOS50D
The results shown above for my newly acquired 50D came as quite a shock, and I wonder how useable the camera will be at anything above ISO 400. Even ISO 400 appears as noisy as ISO 800 on my 350D, which certainly caused me problems in the past on some shots. ISO 1600 was pretty much a setting of last resort for the 350D, which would equate to ISO 800 on the 50D. Not a good prospect!

Out of interest, I thought I would try as an experiment using the raw converter in Canon's free Digital Photo Professional (DPP V3.6.1) instead of my usual Adobe Camera Raw (ACR) in PhotoShop Elements. Imagine my surprise when I found that DPP was giving lower noise levels in the darker areas of the images than ACR, even with all the noise reduction turned off. The graph below shows the results for the 50D at ISO 400 and ISO 800. The improvement obtained using DPP is quite marked. For example at ISO 800, the noise in the darkest parts of the image with DPP is less than the noise at ISO 400 with ACR!

I'm unclear of the reasons for this difference. It could of course be that some hidden noise reduction is going on in DPP, without the user knowing about it. Whatever the reason, it appears better to use DPP than ACR for 50D images, certainly at the higher ISO settings.

To show this makes a difference in practice, have a look at the example below, which shows a 1:1 crop of a somewhat underexposed pic of a Yellow-legged Gull, taken with the 50D at ISO 400. The chroma noise from ACR 5.4 (top) is noticeably worse than with DPP 3.6.1 (bottom) - just look at the darker areas on the back and wing tips. Even after using NeatImage (result not shown), the DPP result is appreciably better - NeatImage in my experience can struggle with chroma noise.

Here is another example, again 1:1 crops, with exposure +0.85 in both ACR and DPP. This was a distant bird, taken against the light, without the sun out, so the underparts were quite dark. Again to my eyes, the chroma noise is appreciably better in DPP (3.6.1) than ACR (5.4).

Hobby with ACR	Hobby with DPP

Of course, I don't have a side-by side comparison obtained with the 350D (or my now ex 40D), but in my experience this sort of 1:1 noise level with the 50D is more characteristic of that seen at ISO 800 with the 350D, which is in accord with the measurements presented above.

Adobe Camera Raw (ACR) compared with Digital Photo Professional (DPP) - on EOS 350D
Given the apparent benefits found above for DPP compared with ACR using the EOS 50D, it seemed interesting to see what happens with the older EOS 350D. I presumed that there would be similar results. But it seems in this field that nothing is simple, or predictable!

The graph below shows the DPP and ACR noise levels at ISO 400, 800 and 1600 for the 350D. As for the 50D, the DPP results show lower noise levels in the darker areas of the image (to the left), but look what happens on the right hand side. Unlike for the 50D, DPP for the 350D gives noticeably higher noise levels than ACR for the brighter areas of the image - almost 1 stop worse in the brightest areas. In fact, over most image grey levels, ACR gives significantly lower noise levels than DPP!

Hence for the 350D, it seems that ACR is preferable to DPP - the reverse of the situation for the 50D. Again I have no idea for the explanation for these differences!

EOS 7D compared with EOS 50D and EOS 350D using Digital Photo Professional (DPP) New March 2011
With my recent (March 2011) acquisition of an EOS 7D, I was interested to see how its noise levels compared with the others. So see below for the noise level plot using Canon's DPP to convert the RAW files, which shows results from the 350D, 50D and 7D.

From the plot below, you can see that the noise levels on the 7D are quite impressive. Despite having slightly smaller pixels than the 50D, the 7D is giving significantly lower noise, especially at ISO 1600. The difference is smaller at the lower ISO settings, but is still noticeable in the darker areas of the image. The prehistoric 350D is similar to the much newer 50D, but of course only has about half the number of pixels of the 50D. Hence the 7D may be performing at a similar noise level to the much older 40D (which of course only had 10 Mpixels)!

These measurements do tend to confirm other reports - that Canon have improved the noise position on the EOS 7D, compared with the 50D. This is some reassurance that the expensive upgrade to the 7D may have been worthwhile! Time will tell if I continue to believe that!

However, what really matters to me is the noise left after the noise reduction software I use on all images - NeatImage. Of course that is a whole new subject in its own right! All I will say on that subject at present is that I often found that quite noisy ISO 800 50D images cleaned up really well with NeatImage, provided they were well exposed (i.e. ETTR).

It is very early days yet, but I'm not convinced NeatImage is doing quite such a good job on the 7D images at ISO 800. These appear to have quite a number of "rogue" pixels, either singly or in small groups, which are largely unaffected by NeatImage. So could it be that after NeatImage, the 50D is "better" than the 7D, despite its higher raw noise levels? Surely not?!

But at ISO 1600, the 7D images appear to clean up quite well with NeatImage, and overall I'm pretty impressed with the results at this high ISO setting.

Conclusions
So what if anything has all this shown, I can hear people saying already? Well its been a very interesting exercise for me, and I think the lessons learned from these results can be summarised as follows:

• Use raw images all the time for best results and greater flexibility.
• Especially at the higher ISO settings, with Adobe Camera Raw (ACR) the noise in the darker areas of a picture will be much higher than in the brighter areas. So the message here is keep the exposure up (ETTR).
• If you clip it a bit (spike at right in histogram) don't worry too much. Just reduce the exposure in your raw converter and it will go away. This can be overdone, though, and very saturated images stay saturated. With the 50D and 7D, there is about a full stop of leeway on this adjustment, but beyond that saturated areas stay saturated no matter what exposure compensation is applied.
• With ACR, the noise levels on the EOS 40D were the lowest measured - but effectively only about 1/4 stop better than those on the 350D, after allowing for differences in the calibrations used for setting the sensitivity of the ISO settings.
• With ACR, the EOS 50D had by far the highest noise levels measured - about 1 stop worse than the 350D and a dismal 1.5 stops worse than the 40D.
• Using the Canon Digital Photo Professional (DPP) instead of Adobe Camera Raw (ACR) for raw conversion gave better control of the noise levels in the darker areas of the 50D images, with no noise reduction set, and similar noise levels in the lighter areas.
• Strangely, for the older EOS 350D camera, ACR was preferable to DPP, for unknown reasons.
• With the new EOS 7D and DPP, the noise levels were measured to be significantly lower than the 50D and 350D especially at ISO 1600 - perhaps roughly back to the 40D levels!
• There are surprisingly significant and unpredictable differences between the noise levels obtained with two very commonly used, and respected, raw converters (ACR, DPP). These appear to vary depending on the camera model.

Resolution
The other key image quality parameter of DSLR images is detail resolution - for some measurements with these cameras and two 400mm lenses, click here. Find out what benefit there is to the smaller sized pixels in the EOS50D!

Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links