Black Clipping in Nikon D7000 Dark

[MarkS]

Mark, is it clear that the raws are coming off the camera like that or could it be an artifact of the file interpreter you're using?

The JPGs and RAWs showed identical (clipping to black) behaviour and it is also the known behaviour previous Nikons.  So I'm as sure as I can be.

Also, where is the light leak you mention as I don't see it at all on the RAWS? Does it only show up with stretching?

It's definitely not a light leak - it is a chip warmer at the corners and sides than in the middle.  You need to stretch the data to see it just like you would be stretching an astroimage.

The same effects (clipping and warm sides/corners)  is seen on these RAWs posted on UKAstroImaging:
http://ukastroimaging.co.uk/forums/index.php?topic=51263.0

Your darks have approx 80% zero-valued pixels whereas his had 70% zero-valued pixels.  Your white balance (according to the EXIF) was R:G:B = 2.16:1.00:1.26.  I think that means the scaling that must be applied to the RAW data to make it look white.

I wonder what the images would be like if they were taken at ISO 200 instead of 800 and with the camera out in the cold rather than in a warm room?

The ISO would make no difference to the data - the sensor will still collect the same number of "thermal" electrons.  Putting the camera in the cold will make a huge difference to the amount of thermal noise but my guess is that you will still see 70-80% zero-valued pixels i.e. clipped.

I really wish I could estimate the dark current then I could relatively easily calculate the amount of light pollution required to bias the pixels away from their clipping point.  I have a couple of ideas that might yield a very rough approximation by attempting to calibrate to the remaining (non-zero) tail of the histogram, knowing what percentage of the histogram has been clipped to zero.   But I need to do a fair bit of maths first to get this right.

BTW, the probability of me buying one, after dropping from 90% to 0% has now increased to around 35%.  

[Ian]

I wonder if you could estimate the amount of clipping by heating the camera up to raise the thermal noise above the clip level? You could then compare that to the normal background sky glow of your favourite observation sites to work out how much of that light you wouldn't be able to measure anyway. Mick, step away from the blow torch...

Mac, does your D3 do this too? I'm a Nikon man (defined as Nikon lenses:5, Canon lenses:0 rather than some sort of religious affiliation).

[mickw]

Mick, step away from the blow torch...

Mmmmm Fire   

[MarkS]

I wonder if you could estimate the amount of clipping by heating the camera up to raise the thermal noise above the clip level?

No, you can't do that.  The clipping level changes with the dark current level.  The camera firmware does this by using the specially blacked out pixels all around the border of the sensor.   Since they receive no light, the only signal they get is from the dark current.  The firmware then calculates an offset to subtract that would make, say, 70-90% of those pixels clip to zero.  This offset is then applied to all the pixels across the whole chip.

In contrast, the Canon approach is to calculate the average signal of the blacked out border pixels and calculate what offset to apply to make this average level the same as the bias level of that camera - usually 512, 1024 or 2048.  This same offset is applied to all pixels across the sensor.  The offset will be positive (so will not clip) unless the thermal signal is so high that it exceeds the camera's bias.

I've no idea what Pentax do.

Mmmmm Fire  

Why did you have to mention fire with Mick around?


I've just been out cycling - I get all my best ideas out on the bike.  Whilst cycling, I was considering the problem of dark frames.

The dark signal across the sensor will have a certain characteristics.  There will be individual bright pixels and bright areas:  there will also be individual darker pixels and darker areas.  If the "clipping level" is set at a point where 70-90% of the pixels are zeroed out then it means that any stucture in the darker areas will always tend to be zeroed out and will never appear in the dark frame.  But when you subsequently take an image in very low light, the the structure in the thermal background will become apparent but impossible to subtract out because the master dark is a clipped master dark - hope that makes sense.

Probability of buying a D7000 has now dropped to 20%.

[RobertM]

The canon approach is effectively doing a bias subtract at the sensor level which makes sense but the Nikon approach is going to make the sensor noise appear much better than it really is.  The latter approach could, arguably, be better than that of the Canons for terrestrial work and eek out a bit more dynamic range.  It could also be why I have not seen a really good D7000 astro images, though I have seen some excellent D300 images but that's a full frame large pixel camera so might be expected.

All in all this is a very depressing topic !

[MarkS]

In a terrestrial context, this is only a issue for long exposure low light images or long exposure very heavily filtered images e.g. infra-red.  Even so, I think there are some photographers would would prefer to have the option to use their own noise reduction techniques to enhance any detail in the really dark shadows than for Nikon to say "it's in a dark shadowy area amongst the thermal noise so we'll erase for you".

[Mike]

It's a shame you can't reverse engineer the Nikon firmware, or get hold of the source code, and amend the code that does the clipping.

[MarkS]

It's a shame you can't reverse engineer the Nikon firmware, or get hold of the source code, and amend the code that does the clipping.

That would be the ideal solution!

I think I can solve one of the problems I identified earlier - I've come up with an idea that would allow a master dark to be constructed.  Suppose you take your darks with a tiny pinhole in the lens cap together with a source of light.  If this creates enough light at the sensor to bias it just enough to prevent clipping then we have succeeded, as long as the light from the pinhole uniformly illuminates the sensor.  The light doesn't need a correct RGB balance either - any overall channel offsets can be dealt with when you set your black point in post-processing.  Just shoot lots of darks in this manner and then average them all in the usual way to create a master dark.

As for dark current estimation - I've almost succeeded here as well by mathematically modelling truncated gaussian distributions.  At a first approximation, the dark current per pixel is slightly less than the Canons which makes sense because it is still CMOS technology.  I'll put my results on the forum once I have a greater level of certainty.  This will allow me to calculate what combinations of ambient temperature, F-ratios and sky conditions will prevent black clipping.

From your RAWs, I've calculated the gain as approx 0.3 electrons/ADU at ISO 800.  So the gain at ISO 200 will be around 1.2 electrons/ADU which is close to unity and almost ideal.  The read noise at ISO 200 is around 2.5 electrons RMS.  Thus makes ISO 200 a real sweetspot for astroimaging - accurately capturing the faintest of detail whilst having an incredible dynamic range.  It might even be the case that you lose nothing by going to ISO 100 - I need to verify the read noise at ISO 100 and do a few extra calculations to compare.

Once we jumped through all these hoops I think we end up with an extraordinarily capable astro-camera.

My probability of buying one has just shot up to 80%.

Mark

[Mac]

It's a shame you can't reverse engineer the Nikon firmware, or get hold of the source code, and amend the code that does the clipping.

You can.

http://simeonpilgrim.com/blog/2011/11/07/how-to-decode-the-nikon-dslr-firmware/

Mac

[MarkS]

I'd like to disable the Hot Pixel Suppression algorithm and the black clipping. 

[MarkS]

I now have some preliminary results on the dark current.

Firstly, here is a graph of how the percentage on non-zero (i.e. non clipped) pixels changes in successive 5 minute dark frames (I used the Red channel only):


Given this, I can model a truncated Normal Distribution with the same percentage of non-zero pixels.  The standard deviation of the pixels in the tail can then be compared with the standard deviation of the original distribution.  This means a correction factor can be applied to standard deviation coming from the camera pixels to determine the original standard deviation before Nikon's truncation.  I've done this approximately using Monte Carlo simulation.  This gives a preliminary dark current graph is as follows:


Mike's data was shot at around 25C ambient so the figures in that graph have been adjusted downwards to approxiamte the situation at 20C

Compare this to the graph for the canons:



If my calculations are correct, it looks like this Exmor sensor has one third of the dark current of the Canons and so the thermal noise is almost halved.

[Rocket Pooch]

Well to me your still proving you should get a cooled one shot colour.

[MarkS]

Well to me your still proving you should get a cooled one shot colour.

Find me a cooled OSC with a read noise of 2.5 electrons then I'll agree with you!  And if you find one, I bet it costs a bit more than £600.

On an average night, the combined read & thermal noise generated by the D7000 is less than the combined read & thermal noise of any cooled one shot colour. 

[MarkS]

So here are my final numbers:



The figures in bold are figures I have calculated (most of the analysis was done at ISO 800) and the rest were inferred by extrapolation.

It's a bit odd that the gain for the Red and Blue channels is different from the Green channel.  But my figures are confirmed by the histograms of pixel values in the RAWs.  The Green channel has every integer represented in the histogram but there are gaps in histograms for Red and Blue.  The frequency of the gaps (14 gaps / 100 values) agrees with the ratio of gains of the Green and R/B channels.

I'm confident that the read noise values at ISO 800 & 400 are correct - they agree well with other sources on the web.  However at ISO 100 & 200 the read noise is too small to meaningfully estimate.  Since the read noise of the sensor is not affected by ISO and because the A/D Converters are on the sensor itself I think it likely that the ISO 100 & 200 read noise should match the others.

Given that the gain at ISO 200 is close to unity, it seems the obvious choice for astrophotography.  Indeed, if you are shooting RAW then there is nothing to gain by using higher ISOs beacuse the read noise is the same.  So you obtain no finer granularity in pixel values but you saturate the top end.  The only reason to use high ISOs on the Canon (and on most earlier Nikons) was to escape the high read noise at low ISOs.

My own guess is that you won't lose contrast in "dark wispy bits" even by going to ISO 100 but I need to investigate this mathematically by simulating the Poisson process for the arrival of photons at the sensor.

Probability of buying a D7000 is now 100%.  I need to get my hands on one of these puppies!

Many thanks to Mike for supplying me with all the RAW frame data I requested.

Mark

[RobertM]

That's a lot of interesting work you've put into this Mark, it would indeed be a bit of a game changer if you managed to prove all this practically at the telescope with a LP filter and our skies.  Your figures aren't too different to those on the Sensogen site so that proves you're on the right path.  I also found this interesting snippet on DPreview:

http://forums.dpreview.com/forums/post/50720449

There is also the issue of photon shot noise which will, for most of us in the UK, swamp read and dark noise for OSC cameras very quickly.  To take advantage you will need to take many shorter exposures.

Robert