Black Clipping in Nikon D7000 Dark

[MarkS]

Here's an odd thing - a 300s dark at ISO 800 from a Nikon D7000 at the end of 2 hours of consecutive images.  It was done as a JPG rather than as a RAW.  I have scaled the size down to 15% and have multiplied the pixel values by 15 to make the noise visible.



The odd thing is that the RGB channels are affected differently.  The Green channel is predominantly clipped to black over the whole image area whereas Red and Blue have large areas unclipped.  The frame should contain only thermal noise and read noise and this should affect pixels of each colour equally.

I can understand that in-camera (or on-sensor) processing will subtract a constant level from the noise - i.e. setting a black point.  I can also understand that the red and blue channels might be multiplied up to achieve a standard white balance.  But I don't understand why the green channel is black (clipped) almost everywhere.  It's as if a higher base level has been subtracted from the green channel than from the red or blue channels.

Any ideas why this should be?

I assume (for now) that this is only a feature of the conversion to JPG.  Hopefully, using RAW shooting, it stores the whole range of pixel values, thus allowing the black level to be manually set in post-processing.

[Mac]

its to do with the fact thats its a Jpg, and not the Raw,
so the camera has to produce a finished image.
Dont forget the Nikons have the additional sensor, which it uses to determine the White balance as well as scene info, and if you have left it on Auto,
the i would have though that the White balance would have been affected by light pollution through the additional sensor,
albeit not that much,
so In the cameras wisdom, it has tried to compensate for the light pol, and has changed the green,
which probably makes scene.

Patent for white balance Nikon.

Just found this.
http://www.google.com/patents/EP1986444A2?dq=white+balance+nikon&ei=J6kbUcX4Gu2Y0QXx-4CoBw&cl=en

Mac

[MarkS]

This was a dark, remember - the lens cap was on.

Where is this additional sensor?  Hidden in the dark behind the lens cap?

[Mac]

Did you cover the rear Eyepiece when you did the darks?
Only you do get light leakage if its not covered.

Just a thought

Mac.

[Mike]

That is a good point actually. It was covered in this thread...

http://forum.orpington-astronomy.org.uk/index.php?topic=8675.msg59638#msg59638

[MarkS]

Whether or not there was light leakage, I've just uncovered a serious problem for astrophotography with a D7000.  Mike has now kindly sent me some RAW darks taken over a 2 hour period.  The purpose was to estimate the dark current and compare it with the Canons.
But what I've noticed is the following:
1) Most of the pixels have value zero
2) When converted to RGB, the non-zero pixels are mainly Red and Blue - there is very little Green

What this means is that the D7000 is performing some serious jiggery-pokery with the data before writing out the raw file.  The raw dark data from the CCD should should a histogram of values with almost none being precisely zero.  The fact that I see around 90% of the pixels being zero means that the histogram is being truncated and only the highest 10% of values remain non-zero.  The green channel is affected worse then the red or blue which means that there is also some RGB channel scaling being applied to the raw data.

The upshot is that it is almost impossible to estimate the dark current of the sensor without resorting to some tricks.  But even if I manage to estimate the dark current there still remains a serious problem - the deliberate truncation of data.  Generally speaking I want to extract as much detail as possible in the faint, wispy bits of nebulosity in the image.  In a single exposure, this detail will be swamped by thermal noise.  But I know that by stacking many frames together, the faint detail sill rise above the noise and become visible.  At least that is the case for any "normal" camera.  But when Nikon truncate the thermal noise histogram, they are also destroying my faint data along with it.  No amount of stacking can ever retrieve it - it is lost for good.

At first sight, the Nikon D7000 looked like the ideal astro-DSLR, having extremely low read noise across all ISOs.  Where previous cameras required the selection of a high ISO to reduce read noise (measured in electrons), the D7000 could be used at ISO 200 giving very high dynamic range with very low read noise.

I was very close indeed to buying one of these to become my main "workhorse".  But now I have very strong reservations.  Unless further beneficial information comes to light I cannot recommend this camera for astrophotography.

Ironically doing astrophotography with my old Nikon D70 (before switching to the superior Canons), I never worried too much about the infamous "star eater" median filter algorithm that Nikon cameras were notorious for.  In truth it was rare for a star to land on a single pixel and be erased.  The greater problem with my ancient D70 was the truncation of the histogram.  Some things never change ...

[MarkS]

The JPG posted at the beginning of this thread exactly matches the RAWs that Mike took later with the viewfinder covered.

This means that the "hot" edges and corners are probably due to the sensor being warmed by other circuitry.  Until I can estimate dark current it's not possible to tell if this is significant or not.

But the fact that 90% of the pixels in a long exposure dark frame have value zero tells me that the camera is setting it's own black level before writing out the raw image and in the process it is destroying genuine faint data along with the noise so it can never be teased out from the background noise by stacking.

I've started a thread on DPReview to discuss the problem:
http://forums.dpreview.com/forums/thread/3384710

[MarkS]

I just found this article online where the authors describe exactly the same problem with the Nikon D300:
http://theory.uchicago.edu/~ejm/pix/20d/posts/tests/D300_40D_tests/

Their conclusion on the D300 completely agrees with my conclusion on the D7000:
"The nonlinear distortion of deep shadows in [Nikon] D300 raw data by clipping means that it will be a poor choice for astrophotography, or any application where pulling weak signals out of the noise at very low illumination levels is of importance."

[Mike]

This is very disappointing.

[RobertM]

That's put a spanner in the works.

What are you going to do now - try a Canon 7D or 60D ?

Robert

[MarkS]

I don't know.  I was 90% cetain of buying the D7000 until those test results.  I might look at the Pentax K5 which uses the same excellent sensor which gives identical low read noise across the whole ISO range.

The problem with the whole Canon range is you have to set a high ISO (e.g. 800 or 1600) to obtain the low read noise and then the highlights get saturated very quickly. 

The Sony IMX071 sensor is a great leap forward - such a shame Nikon then went and crippled it for astro use.

[RobertM]

But then you won't quite know whether the K5 has its own quirks until you get one.  Have you thought of hiring one for a couple of days ?

[MarkS]

Maybe, just maybe, this clipping of the low level signal isn't critical in practice.  

If there is sufficient unwanted background light (i.e. light pollution or moon glow) to bias the pixel values above the level at which the clipping of the thermal noise takes place then no desired information is lost because you would subtract the unwanted signal in post-processing.

So, what I am saying is that as long as the unwanted background light sufficiently exceeds the dark current then the final image will be no different to an image taken with an equivalent camera that does not clip.

How much is sufficiently?  I now need to analyse this mathematically  
I also need to think carefully about whether such an image can be properly calibrated with clipped darks.

The camera is still not suitable for astroimaging in a truly dark place but it is possible round here it actually doesn't matter (unless you start trying to do narrowband).

[Ian]

Mark, is it clear that the raws are coming off the camera like that or could it be an artifact of the file interpreter you're using?

[Mike]

Also, where is the light leak you mention as I don't see it at all on the RAWS? Does it only show up with stretching?

I wonder what the images would be like if they were taken at ISO 200 instead of 800 and with the camera out in the cold rather than in a warm room?