Black Clipping in Nikon D7000 Dark

[Rocket Pooch]

You can have 1e read out noise but a thermally unstable camera is not good.  And, by the time you have hacked up4 DSLR's you could have a proper Astro Cam, and a good one to boot.

[RobertM]

Another snippet from DPreview ...

There is no processing of RAW data related to black level, i.e., there is no artificial black clipping added. Nikon set the DN=0 level to be at the analog 0V level. However, since noise is bipolar (averages to 0, but swings both above and below), half of the noise is suppressed (truncated) when the A/D converter assigns DN=0 to negative voltages, and there is no signal present (black frame).

In blacks or very dark shadows, the noise truncation produces a false signal since the remaining noise no longer averages to 0. We're talking about a very small signal, though, and most users would never notice it. It's of interest to those who study sensor performance, and a few critical users such as astrophotographers who are trying to stack images, etc.

Sounds like what you're experiencing.

Don't forget that cooled OSC cameras give you the real raw data and don't eat stars.

Robert

[MarkS]

You can have 1e read out noise but a thermally unstable camera is not good.  

You can have have rock solid set point cooling, maybe even with no discernible thermal noise, but 7-11e of read noise is not good

Another snippet from DPreview ...

Sounds like what you're experiencing.

Not exactly.  That implies the average level of thermal noise is subtracted i.e. 50% of pixels in a dark frame would have a value of zero.  What I am seeing (from 2 examples of D7000 camera) is 80-85% of pixels in a dark frame with a value of zero.  I can see a good defence for subtracting the average level (though it is still destructive to faint data) but they are actually subtracting much more than the average level.

Anyway this is a moot point.  Unless you are shooting in really, really dark conditions of using a narrowband filter this will never happen in practice.

But it does make the creation of proper dark frames into an intellectual challenge!

[Rocket Pooch]

I completely disagree, I think your getting sensitivity and readout confused with usability.

[MarkS]

I completely disagree, I think your getting sensitivity and readout confused with usability.

My goal is the quality of the final image.  That means controlling all sources of noise.  From the information I have available, the sensor in this camera does this extremely well - at least in theory.  If this sensor was available in nice cylindrical package with set point cooling I would be the first to buy one.

Mark

[Fay]

could you not dither instead of doing darks

[MarkS]

could you not dither instead of doing darks

Dithering is important but it works even better if darks are applied first.

[RobertM]

True but applying darks is a law of diminishing returns just like stacking subs.  There will be a point where enough subs are stacked that the difference may be only statistical and invisible to the eye.  If you agree with that statement then the issue is finding what point that is.  For light polluted situations that number may be relatively low so you 'might' get away without darks but for darker sites the number of subs needed to get to that point will be considerably more.  There must be a way of calculating those numbers but the point here is that given enough subs then darks will not be necessary.  I'm assuming dithered subs for the purposes of this discussion.

Robert

[Mac]

you might want a look at this.

http://nikonhacker.com/wiki/NikonEmulator

It allows you to emulate the firmware.

Mac

[Rocket Pooch]

How you going to get around the lack of dynamic range on the camera?

Just taking more and more darks and subs on a variable platform is the answer to total image quality, nor having square stars.  There are too many compromised with a DSLR, total noise, stability, light leaks, software support, dynamic range, overall sensitivity etc.  And that's without the inflexibility for things like narrow band imaging etc. I do think they are good at taking quick colour images though.

But I guess each to their own.

[Rocket Pooch]

Mark,

I have taken a couple of technical points from your thread with some notes below;

"The clipping level changes with the dark current level.  The camera firmware does this by using the specially blacked out pixels all around the border of the sensor.   Since they receive no light, the only signal they get is from the dark current.  The firmware then calculates an offset to subtract that would make, say, 70-90% of those pixels clip to zero.  This offset is then applied to all the pixels across the whole chip."

If you think about it these pixels sit at the edge of the CCD and will be warmer than the pixels in the image. If they are hotter then the base ADU within the CCD will be a minus value, this could explain why the dark is dark clipped.  This camera is not designed for astro ccd and therefore there is no reason why this is an issue for normal photography taking a picture of a black thing in a black room with no light rarely happens.


"Anyway this is a moot point.  Unless you are shooting in really, really dark conditions of using a narrowband filter this will never happen in practice."

You will be gutted if you goto really really dark skies and the camera subtracts a lot of the background image is missing, think about images when what you are imaging is essentially black in the middle, what would happen there?  Also it seems narrow band is also off the cards.

Also do you want a set of RAW 1000D darks for your comparison, I can send them if you like.

[MarkS]

How you going to get around the lack of dynamic range on the camera?

Dynamic range is (saturation level) / (noise level) and then quoted in weird units.  

The D7000 dynamic range far exceeds that of a sensor crippled by 7-11e of read noise.  My early calculations suggest that ISO 100 would be ideal for stacked astrophotograpy on this beast.

The Canon 350d has a dynamic range problem because a high ISO is required to get the low read noise.  So I just bracket exposure to not lose the brighter stars.  Therefore dynamic range is not a big issue in any case.

As for your other points:
1) Total noise - the combined read noise and thermal noise of D7000 is less than that of most OSCs.  If there is any fixed pattern noise it should easily be easily calibrated out.
2) Light leaks - none.  But if it is ever a problem just cover with a black cloth!
3) Software support - granted, this is something I need to look into. But it's unlikely to be a determining factor.
4) Overall sensitivity - probably slightly less sensitive than the best OSC (SXVR-H694)out here but better than all the rest.  But to bring up dynamic range again, the  SXVR-H694 has a full well capacity of 20,000e and a read noise of 5-7e.  D7000 has full well capacity of 40,000e and read noise of 2.5e  which demonstrates the vastly superior dynamic range of the D7000.  But, as I said earlier, DR is a red herring.

Mark

[MarkS]

Our replies crossed.

Answering your other points:

If you think about it these pixels sit at the edge of the CCD and will be warmer than the pixels in the image. If they are hotter then the base ADU within the CCD will be a minus value, this could explain why the dark is dark clipped.

Yes I think you are right.  In the warmer corners only 50% of the pixels are clipped in a dark frame.  My analysis indicates that the dark current in the corners is only 10% greater than the rest of the chip which menas they are approx 1C warmer.  But this is enough to drive the pixel clipping form 50% up to 85% across the chip.

You will be gutted if you goto really really dark skies and the camera subtracts a lot of the background image is missing, think about images when what you are imaging is essentially black in the middle, what would happen there?  Also it seems narrow band is also off the cards.

Point accepted. I still need to do the calculations on this.

Also do you want a set of RAW 1000D darks for your comparison, I can send them if you like.

That would be great.  I'll PM you what additional exposures I would need for a proper analysis.

[Rocket Pooch]

"The D7000 dynamic range far exceeds that of a sensor crippled by 7-11e of read noise.  My early calculations suggest that ISO 100 would be ideal for stacked astrophotograpy on this beast.
The Canon 350d has a dynamic range problem because a high ISO is required to get the low read noise.  So I just bracket exposure to not lose the brighter stars.  Therefore dynamic range is not a big issue in any case."

I thought it was common knowledge that ISO400 for the Canon cameras gave the best dynamic range at the cost of some readout noise?

Why bracket the image, thats more flats and darks to do, with the QSI, unless its a real high cintrast image aka M42 this is not needed at all.


"2) Light leaks - none.  But if it is ever a problem just cover with a black cloth!"

That rarely works, even cloth lets in light, but stick it in a box and it will be fine, although it will warm up a bit.


"4) Overall sensitivity - probably slightly less sensitive than the best OSC (SXVR-H694) out here but better than all the rest.  But to bring up dynamic range again, the SXVR-H694 has a full well capacity of 20,000e and a read noise of 5-7e.  D7000 has full well capacity of 40,000e and read noise of 2.5e  which demonstrates the vastly superior dynamic range of the D7000.  But, as I said earlier, DR is a red herring."

Conversion into a 10, 12, 14 or 16 bit file is not that same as the well depth within the camera itself.  Nor is it the same as the representation of the data when you put it into whatever you're going to process it with for your final image.

For example, if you want to get that every little bit of detail out of an image which sits between your FWD and histogram you need to consider the file format it is down loaded into.  Why if the DR a red herring?  If you bracket the exposures are you do then you lose information from other subs as well during the combination to a HDR type image.  Great if you want somethign pretty I guess.

I think your math's is looking at the chip only and not what's involved in the whole image setup, from start to finish.  If the camera has 2e readout and is thermally stable then I'll be selling the QSI and getting a DSLR.

Also think about what you have agreed with yourself;
      You have accepted the boarder in the chip cause problems with the dark point
      You know that the thermal stability of the camera is at best ropey
      You have to bracket to get a range of exposure more than you would have to with an astro cam
      You will have to over calibrate the camera to get the noise out of it
      You will have to hack it for IR no doubt, bang goes £600

Also
      No narrowband of any real quality
      No ability to do any science, even photometry accurately

How are you going to cope with heat, say a typical UK summer night which will start at 15c and go down to 7c (I'm missing June and July), the camera chip will be hot then cool, and the camera will get hotter, making a nonsense of you bias and dark frames unless you periodically stop and start darks and bias during the night.  You can't even keep accurate reference darks or bias frames with a DSLR?

These kind of threads to make me smile, they are all over the net and they miss one point, people with DSLR do it because a) it's cheap and b) you get colour images quickly.  Other than that DSLR are a right pain.

The good thing about this thread though I know if I keep at it you will but a D7000 anyway because you're as stubborn as me. 

But then when I bought my camera I only wanted to buy one which works for a long time, hopefully, which I could calibrate properly and could be used for all aspects of Deep Sky imaging including Photometry and Spectroscopy as well and making pretty images.

Having said all that, only if you have a 16 or 32 bit screen and file format can you actually see what you have captured anyway.

Change the SCOPE, they stars are horrible with the Bananascope, Hyperstar the C11.

Lastly, how are you working out dark current?

And, can you make the skies clear.

[MarkS]

Only have time for a very quick response.

1) At ISO 400 (though the precise ISO changes according to Canon model) you get the best compromise between dynamic range and noise.  If you want dynamic range you should go for ISO 100 every time but you will end up with more noise in the faint regions.

4) Bit depth is a very interesting question.  What really matters is the final stacked image and signal-to-noise ratio in the faint (i.e. darker) areas of that image.  There is no advantage in 16 bits if your read noise is 7-11e because you are swamping the fine granularity of the file representation by the noise.

Also interestingly, even if read noise is zero, there is no discernible difference between using a gain of 1 electron/ADU and using a gain of 2.5 electrons/ADU.  The "missing steps" in the "staircase" actually get filled in statistically as you stack multiple images together.  I really didn't expect this to be the case but the maths I did yesterday afternoon proved it.  The signal-to-noise ratios in the darker areas are more or less identical in each case but you do end up with a very slight non-linearity in the recorded response in the regions of the image where the photon arrival rate is less than 5 electrons per pixel per subexposure.  I'm still working on the theory of this.

As for the choice of scope - I totally agree with you.  A fast scope often compromises the image aesthetically for various reasons.  It is a very interesting area for future discussion.  By the way, what do you think of the Offina Stellare Riccardi Honders f/3