Orpington Astronomical Society

Astronomy => Technical => Topic started by: MarkS on Feb 13, 2013, 00:03:35

Title: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 13, 2013, 00:03:35
Here's an odd thing - a 300s dark at ISO 800 from a Nikon D7000 at the end of 2 hours of consecutive images.  It was done as a JPG rather than as a RAW.  I have scaled the size down to 15% and have multiplied the pixel values by 15 to make the noise visible.

(http://www.markshelley.co.uk/webdisk/d7000dark.jpg)

The odd thing is that the RGB channels are affected differently.  The Green channel is predominantly clipped to black over the whole image area whereas Red and Blue have large areas unclipped.  The frame should contain only thermal noise and read noise and this should affect pixels of each colour equally.

I can understand that in-camera (or on-sensor) processing will subtract a constant level from the noise - i.e. setting a black point.  I can also understand that the red and blue channels might be multiplied up to achieve a standard white balance.  But I don't understand why the green channel is black (clipped) almost everywhere.  It's as if a higher base level has been subtracted from the green channel than from the red or blue channels.

Any ideas why this should be?

I assume (for now) that this is only a feature of the conversion to JPG.  Hopefully, using RAW shooting, it stores the whole range of pixel values, thus allowing the black level to be manually set in post-processing.

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mac on Feb 13, 2013, 14:55:53
its to do with the fact thats its a Jpg, and not the Raw,
so the camera has to produce a finished image.
Dont forget the Nikons have the additional sensor, which it uses to determine the White balance as well as scene info, and if you have left it on Auto,
the i would have though that the White balance would have been affected by light pollution through the additional sensor,
albeit not that much,
so In the cameras wisdom, it has tried to compensate for the light pol, and has changed the green,
which probably makes scene.

Patent for white balance Nikon.

Just found this.
http://www.google.com/patents/EP1986444A2?dq=white+balance+nikon&ei=J6kbUcX4Gu2Y0QXx-4CoBw&cl=en (http://www.google.com/patents/EP1986444A2?dq=white+balance+nikon&ei=J6kbUcX4Gu2Y0QXx-4CoBw&cl=en)

Mac
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 13, 2013, 15:10:57
This was a dark, remember - the lens cap was on.

Where is this additional sensor?  Hidden in the dark behind the lens cap?
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mac on Feb 13, 2013, 17:44:55
Did you cover the rear Eyepiece when you did the darks?
Only you do get light leakage if its not covered.

Just a thought

Mac.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mike on Feb 13, 2013, 19:41:32
That is a good point actually. It was covered in this thread...

http://forum.orpington-astronomy.org.uk/index.php?topic=8675.msg59638#msg59638

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 14, 2013, 22:01:18
Whether or not there was light leakage, I've just uncovered a serious problem for astrophotography with a D7000.  Mike has now kindly sent me some RAW darks taken over a 2 hour period.  The purpose was to estimate the dark current and compare it with the Canons.
But what I've noticed is the following:
1) Most of the pixels have value zero
2) When converted to RGB, the non-zero pixels are mainly Red and Blue - there is very little Green

What this means is that the D7000 is performing some serious jiggery-pokery with the data before writing out the raw file.  The raw dark data from the CCD should should a histogram of values with almost none being precisely zero.  The fact that I see around 90% of the pixels being zero means that the histogram is being truncated and only the highest 10% of values remain non-zero.  The green channel is affected worse then the red or blue which means that there is also some RGB channel scaling being applied to the raw data.

The upshot is that it is almost impossible to estimate the dark current of the sensor without resorting to some tricks.  But even if I manage to estimate the dark current there still remains a serious problem - the deliberate truncation of data.  Generally speaking I want to extract as much detail as possible in the faint, wispy bits of nebulosity in the image.  In a single exposure, this detail will be swamped by thermal noise.  But I know that by stacking many frames together, the faint detail sill rise above the noise and become visible.  At least that is the case for any "normal" camera.  But when Nikon truncate the thermal noise histogram, they are also destroying my faint data along with it.  No amount of stacking can ever retrieve it - it is lost for good.

At first sight, the Nikon D7000 looked like the ideal astro-DSLR, having extremely low read noise across all ISOs.  Where previous cameras required the selection of a high ISO to reduce read noise (measured in electrons), the D7000 could be used at ISO 200 giving very high dynamic range with very low read noise.

I was very close indeed to buying one of these to become my main "workhorse".  But now I have very strong reservations.  Unless further beneficial information comes to light I cannot recommend this camera for astrophotography.

Ironically doing astrophotography with my old Nikon D70 (before switching to the superior Canons), I never worried too much about the infamous "star eater" median filter algorithm that Nikon cameras were notorious for.  In truth it was rare for a star to land on a single pixel and be erased.  The greater problem with my ancient D70 was the truncation of the histogram.  Some things never change ...
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 15, 2013, 08:08:55
The JPG posted at the beginning of this thread exactly matches the RAWs that Mike took later with the viewfinder covered.

This means that the "hot" edges and corners are probably due to the sensor being warmed by other circuitry.  Until I can estimate dark current it's not possible to tell if this is significant or not.

But the fact that 90% of the pixels in a long exposure dark frame have value zero tells me that the camera is setting it's own black level before writing out the raw image and in the process it is destroying genuine faint data along with the noise so it can never be teased out from the background noise by stacking.

I've started a thread on DPReview to discuss the problem:
http://forums.dpreview.com/forums/thread/3384710

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 15, 2013, 12:57:39

I just found this article online where the authors describe exactly the same problem with the Nikon D300:
http://theory.uchicago.edu/~ejm/pix/20d/posts/tests/D300_40D_tests/

Their conclusion on the D300 completely agrees with my conclusion on the D7000:
"The nonlinear distortion of deep shadows in [Nikon] D300 raw data by clipping means that it will be a poor choice for astrophotography, or any application where pulling weak signals out of the noise at very low illumination levels is of importance."

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mike on Feb 15, 2013, 14:11:07
This is very disappointing.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: RobertM on Feb 15, 2013, 15:17:25
That's put a spanner in the works.

What are you going to do now - try a Canon 7D or 60D ?

Robert
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 15, 2013, 19:57:32

I don't know.  I was 90% cetain of buying the D7000 until those test results.  I might look at the Pentax K5 which uses the same excellent sensor which gives identical low read noise across the whole ISO range.

The problem with the whole Canon range is you have to set a high ISO (e.g. 800 or 1600) to obtain the low read noise and then the highlights get saturated very quickly. 

The Sony IMX071 sensor is a great leap forward - such a shame Nikon then went and crippled it for astro use.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: RobertM on Feb 15, 2013, 21:48:42
But then you won't quite know whether the K5 has its own quirks until you get one.  Have you thought of hiring one for a couple of days ?
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 16, 2013, 07:45:46
Maybe, just maybe, this clipping of the low level signal isn't critical in practice.  

If there is sufficient unwanted background light (i.e. light pollution or moon glow) to bias the pixel values above the level at which the clipping of the thermal noise takes place then no desired information is lost because you would subtract the unwanted signal in post-processing.

So, what I am saying is that as long as the unwanted background light sufficiently exceeds the dark current then the final image will be no different to an image taken with an equivalent camera that does not clip.

How much is sufficiently?  I now need to analyse this mathematically   :cheesy:
I also need to think carefully about whether such an image can be properly calibrated with clipped darks.

The camera is still not suitable for astroimaging in a truly dark place but it is possible round here it actually doesn't matter (unless you start trying to do narrowband).

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Ian on Feb 16, 2013, 10:42:15
Mark, is it clear that the raws are coming off the camera like that or could it be an artifact of the file interpreter you're using?
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mike on Feb 16, 2013, 11:44:59
Also, where is the light leak you mention as I don't see it at all on the RAWS? Does it only show up with stretching?

I wonder what the images would be like if they were taken at ISO 200 instead of 800 and with the camera out in the cold rather than in a warm room?

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 16, 2013, 12:27:58
Quote from: Ian
Mark, is it clear that the raws are coming off the camera like that or could it be an artifact of the file interpreter you're using?

The JPGs and RAWs showed identical (clipping to black) behaviour and it is also the known behaviour previous Nikons.  So I'm as sure as I can be.

Quote from: Mike
Also, where is the light leak you mention as I don't see it at all on the RAWS? Does it only show up with stretching?

It's definitely not a light leak - it is a chip warmer at the corners and sides than in the middle.  You need to stretch the data to see it just like you would be stretching an astroimage.

The same effects (clipping and warm sides/corners)  is seen on these RAWs posted on UKAstroImaging:
http://ukastroimaging.co.uk/forums/index.php?topic=51263.0

Your darks have approx 80% zero-valued pixels whereas his had 70% zero-valued pixels.  Your white balance (according to the EXIF) was R:G:B = 2.16:1.00:1.26.  I think that means the scaling that must be applied to the RAW data to make it look white.

Quote from: Mike
I wonder what the images would be like if they were taken at ISO 200 instead of 800 and with the camera out in the cold rather than in a warm room?

The ISO would make no difference to the data - the sensor will still collect the same number of "thermal" electrons.  Putting the camera in the cold will make a huge difference to the amount of thermal noise but my guess is that you will still see 70-80% zero-valued pixels i.e. clipped.

I really wish I could estimate the dark current then I could relatively easily calculate the amount of light pollution required to bias the pixels away from their clipping point.  I have a couple of ideas that might yield a very rough approximation by attempting to calibrate to the remaining (non-zero) tail of the histogram, knowing what percentage of the histogram has been clipped to zero.   But I need to do a fair bit of maths first to get this right.

BTW, the probability of me buying one, after dropping from 90% to 0% has now increased to around 35%.  
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Ian on Feb 16, 2013, 13:15:07
I wonder if you could estimate the amount of clipping by heating the camera up to raise the thermal noise above the clip level? You could then compare that to the normal background sky glow of your favourite observation sites to work out how much of that light you wouldn't be able to measure anyway. Mick, step away from the blow torch...

Mac, does your D3 do this too? I'm a Nikon man (defined as Nikon lenses:5, Canon lenses:0 rather than some sort of religious affiliation).
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: mickw on Feb 16, 2013, 13:18:46
QuoteMick, step away from the blow torch...

Mmmmm Fire   :flame: :flame:
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 16, 2013, 14:37:35
Quote from: Ian
I wonder if you could estimate the amount of clipping by heating the camera up to raise the thermal noise above the clip level?

No, you can't do that.  The clipping level changes with the dark current level.  The camera firmware does this by using the specially blacked out pixels all around the border of the sensor.   Since they receive no light, the only signal they get is from the dark current.  The firmware then calculates an offset to subtract that would make, say, 70-90% of those pixels clip to zero.  This offset is then applied to all the pixels across the whole chip.

In contrast, the Canon approach is to calculate the average signal of the blacked out border pixels and calculate what offset to apply to make this average level the same as the bias level of that camera - usually 512, 1024 or 2048.  This same offset is applied to all pixels across the sensor.  The offset will be positive (so will not clip) unless the thermal signal is so high that it exceeds the camera's bias.

I've no idea what Pentax do.

Quote from: Mick
Mmmmm Fire   :flame: :flame:

Why did you have to mention fire with Mick around?


I've just been out cycling - I get all my best ideas out on the bike.  Whilst cycling, I was considering the problem of dark frames.

The dark signal across the sensor will have a certain characteristics.  There will be individual bright pixels and bright areas:  there will also be individual darker pixels and darker areas.  If the "clipping level" is set at a point where 70-90% of the pixels are zeroed out then it means that any stucture in the darker areas will always tend to be zeroed out and will never appear in the dark frame.  But when you subsequently take an image in very low light, the the structure in the thermal background will become apparent but impossible to subtract out because the master dark is a clipped master dark - hope that makes sense.

Probability of buying a D7000 has now dropped to 20%.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: RobertM on Feb 16, 2013, 14:56:20
The canon approach is effectively doing a bias subtract at the sensor level which makes sense but the Nikon approach is going to make the sensor noise appear much better than it really is.  The latter approach could, arguably, be better than that of the Canons for terrestrial work and eek out a bit more dynamic range.  It could also be why I have not seen a really good D7000 astro images, though I have seen some excellent D300 images but that's a full frame large pixel camera so might be expected.

All in all this is a very depressing topic !

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 16, 2013, 15:13:52
In a terrestrial context, this is only a issue for long exposure low light images or long exposure very heavily filtered images e.g. infra-red.  Even so, I think there are some photographers would would prefer to have the option to use their own noise reduction techniques to enhance any detail in the really dark shadows than for Nikon to say "it's in a dark shadowy area amongst the thermal noise so we'll erase for you".
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mike on Feb 16, 2013, 23:27:31
It's a shame you can't reverse engineer the Nikon firmware, or get hold of the source code, and amend the code that does the clipping.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 17, 2013, 00:35:06
Quote from: Mike
It's a shame you can't reverse engineer the Nikon firmware, or get hold of the source code, and amend the code that does the clipping.

That would be the ideal solution!

I think I can solve one of the problems I identified earlier - I've come up with an idea that would allow a master dark to be constructed.  Suppose you take your darks with a tiny pinhole in the lens cap together with a source of light.  If this creates enough light at the sensor to bias it just enough to prevent clipping then we have succeeded, as long as the light from the pinhole uniformly illuminates the sensor.  The light doesn't need a correct RGB balance either - any overall channel offsets can be dealt with when you set your black point in post-processing.  Just shoot lots of darks in this manner and then average them all in the usual way to create a master dark.

As for dark current estimation - I've almost succeeded here as well by mathematically modelling truncated gaussian distributions.  At a first approximation, the dark current per pixel is slightly less than the Canons which makes sense because it is still CMOS technology.  I'll put my results on the forum once I have a greater level of certainty.  This will allow me to calculate what combinations of ambient temperature, F-ratios and sky conditions will prevent black clipping.

From your RAWs, I've calculated the gain as approx 0.3 electrons/ADU at ISO 800.  So the gain at ISO 200 will be around 1.2 electrons/ADU which is close to unity and almost ideal.  The read noise at ISO 200 is around 2.5 electrons RMS.  Thus makes ISO 200 a real sweetspot for astroimaging - accurately capturing the faintest of detail whilst having an incredible dynamic range.  It might even be the case that you lose nothing by going to ISO 100 - I need to verify the read noise at ISO 100 and do a few extra calculations to compare.

Once we jumped through all these hoops I think we end up with an extraordinarily capable astro-camera.

My probability of buying one has just shot up to 80%.

Mark
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mac on Feb 17, 2013, 07:04:43
QuoteIt's a shame you can't reverse engineer the Nikon firmware, or get hold of the source code, and amend the code that does the clipping.

You can.

http://simeonpilgrim.com/blog/2011/11/07/how-to-decode-the-nikon-dslr-firmware/ (http://simeonpilgrim.com/blog/2011/11/07/how-to-decode-the-nikon-dslr-firmware/)

Mac :cheesy:
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 17, 2013, 08:56:01

I'd like to disable the Hot Pixel Suppression algorithm and the black clipping.  :cheesy:
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 17, 2013, 10:02:21
I now have some preliminary results on the dark current.

Firstly, here is a graph of how the percentage on non-zero (i.e. non clipped) pixels changes in successive 5 minute dark frames (I used the Red channel only):
(http://www.markshelley.co.uk/webdisk/NikonD7000_2.jpg)

Given this, I can model a truncated Normal Distribution with the same percentage of non-zero pixels.  The standard deviation of the pixels in the tail can then be compared with the standard deviation of the original distribution.  This means a correction factor can be applied to standard deviation coming from the camera pixels to determine the original standard deviation before Nikon's truncation.  I've done this approximately using Monte Carlo simulation.  This gives a preliminary dark current graph is as follows:
(http://www.markshelley.co.uk/webdisk/NikonD7000_1.jpg)

Mike's data was shot at around 25C ambient so the figures in that graph have been adjusted downwards to approxiamte the situation at 20C

Compare this to the graph for the canons:
(http://www.markshelley.co.uk/webdisk/CanonDarkCurrent_v2.jpg)


If my calculations are correct, it looks like this Exmor sensor has one third of the dark current of the Canons and so the thermal noise is almost halved.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Rocket Pooch on Feb 17, 2013, 13:24:29
Well to me your still proving you should get a cooled one shot colour.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 17, 2013, 13:46:15
Quote from: Rocket Pooch
Well to me your still proving you should get a cooled one shot colour.

Find me a cooled OSC with a read noise of 2.5 electrons then I'll agree with you!  And if you find one, I bet it costs a bit more than £600.

On an average night, the combined read & thermal noise generated by the D7000 is less than the combined read & thermal noise of any cooled one shot colour. 



Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 17, 2013, 14:09:56
So here are my final numbers:

(http://www.markshelley.co.uk/webdisk/NikonD7000_3.jpg)

The figures in bold are figures I have calculated (most of the analysis was done at ISO 800) and the rest were inferred by extrapolation.

It's a bit odd that the gain for the Red and Blue channels is different from the Green channel.  But my figures are confirmed by the histograms of pixel values in the RAWs.  The Green channel has every integer represented in the histogram but there are gaps in histograms for Red and Blue.  The frequency of the gaps (14 gaps / 100 values) agrees with the ratio of gains of the Green and R/B channels.

I'm confident that the read noise values at ISO 800 & 400 are correct - they agree well with other sources on the web.  However at ISO 100 & 200 the read noise is too small to meaningfully estimate.  Since the read noise of the sensor is not affected by ISO and because the A/D Converters are on the sensor itself I think it likely that the ISO 100 & 200 read noise should match the others.

Given that the gain at ISO 200 is close to unity, it seems the obvious choice for astrophotography.  Indeed, if you are shooting RAW then there is nothing to gain by using higher ISOs beacuse the read noise is the same.  So you obtain no finer granularity in pixel values but you saturate the top end.  The only reason to use high ISOs on the Canon (and on most earlier Nikons) was to escape the high read noise at low ISOs.

My own guess is that you won't lose contrast in "dark wispy bits" even by going to ISO 100 but I need to investigate this mathematically by simulating the Poisson process for the arrival of photons at the sensor.

Probability of buying a D7000 is now 100%.  I need to get my hands on one of these puppies!

Many thanks to Mike for supplying me with all the RAW frame data I requested.

Mark
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: RobertM on Feb 17, 2013, 14:57:27
That's a lot of interesting work you've put into this Mark, it would indeed be a bit of a game changer if you managed to prove all this practically at the telescope with a LP filter and our skies.  Your figures aren't too different to those on the Sensogen site so that proves you're on the right path.  I also found this interesting snippet on DPreview:

http://forums.dpreview.com/forums/post/50720449 (http://forums.dpreview.com/forums/post/50720449)

There is also the issue of photon shot noise which will, for most of us in the UK, swamp read and dark noise for OSC cameras very quickly.  To take advantage you will need to take many shorter exposures.

Robert

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Rocket Pooch on Feb 17, 2013, 16:12:29
You can have 1e read out noise but a thermally unstable camera is not good.  And, by the time you have hacked up4 DSLR's you could have a proper Astro Cam, and a good one to boot.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: RobertM on Feb 17, 2013, 16:28:38
Another snippet from DPreview ...

QuoteThere is no processing of RAW data related to black level, i.e., there is no artificial black clipping added. Nikon set the DN=0 level to be at the analog 0V level. However, since noise is bipolar (averages to 0, but swings both above and below), half of the noise is suppressed (truncated) when the A/D converter assigns DN=0 to negative voltages, and there is no signal present (black frame).

In blacks or very dark shadows, the noise truncation produces a false signal since the remaining noise no longer averages to 0. We're talking about a very small signal, though, and most users would never notice it. It's of interest to those who study sensor performance, and a few critical users such as astrophotographers who are trying to stack images, etc.

Sounds like what you're experiencing.

Don't forget that cooled OSC cameras give you the real raw data and don't eat stars.

Robert
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 17, 2013, 16:39:40
Quote from: Rocket Pooch
You can have 1e read out noise but a thermally unstable camera is not good.  

You can have have rock solid set point cooling, maybe even with no discernible thermal noise, but 7-11e of read noise is not good ;-)

Quote from: RobertM
Another snippet from DPreview ...

Sounds like what you're experiencing.


Not exactly.  That implies the average level of thermal noise is subtracted i.e. 50% of pixels in a dark frame would have a value of zero.  What I am seeing (from 2 examples of D7000 camera) is 80-85% of pixels in a dark frame with a value of zero.  I can see a good defence for subtracting the average level (though it is still destructive to faint data) but they are actually subtracting much more than the average level.

Anyway this is a moot point.  Unless you are shooting in really, really dark conditions of using a narrowband filter this will never happen in practice.

But it does make the creation of proper dark frames into an intellectual challenge!
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Rocket Pooch on Feb 17, 2013, 16:56:12
I completely disagree, I think your getting sensitivity and readout confused with usability.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 17, 2013, 17:12:01
Quote from: Rocket Pooch
I completely disagree, I think your getting sensitivity and readout confused with usability.

My goal is the quality of the final image.  That means controlling all sources of noise.  From the information I have available, the sensor in this camera does this extremely well - at least in theory.  If this sensor was available in nice cylindrical package with set point cooling I would be the first to buy one.

Mark
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Fay on Feb 17, 2013, 18:23:28
could you not dither instead of doing darks
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 17, 2013, 18:45:02
Quote from: Fay
could you not dither instead of doing darks

Dithering is important but it works even better if darks are applied first.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: RobertM on Feb 17, 2013, 19:37:25
True but applying darks is a law of diminishing returns just like stacking subs.  There will be a point where enough subs are stacked that the difference may be only statistical and invisible to the eye.  If you agree with that statement then the issue is finding what point that is.  For light polluted situations that number may be relatively low so you 'might' get away without darks but for darker sites the number of subs needed to get to that point will be considerably more.  There must be a way of calculating those numbers but the point here is that given enough subs then darks will not be necessary.  I'm assuming dithered subs for the purposes of this discussion.

Robert
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mac on Feb 18, 2013, 03:18:54
you might want a look at this.

http://nikonhacker.com/wiki/NikonEmulator (http://nikonhacker.com/wiki/NikonEmulator)

It allows you to emulate the firmware.

Mac
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Rocket Pooch on Feb 18, 2013, 07:08:44
How you going to get around the lack of dynamic range on the camera?

Just taking more and more darks and subs on a variable platform is the answer to total image quality, nor having square stars.  There are too many compromised with a DSLR, total noise, stability, light leaks, software support, dynamic range, overall sensitivity etc.  And that's without the inflexibility for things like narrow band imaging etc. I do think they are good at taking quick colour images though.

But I guess each to their own.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Rocket Pooch on Feb 18, 2013, 08:03:48
Mark,

I have taken a couple of technical points from your thread with some notes below;

"The clipping level changes with the dark current level.  The camera firmware does this by using the specially blacked out pixels all around the border of the sensor.   Since they receive no light, the only signal they get is from the dark current.  The firmware then calculates an offset to subtract that would make, say, 70-90% of those pixels clip to zero.  This offset is then applied to all the pixels across the whole chip."

If you think about it these pixels sit at the edge of the CCD and will be warmer than the pixels in the image. If they are hotter then the base ADU within the CCD will be a minus value, this could explain why the dark is dark clipped.  This camera is not designed for astro ccd and therefore there is no reason why this is an issue for normal photography taking a picture of a black thing in a black room with no light rarely happens.


"Anyway this is a moot point.  Unless you are shooting in really, really dark conditions of using a narrowband filter this will never happen in practice."

You will be gutted if you goto really really dark skies and the camera subtracts a lot of the background image is missing, think about images when what you are imaging is essentially black in the middle, what would happen there?  Also it seems narrow band is also off the cards.

Also do you want a set of RAW 1000D darks for your comparison, I can send them if you like.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 18, 2013, 08:14:05
Quote from: Rocket Pooch
How you going to get around the lack of dynamic range on the camera?

Dynamic range is (saturation level) / (noise level) and then quoted in weird units.  

The D7000 dynamic range far exceeds that of a sensor crippled by 7-11e of read noise.  My early calculations suggest that ISO 100 would be ideal for stacked astrophotograpy on this beast.

The Canon 350d has a dynamic range problem because a high ISO is required to get the low read noise.  So I just bracket exposure to not lose the brighter stars.  Therefore dynamic range is not a big issue in any case.

As for your other points:
1) Total noise - the combined read noise and thermal noise of D7000 is less than that of most OSCs.  If there is any fixed pattern noise it should easily be easily calibrated out.
2) Light leaks - none.  But if it is ever a problem just cover with a black cloth!
3) Software support - granted, this is something I need to look into. But it's unlikely to be a determining factor.
4) Overall sensitivity - probably slightly less sensitive than the best OSC (SXVR-H694)out here but better than all the rest.  But to bring up dynamic range again, the  SXVR-H694 has a full well capacity of 20,000e and a read noise of 5-7e.  D7000 has full well capacity of 40,000e and read noise of 2.5e  which demonstrates the vastly superior dynamic range of the D7000.  But, as I said earlier, DR is a red herring.

Mark
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 18, 2013, 08:42:50
Our replies crossed.

Answering your other points:

Quote from: Rocket Pooch
If you think about it these pixels sit at the edge of the CCD and will be warmer than the pixels in the image. If they are hotter then the base ADU within the CCD will be a minus value, this could explain why the dark is dark clipped.

Yes I think you are right.  In the warmer corners only 50% of the pixels are clipped in a dark frame.  My analysis indicates that the dark current in the corners is only 10% greater than the rest of the chip which menas they are approx 1C warmer.  But this is enough to drive the pixel clipping form 50% up to 85% across the chip.


Quote from: Rocket Pooch
You will be gutted if you goto really really dark skies and the camera subtracts a lot of the background image is missing, think about images when what you are imaging is essentially black in the middle, what would happen there?  Also it seems narrow band is also off the cards.

Point accepted. I still need to do the calculations on this.

Quote from: Rocket Pooch
Also do you want a set of RAW 1000D darks for your comparison, I can send them if you like.

That would be great.  I'll PM you what additional exposures I would need for a proper analysis.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Rocket Pooch on Feb 18, 2013, 10:14:25
"The D7000 dynamic range far exceeds that of a sensor crippled by 7-11e of read noise.  My early calculations suggest that ISO 100 would be ideal for stacked astrophotograpy on this beast.
The Canon 350d has a dynamic range problem because a high ISO is required to get the low read noise.  So I just bracket exposure to not lose the brighter stars.  Therefore dynamic range is not a big issue in any case."

I thought it was common knowledge that ISO400 for the Canon cameras gave the best dynamic range at the cost of some readout noise?

Why bracket the image, thats more flats and darks to do, with the QSI, unless its a real high cintrast image aka M42 this is not needed at all.


"2) Light leaks - none.  But if it is ever a problem just cover with a black cloth!"

That rarely works, even cloth lets in light, but stick it in a box and it will be fine, although it will warm up a bit.


"4) Overall sensitivity - probably slightly less sensitive than the best OSC (SXVR-H694) out here but better than all the rest.  But to bring up dynamic range again, the SXVR-H694 has a full well capacity of 20,000e and a read noise of 5-7e.  D7000 has full well capacity of 40,000e and read noise of 2.5e  which demonstrates the vastly superior dynamic range of the D7000.  But, as I said earlier, DR is a red herring."

Conversion into a 10, 12, 14 or 16 bit file is not that same as the well depth within the camera itself.  Nor is it the same as the representation of the data when you put it into whatever you're going to process it with for your final image.

For example, if you want to get that every little bit of detail out of an image which sits between your FWD and histogram you need to consider the file format it is down loaded into.  Why if the DR a red herring?  If you bracket the exposures are you do then you lose information from other subs as well during the combination to a HDR type image.  Great if you want somethign pretty I guess.

I think your math's is looking at the chip only and not what's involved in the whole image setup, from start to finish.  If the camera has 2e readout and is thermally stable then I'll be selling the QSI and getting a DSLR.

Also think about what you have agreed with yourself;
      You have accepted the boarder in the chip cause problems with the dark point
      You know that the thermal stability of the camera is at best ropey
      You have to bracket to get a range of exposure more than you would have to with an astro cam
      You will have to over calibrate the camera to get the noise out of it
      You will have to hack it for IR no doubt, bang goes £600

Also
      No narrowband of any real quality
      No ability to do any science, even photometry accurately

How are you going to cope with heat, say a typical UK summer night which will start at 15c and go down to 7c (I'm missing June and July), the camera chip will be hot then cool, and the camera will get hotter, making a nonsense of you bias and dark frames unless you periodically stop and start darks and bias during the night.  You can't even keep accurate reference darks or bias frames with a DSLR?

These kind of threads to make me smile, they are all over the net and they miss one point, people with DSLR do it because a) it's cheap and b) you get colour images quickly.  Other than that DSLR are a right pain.

The good thing about this thread though I know if I keep at it you will but a D7000 anyway because you're as stubborn as me. 

But then when I bought my camera I only wanted to buy one which works for a long time, hopefully, which I could calibrate properly and could be used for all aspects of Deep Sky imaging including Photometry and Spectroscopy as well and making pretty images.

Having said all that, only if you have a 16 or 32 bit screen and file format can you actually see what you have captured anyway.

Change the SCOPE, they stars are horrible with the Bananascope, Hyperstar the C11.

Lastly, how are you working out dark current?

And, can you make the skies clear.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 18, 2013, 10:49:49
Only have time for a very quick response.

1) At ISO 400 (though the precise ISO changes according to Canon model) you get the best compromise between dynamic range and noise.  If you want dynamic range you should go for ISO 100 every time but you will end up with more noise in the faint regions.

4) Bit depth is a very interesting question.  What really matters is the final stacked image and signal-to-noise ratio in the faint (i.e. darker) areas of that image.  There is no advantage in 16 bits if your read noise is 7-11e because you are swamping the fine granularity of the file representation by the noise.

Also interestingly, even if read noise is zero, there is no discernible difference between using a gain of 1 electron/ADU and using a gain of 2.5 electrons/ADU.  The "missing steps" in the "staircase" actually get filled in statistically as you stack multiple images together.  I really didn't expect this to be the case but the maths I did yesterday afternoon proved it.  The signal-to-noise ratios in the darker areas are more or less identical in each case but you do end up with a very slight non-linearity in the recorded response in the regions of the image where the photon arrival rate is less than 5 electrons per pixel per subexposure.  I'm still working on the theory of this.

As for the choice of scope - I totally agree with you.  A fast scope often compromises the image aesthetically for various reasons.  It is a very interesting area for future discussion.  By the way, what do you think of the Offina Stellare Riccardi Honders f/3  ;-)
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Rocket Pooch on Feb 18, 2013, 11:27:17
Hi,

That scope looks nice, but no good for me, I need F4 at max or I'll have image train issues without changing all my filters.

Chris
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 18, 2013, 21:51:33
To continue ...

Bracketing - the D7000 will not need bracketing - you can use the full dynamic range because the read noise is unusually low at ISO 100 & 200.  This is mainly due to the revolutionary sensor design - the A/D Converters are on the sensor so you get the same read noise (in electrons) at all ISOs.

I totally agree that DSLRs are not thermally stable.  The way to cope with this is to create a good quality master dark at an average ambient temperature and then during processing of subs a program such as IRIS or PixInsight (and probably most others) will decide what multiplier to apply to the dark before subtraction to minimise noise.  It's not ideal situation but the real issue is how much noise does this generate in the final stack - a difficult one to answer.

On the subject of darks, you asked how I calculate dark current.  The level can't be measured directly on DSLRs because some bias is always applied. However, on the Canons you shoot two successive long exposures and calculate the standard deviation in the difference frame - divide it by square root of 2 to get the standard deviation of a single frame.  Since the arrival of thermal slectrons (the dark current) is a Poisson process you can calculate the average level as the square of this because for a Poisson process the standard deviation is always the square root of the mean.  The average level is then converted to electrons by using the gain.  From the number of electrons and the exposure length you get the dark current.  For the Nikon, with its black clipped pixels, the calculation is similar but adjustments have to be made for the fact you only have part of the distribution.

Going back to cameras, I've so far been comparing the D7000 against one-shot-colour cameras.  But we all know I would get better images by using a mono camera and LRGB - broadly speaking it would halve for imaging time.  The truth is it's very much a question of convenience and of cost.  With a DSLR or OSC you get a complete colour image in one go.  Shooting LRGB there is always a risk that clouds will prevent the acquisition of all the channels required.  A good compromise might be to use a DSLR or OSC for the colour component and a mono camera to grab the Luminance and/or H-alpha.

One other thing, I said earlier that dynamic range is a bit of a red herring.  For wide band imaging around here read noise is also a red herring.  Light pollution is by far the biggest limitation on RGB imaging in all parts of Kent, Sussex and Norfolk.  It swamps the read noise and for fast F-ratios it even swamps the thermal noise of DSLRs.  

However, for short exposures, read noise tends to be the limiting factor and this is one place where a low read noise DSLR comes into it's own.

For one reason or another I've ended up specialising in RGB imaging and I further admit that using a DSLR is the lazy and cheap route to do so, but also very versatile  ;-)
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: JohnP on Feb 19, 2013, 06:48:32
Well I'm waiting for input from the expert.. Mr. G where are you. ?     
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 19, 2013, 07:37:08
Quote from: JohnP
Well I'm waiting for input from the expert.. Mr. G where are you. ?     

No, this is a colour camera - Mr. G. is the mono afficionado.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: JohnP on Feb 19, 2013, 09:01:14
Ahhh I see...... obviously why he hasn't participated in this extremely interesting thread.. my excuse is that I haven't got a clue what you are talking about..... Wooossssshhhhhhhhh!!!!!!!

Mark - Why don't you apply for a job as technical head of Nikon & you can tell them how they should be designing their camera's.......?
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Fay on Feb 19, 2013, 09:09:47
I was going to make a substantial, relevant participation in this thread.........................but decided not to, dont want to confuse the discussion!
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 19, 2013, 12:32:17
Quote from: JohnP
.. my excuse is that I haven't got a clue what you are talking about..... Wooossssshhhhhhhhh!!!!!!!

That's useful feedback.  I need to summarise the pertinent information clearly, together with a list of pros and cons and then add some tips for those unfortunate enough and/or crazy enough to ever want to use one for astrophotography.

Quote from: JohnP
Mark - Why don't you apply for a job as technical head of Nikon & you can tell them how they should be designing their camera's.......?

I think they would boot me out as soon as I tell them to:
1) Change-over the IR filter to allow H-alpha
2) Add a peltier for set point cooling the sensor
3) Stop destroying valuable data by pre-processing the RAWs
4) Produce an optional mono version (for Tony G. of course) but with a built in filter wheel
5) All whilst maintaining the same competitive price!

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mac on Feb 19, 2013, 21:06:02
Just a thought, does the D7000 still have the old mode 3 raw hack?
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Mike on Feb 19, 2013, 21:13:18
Yes but that's not exactly practical for multiple exposures.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 19, 2013, 21:17:29
Quote from: Mac
Just a thought, does the D7000 still have the old mode 3 raw hack?

I thought it was disabled on the D7000.  A mode 3 image gives exactly the same as a normal raw.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Tony G on Feb 19, 2013, 22:06:26
Quote from: JohnP on Feb 19, 2013, 06:48:32
Well I'm waiting for input from the expert.. Mr. G where are you. ?     

John, it started whoooossssshhhhing over my head at about this point in the thread.....................................

Quote from: MarkS on Feb 13, 2013, 00:03:35
Here's an odd thing - a 300s dark at ISO 800

:o :o :o :o :o :o :o :o :o

Tony G
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Fay on Feb 20, 2013, 14:02:58
Did you borrow the camera Mark?
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Carole on Feb 20, 2013, 15:27:57
This has been an interesting thread.  Some of it (with the statistics) went completely over my head, but I got the jist of enough of it.

I agree with Mark's comments about lack of clear skies limiting LRGB imaging which is why I am doing precisely what Mark suggests below.  
QuoteA good compromise might be to use a DSLR or OSC for the colour component and a mono camera to grab the Luminance and/or H-alpha.

I'm adding Ha or Luminance to existing DSLR images for the RGB component (blurring the DSLR images to smooth out the noise).  Or using my OSC for the colour component and using the Ha or luminance for the detail.
I also have LRGB filters which I will use when the weather gets more settled and we have clearer nights.  

Having had the experience of both DSLR and CCD cameras, I would definitely recommend a Mono camera for the fine detail, and use whatever is most convenient for the colour.  There is also the option of CCD narrowband as well.

Also having set point cooling makes life a lot easier.  

If only we could get some decent imaging time!!!!

Carole


Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 20, 2013, 16:47:36
Quote from: Fay
Did you borrow the camera Mark?

Mike did all the hard work, taking all the test frames and sending them to me.  Thanks Mike!
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 20, 2013, 21:05:51
Quote from: Carole
I agree with Mark's comments about lack of clear skies limiting LRGB imaging which is why I am doing precisely what Mark suggests below.  
QuoteA good compromise might be to use a DSLR or OSC for the colour component and a mono camera to grab the Luminance and/or H-alpha.

In terms of overall noise reduction it is very slightly better to shoot the R,G & B through separate filters with 2x2 on chip binning.
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Carole on Feb 21, 2013, 09:32:58
QuoteIn terms of overall noise reduction it is very slightly better to shoot the R,G & B through separate filters with 2x2 on chip binning
I agree but needs must with so little clear skies.  
So 2 x 2 binning is better for noise Mark?  

I tried binning once but found it too much faff with having to do separate calibration files and then re-sizing for combining with Luminance.  Not had a chance to use NB or LRGB since. :(

Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Feb 27, 2013, 08:06:38
Quote from: Carole
QuoteIn terms of overall noise reduction it is very slightly better to shoot the R,G & B through separate filters with 2x2 on chip binning
I agree but needs must with so little clear skies.  
So 2 x 2 binning is better for noise Mark?  

Actually, there is only one good reason for using 2x2 hardware binning - do it when the the read noise is the dominant source of noise.  If you shoot normally (i.e. without binning), then you always have the flexibility of performing a Gaussian blur on the R,G,B layers during processing, if that's what you wish to do.

Mark
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Mar 03, 2013, 17:38:53

I've been oscillating over this Nikon D7000 for a while now but I've finally come to the conclusion that it's really not going to suit my purposes i.e. stacked exposure astrophotography.  It's a shame because I really wanted to like this camera.  The sensor is noticeably superior to the entry level Canons, having higher QE, very low read noise across the ISO range and much lower dark current.

So I'm going to get a Canon 550D (they're going very cheap now) which I'll mod for H-alpha and almost certainly later I'll do a Peltier cooling mod to deal with the dark current.

Mark
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Carole on Mar 03, 2013, 19:08:05
What will be the advantage of this Mark, a better sensor?

Also are you aware they are selling already modified and peltier cooled DSLRs, but they are not cheap?
http://www.centralds.net/en/index.htm

Carole
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Mar 03, 2013, 19:44:49
The 550D is a better sensor than my present 350D and will give me plenty of megapixels to play with.

Its sensor is not as good as the Nikon D7000 but the Canon doesn't clip the pixels in dark frames or in light frames that are very dark (e.g. narrowband or very slow F-ratios).

Actually, that CDS-600D on the Centralds site is very tempting:
http://www.centralds.net/cam/?product=cds-600dx-cooled-eos-600d-2

USD $1000 for this "Frankenstein" conversion or USD $1800 brand new (with Baader filter) is not an unreasonable price compared with dedicated OSC astro-cameras.  The "Frankenstein" conversion is very similar in appearance to Julian's !

What makes me really laugh is that they charge extra to fit a "lens adaptor" i.e. the ability to use a standard Canon lens!
The downside is that they are removing a lot of what make it a real camera (look at the list of things removed).

Mark
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Carole on Mar 04, 2013, 09:50:55
Yes I guess it's really intended for astro use.  I know some-one who has bought one, but like us is besieged by clouds so hasn't had a chance to try it out yet. 

Carole
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Mar 04, 2013, 11:52:08
Quote from: Carole
I know some-one who has bought one

What, a CDS-600D ?
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Carole on Mar 04, 2013, 13:49:13
One of the Cooled and modded DSLRs not sure which model, do you want me to ask?

She hasn't had a chance to use it yet due to cloud same as us, but lives in Herne Bay, so if you wanted to take a look I am sure she would be obliging.

Carole
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Mar 06, 2013, 19:43:03
My Canon 550D arrived today so I'll shortly put it through the usual batch of tests before I pull it to bits to mod it and to investigate how to add a cold finger.

I've also got a Starlight Xpress SXVF-M25C to test.  I won't be pulling that one to bits though!

BTW - someone on the Cloudy Nights forum sent me some Pentax raws from the K-01 which uses the same IMX071 sensor as the Nikon D7000.  The Pentax is clipping pixels to zero in a very similar way to the Nikon. 

So it really does appear that Canons are the only mainstream DSLRs that are suitable for long exposure stacked astrophotography.

Mark
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: Rocket Pooch on Mar 06, 2013, 20:41:28
QED?
Title: Re: Black Clipping in Nikon D7000 Dark
Post by: MarkS on Mar 06, 2013, 22:04:33
Quote from: Rocket Pooch
QED?

No - the exact opposite in fact. 

I set out to prove that Nikons had got an edge over the Canons by using superior sensor technology and that it was about time we seriously considered using them for astrophotography.  I really didn't expect to discover the "clipping to zero" problem.  So I'm forced to conclude that the Canons are still overall the best choice for entry level DSLRs - at least for stacked astrophotography.

But for single long exposures and for making astro-videos the low noise of the Nikon sensor definitely gives it the edge.