Orpington Astronomical Society

Astronomy => Astrophotography => Topic started by: Ivor on Nov 06, 2011, 08:25:41

Title: Let's start at the very begining... (well the processing part anyway)
Post by: Ivor on Nov 06, 2011, 08:25:41
Morning,

Well as discussed in the intro section before I try and process my next image I want to check the quality of my darks, flats, offset and dark flat to eliminate them from the processing issues I'm struggling with. I've put a  copy of my masters here
https://picasaweb.google.com/103066655176609881871/Astrophotography , my understanding of the definition of these files is:
 
Lights — frames taken with the imaging camera through the imaging scope, with the dust cap off.
Darks — frames taken at the same ISO, exposure time, and temperature as the Lights, but with the telescope cap on.
Offsets/Bias — frames taken at the same ISO, but with as short an exposure time as the camera allows (1/4000th in my case) and with the camera's body cap in place.
Flats — frames taken of an evenly illuminated target, through the telescope @ 1/250th temperature doesn't matter.
Flat Darks — frames taken at the same ISO, exposure timeas the Flats, but with the camera's body cap in place.
 
Please let me know if my understanding of these files is correct and the images look as expected.

Thanks

Ivor
  
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 06, 2011, 10:20:07
Your understanding looks pretty much correct, but I had a look at your files - very nice M27 and M33 by the way.  
I am not sure if the Master flat is right as it doesn't appear to show any vignetting or dust particles when stretched.  It might be because of copying a file from your attached site.

The flats need to be half well depth, or roughly half way accross the histogram.  I was doing mine wrong for quite a while before some-one kindly pointed out to me that my flats were not doing anything to my images.  I had been over exposing them and washing them out.  

I now use 100ISO and switch to AV which gives a nice short exposure and point at a dull daytime sky.  There are other methods of doing this, depending on your circumstances and what suits you.  

I am sure the other more expert imagers will give you further advice.

Carole

Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Mac on Nov 06, 2011, 12:03:20
Hi Ivor

Your understanding of the definition of the files is correct, but the one that always makes me chuckle is the flat dark, why? anyway.

A Light frame is your image file plus all the other junk.

The idea of the flat frame is to take an image of the optical patch and photograph all of the debris within the path,
i.e all the junk on your filters, ccd lenses, ect. so to this you need light. As carol said, this is normally taken so the well depth is half full, so check your histogram on the back of the camera, and make sure that you have not
overexposed your image, even underexposing by 2 stops should be ok.

A dark frame is an image that records all the noise present within the camera/ccd. As the camera heats up during the exposure it produces noise,
as you said this, is normally done at the same iso, and length of exposure, so these should be ok.

A bias frame is the frame that records the readout noise from the ccd/camera.

All you need are these four files to process your image,

however!!, you need to take your flats at the same time as your lights, so that any debris is in the same place optically.

If you take them at a later date, the debris will have moved and you will get strange processing problems.
If you look at this flat that has been stretched you can see the junk in the image train.

(http://farm7.static.flickr.com/6099/6318183624_677a3e6360_z.jpg)

If you set your camera up next time, the junk will have moved, so any processing that takes place using the flat will affect the image.


Bias files contain only read noise.
Dark files contain read noise and exposure noise, (no illumination problems as there is no light)
flat files contain all the uneven illumination noise.
light contain all of the above plus your image

by subtracting each of the bias and dark above from your light frame, you are just left with the image and the illumination problems
by then adjusting the exposure with the flat frame, you are just left with your image.

Now by stacking your image files you reduce the ratio of signal to noise,
so the more frames you take, the better the final image, upto a point,
adding more frames past this limit will not increase the image noticeably.

then there is the subject of colours, but thats all down to you and artistic licence.  ;)

Mac




Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Fay on Nov 06, 2011, 14:44:46
I always do flat darks.

In DSS and Astroart, there is a place for them as well, when stacking or processing. I think you do not have to do bias, if you do flat darks, not sure.


Fay
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Ivor on Nov 06, 2011, 20:08:04
thanks for the replies I hadn't realised the flats had to be taken at the same time, I've been used a set I created at the begining, so they probably need redoing. Your comments have left me the three questions:

I don't understand what you mean by "well depth is half full" I've looked at the histogram of Mac's example and it is clearly broader than mine but I thought that was also down to the diameter of the telescope and the amount of light getting in.

My next question is logistical how do yo manage to do the lights as, At the time of year I'm home after dark so doing flats would provide difficult without a light box. Do you have a light box or a cunning alternative?

As it would appear my flats are rejects is there a way I can get around this and not have to throw away my lights?
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 06, 2011, 20:56:02
Unfortunately you can't do flats after the event unless the camera and optics haven't been moved.
Mark has a way of reusing flats, but I am not sure what his method is.

You might be able to get away with doing flats purely for vignetting, provided you have the same orientation, and same focus etc, but the chances of you lining up the dust is pretty impossible.

Doing flats in the dark you can either use a light box, or a flats panel:
http://www.gerdneumann.net/v2/download/Pricelist%20No%2027%20Brutto_Netto%20EN.pdf

Or leave the optics assembled until the following day and take them in daylight.  Either put a cover over your set up, or disassemble the telescope and camera from the mount without moving any of the optical light path. 
Quote
I always do flat darks.
I keep getting conflicting advice about this, some people say it is not necessary and some say it is.  

Carole




Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: RobertM on Nov 06, 2011, 22:02:28
Ivor,

You can take flats against any evenly illuminated smooth white surface, I'm using a white card on a wall in the observatory - akin to what the professionals call dome flats.   I illuminate the white card with some white LEDs through a diffuser.  You can do them any time but, as Carole mentioned', it must be before you change the optical configuration in any way.

A light box or flats (ELP) panel isn't needed but they do make life much much easier if you haven't got a permanent setup.  If you have a smallish refractor then an option is to keep the camera/optics together as a unit then you could do the flats indoors in the warm.

Carole,
Dark Flats - take darks for your flats so you have another set of darks to process.  I've never done that as I take sufficient flats (30-50) to bring the noise down but technically dark flats should be done.  Trouble is I don't know anyone who uses them apart from Fay and perhaps JP and Chris ?  I would say that if your Flats are of short duration then noise due to dark current from the sensor is not an issue and can be ignored, thus Dark Flats are not needed.

Mac - 2/3rd well is best, half well will lead to more nose in areas with strong vignetting.

Robert
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Mac on Nov 07, 2011, 06:56:28
can some one explain how you take a dark flat, because they are of no use.

To take a dark you need to cover the optics, thats why its called a dark.
To take a flat you need light, so you cant take a light with the cover on.

you cant take a DARK WITH LIGHT,
you cant take a light with dark.

so it is IMPOSSIBLE to take a dark flat.
anyway.

You have the following.

Dark  all the noise from the CCD due to heating ect.
Flat   all the noise created by dust in the optical train.
Bias  all the noise created by reading out the data from the chip
Light everything above including your image.

dont forget though,

your dark will have bias noise.
your flat will also have bias noise.

To get a clean light, you need to remove the bias noise and the dark noise and then adjust the exposure to compensate for the dust by using the flat.
this is done automatically by most software.

If you use incorrect flats, then the adjustments made to your image will give you processing errors.

Mac.

ps. the image of my flat has been stretched so that you can see the debris.










Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 07:19:45
Quote from: Mac
can some one explain how you take a dark flat, because they are of no use.

They are not "dark flats" but "flat darks".  

They are only required if your flats require you to use a long exposure.  Flat darks are darks taken with the same exposure time (and ISO setting) as the flats. I never use them.


Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 07:30:39
Ivor,

Flats - my take on flats is the following.

For a refractor and DSLR combination, just point the scope at your LCD monitor using a low ISO (ISO 100).  Adjust the exposure so the pixel values are around half the max possible.  Newtonians and SCTs tend to be more problematic.

I disagree with Carole:
Quote
You might be able to get away with doing flats purely for vignetting, provided you have the same orientation, and same focus etc, but the chances of you lining up the dust is pretty impossible.

The truth is that for a DSLR user, dust particles only show up in the flat only if they are on the CCD itself.  Any filters being used are too far away from the CCD to create a noticable shadow.  So there is no need to get stressed over the filters and optics all being in the same precise orientation as they were during shooting. Also, the exact position of the focuser makes almost no difference whatsoever.  However, the position of the dewshield might affect things so make sure this correctly extended/retracted.

Moreover, if you treat the camera carefully (i.e. no not leave it with the insides exposed) dust particles don't tend to move around much.  The dust specks on my Canon 350D have not moved for 18 months and the camera has travelled around literally thousands of miles in the boot of my car - so I'm still using the same flats I took 18 months ago.  If you are using a more modern camera that vibrates the CCD to remove dust then SWITCH OFF the vibration permanently because otherwise it will tend to move the dust around and you will need new flats each time.

Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 07:49:31
Quote from: Carole
Quote
I always do flat darks.
I keep getting conflicting advice about this, some people say it is not necessary and some say it is.  

Well ignore the conflicting advice and just listen to me  8)
You don't need flat darks unless you are taking flats using long exposure times.  If so, your flat dark needs the same exposure time and ISO as your flats.
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 07, 2011, 08:55:16
Thanks for clearing that up Mark regarding flat darks.

I am pleased to hear about the optical train regarding dust on filters etc with a DSLR I was just quoting what had been told to me when I was learning. 

BUT, as a prime example, if you remember when I came to High Halden on 1st October, my M31 which I took, I had left the camera in place to take the flats the following morning.  Unfortunately the camera was not tightened sufficiently to the flattener and in pressing the on/off switch on the DSLR it caused it to rotate.  I attempted to put the camera back to the same orientation, but did not succeed, with the result that the dust (which according to your explanation above must have been on the CCD) did not line up and I got two dust bunnies which have spoilt my image.

I plan to mark the flattener and camera positions for the future - AND make sure the camera and flattener are tight before imaging.  

Carole

Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: RobertM on Nov 07, 2011, 09:10:52
Carole, I'm afraid your're getting to be a bit of a Forum junkie.  If you will insist on getting advice from numerous forums on the same topics they you're bound to get conflicting advice.

In this case, because of the longer focal ratio, those dust specks will in fact cast a shadow and cause a dust bunny.  Mark is entirely correct at short focal ratios but at longer focal ratios you will still need to take flats to remove any bunnies.  Of course if your optics were free of dust then that would be be the ideal scenario  8)

Robert
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 10:01:28
Quote from: Carole
BUT, as a prime example, if you remember when I came to High Halden on 1st October, my M31 which I took, I had left the camera in place to take the flats the following morning.  Unfortunately the camera was not tightened sufficiently to the flattener and in pressing the on/off switch on the DSLR it caused it to rotate.  I attempted to put the camera back to the same orientation, but did not succeed, with the result that the dust (which according to your explanation above must have been on the CCD) did not line up and I got two dust bunnies which have spoilt my image.

I'm happy to be proved wrong but I need to see the evidence. 

My calculations tell me the following:
The Canon CCD is 44mm behind the front mating surface of the camera.
The focal reducer must therefore be at least 44mm in front of the camera.
Even at F/10 a dust particle on the reducer would create a dust bunny on the CCD 44/10 = 4.4mm across
The 450D pixel pitch is around 5 microns so your dust bunny would be nearly 1000 pixels in diameter at F/10

But you were probably imaging at F/6 which means the dust bunny would be 1500 pixels across.
I seriously don't believe a 1500 pixel dust bunny would be noticeable.

If the dust was on the clip-in filter or on the CCD then when you rotated the camera, the dust bunny would remain in the same place on the image.

Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 10:02:53
Quote from: RobertM
In this case, because of the longer focal ratio, those dust specks will in fact cast a shadow and cause a dust bunny.  Mark is entirely correct at short focal ratios but at longer focal ratios you will still need to take flats to remove any bunnies. 

See my calculations in previous post.  Would anyone notice a 1500 pixel dust bunny?

Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 07, 2011, 10:14:14
Quote
If the dust was on the clip-in filter or on the CCD then when you rotated the camera, the dust bunny would remain in the same place on the image. 
the images had already been done Mark, the rotation took place between the images and the flats.  I'll post up the images and the flats.

Carole
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 10:34:36
Quote from: Carole
I'll post up the images and the flats.

I'm looking forward to seeing them.
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: mickw on Nov 07, 2011, 11:31:33
Round and round we go, where it stops etc................

There seems to be the assumption that Carole's dust is on the optics at the camera end of the scope, which the maths say will not noticeable.

If the camera rotates and the bunny is in a different place, the dust, bird shit or glow-worm is on the scope and will be noticeable.
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 07, 2011, 11:37:20
I agree with Mick, and here's the proof.
I don't really care where the dust particle is, so long as they get subtracted from my image. 

This is a partly processed image with no flats or darks applied to show where the dust is on the light frames. N.B. there is one prominent dust particle near M110 and no dust particle near the galaxy edge on the right.

(http://www.caroleastronomy.webspace.virginmedia.com/_wp_generated/wp555e86f8_0f.jpg)

this is the flat (after DSLR had rotated) as you can see the prominent dust particle is not where it is on the light frame but close to the galaxy edge on the right.

(http://www.caroleastronomy.webspace.virginmedia.com/_wp_generated/wpfc414eb7_0f.jpg)

Here is the stacked and partly processed image with flats and darks.
As you can see the dust particle near M110 has not been substracted but a non existent dust particle has been subtracted near the galaxy edge on the right.

(http://www.caroleastronomy.webspace.virginmedia.com/_wp_generated/wp95703c33_0f.jpg)
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 07, 2011, 11:45:20
There are other dust particles there as well, but not as noticeable.

Carole
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: mickw on Nov 07, 2011, 12:02:42
Quote
I agree with Mick, and here's the proof.

You are both talking about different locations for the dust
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 12:10:12
Carole,

Thanks for posting the images.  

From them I deduce the following:
Scaled up to full size (4272 x 2848) the dust bunny is approx 100 pixels across i.e. 0.52mm using the 5.2 micron pixel pitch of the 450D.
Assuming an F-ratio of around F/5 or F/6 this places the dust particle less than 3mm in front of the CCD i.e. the dust is sitting on the CCD cover glass (notice it's the same size as some of the other dust bunnies).
Unfortunately for you, for some reason this dust particle moved between shooting M31 and shooting your flat (I hope you don't have your dust removal vibrator switched on).

Also, I think you'll find that if you rotate your flat (approx 80degrees) , the dust particle doesn't end up sitting in the right place on M31.  So that rules out rotation as the cause in any case.

Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: mickw on Nov 07, 2011, 12:38:08
Quote
Also, I think you'll find that if you rotate your flat (approx 80degrees) , the dust particle doesn't end up sitting in the right place on M31.  So that rules out rotation as the cause in any case.

If the camera wasn't firmly attached and seated squarely to the scope, the combination of rotation and misalignment could possibly be the cause ?
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 13:46:03
Quote from: mickw
If the camera wasn't firmly attached and seated squarely to the scope, the combination of rotation and misalignment could possibly be the cause ?

Possibly.  Except for the inconvenient truth that the dust particle in question is sitting on the CCD cover glass.

Ivor - look what you started!!!
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Mac on Nov 07, 2011, 13:59:06
Quote
Mark is entirely correct at short focal ratios but at longer focal ratios you will still need to take flats to remove any bunnies
#Open frying pan and into the fire#

Question regarding the above. (not the frying pan bit)
My take is that the flats are only for the dust closest to the chip, ie, the top of the chip, and filter wheels,
you would never notice any dust on the front of the lens.
Why,
Mark, do you not have a permanent dust bunny on the front of your C11 in the shape of the secondary mirror as do i on the 10"
as do any newt's ect.
does this mirror shadow show up on a flat?
If so can i see one please.

#close can of worms#
(http://www.macrhon.co.uk/oas/smiley/signs/sign0163.gif)
(http://www.macrhon.co.uk/oas/smiley/signs/sign0141.gif)

Mac
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: mickw on Nov 07, 2011, 14:29:20
That's because the secondary in a newt is made from very expensive magic glass  :roll:

or

I think I can understand the logic behind that as the CCD is focussed on an object at infinity.  Dust on or near the CCD will be closer to focus than dust on the scope so would be more noticeable.

At a risk of invoking a previous thread about reality
http://forum.orpington-astronomy.org.uk/index.php?topic=7096.msg44801#msg44801 (http://forum.orpington-astronomy.org.uk/index.php?topic=7096.msg44801#msg44801)

So why can't a star hide behind the secondary if the photons move in a straight line - I will not accept gravity lensing  8)
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 14:32:53
Quote from: Mac
Mark, do you not have a permanent dust bunny on the front of your C11 in the shape of the secondary mirror as do i on the 10"
as do any newt's ect.
does this mirror shadow show up on a flat?

A sensible question!
The primary mirror will certainly create an image of the secondary.  Using the lens formula (1/u + 1/v = 1/f) and remembering to use the correct signs for u,v and f you can calculate where this image plane will be.  In fact it will be nowhere near the CCD and certainly nowhere near enough to produce a dust bunny.  But if you defocus your camera you will see that every defocused star is, in fact, an image of the front opening of the scope (i.e. usually a circle) with a hole for the secondary.

However, there is one other factor to consider.  When you add a focal reducer this changes things.  The reducer is a lens that will create a giant out of focus image of the secondary.  This can sometimes cause havoc like in this image:

http://www.markshelley.co.uk/Astronomy/2008/horsehead2008.html

The central disc is the image of the secondary and the bright ring round the outside is stray light either coming straight into the scope or being difused by the corrector plate of the SCT. In a sense, it is a kind of giant dust bunny but it is more or less impossible to remove using flats.

Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 14:34:36
Quote from: mickw
Dust on or near the CCD will be closer to focus than dust on the scope so would be more noticeable.

Precisely!
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: mickw on Nov 07, 2011, 14:43:42
So having a grot splattered primary, secondary or front element will only degrade the image rather than add bunnies ?
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Ivor on Nov 07, 2011, 14:52:01
I can see if you step away from a thread for a while it quickly moves on. It's great to see a practical example of a flat and low level details  you have all started discussing but I'd like to come back a couple of steps (well maybe a couple of miles) to make sure I understand the basics first.

Going back to the logistics of taking flats, I like the idea of Carole's laminated light panel, however I prefer the cost of pointing my telescope at my LCD monitor  :D Unfortunately my garden isn't secure so I'm reluctant to leave the kit outside until the morning plus I don't a permanent site, so practical I'm looking to create the flats at the start of session.
 
Quote
Flats - my take on flats is the following.
For a refractor and DSLR combination, just point the scope at your LCD monitor using a low ISO (ISO 100).  Adjust the exposure so the pixel values are around half the max possible.

When you do this what do you have as a background a pure white screen?

The other bit I’m still unclear on is the well depth and whether it is either half full or 2/3 full and more to the point the histogram is a bell curve so where does the well come into it?

Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 15:23:02
Quote from: Ivor
When you do this what do you have as a background a pure white screen?

That's right - a plain white screen

Quote from: Ivor
The other bit I’m still unclear on is the well depth and whether it is either half full or 2/3 full and more to the point the histogram is a bell curve so where does the well come into it?

Well depth is how many electrons one pixel on the CCD can hold.

I find the histograms displayed by a DSLR quite confusing.  So I simply choose a low ISO and adjust the exposure so that some parts of the image just begins to saturate.  Then halve the exposure and use that halved exposure for creating the flats.
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 15:24:16
Quote from: mickw
So having a grot splattered primary, secondary or front element will only degrade the image rather than add bunnies ?

Precisely!
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Ivor on Nov 07, 2011, 16:25:41
Another question springs to mind

On the basis

Bias = read noise.
Dark = read noise and exposure noise
Flat = illumination noise.
light = read, exposure & illumination noise and object

Therefore Object =  Light - (Dark + Flat)


Why do you need the bias?
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 07, 2011, 16:46:53
Quote
I can see if you step away from a thread for a while it quickly moves on.
I can agree with Ivor on that, have been out all day.

Thanks for the analysis Mark, I have now done what you said and rotated the flat and although it is fairly close, it does not actually hit the same spot, so damn it, the dust must have moved.  I hope you can see with my less mathematical mind why I came to the conclusion that it was due to rotation.

However I find the idea that the dust particle had moved almost incomprehensible since the flattener was covering the DSLR and apart from rotating there was no other movement or air movement around the sensor.  Still at least if it moves then there is a chance I can blow it off altogether with my squeezy airball (whatever it is called). 

Ivor you have started a very interesting discussion here and hopefully we are also answering your questions and you have learnt a bit more than you had intended.

Carole

Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 17:57:37
Quote from: Ivor
Another question springs to mind

On the basis

Bias = read noise.
Dark = read noise and exposure noise
Flat = illumination noise.
light = read, exposure & illumination noise and object

Therefore Object =  Light - (Dark + Flat)

Why do you need the bias?

Good question.
Definitions of Flats and Darks can differ.  But let's take the definition where the Flats and Darks are what comes out of the camera without further adjustments i.e. without the bias subtracted.  Then the Lights, Flats and Darks all contain bias.

So,
Light = (Object x (Flat - Bias)) + Dark
Hence,
Object = (Light - Dark)/(Flat - Bias)

Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 18:05:43
Quote from: Carole
However I find the idea that the dust particle had moved almost incomprehensible since the flattener was covering the DSLR and apart from rotating there was no other movement or air movement around the sensor. 

The CCD cleaning cycle takes place every time you switch the camera on.  It vibrates the sensor at an ultra-sonic frequency in order to shake off the dust. Sometimes it will only move the dust around, instead.  For astro-photography this feature really must be switched off.  Are you sure you have it switched off?

Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Rocket Pooch on Nov 07, 2011, 20:11:56
I always do flat darks.

 :o

 :ferret:
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Ivor on Nov 07, 2011, 20:26:33
Based on what you've said I pondered some more, I'm going to risk it all and challenge your equation (I think this could be suicidal based on this thread and having only been here 5 mins)

Quote
So,
Light = (Object x (Flat - Bias)) + Dark
Hence,
Object = (Light - Dark)/(Flat - Bias)


The purpose of the bias frame is to capture the read noise, from what I understand the read noise is produced during the signal conversion from analogue to digital. Therefore the read noise must exist in all frames and won't increase with length of exposure. It might however increase with time as the CCD ages.

As the frames we take are meant to enable us to eliminate the different types of noise so the different frames must contain the following:
Bias frame [BF] = read noise.
Dark frame [DF] = read and exposure noise.
Flat frame [FF] = read and illumination noise.
Light frame [LF] = read, exposure & illumination noise and object
The goal is to eliminate the read, exposure and illumination noise from the light frames at the same time we try to reduce the signal to noise ratio by taking multiple frames.
So
Master Bias [MB] = ∑ BF
Master Dark [MD] = ∑ DF - MB
Master Flat [MF] = ∑ FF -MB
Master Light [MB] = ∑Object + MD + MF + MB
Therefore
∑Object = MB – (MD + MF + MB)

Discuss...

Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Rocket Pooch on Nov 07, 2011, 20:31:40
Off you go Ivor and Mark :-)

Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 20:53:49
Quote from: Ivor
Discuss ...

Firstly I'll correct your typo - I think you meant:
Master Light [ML] = ∑Object + MD + MF + MB
Therefore
∑Object = ML – (MD + MF + MB)

However, the master flat is multiplicative.  It corrects for vignetting, differential pixel sensitivity and for dust bunnies.  All these are multiplicative effects.  For instance, in the case of serious vignetting, the light intensity at the corners of an image might be 0.5x the intensity at the image centre.  Similarly, a certain pixel might be 1.1x as sensitive as its neighbour and similarly a dust bunny may reduce the light intensity by 0.7x

So, using your notation,

Master Bias [MB] = ∑ BF
Master Dark [MD] = ∑ DF - MB
Master Flat [MF] = ∑ FF - MB
Master Light [ML] = ∑Object x MF/constant  + MD + MB

The constant is chosen to scale most of the MF pixel values into the range 0 - 1.0

Therefore
∑Object = (ML – (MD + MB) ) / MF x constant

Mark
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 07, 2011, 20:55:40
Quote from: Rocket Pooch
Off you go Ivor and Mark :-)

I've clearly got too much spare time on my hands  :evil:
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Fay on Nov 07, 2011, 22:02:29
Chris, are you critiscising my contribution to the thread?


Mark, you have really warmed to your subject  :D
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 07, 2011, 23:39:16
Looking at the technical conversation between Ivor and Mark I think it is time to duck out of this thread and leave them to it.

Carole
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Mac on Nov 08, 2011, 00:32:22
getting back to the histogram on the back of your camera.

(http://www.basic-digital-photography.com/image-files/how-to-use-the-camera-histogram-2.jpg)

As you look at it from left to right.

the very left hand side is the colour black,
the very right hand side is the colour white
on a grey scale, each channel will have its own histogram, the one on the back of the camer is
normally an averaged out grey scale of your image.

the height of the graph is the amount of that particular colour.
so looking at the histogram you can guess that the image hame more dark colours then light colours.

If you look at this one
(http://www.usernetsite.com/photography/guide-to-understanding-and-reading-histograms/histogram-exposed.jpg)

you can clearly see the explanations for over exposing and underexposing and how the appear on the hisogram.
if you look at the overexposed one you can see that the graph is over to the right hand side and you have lost all the detail in the bright parts,
If this was your flat then you would loose all the flat detail in this area.
If you look at the OK one, again there is some over exposure on the right as the graph goes up to the right hand side.

wha you want is for the graph to go about 2/3 across the screen so that you dont loose any detail.

If you are doing a flat image then you will probably have a graph like this.
(http://www.nobsphotosuccess.com/images/GreyONE.jpg)

Mac.
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 08, 2011, 07:23:39
Quote
what you want is for the graph to go about 2/3 across the screen so that you dont loose any detail.

If you are doing a flat image then you will probably have a graph like this.
(http://www.nobsphotosuccess.com/images/GreyONE.jpg)
I strongly disagree - if you are talking about the histogram on the back of the camera. Here's why:

I've just done an experiment with the histogram on the back of my Canon 400D and with the highlight display switched on (which flashes the over saturated pixels) pointing at at flat white screen.  At 1/20sec no pixels are saturated but at 1/15sec many pixels are saturated. So I halve the 1/20sec exposure time further to 1/40sec to get a decent flat. This works and gives me the correct exposure for a flat.

But when I now look at the histogram on the back of the camera for 1/40sec, ALL the histogram is in the extreme right hand 20% of the chart.  If I selected the exposure time so the histogram was 2/3 along or 1/2 along then I would be severly underexposing my flats.  That's what I mean when I say I don't know how to interpret the histogram on the camera.  It is demonstrably not a linear scale.

For this reason, my advice is to never use the camera histogram to determine the correct exposure for flats.  There is a lot of bad advice on astro-forums concerning this issue.

Mark

P.S. I should also have said that I'm using my slowest ISO for flats.
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Fay on Nov 08, 2011, 08:30:04
a lot of people say use AV setting for correct flat exposure. What do you think of that method?
I always find it hard to get the correct exposure for DSLR flats
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 08, 2011, 09:00:19
Quote from: Fay
a lot of people say use AV setting for correct flat exposure. What do you think of that method?
I always find it hard to get the correct exposure for DSLR flats

I think the AV setting is the worst way of doing it.  On my Canon 350D and 400D this method gives a severely underexposed flat.  You also run the risk that the camera may change the shutter speed slightly from frame to frame during your sequence of flat acquisitions which will result in flats with different exposures.

My advice for taking flats is:
1)  Always use manual shutter speed
2)  Adjust shutter speed until it just begins to saturate part of the image (according to the "highlight" display) and then double the shutter speed (i.e. halve the exposure time) to produce your flats. Use the same shutter speed for the whole sequence of flats.  

This will give you well-exposed but unsaturated flats.

Mark

P.S.  If you want to be really an*l about this (and I often am) then take one set of flats to correctly expose the red pixels then another to correctly expose the green pixels and then another set to correctly expose the blue pixels.  Then using my favorite processing software (IRIS!!!) these raw images can be combined into one "super-flat" where the red, green and blue pixels are all correctly exposed.

Otherwise, typically the flat will contain green pixels that are correctly exposed and red/blue pixels that are underexposed.  This results in greater noise in your red/blue channels in your final image.  I'm giving away my trade secrets here!
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Ivor on Nov 08, 2011, 09:15:21
As vignetting isn't really a problem with my scope I'd not considered it, I've thought some more I’ve nearly got the equation bit wrapped up, so to recap if were to map the flat to a 3 x 3 grid it would look something like this

0.20.50.9
0.21.01.1
0.10.71.0

BTW How do you get the boarders on a table to appear?

In a 3D graph with a larger pixel map the stronger the vignetting the more the graph would look like a bell shape with a few dips for bunnies. This is scaled back to 0 - 1 range by the constant.

Now I've hopefully got that straight in my head I need to go back and understand the rest of this thread   :)
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Fay on Nov 08, 2011, 09:41:00
Have we got two Einstiens in collaboration here? :lol: :lol: :lol:
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Carole on Nov 08, 2011, 10:46:25
I was thinking the same thing Fay  :cheesy:
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Rick on Nov 08, 2011, 10:46:30
BTW How do you get the boarders on a table to appear?

You sit back and wait for the pirates!  :police:  Sooner or later there'll be a boarding party headed your way!  :police: Probably with  :parrot:  :squirrel: :ferret:  :bug:  :baa:  :chase: and  :flame: for company.  :abducted:

I can't see any options in the basic BBCode that might turn borders on or off. I'm afraid I just tend to fall back on the fixed format tags
Teletype
or
Code: [Select]
code for anything vaguely tabular...
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Ivor on Nov 08, 2011, 12:13:00
Going back to the practical acquisition of the flats

[I think the AV setting is the worst way of doing it.  On my Canon 350D and 400D this method gives a severely underexposed flat.  You also run the risk that the camera may change the shutter speed slightly from frame to frame during your sequence of flat acquisitions which will result in flats with different exposures.

My advice for taking flats is:
1)  Always use manual shutter speed
2)  Adjust shutter speed until it just begins to saturate part of the image (according to the "highlight" display) and then double the shutter speed (i.e. halve the exposure time) to produce your flats. Use the same shutter speed for the whole sequence of flats. 

This will give you well-exposed but unsaturated flats.

Mark

P.S.  If you want to be really an*l about this (and I often am) then take one set of flats to correctly expose the red pixels then another to correctly expose the green pixels and then another set to correctly expose the blue pixels.  Then using my favorite processing software (IRIS!!!) these raw images can be combined into one "super-flat" where the red, green and blue pixels are all correctly exposed.

Mark if you’re saying the histogram on the camera is inaccurate what application are you using to capture the flats and review the histogram? I've tried doing this in nebulosity and it won't let me reduce the exposure time sufficiently.

Also now you're started you can't leave it there, how are you isolating the RGB channels when creating the flats?

It strikes me that approach above where effective and probably the best, like the others required a skilled eye unlike the other frames required. Surely there must be a more scientific approach to acquiring the flats especially if we consider the large scopes in observatories they clearly can’t be using the approaches we are discussing.


Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: MarkS on Nov 08, 2011, 13:31:17
Quote
Mark if you’re saying the histogram on the camera is inaccurate
I'm not saying it is inaccurate.  The histogram is definitely accurate - you just need to know how to interpret it.  I think most people assume the histogram axis on the camera is linear but it is not.

Quote
what application are you using to capture the flats
I don't use an application to capture them but I do review them later in IRIS

Quote
how are you isolating the RGB channels when creating the flats
IRIS - the split_cfa and merge_cfa commands - these act on a raw frame and will isolate the R, G1, G2, B channels and then recreate a raw frame from those channels.  However, I wouldn't recommend going to this level of sophistication when you're starting out.  There are far more important things to get right first.

Quote
if we consider the large scopes in observatories they clearly can’t be using the approaches we are discussing.
They don't use DSLRs!!
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Mac on Nov 08, 2011, 14:22:02
Quote
They don't use DSLRs!!
:cheesy:
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: Ivor on Nov 08, 2011, 14:54:14
Clearly the don't use DSLRs, but wouldn't you need Flats for all CCDs as illumination noise could exist on any visual scope?
Title: Re: Let's start at the very begining... (well the processing part anyway)
Post by: RobertM on Nov 08, 2011, 15:28:30
Ivor,

Your term 'illumination noise' is misleading and incorrect - they are called Flat field frames.  Flat field frames are needed to characterise the illumination of a sensor by the optical system.  It is true they do contain all sorts of noise components but those are minimised by taking many sub exposures together with Flat Darks if you think they are necessary.  Ideally they should contain no noise.

Professional observatories will take Flat Field frames through each filter either at twilight or during the day as they can't afford to waste valuable observing time !

Robert