QHY8L OSC artifact problem

[Carole]

I've been struggling with my QHY8L with 4 problems, which is why I haven't used it much.  I know you guys like a bit of analysis so I was hoping you might be able to tell me what's going on here.  

1. I've cured the regularly disconnecting cables problem by fitting a retainer clip (poor design fault).
2. I think I have sorted the focus/looping/live view problem, as Ivo has brought out a looping facility with the latest version of APT, though I have yet to try it with a live sky.
3. N.B. the original problem of the faint lines was (I think) due to using the wrong download speed, I have not had the problem since as far as I can see using normal download speed.
4. My final problem is that of dirt/artifacts not getting removed by flats.

I wonder whether you guys could give an opinion as to what is causing this and why aren't flats removing the problem.

Should I get some lens cleaning tissue and try cleaning the lens?

It's not the scope as I have used (on the same evening) a different camera on same scope, and no artifacts there, ditto reducer/flattener. It's not the LP filter as I get the same with or without. I have tried cleaning the glass in the front of the camera, but maybe it could be on the underside. But even if it is a dirty sensor, why aren't flats removing it? In fact it looks like flats are possibly over-compensating, really not sure what is going on.

Don't try to fit the flat exactly over the image as I've cropped them to make it easier to see, and the crops are not necessarily precisely the same.  

Any ideas?



Carole

[RobertM]

It looks a little like your calibration is going wrong somewhere.  As a first step I would take one of your flats and calibrate that as if it were your image.  That would show whether it's the calibration process since it should produce a perfect flat frame.  If it doesn't then you'll have to check the calibration process as something is going wrong (for instance the flats aren't being scaled to the image correctly).

I'll be out the rest of the weekend so can't reply further but I'm sure someone else will chip in somewhere along the line.

HTH
Robert

[MarkS]

As you said, the flat is clearly "over-compensating".
There are two possible explanations:
1) The flats are not being applied properly by the processing - this is unlikely.
2) The most likely explanation is that more scattered light was reaching the sensor when you were imaging than when you were taking the flats.

Is it possible the flats were taken with a dewshield on and the imaging was done with it off?
Was the moon or some other light getting into the scope?

There's some useful (fairly technical) background reading on flats here:
http://forum.orpington-astronomy.org.uk/index.php?topic=9342.0

[RobertM]

Just had a look at the flat in Maxim and it looks a little too flat, can you check a single flat against the master flat.  Flats generally look quite a bit brighter in the middle and not very uniform like that master.  The centre to edge brightness usually varies by around 8-20% depending on optics and sensor size.

Robert

[Carole]

In quick reply Mark and Robert (thanks). 

2) The most likely explanation is that more scattered light was reaching the sensor when you were imaging than when you were taking the flats.

I have had this problem ever since I got this camera, so I can't imagine this would happen every time in differing conditions. 

Is it possible the flats were taken with a dewshield on and the imaging was done with it off?

The flats were done with an EL panel and light box.  I don't have this problem with my Atik camera using the same method.

Just had a look at the flat in Maxim and it looks a little too flat, can you check a single flat against the master flat.  Flats generally look quite a bit brighter in the middle and not very uniform like that master.  The centre to edge brightness usually varies by around 8-20% depending on optics and sensor size.

The flat I posted up was only slightly stretched Robert, this is it stretched quite a lot, and as you can see there is vignetting and it has a lot of dust that it has removed. 



Normally I just process out these artifacts, but it's not the answer really, and I would like to be able to remove the worst bits but do you think they are on the sensor itself?  QHY talk about cleaning it:

You can open the front cover of CCD to clean the surface of chip. You don't have to open the front
cover of CCD( this can be delete because ccd front cover can not be open)
Screw the front cover of CCD counterclockwise, and clean the dust off of the surface of CCD with a manual air pump. If there is dirt that doesn't come off with the air pump, you can use lens paper or a commercially available SLR camera cleaning kit.
Here is the correct procedure to follow with the lens
paper:
1. Wash your hands with soap.
2. Take a piece of lens paper,
and fold it once or twice
(do not fold too many times as
the lens paper will get very
sharp and the edges may
scratch the CCD glass surface)
Cleaning tips
3. Blow at CCD in a breath and use lens paper to
clean the CCD surface. Make sure to maintain an
appropriate pressure for CCD with your hand.
4. Finish cleaning and reinstall the front cover of
CCD. If there is higher relative humidity in your
environment- it is necessary to dry the inside of the
CCD sealed chamber.

I have tried the puff ball method. 

Carole

[Carole]

As a first step I would take one of your flats and calibrate that as if it were your image.

I've done that Robert and the result is clean as a whistle.

I'm mystified.

I have also had a clean of the glass in front of the CCD and it was a bit dirty even though this did not appear obvious to the naked eye, however this still doesn't solve the cause of flats not subtracting these artifacts. 

Carole

[MarkS]

Do you use an IR filter when imaging with this camera?

[Carole]

Do you use an IR filter when imaging with this camera?

I don't add any filters to it Mark except a CLS filter when at home.  However the glass in front of the chip seems to be IR block.

Carole

[Mike]

Is the histogram for the flat clipped?

[Carole]

Is the histogram for the flat clipped?

The unstretched flat isn't clipped Mike.

The one above which I stretched a lot for demonstration purposes I am not sure about:

[MarkS]

Yes, it does appear the QHY8L has IR block:
http://qhyccd.com/ccdbbs/index.php?topic=2972.0

This is a bit of a mystery. 

I'll take a look if you can post a light or two and the master calibration files (flat, dark, bias) in Dropbox or similar.  Make them FITS or TIF files so that there is no loss of data.

Mark

[Carole]

I think they are already in dropbox Mark as some-one else wanted to practice on them on Astronomy shed.

I've shared the folders with you on Dropbox. 

Carole

[RobertM]

That flat does not look right !  Could you post an UNSTRETCED version of both master and a flat I'll take a look at it, , if not resolved before I get back, tomorrow night.

[MarkS]

Carole,

There's a problem with your master dark.

The individual darks have 16 bit integer pixel values of around 2600 but the master dark has 16 bit pixel values of around only 300

The master bias is fine - it has values of around 2300 which is the same as found in the individual offset frames

Using that faulty master dark causes the over-compensation you are seeing.

You need to review what you did to create the master dark.

Mark

[MarkS]

That flat does not look right !  Could you post an UNSTRETCED version of both master and a flat I'll take a look at it, , if not resolved before I get back, tomorrow night.

The flat was OK - it was simply stretched to show the dust.

[MarkS]

Thinking about this again, it appears your master dark has already had the master bias subtracted from it, since 2600 - 2300 = 300.  That might be right or it might be wrong - it depends on what your stacking program expects.

What program or programs are you using to create the masters and perform the calibration and stacking?
Do you process using the masters or the original bias, darks and flats?

[Carole]

I used DSS Mark.  I don't use masters to stack always the individual fits files.  

Sorry, I didn't realise there were master stacks in the folders, it could be due to the person I previously shared the files with uploading them to their stacking software direct from Dropbox, in which case ignore those and just use the individual files as the masters are not the result of my stacking.  

Carole

[The Thing]

I used DSS Mark.  I don't use masters to stack always the individual fits files. 

Sorry, I didn't realise there were master stacks in the folders, it could be due to the person I previously shared the files with uploading them to their stacking software direct from Dropbox, in which case ignore those and just use the individual files as the masters are not the result of my stacking. 

Carole

With DSS you should stack with all your flats and darks and bias frames for the first pass only. Next time you deselect those files (make sure you save a file list) and select the Masters that are always created by DSS (and which are always TIF files) to use instead. It makes the subsequent stacking runs where you are tweaking kappa-sigma etc. much much faster as all the calibration preprocessing has been done.

[Carole]

I rarely need to stack more than once Duncan and speed doesn't bother me.  I am gradually going over to AstroArt now in any case. 

Carole

[MarkS]

Carole,

I put your files into DSS and it all worked fine - no artifacts.  So I wonder what the difference is?

Mark

[Fay]

Gosh Carole all that dirt in your imaging train, i think you need a hoover on it!!!

[Carole]

I put your files into DSS and it all worked fine - no artifacts.  So I wonder what the difference is?

I have no idea Mark, unless you used different settings to me. 

I know you don't normally use DSS, but do you know what settings you used, what recommended settings and what stacking parameters you used?

I have had this problem consistently using this camera and there has to be something different if we're using the same data and same software.  I have been told to use GBRG for the generic settings for this camera.  Also what version of DSS did you use?

Carole

[MarkS]

I was using version 3.0.1 and just treated them as mono images.  But that is sufficient to check your calibration files are OK.

I'll download a more recent version and try again.

[Carole]

I've got version 3.3.2.  I just did a mono stack, it's hard to see the artifacts in Mono but they are still there.  I accidentally (for the first time ever) loaded the darks as dark flats.

I am going to try to make some new flats with the camera today (complicated to set up as everything is indoors for the storm) and see if the clean I had of the IR glass in front of the chip makes any difference.  It would be lovely if I have removed the offending particles, but still doesn't solve the mystery.

Carole

[Carole]

Also stacked in AstroArt and the artifacts are still there.

Carole

[MarkS]

I downloaded DSS 3.3.2 and it gives complete rubbish (after multiple attempts) with severely clipped and posterised results as soon as I try to calibrate one frame (using a single flat, single dark and single bias).
However, GBRG is definitely the correct colour balance - I checked this by not using the calibration frames.

So I checked in more detail using another program and an odd thing I have noticed about your files is that the lights and flats have a dark stripe down the left side but the darks and bias don't have these stripes.  The fact that some files have stripes and others don't, causes negative numbers at some points during the calibration of frames and maybe this is what upsets DSS.  It might also upset other software but IRIS copes OK.

It would be interesting to know why some frames have the dark stripe and others don't.  It might be a question for the manufacturers.  It makes no sense to me.  I'm sure this is what is causing your calibration issues.

Mark

[Carole]

severely clipped and posterised results as soon as I try to calibrate one frame

That's very interesting Mark as I have been recently having this problem myself, and indeed if you may recall when I was in France I was unable to process some of my Ha images because of posterisation using DSS (v 3.3.0) and Olly had to stack them for me in AstroArt.  Since I came back I have had similar experiences with some of the filters with my desktop version of DSS V 3.3.2, had to use AstroArt, but this is with a mono Atik camera and not the OSC.  Yet I have never had this problem before, and I am not prone to updating my software (I always think if it works don't risk messing things up), so I wonder why this version has suddenly start producing posterisation problems, though I in the past used to have problems with DSS not subtracting flats from my DSLR images.

Regarding the stripe down the side, this is a feature of the QHY8L it's called overscan, I just checked the fits files for darks and Bias and I think it is there but less obvious because they are black.  

I wonder whether I should update DSS but the fact that I still get the artifacts left behind in Astroart doesn't suggest this particular issue is a DSS problem.

I've tried flats again after cleaning the IR glass but the artifacts are still present, so I took the bull by the horns and cleaned the CCD chip using the QHY instructions.  I haven;t yet re-tested it.

Carole

[Carole]

Hmmm, it seems DSS v 3.3.2 is the latest version and no previous versions on offer.

Carole

[MarkS]

I don't understand why the lights and flats suffer from obvious overscan but the darks and bias don't.

My suspicion is that DSS 3.2.2 is not coping with the overscan region.  I'll run some tests to test this hypothesis.  IRIS has no trouble and nor does PixInsight and if I do it fully manually there is no problem either.

[MarkS]

I cropped the FITS files in PixInsight to remove all the overscanned regions and ran them through DSS again.

I got exactly the same clipping and posterisation problems again.  I've tried setting both "Generic GBRG" and "QHY8" as the camera type for the FITS files but it made no difference. 

I've run out of ideas and patience with DSS now!

Mark

[Carole]

Thanks for all your help Mark, I think if you can't solve it then it is unlikely I am going to find the answer.  Gonna have to find a way around it somehow, or just carry on processing out those artifacts.

Carole

[MarkS]

Sorry - I meant I was using 3.3.2

I've now downloaded 3.3.3 Beta 51 and still see the same posterisation.  I think this is a combination of 2 problems:

Problem 1:
Carole, I think you have a problem with your darks.  The ADU count in the dark files is 2600 whilst the bias files are around 2300.  So the dark current is giving an ADU count of 2600-2300 = 300 in 600seconds.  I think the gain on the QHY8 is around 0.5e/ADU so that means the dark current is 150 electrons in 600 seconds.  That is about the same as my uncooled Canon 350D.  It tells me that the camera's cooling was not operating when the darks were taken.

Problem 2:
I'm beginning to think there is something strange happening in DSS's calibration calculations.  Maybe I've got a setting wrong somewhere because I now have a set of files that ought to produce sensible results but I'm still seeing clipping to black and posterisation. I might design some test files to put through DSS to reverse engineer its calculations and determine once and for all if anything is amiss.

Mark

[Carole]

Thanks for all your work Mark.  The darks are old ones from the library, I could run another set and see what I get. 

APT temperature control wasn't as good as Artemis when I did these and normally I watch the exif readings, but being darks I might have walked away.

Carole

[MarkS]

I forgot to say Carole, that in addition, I don't understand why AstroArt is giving you those bright patches.   I suspect that it is being upset by your faulty darks and is trying to cope as best it can.  Shoot a new set of darks and you might see a big improvement.

BTW,  your file names indicate that your bias frames are shot at 0.1s.  Is that the fastest shutter speed available?  If not then you should also shoot a new set of bias frames at a faster speed.

I don't think this will solve the DSS problem but it might give you much better calibration in AA.

Mark

[Carole]

Thanks Mark, I'll see how low APT will go in shooting Bias. 

Not had a chance yet to do another set of darks, I was puzzled too by the bright spot in the centre as I didn't get that with the first stack I did with DSS on the laptop.  All very weird.

I also don't understand why I am getting the posterization problems with various bits of stacking in DSS with the Atik camera, but that's a different issue.  I'd switch over permanently to AA except sometimes strangely DSS does a better job than AA, so I use whichever one works best on each object.

Carole

 

[MarkS]

I also don't understand why I am getting the posterization problems with various bits of stacking in DSS with the Atik camera, but that's a different issue. 

I'm hoping to get some time tonight to investigate the DSS "background clipping" and "star posterisation" problems by generating some synthetic test files.  It will allow me to check the calibration calculations that DSS is performing.

[Mike]

I'm sure APT automatically chooses the shortest exposure for Bias frames. Same as when doing flts it automatically chooses AV mode.

[JohnP]

Blimey Mark its a good job you don't charge by the hour Carole would owe you £1,000's...

Carole - no idea what's causing all your problems but if Mark can't solve it I doubt there's a man that can - have to agree with Fay though need to give your kit a bit of a clean...

[Carole]

I've cleaned it John, I need to take some more flats to see if it's made any difference.  Cleaning the IR glass did not get rid of the artifacts, so I cleaned the chip as per QHY instructions. 

Carole

[MarkS]

I've solved the " background clipping" and posterisation problem I was seeing with DSS.

Experiment shows that if you provide only one dark, bias or flat then DSS assumes they must be the Master dark, bias or flat.  The trouble is that it assumes the Master dark has already had the Master bias subtracted because when DSS creates Master darks, it subtracts the Master bias.

So if you provide only one of each file it does the following:
  calibrated light = (light - dark - bias)/(flat-bias)

So the bias gets subtracted twice (because the bias is already inherent in the dark) and hence the background clipping.

I was providing single calibration files for speed!

However, Carole was seeing background clipping for a different reason:  the darks had higher pixel values than the lights - possibly because the cooling was not operational for one reason or another or possibly for some other odd reason, yet to be determined.

So that's that almost sorted!

[Fay]

 

[Carole]

You are so good Mark, you never let a problem beat you.  I thought something else was going on because I kept getting a message from Dropbox saying various master files had been updated, I had a feeling you were doing some more experimentation.

Experiment shows that if you provide only one dark, bias or flat then DSS assumes they must be the Master dark, bias or flat.  The trouble is that it assumes the Master dark has already had the Master bias subtracted because when DSS creates Master darks, it subtracts the Master bias.

when I was in France I was using Ollys stock master darks (probably claibrated in AA), so I have no idea whether bias had already been subtracted, however it was only the Ha I had problems with posterisation, no problems with any of the other filters. 

I've had rather a hectic few days and still have for the next couple of days, so will re-shoot those flats and bias over the weekend. 

I'm sure APT automatically chooses the shortest exposure for Bias frames. Same as when doing flts it automatically chooses AV mode.

I have a feeling that I forgot how to create a new file from the list in APT (i.e. when no imaging plan has been set before in the Bias) and just experimented with length of subs for the Bias in the darks menu, I've now remembered how to do that so will do them again.

Carole

[MarkS]

You are so good Mark, you never let a problem beat you. 

I'm obstinate, that's all.  Sometimes it's a good thing and sometimes it's a bad thing. 

[RobertM]

So, let me get this straight - it was operator error 

[Carole]

So, let me get this straight - it was operator error

No Robert, Mark hasn't got to the bottom of the artifacts problem, only another DSS stacking problem he found along the way.

Carole

[MarkS]

So, let me get this straight - it was operator error 

In my defence, nowhere does it say or warn you that it treats single calibration files any differently to sets of calibration files.

[RobertM]

True, but it should do. I think I tripped over a similar thing when I used DSS for calibration but it never gave me good results and took too long so I binned it in frustration.  I'd have expected the software to have improved a lot since then but even so the documentation of Astro software (especially freeware) leaves a lot to be desired.

We all fall into these traps from time to time, it's part of the fun of it

Robert

[MarkS]

Similarly with me.  I stopped using it a long time time ago because of inconsistent and incorrect results.   Maybe this partly explains those previous problems.