Coverage for python/lsst/ip/diffim/zogy.py : 84%

Hot-keys on this page
r m x p toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
# # LSST Data Management System # Copyright 2016 AURA/LSST. # # This product includes software developed by the # LSST Project (http://www.lsst.org/). # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the LSST License Statement and # the GNU General Public License along with this program. If not, # see <https://www.lsstcorp.org/LegalNotices/>. #
ImageMapReduceTask) subtractAlgorithmRegistry)
"ZogyMapper", "ZogyMapReduceConfig", "ZogyImagePsfMatchConfig", "ZogyImagePsfMatchTask"]
"""Tasks for performing the "Proper image subtraction" algorithm of Zackay, et al. (2016), hereafter simply referred to as 'ZOGY (2016)'.
`ZogyTask` contains methods to perform the basic estimation of the ZOGY diffim `D`, its updated PSF, and the variance-normalized likelihood image `S_corr`. We have implemented ZOGY using the proscribed methodology, computing all convolutions in Fourier space, and also variants in which the convolutions are performed in real (image) space. The former is faster and results in fewer artifacts when the PSFs are noisy (i.e., measured, for example, via `PsfEx`). The latter is presumed to be preferred as it can account for masks correctly with fewer "ringing" artifacts from edge effects or saturated stars, but noisy PSFs result in their own smaller artifacts. Removal of these artifacts is a subject of continuing research. Currently, we "pad" the PSFs when performing the subtractions in real space, which reduces, but does not entirely eliminate these artifacts.
All methods in `ZogyTask` assume template and science images are already accurately photometrically and astrometrically registered.
`ZogyMapper` is a wrapper which runs `ZogyTask` in the `ImageMapReduce` framework, computing of ZOGY diffim's on small, overlapping sub-images, thereby enabling complete ZOGY diffim's which account for spatially-varying noise and PSFs across the two input exposures. An example of the use of this task is in the `testZogy.py` unit test. """
"""Configuration parameters for the ZogyTask """ dtype=bool, default=False, doc="Perform all convolutions in real (image) space rather than Fourier space. " "Currently if True, this results in artifacts when using real (noisy) PSFs." )
dtype=int, default=7, doc="Number of pixels to pad PSFs to avoid artifacts (when inImageSpace is True)" )
dtype=float, default=1., doc="Template flux scaling factor (Fr in ZOGY paper)" )
dtype=float, default=1., doc="Science flux scaling factor (Fn in ZOGY paper)" )
dtype=bool, default=False, doc="Trim kernels for image-space ZOGY. Speeds up convolutions and shrinks artifacts. " "Subject of future research." )
dtype=bool, default=True, doc="Filter PSFs for image-space ZOGY. Aids in reducing artifacts. " "Subject of future research." )
dtype=str, default=("INTRP", "EDGE", "DETECTED", "SAT", "CR", "BAD", "NO_DATA", "DETECTED_NEGATIVE"), doc="Mask planes to ignore for statistics" )
"""Task to perform ZOGY proper image subtraction. See module-level documentation for additional details.
In all methods, im1 is R (reference, or template) and im2 is N (new, or science). """
psf1=None, psf2=None, *args, **kwargs): """Create the ZOGY task.
Parameters ---------- templateExposure : `lsst.afw.image.Exposure` Template exposure ("Reference image" in ZOGY (2016)). scienceExposure : `lsst.afw.image.Exposure` Science exposure ("New image" in ZOGY (2016)). Must have already been registered and photmetrically matched to template. sig1 : `float` (Optional) sqrt(variance) of `templateExposure`. If `None`, it is computed from the sqrt(mean) of the `templateExposure` variance image. sig2 : `float` (Optional) sqrt(variance) of `scienceExposure`. If `None`, it is computed from the sqrt(mean) of the `scienceExposure` variance image. psf1 : 2D `numpy.array` (Optional) 2D array containing the PSF image for the template. If `None`, it is extracted from the PSF taken at the center of `templateExposure`. psf2 : 2D `numpy.array` (Optional) 2D array containing the PSF image for the science img. If `None`, it is extracted from the PSF taken at the center of `scienceExposure`. args : additional arguments to be passed to `lsst.pipe.base.task.Task.__init__` kwargs : additional keyword arguments to be passed to `lsst.pipe.base.task.Task.__init__` """ sig1=sig1, sig2=sig2, psf1=psf1, psf2=psf2, *args, **kwargs)
psf1=None, psf2=None, correctBackground=False, *args, **kwargs): """Set up the ZOGY task.
Parameters ---------- templateExposure : `lsst.afw.image.Exposure` Template exposure ("Reference image" in ZOGY (2016)). scienceExposure : `lsst.afw.image.Exposure` Science exposure ("New image" in ZOGY (2016)). Must have already been registered and photmetrically matched to template. sig1 : `float` (Optional) sqrt(variance) of `templateExposure`. If `None`, it is computed from the sqrt(mean) of the `templateExposure` variance image. sig2 : `float` (Optional) sqrt(variance) of `scienceExposure`. If `None`, it is computed from the sqrt(mean) of the `scienceExposure` variance image. psf1 : 2D `numpy.array` (Optional) 2D array containing the PSF image for the template. If `None`, it is extracted from the PSF taken at the center of `templateExposure`. psf2 : 2D `numpy.array` (Optional) 2D array containing the PSF image for the science img. If `None`, it is extracted from the PSF taken at the center of `scienceExposure`. correctBackground : `bool` (Optional) subtract sigma-clipped mean of exposures. Zogy doesn't correct nonzero backgrounds (unlike AL) so subtract them here. args : additional arguments to be passed to `lsst.pipe.base.task.Task.__init__` kwargs : additional keyword arguments to be passed to `lsst.pipe.base.task.Task.__init__` """ return return
self.config.ignoreMaskPlanes))
else:
# Make sure PSFs are the same size. Messy, but should work for all cases.
# PSFs' centers may be offset relative to each other; now fix that! # *Very* rarely happens but if they're off by >1 pixel, do it more than once.
# Make sure there are no div-by-zeros
# if sig1 or sig2 are NaN, then the entire region being Zogy-ed is masked. # Don't worry about it - the result will be masked but avoid warning messages. self.sig1 = 1. self.sig2 = 1.
# Zogy doesn't correct nonzero backgrounds (unlike AL) so subtract them here. def _subtractImageMean(exposure): """Compute the sigma-clipped mean of the image of `exposure`.""" mi = exposure.getMaskedImage() statObj = afwMath.makeStatistics(mi.getImage(), mi.getMask(), afwMath.MEANCLIP, self.statsControl) mean = statObj.getValue(afwMath.MEANCLIP) if not np.isnan(mean): mi -= mean
_subtractImageMean(self.template) _subtractImageMean(self.science)
"""Compute the sigma-clipped mean of the variance image of `exposure`. """ exposure.getMaskedImage().getMask(), afwMath.MEANCLIP, self.statsControl)
def _padPsfToSize(psf, size): """Zero-pad `psf` to the dimensions given by `size`.
Parameters ---------- psf : 2D `numpy.array` Input psf to be padded size : `list` Two element list containing the dimensions to pad the `psf` to
Returns ------- psf : 2D `numpy.array` The padded copy of the input `psf`. """
"""Compute standard ZOGY quantities used by (nearly) all methods.
Many of the ZOGY calculations require similar quantities, including FFTs of the PSFs, and the "denominator" term (e.g. in eq. 13 of ZOGY manuscript (2016). This function consolidates many of those operations.
Parameters ---------- psf1 : 2D `numpy.array` (Optional) Input psf of template, override if already padded psf2 : 2D `numpy.array` (Optional) Input psf of science image, override if already padded
Returns ------- A lsst.pipe.base.Struct containing: - Pr : 2D `numpy.array`, the (possibly zero-padded) template PSF - Pn : 2D `numpy.array`, the (possibly zero-padded) science PSF - Pr_hat : 2D `numpy.array`, the FFT of `Pr` - Pn_hat : 2D `numpy.array`, the FFT of `Pn` - denom : 2D `numpy.array`, the denominator of equation (13) in ZOGY (2016) manuscript - Fd : `float`, the relative flux scaling factor between science and template """ # Make sure there are no div-by-zeros
Pr=Pr, Pn=Pn, Pr_hat=Pr_hat, Pn_hat=Pn_hat, denom=denom, Fd=Fd )
# In all functions, im1 is R (reference, or template) and im2 is N (new, or science) """Compute ZOGY diffim `D` as proscribed in ZOGY (2016) manuscript
Compute the ZOGY eqn. (13): $$ \widehat{D} = \frac{Fr\widehat{Pr}\widehat{N} - F_n\widehat{Pn}\widehat{R}}{\sqrt{\sigma_n^2 Fr^2 |\widehat{Pr}|^2 + \sigma_r^2 F_n^2 |\widehat{Pn}|^2}} $$ where $D$ is the optimal difference image, $R$ and $N$ are the reference and "new" image, respectively, $Pr$ and $P_n$ are their PSFs, $Fr$ and $Fn$ are their flux-based zero-points (which we will set to one here), $\sigma_r^2$ and $\sigma_n^2$ are their variance, and $\widehat{D}$ denotes the FT of $D$.
Returns ------- A `lsst.pipe.base.Struct` containing: - D : 2D `numpy.array`, the proper image difference - D_var : 2D `numpy.array`, the variance image for `D` """ # Do all in fourier space (needs image-sized PSFs)
# Filter the wings of Kn, Kr, set to zero ps = trim_amount K[:ps, :] = K[-ps:, :] = 0 K[:, :ps] = K[:, -ps:] = 0 return K
# Suggestion from Barak to trim Kr and Kn to remove artifacts # Here we just filter them (in image space) to keep them the same size ps = (Kn_hat.shape[1] - 80)//2 Kn = _filterKernel(np.fft.ifft2(Kn_hat), ps) Kn_hat = np.fft.fft2(Kn) Kr = _filterKernel(np.fft.ifft2(Kr_hat), ps) Kr_hat = np.fft.fft2(Kr)
# Some masked regions are NaN or infinite!, and FFTs no likey.
else:
R = np.fft.ifft2(D_hat_R) R = np.fft.ifftshift(R.real) / preqs.Fd
# First do the image # Do the exact same thing to the var images, except add them
"""! Convolve an Exposure with a decorrelation convolution kernel.
Parameters ---------- exposure : `lsst.afw.image.Exposure` to be convolved. kernel : 2D `numpy.array` to convolve the image with
Returns ------- A new `lsst.afw.image.Exposure` with the convolved pixels and the (possibly re-centered) kernel.
Notes ----- - We optionally re-center the kernel if necessary and return the possibly re-centered kernel """ maxloc = np.unravel_index(np.argmax(kernel), kernel.shape) kern.setCtrX(maxloc[0]) kern.setCtrY(maxloc[1]) maxInterpolationDistance=0) # Allow exposure to actually be an image/maskedImage # (getMaskedImage will throw AttributeError in that case)
"""Compute ZOGY diffim `D` using image-space convlutions
This method is still being debugged as it results in artifacts when the PSFs are noisy (see module-level docstring). Thus there are several options still enabled by the `debug` flag, which are disabled by defult.
Parameters ---------- padSize : `int`, the amount to pad the PSFs by debug : `bool`, flag to enable debugging tests and options
Returns ------- D : `lsst.afw.Exposure` the proper image difference, including correct variance, masks, and PSF """
delta = 1. # Regularize the ratio, a possible option to remove artifacts
# Trim out the wings of Kn, Kr (see notebook #15) # only necessary if it's from a measured psf and PsfEx seems to always make PSFs of size 41x41 ps = trim_amount K = K[ps:-ps, ps:-ps] return K
# Enabling this block (debug=True) makes it slightly faster, but ~25% worse artifacts: # Filtering also makes it slightly faster (zeros are ignored in convolution) # but a bit worse. Filter the wings of Kn, Kr (see notebook #15) Kn = _trimKernel(Kn, padSize) Kr = _trimKernel(Kr, padSize)
# Note these are reverse-labelled, this is CORRECT!
"""Utility method to set an exposure's PSF when provided as a 2-d numpy.array """
returnMatchedTemplate=False, **kwargs): """Wrapper method to compute ZOGY proper diffim
This method should be used as the public interface for computing the ZOGY diffim.
Parameters ---------- inImageSpace : `bool` Override config `inImageSpace` parameter padSize : `int` Override config `padSize` parameter returnMatchedTemplate : bool Include the PSF-matched template in the results Struct **kwargs : `dict` additional keyword arguments to be passed to `computeDiffimFourierSpace` or `computeDiffimImageSpace`.
Returns ------- An lsst.pipe.base.Struct containing: - D : `lsst.afw.Exposure` the proper image difference, including correct variance, masks, and PSF - R : `lsst.afw.Exposure` If `returnMatchedTemplate` is True, the PSF-matched template exposure """ R = res.R else: R = self.science.clone() R.getMaskedImage().getImage().getArray()[:, :] = res.R R.getMaskedImage().getVariance().getArray()[:, :] = res.R_var
"""Compute the ZOGY diffim PSF (ZOGY manuscript eq. 14)
Parameters ---------- padSize : `int` Override config `padSize` parameter keepFourier : `bool` Return the FFT of the diffim PSF (do not inverse-FFT it) psf1 : 2D `numpy.array` (Optional) Input psf of template, override if already padded psf2 : 2D `numpy.array` (Optional) Input psf of science image, override if already padded
Returns ------- Pd : 2D `numpy.array`, the diffim PSF (or FFT of PSF if `keepFourier=True`) """
R_hat=None, Kr_hat=None, Kr=None, N_hat=None, Kn_hat=None, Kn=None): """Compute the astrometric noise correction terms
Compute the correction for estimated astrometric noise as proscribed in ZOGY (2016), section 3.3. All convolutions performed either in real (image) or Fourier space.
Parameters ---------- xVarAst, yVarAst : `float` estimated astrometric noise (variance of astrometric registration errors) inImageSpace : `bool` Perform all convolutions in real (image) space rather than Fourier space R_hat : 2-D `numpy.array` (Optional) FFT of template image, only required if `inImageSpace=False` Kr_hat : 2-D `numpy.array` FFT of Kr kernel (eq. 28 of ZOGY (2016)), only required if `inImageSpace=False` Kr : 2-D `numpy.array` Kr kernel (eq. 28 of ZOGY (2016)), only required if `inImageSpace=True`. Kr is associated with the template (reference). N_hat : 2-D `numpy.array` FFT of science image, only required if `inImageSpace=False` Kn_hat : 2-D `numpy.array` FFT of Kn kernel (eq. 29 of ZOGY (2016)), only required if `inImageSpace=False` Kn : 2-D `numpy.array` Kn kernel (eq. 29 of ZOGY (2016)), only required if `inImageSpace=True`. Kn is associated with the science (new) image.
Returns ------- VastSR, VastSN : 2-D `numpy.array`s containing the values in eqs. 30 and 32 of ZOGY (2016). """ else:
else:
"""Compute corrected likelihood image, optimal for source detection
Compute ZOGY S_corr image. This image can be thresholded for detection without optimal filtering, and the variance image is corrected to account for astrometric noise (errors in astrometric registration whether systematic or due to effects such as DCR). The calculations here are all performed in Fourier space, as proscribed in ZOGY (2016).
Parameters ---------- xVarAst, yVarAst : `float` estimated astrometric noise (variance of astrometric registration errors)
Returns ------- A lsst.pipe.base.Struct containing: - S : `numpy.array`, the likelihood image S (eq. 12 of ZOGY (2016)) - S_var : the corrected variance image (denominator of eq. 25 of ZOGY (2016)) - Dpsf : the PSF of the diffim D, likely never to be used. """ # Some masked regions are NaN or infinite!, and FFTs no likey. """Replace any NaNs or Infs with the mean of the image.""" im[isbad] = np.nan im[isbad] = np.nanmean(im)
# Do all in fourier space (needs image-sized PSFs)
# Compute D_hat here (don't need D then, for speed)
# Adjust the variance planes of the two images to contribute to the final detection # (eq's 26-29).
# Do the astrometric variance correction R_hat=R_hat, Kr_hat=Kr_hat, N_hat=N_hat, Kn_hat=Kn_hat)
"""Compute corrected likelihood image, optimal for source detection
Compute ZOGY S_corr image. This image can be thresholded for detection without optimal filtering, and the variance image is corrected to account for astrometric noise (errors in astrometric registration whether systematic or due to effects such as DCR). The calculations here are all performed in Real (image) space.
Parameters ---------- xVarAst, yVarAst : `float` estimated astrometric noise (variance of astrometric registration errors)
Returns ------- a tuple containing: - S : `lsst.afw.image.Exposure`, the likelihood exposure S (eq. 12 of ZOGY (2016)), including corrected variance, masks, and PSF - D : `lsst.afw.image.Exposure`, the proper image difference, including correct variance, masks, and PSF """ # Do convolutions in image space
# Adjust the variance planes of the two images to contribute to the final detection # (eq's 26-29).
# Do the astrometric variance correction Kr=Kr, Kn=Kn)
# also return diffim since it was calculated and might be desired
"""Wrapper method to compute ZOGY corrected likelihood image, optimal for source detection
This method should be used as the public interface for computing the ZOGY S_corr.
Parameters ---------- xVarAst, yVarAst : float estimated astrometric noise (variance of astrometric registration errors) inImageSpace : bool Override config `inImageSpace` parameter padSize : int Override config `padSize` parameter
Returns ------- S : lsst.afw.image.Exposure, the likelihood exposure S (eq. 12 of ZOGY (2016)), including corrected variance, masks, and PSF """ else:
"""Task to be used as an ImageMapper for performing ZOGY image subtraction on a grid of subimages. """
**kwargs): """Perform ZOGY proper image subtraction on sub-images
This method performs ZOGY proper image subtraction on `subExposure` using local measures for image variances and PSF. `subExposure` is a sub-exposure of the science image. It also requires the corresponding sub-exposures of the template (`template`). The operations are actually performed on `expandedSubExposure` to allow for invalid edge pixels arising from convolutions, which are then removed.
Parameters ---------- subExposure : lsst.afw.image.Exposure the sub-exposure of the diffim expandedSubExposure : lsst.afw.image.Exposure the expanded sub-exposure upon which to operate fullBBox : lsst.afw.geom.BoundingBox the bounding box of the original exposure template : lsst.afw.image.Exposure the template exposure, from which a corresponding sub-exposure is extracted kwargs : additional keyword arguments propagated from `ImageMapReduceTask.run`. These include: - doScorr : bool Compute and return the corrected likelihood image S_corr rather than the proper image difference - inImageSpace : bool Perform all convolutions in real (image) space rather than in Fourier space. This option currently leads to artifacts when using real (measured and noisy) PSFs, thus it is set to `False` by default. These kwargs may also include arguments to be propagated to `ZogyTask.computeDiffim` and `ZogyTask.computeScorr`.
Returns ------- A `lsst.pipe.base.Struct` containing the result of the `subExposure` processing, labelled 'subExposure'. In this case it is either the subExposure of the proper image difference D, or (if `doScorr==True`) the corrected likelihood exposure `S`.
Notes ----- This `run` method accepts parameters identical to those of `ImageMapper.run`, since it is called from the `ImageMapperTask`. See that class for more information. """
# Psf and image for science img (index 2)
# Psf and image for template img (index 1)
else: # for testing, can use the input sigma (e.g., global value for entire exposure) sig1, sig2 = sigmas[0], sigmas[1]
# Sometimes CoaddPsf does this. Make it square. constant_values=0.) constant_values=0.)
# from diffimTests.diffimTests ... return pipeBase.Struct(subExposure=subExposure)
"""Filter a noisy Psf to remove artifacts. Subject of future research.""" # only necessary if it's from a measured psf and PsfEx seems to always make PSFs of size 41x41 psf = psf.copy() psf[psf < 0] = 0 psf[0:10, :] = psf[:, 0:10] = psf[31:41, :] = psf[:, 31:41] = 0 psf /= psf.sum()
# Note this *really* helps for measured psfs.
sig1=sig1, sig2=sig2, psf1=psf1b, psf2=psf2b, config=config)
else:
"""Config to be passed to ImageMapReduceTask
This config targets the imageMapper to use the ZogyMapper. """ doc='Zogy task to run on each sub-image', target=ZogyMapper )
"""Config for the ZogyImagePsfMatchTask"""
dtype=ZogyConfig, doc='ZogyTask config to use when running on complete exposure (non spatially-varying)', )
dtype=ZogyMapReduceConfig, doc='ZogyMapReduce config to use when running Zogy on each sub-image (spatially-varying)', )
"""Task to perform Zogy PSF matching and image subtraction.
This class inherits from ImagePsfMatchTask to contain the _warper subtask and related methods. """
"""Compute the sigma-clipped mean of the pixels image of `exposure`. """ exposure.getMaskedImage().getMask(), afwMath.MEANCLIP | afwMath.MEDIAN, statsControl)
doWarping=True, spatiallyVarying=True, inImageSpace=False, doPreConvolve=False): """Register, PSF-match, and subtract two Exposures using the ZOGY algorithm.
Do the following, in order: - Warp templateExposure to match scienceExposure, if their WCSs do not already match - Compute subtracted exposure ZOGY image subtraction algorithm on the two exposures
Parameters ---------- templateExposure : `lsst.afw.image.Exposure` exposure to PSF-match to scienceExposure. The exposure's mean value is subtracted in-place. scienceExposure : `lsst.afw.image.Exposure` reference Exposure. The exposure's mean value is subtracted in-place. doWarping : `bool` what to do if templateExposure's and scienceExposure's WCSs do not match: - if True then warp templateExposure to match scienceExposure - if False then raise an Exception spatiallyVarying : bool If True, perform the operation over a grid of patches across the two exposures inImageSpace : `bool` If True, perform the Zogy convolutions in image space rather than in frequency space. doPreConvolve : `bool` ***Currently not implemented.*** If True assume we are to compute the match filter-convolved exposure which can be thresholded for detection. In the case of Zogy this would mean we compute the Scorr image.
Returns ------- A `lsst.pipe.base.Struct` containing these fields: - subtractedExposure: subtracted Exposure - warpedExposure: templateExposure after warping to match scienceExposure (if doWarping true) """
mi = templateExposure.getMaskedImage() mi -= mn1[0] mi = scienceExposure.getMaskedImage() mi -= mn2[0]
if doWarping: self.log.info("Astrometrically registering template to science image") # Also warp the PSF xyTransform = afwGeom.makeWcsPairTransform(templateExposure.getWcs(), scienceExposure.getWcs()) psfWarped = measAlg.WarpedPsf(templateExposure.getPsf(), xyTransform) templateExposure = self._warper.warpExposure(scienceExposure.getWcs(), templateExposure, destBBox=scienceExposure.getBBox())
templateExposure.setPsf(psfWarped) else: self.log.error("ERROR: Input images not registered") raise RuntimeError("Input images not registered")
return exp.getMaskedImage().getMask()
return exp.getMaskedImage().getImage().getArray()
inImageSpace = True # Override doScorr=doPreConvolve, forceEvenSized=False) # The CoaddPsf, when used for detection does not utilize its spatially-varying # properties; it simply computes the PSF at its getAveragePosition(). # TODO: we need to get it to return the matchedExposure (convolved template) # too, for dipole fitting; but the imageMapReduce task might need to be engineered # for this purpose. else: config=config) else: results = task.computeScorr(inImageSpace=inImageSpace) results.D = results.S
# Make sure masks of input images are propagated to diffim
doWarping=True, spatiallyVarying=True, inImageSpace=False, doPreConvolve=False): raise NotImplementedError
|