lsst.pipe.tasks  16.0-26-g0359d09d+1
propagateVisitFlags.py
Go to the documentation of this file.
1 #!/usr/bin/env python
2 #
3 # LSST Data Management System
4 # Copyright 2014-2015 LSST/AURA
5 #
6 # This product includes software developed by the
7 # LSST Project (http://www.lsst.org/).
8 #
9 # This program is free software: you can redistribute it and/or modify
10 # it under the terms of the GNU General Public License as published by
11 # the Free Software Foundation, either version 3 of the License, or
12 # (at your option) any later version.
13 #
14 # This program is distributed in the hope that it will be useful,
15 # but WITHOUT ANY WARRANTY; without even the implied warranty of
16 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
17 # GNU General Public License for more details.
18 #
19 # You should have received a copy of the LSST License Statement and
20 # the GNU General Public License along with this program. If not,
21 # see <http://www.lsstcorp.org/LegalNotices/>.
22 #
23 import numpy
24 from lsst.pex.config import Config, Field, DictField
25 from lsst.pipe.base import Task
26 import lsst.afw.geom as afwGeom
27 import lsst.afw.table as afwTable
28 
29 
31  """!Configuration for propagating flags to coadd"""
32  flags = DictField(keytype=str, itemtype=float,
33  default={"calib_psf_candidate": 0.2, "calib_psf_used": 0.2, "calib_psf_reserved": 0.2,
34  "calib_astrometry_used": 0.2, "calib_photometry_used": 0.2,
35  "calib_photometry_reserved": 0.2, },
36  doc=("Source catalog flags to propagate, with the threshold of relative occurrence "
37  "(valid range: [0-1], default is 0.2). Coadd object will have flag set if the "
38  "fraction of input visits in which it is flagged is greater than the threshold."))
39  matchRadius = Field(dtype=float, default=0.2, doc="Source matching radius (arcsec)")
40 
41 
42 
48 
50  r"""!Task to propagate flags from single-frame measurements to coadd measurements
51 
52 \anchor PropagateVisitFlagsTask_
53 
54 \brief Propagate flags from individual visit measurements to coadd measurements
55 
56 \section pipe_tasks_propagateVisitFlagsTask_Contents Contents
57 
58  - \ref pipe_tasks_propagateVisitFlagsTask_Description
59  - \ref pipe_tasks_propagateVisitFlagsTask_Initialization
60  - \ref pipe_tasks_propagateVisitFlagsTask_Config
61  - \ref pipe_tasks_propagateVisitFlagsTask_Use
62  - \ref pipe_tasks_propagateVisitFlagsTask_Example
63 
64 \section pipe_tasks_propagateVisitFlagsTask_Description Description
65 
66 \copybrief PropagateVisitFlagsTask
67 
68 We want to be able to set a flag for sources on the coadds using flags
69 that were determined from the individual visits. A common example is sources
70 that were used for PSF determination, since we do not do any PSF determination
71 on the coadd but use the individual visits. This requires matching the coadd
72 source catalog to each of the catalogs from the inputs (see
73 PropagateVisitFlagsConfig.matchRadius), and thresholding on the number of
74 times a source is flagged on the input catalog.
75 
76 An important consideration in this is that the flagging of sources in the
77 individual visits can be somewhat stochastic, e.g., the same stars may not
78 always be used for PSF determination because the field of view moves slightly
79 between visits, or the seeing changed. We there threshold on the relative
80 occurrence of the flag in the visits (see PropagateVisitFlagsConfig.flags).
81 Flagging a source that is always flagged in inputs corresponds to a threshold
82 of 1, while flagging a source that is flagged in any of the input corresponds
83 to a threshold of 0. But neither of these extrema are really useful in
84 practise.
85 
86 Setting the threshold too high means that sources that are not consistently
87 flagged (e.g., due to chip gaps) will not have the flag propagated. Setting
88 that threshold too low means that random sources which are falsely flagged in
89 the inputs will start to dominate. If in doubt, we suggest making this
90 threshold relatively low, but not zero (e.g., 0.1 to 0.2 or so). The more
91 confidence in the quality of the flagging, the lower the threshold can be.
92 
93 The relative occurrence accounts for the edge of the field-of-view of the
94 camera, but does not include chip gaps, bad or saturated pixels, etc.
95 
96 \section pipe_tasks_propagateVisitFlagsTask_Initialization Initialization
97 
98 Beyond the usual Task initialization, PropagateVisitFlagsTask also requires
99 a schema for the catalog that is being constructed.
100 
101 \section pipe_tasks_propagateVisitFlagsTask_Config Configuration parameters
102 
103 See \ref PropagateVisitFlagsConfig
104 
105 \section pipe_tasks_propagateVisitFlagsTask_Use Use
106 
107 The 'run' method (described below) is the entry-point for operations. The
108 'getCcdInputs' staticmethod is provided as a convenience for retrieving the
109 'ccdInputs' (CCD inputs table) from an Exposure.
110 
111 \copydoc run
112 
113 \section pipe_tasks_propagateVisitFlagsTask_Example Example
114 
115 \code{.py}
116 # Requires:
117 # * butler: data butler, for retrieving the CCD catalogs
118 # * coaddCatalog: catalog of source measurements on the coadd (lsst.afw.table.SourceCatalog)
119 # * coaddExposure: coadd (lsst.afw.image.Exposure)
120 from lsst.pipe.tasks.propagateVisitFlags import PropagateVisitFlagsTask, PropagateVisitFlagsConfig
121 config = PropagateVisitFlagsConfig()
122 config.flags["calib_psf_used"] = 0.3 # Relative threshold for this flag
123 config.matchRadius = 0.5 # Matching radius in arcsec
124 task = PropagateVisitFlagsTask(coaddCatalog.schema, config=config)
125 ccdInputs = task.getCcdInputs(coaddExposure)
126 task.run(butler, coaddCatalog, ccdInputs, coaddExposure.getWcs())
127 \endcode
128 """
129  ConfigClass = PropagateVisitFlagsConfig
130 
131  def __init__(self, schema, **kwargs):
132  Task.__init__(self, **kwargs)
133  self.schema = schema
134  self._keys = dict((f, self.schema.addField(f, type="Flag", doc="Propagated from visits")) for
135  f in self.config.flags)
136 
137  @staticmethod
138  def getCcdInputs(coaddExposure):
139  """!Convenience method to retrieve the CCD inputs table from a coadd exposure"""
140  return coaddExposure.getInfo().getCoaddInputs().ccds
141 
142  def run(self, butler, coaddSources, ccdInputs, coaddWcs):
143  """!Propagate flags from individual visit measurements to coadd
144 
145  This requires matching the coadd source catalog to each of the catalogs
146  from the inputs, and thresholding on the number of times a source is
147  flagged on the input catalog. The threshold is made on the relative
148  occurrence of the flag in each source. Flagging a source that is always
149  flagged in inputs corresponds to a threshold of 1, while flagging a
150  source that is flagged in any of the input corresponds to a threshold of
151  0. But neither of these extrema are really useful in practise.
152 
153  Setting the threshold too high means that sources that are not consistently
154  flagged (e.g., due to chip gaps) will not have the flag propagated. Setting
155  that threshold too low means that random sources which are falsely flagged in
156  the inputs will start to dominate. If in doubt, we suggest making this threshold
157  relatively low, but not zero (e.g., 0.1 to 0.2 or so). The more confidence in
158  the quality of the flagging, the lower the threshold can be.
159 
160  The relative occurrence accounts for the edge of the field-of-view of
161  the camera, but does not include chip gaps, bad or saturated pixels, etc.
162 
163  @param[in] butler Data butler, for retrieving the input source catalogs
164  @param[in,out] coaddSources Source catalog from the coadd
165  @param[in] ccdInputs Table of CCDs that contribute to the coadd
166  @param[in] coaddWcs Wcs for coadd
167  """
168  if len(self.config.flags) == 0:
169  return
170 
171  flags = self._keys.keys()
172  visitKey = ccdInputs.schema.find("visit").key
173  ccdKey = ccdInputs.schema.find("ccd").key
174  radius = self.config.matchRadius*afwGeom.arcseconds
175 
176  self.log.info("Propagating flags %s from inputs" % (flags,))
177 
178  counts = dict((f, numpy.zeros(len(coaddSources), dtype=int)) for f in flags)
179  indices = numpy.array([s.getId() for s in coaddSources]) # Allowing for non-contiguous data
180 
181  # Accumulate counts of flags being set
182  for ccdRecord in ccdInputs:
183  v = ccdRecord.get(visitKey)
184  c = ccdRecord.get(ccdKey)
185  ccdSources = butler.get("src", visit=int(v), ccd=int(c), immediate=True)
186  for sourceRecord in ccdSources:
187  sourceRecord.updateCoord(ccdRecord.getWcs())
188  for flag in flags:
189  # We assume that the flags will be relatively rare, so it is more efficient to match
190  # against a subset of the input catalog for each flag than it is to match once against
191  # the entire catalog. It would be best to have built a kd-tree on coaddSources and
192  # keep reusing that for the matching, but we don't have a suitable implementation.
193  mc = afwTable.MatchControl()
194  mc.findOnlyClosest = False
195  matches = afwTable.matchRaDec(coaddSources, ccdSources[ccdSources.get(flag)], radius, mc)
196  for m in matches:
197  index = (numpy.where(indices == m.first.getId()))[0][0]
198  counts[flag][index] += 1
199 
200  # Apply threshold
201  for f in flags:
202  key = self._keys[f]
203  for s, num in zip(coaddSources, counts[f]):
204  numOverlaps = len(ccdInputs.subsetContaining(s.getCentroid(), coaddWcs, True))
205  s.setFlag(key, bool(num > numOverlaps*self.config.flags[f]))
206  self.log.info("Propagated %d sources with flag %s" % (sum(s.get(key) for s in coaddSources), f))
Task to propagate flags from single-frame measurements to coadd measurements.
def getCcdInputs(coaddExposure)
Convenience method to retrieve the CCD inputs table from a coadd exposure.
Configuration for propagating flags to coadd.
def run(self, butler, coaddSources, ccdInputs, coaddWcs)
Propagate flags from individual visit measurements to coadd.