Coverage for python/lsst/pipe/tasks/propagateVisitFlags.py: 23%

60 statements  

« prev     ^ index     » next       coverage.py v6.4.4, created at 2022-08-18 20:09 +0000

1#!/usr/bin/env python 

2# 

3# LSST Data Management System 

4# Copyright 2014-2015 LSST/AURA 

5# 

6# This product includes software developed by the 

7# LSST Project (http://www.lsst.org/). 

8# 

9# This program is free software: you can redistribute it and/or modify 

10# it under the terms of the GNU General Public License as published by 

11# the Free Software Foundation, either version 3 of the License, or 

12# (at your option) any later version. 

13# 

14# This program is distributed in the hope that it will be useful, 

15# but WITHOUT ANY WARRANTY; without even the implied warranty of 

16# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 

17# GNU General Public License for more details. 

18# 

19# You should have received a copy of the LSST License Statement and 

20# the GNU General Public License along with this program. If not, 

21# see <http://www.lsstcorp.org/LegalNotices/>. 

22# 

23import numpy 

24from lsst.pex.config import Config, Field, DictField 

25from lsst.pipe.base import Task 

26import lsst.geom as geom 

27import lsst.afw.table as afwTable 

28import lsst.pex.exceptions as pexExceptions 

29from deprecated.sphinx import deprecated 

30 

31 

32class PropagateVisitFlagsConfig(Config): 

33 """!Configuration for propagating flags to coadd""" 

34 flags = DictField(keytype=str, itemtype=float, 

35 default={"calib_psf_candidate": 0.2, "calib_psf_used": 0.2, "calib_psf_reserved": 0.2, 

36 "calib_astrometry_used": 0.2, "calib_photometry_used": 0.2, 

37 "calib_photometry_reserved": 0.2, }, 

38 doc=("Source catalog flags to propagate, with the threshold of relative occurrence " 

39 "(valid range: [0-1], default is 0.2). Coadd object will have flag set if the " 

40 "fraction of input visits in which it is flagged is greater than the threshold.")) 

41 matchRadius = Field(dtype=float, default=0.2, doc="Source matching radius (arcsec)") 

42 ccdName = Field(dtype=str, default='ccd', doc="Name of ccd to give to butler") 

43 

44 

45## \addtogroup LSST_task_documentation 

46## \{ 

47## \page page_PropagateVisitFlagsTask PropagateVisitFlagsTask 

48## \ref PropagateVisitFlagsTask_ "PropagateVisitFlagsTask" 

49## \copybrief PropagateVisitFlagsTask 

50## \} 

51 

52@deprecated(reason="This task has been replaced with PropagateSourceFlagsTask", 

53 version="v24.0", category=FutureWarning) 

54class PropagateVisitFlagsTask(Task): 

55 r"""!Task to propagate flags from single-frame measurements to coadd measurements 

56 

57\anchor PropagateVisitFlagsTask_ 

58 

59\brief Propagate flags from individual visit measurements to coadd measurements 

60 

61\section pipe_tasks_propagateVisitFlagsTask_Contents Contents 

62 

63 - \ref pipe_tasks_propagateVisitFlagsTask_Description 

64 - \ref pipe_tasks_propagateVisitFlagsTask_Initialization 

65 - \ref pipe_tasks_propagateVisitFlagsTask_Config 

66 - \ref pipe_tasks_propagateVisitFlagsTask_Use 

67 - \ref pipe_tasks_propagateVisitFlagsTask_Example 

68 

69\section pipe_tasks_propagateVisitFlagsTask_Description Description 

70 

71\copybrief PropagateVisitFlagsTask 

72 

73We want to be able to set a flag for sources on the coadds using flags 

74that were determined from the individual visits. A common example is sources 

75that were used for PSF determination, since we do not do any PSF determination 

76on the coadd but use the individual visits. This requires matching the coadd 

77source catalog to each of the catalogs from the inputs (see 

78PropagateVisitFlagsConfig.matchRadius), and thresholding on the number of 

79times a source is flagged on the input catalog. 

80 

81An important consideration in this is that the flagging of sources in the 

82individual visits can be somewhat stochastic, e.g., the same stars may not 

83always be used for PSF determination because the field of view moves slightly 

84between visits, or the seeing changed. We there threshold on the relative 

85occurrence of the flag in the visits (see PropagateVisitFlagsConfig.flags). 

86Flagging a source that is always flagged in inputs corresponds to a threshold 

87of 1, while flagging a source that is flagged in any of the input corresponds 

88to a threshold of 0. But neither of these extrema are really useful in 

89practise. 

90 

91Setting the threshold too high means that sources that are not consistently 

92flagged (e.g., due to chip gaps) will not have the flag propagated. Setting 

93that threshold too low means that random sources which are falsely flagged in 

94the inputs will start to dominate. If in doubt, we suggest making this 

95threshold relatively low, but not zero (e.g., 0.1 to 0.2 or so). The more 

96confidence in the quality of the flagging, the lower the threshold can be. 

97 

98The relative occurrence accounts for the edge of the field-of-view of the 

99camera, but does not include chip gaps, bad or saturated pixels, etc. 

100 

101\section pipe_tasks_propagateVisitFlagsTask_Initialization Initialization 

102 

103Beyond the usual Task initialization, PropagateVisitFlagsTask also requires 

104a schema for the catalog that is being constructed. 

105 

106\section pipe_tasks_propagateVisitFlagsTask_Config Configuration parameters 

107 

108See \ref PropagateVisitFlagsConfig 

109 

110\section pipe_tasks_propagateVisitFlagsTask_Use Use 

111 

112The 'run' method (described below) is the entry-point for operations. The 

113'getCcdInputs' staticmethod is provided as a convenience for retrieving the 

114'ccdInputs' (CCD inputs table) from an Exposure. 

115 

116\copydoc run 

117 

118\section pipe_tasks_propagateVisitFlagsTask_Example Example 

119 

120\code{.py} 

121# Requires: 

122# * butler: data butler, for retrieving the CCD catalogs 

123# * coaddCatalog: catalog of source measurements on the coadd (lsst.afw.table.SourceCatalog) 

124# * coaddExposure: coadd (lsst.afw.image.Exposure) 

125from lsst.pipe.tasks.propagateVisitFlags import PropagateVisitFlagsTask, PropagateVisitFlagsConfig 

126config = PropagateVisitFlagsConfig() 

127config.flags["calib_psf_used"] = 0.3 # Relative threshold for this flag 

128config.matchRadius = 0.5 # Matching radius in arcsec 

129task = PropagateVisitFlagsTask(coaddCatalog.schema, config=config) 

130ccdInputs = task.getCcdInputs(coaddExposure) 

131task.run(butler, coaddCatalog, ccdInputs, coaddExposure.getWcs()) 

132\endcode 

133""" 

134 ConfigClass = PropagateVisitFlagsConfig 

135 

136 def __init__(self, schema, **kwargs): 

137 Task.__init__(self, **kwargs) 

138 self.schema = schema 

139 self._keys = dict((f, self.schema.addField(f, type="Flag", doc="Propagated from visits")) for 

140 f in self.config.flags) 

141 

142 @staticmethod 

143 def getCcdInputs(coaddExposure): 

144 """!Convenience method to retrieve the CCD inputs table from a coadd exposure""" 

145 return coaddExposure.getInfo().getCoaddInputs().ccds 

146 

147 def run(self, butler, coaddSources, ccdInputs, coaddWcs, visitCatalogs=None, wcsUpdates=None): 

148 """!Propagate flags from individual visit measurements to coadd 

149 

150 This requires matching the coadd source catalog to each of the catalogs 

151 from the inputs, and thresholding on the number of times a source is 

152 flagged on the input catalog. The threshold is made on the relative 

153 occurrence of the flag in each source. Flagging a source that is always 

154 flagged in inputs corresponds to a threshold of 1, while flagging a 

155 source that is flagged in any of the input corresponds to a threshold of 

156 0. But neither of these extrema are really useful in practise. 

157 

158 Setting the threshold too high means that sources that are not consistently 

159 flagged (e.g., due to chip gaps) will not have the flag propagated. Setting 

160 that threshold too low means that random sources which are falsely flagged in 

161 the inputs will start to dominate. If in doubt, we suggest making this threshold 

162 relatively low, but not zero (e.g., 0.1 to 0.2 or so). The more confidence in 

163 the quality of the flagging, the lower the threshold can be. 

164 

165 The relative occurrence accounts for the edge of the field-of-view of 

166 the camera, but does not include chip gaps, bad or saturated pixels, etc. 

167 

168 @param[in] butler Data butler, for retrieving the input source catalogs 

169 @param[in,out] coaddSources Source catalog from the coadd 

170 @param[in] ccdInputs Table of CCDs that contribute to the coadd 

171 @param[in] coaddWcs Wcs for coadd 

172 @param[in] visitCatalogs List of loaded source catalogs for each input ccd in 

173 the coadd. If provided this is used instead of this 

174 method loading in the catalogs itself 

175 @param[in] wcsUpdates optional, If visitCatalogs is a list of ccd catalogs, this 

176 should be a list of updated wcs to apply 

177 """ 

178 if len(self.config.flags) == 0: 

179 return 

180 

181 flags = self._keys.keys() 

182 counts = dict((f, numpy.zeros(len(coaddSources), dtype=int)) for f in flags) 

183 indices = numpy.array([s.getId() for s in coaddSources]) # Allowing for non-contiguous data 

184 radius = self.config.matchRadius*geom.arcseconds 

185 

186 def processCcd(ccdSources, wcsUpdate): 

187 for sourceRecord in ccdSources: 

188 sourceRecord.updateCoord(wcsUpdate) 

189 for flag in flags: 

190 # We assume that the flags will be relatively rare, so it is more efficient to match 

191 # against a subset of the input catalog for each flag than it is to match once against 

192 # the entire catalog. It would be best to have built a kd-tree on coaddSources and 

193 # keep reusing that for the matching, but we don't have a suitable implementation. 

194 mc = afwTable.MatchControl() 

195 mc.findOnlyClosest = False 

196 matches = afwTable.matchRaDec(coaddSources, ccdSources[ccdSources.get(flag)], radius, mc) 

197 for m in matches: 

198 index = (numpy.where(indices == m.first.getId()))[0][0] 

199 counts[flag][index] += 1 

200 

201 if visitCatalogs is not None: 

202 if wcsUpdates is None: 

203 raise pexExceptions.ValueError("If ccdInputs is a list of src catalogs, a list of wcs" 

204 " updates for each catalog must be supplied in the " 

205 "wcsUpdates parameter") 

206 for i, ccdSource in enumerate(visitCatalogs): 

207 processCcd(ccdSource, wcsUpdates[i]) 

208 else: 

209 if ccdInputs is None: 

210 raise pexExceptions.ValueError("The visitCatalogs and ccdInput parameters can't both be None") 

211 visitKey = ccdInputs.schema.find("visit").key 

212 ccdKey = ccdInputs.schema.find("ccd").key 

213 

214 self.log.info("Propagating flags %s from inputs", flags) 

215 

216 # Accumulate counts of flags being set 

217 for ccdRecord in ccdInputs: 

218 v = ccdRecord.get(visitKey) 

219 c = ccdRecord.get(ccdKey) 

220 dataId = {"visit": int(v), self.config.ccdName: int(c)} 

221 ccdSources = butler.get("src", dataId=dataId, immediate=True) 

222 processCcd(ccdSources, ccdRecord.getWcs()) 

223 

224 # Apply threshold 

225 for f in flags: 

226 key = self._keys[f] 

227 for s, num in zip(coaddSources, counts[f]): 

228 numOverlaps = len(ccdInputs.subsetContaining(s.getCentroid(), coaddWcs, True)) 

229 s.setFlag(key, bool(num > numOverlaps*self.config.flags[f])) 

230 self.log.info("Propagated %d sources with flag %s", sum(s.get(key) for s in coaddSources), f)