lsst.meas.base  14.0-19-ga9fd870
forcedMeasurement.py
Go to the documentation of this file.
1 #!/usr/bin/env python
2 #
3 # LSST Data Management System
4 # Copyright 2008-2015 LSST Corporation.
5 #
6 # This product includes software developed by the
7 # LSST Project (http://www.lsst.org/).
8 #
9 # This program is free software: you can redistribute it and/or modify
10 # it under the terms of the GNU General Public License as published by
11 # the Free Software Foundation, either version 3 of the License, or
12 # (at your option) any later version.
13 #
14 # This program is distributed in the hope that it will be useful,
15 # but WITHOUT ANY WARRANTY; without even the implied warranty of
16 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
17 # GNU General Public License for more details.
18 #
19 # You should have received a copy of the LSST License Statement and
20 # the GNU General Public License along with this program. If not,
21 # see <http://www.lsstcorp.org/LegalNotices/>.
22 #
23 """Base classes for forced measurement plugins and the driver task for these.
24 
25 In forced measurement, a reference catalog is used to define restricted measurements (usually just fluxes)
26 on an image. As the reference catalog may be deeper than the detection limit of the measurement image, we
27 do not assume that we can use detection and deblend information from the measurement image. Instead, we
28 assume this information is present in the reference catalog and can be "transformed" in some sense to
29 the measurement frame. At the very least, this means that Footprints from the reference catalog should
30 be transformed and installed as Footprints in the output measurement catalog. If we have a procedure that
31 can transform HeavyFootprints, we can then proceed with measurement as usual, but using the reference
32 catalog's id and parent fields to define deblend families. If this transformation does not preserve
33 HeavyFootprints (this is currently the case, at least for CCD forced photometry), then we will only
34 be able to replace objects with noise one deblend family at a time, and hence measurements run in
35 single-object mode may be contaminated by neighbors when run on objects with parent != 0.
36 
37 Measurements are generally recorded in the coordinate system of the image being measured (and all
38 slot-eligible fields must be), but non-slot fields may be recorded in other coordinate systems if necessary
39 to avoid information loss (this should, of course, be indicated in the field documentation). Note that
40 the reference catalog may be in a different coordinate system; it is the responsibility of plugins
41 to transform the data they need themselves, using the reference WCS provided. However, for plugins
42 that only require a position or shape, they may simply use output SourceCatalog's centroid or shape slots,
43 which will generally be set to the transformed position of the reference object before any other plugins are
44 run, and hence avoid using the reference catalog at all.
45 
46 Command-line driver tasks for forced measurement can be found in forcedPhotImage.py, including
47 ForcedPhotImageTask, ForcedPhotCcdTask, and ForcedPhotCoaddTask.
48 """
49 from builtins import zip
50 
51 import lsst.pex.config
52 import lsst.pipe.base
53 
54 from .pluginRegistry import PluginRegistry
55 from .baseMeasurement import (BaseMeasurementPluginConfig, BaseMeasurementPlugin,
56  BaseMeasurementConfig, BaseMeasurementTask)
57 from .noiseReplacer import NoiseReplacer, DummyNoiseReplacer
58 
59 __all__ = ("ForcedPluginConfig", "ForcedPlugin",
60  "ForcedMeasurementConfig", "ForcedMeasurementTask")
61 
62 
64  """Base class for configs of forced measurement plugins."""
65  pass
66 
67 
69 
70  # All subclasses of ForcedPlugin should be registered here
71  registry = PluginRegistry(ForcedPluginConfig)
72 
73  ConfigClass = ForcedPluginConfig
74 
75  def __init__(self, config, name, schemaMapper, metadata, logName=None):
76  """Initialize the measurement object.
77 
78  @param[in] config An instance of this class's ConfigClass.
79  @param[in] name The string the plugin was registered with.
80  @param[in,out] schemaMapper A SchemaMapper that maps reference catalog fields to output
81  catalog fields. Output fields should be added to the
82  output schema. While most plugins will not need to map
83  fields from the reference schema, if they do so, those fields
84  will be transferred before any plugins are run.
85  @param[in] metadata Plugin metadata that will be attached to the output catalog
86  """
87  BaseMeasurementPlugin.__init__(self, config, name, logName=logName)
88 
89  def measure(self, measRecord, exposure, refRecord, refWcs):
90  """Measure the properties of a source on a single image, given data from a
91  reference record.
92 
93  @param[in] exposure lsst.afw.image.ExposureF, containing the pixel data to
94  be measured and the associated Psf, Wcs, etc. All
95  other sources in the image will have been replaced by
96  noise according to deblender outputs.
97  @param[in,out] measRecord lsst.afw.table.SourceRecord to be filled with outputs,
98  and from which previously-measured quantities can be
99  retreived.
100  @param[in] refRecord lsst.afw.table.SimpleRecord that contains additional
101  parameters to define the fit, as measured elsewhere.
102  @param[in] refWcs The coordinate system for the reference catalog values.
103  An afw.geom.Angle may be passed, indicating that a
104  local tangent Wcs should be created for each object
105  using afw.image.makeLocalWcs and the given angle as
106  a pixel scale.
107 
108  In the normal mode of operation, the source centroid will be set to the
109  WCS-transformed position of the reference object, so plugins that only
110  require a reference position should not have to access the reference object
111  at all.
112  """
113  raise NotImplementedError()
114 
115  def measureN(self, measCat, exposure, refCat, refWcs):
116  """Measure the properties of a group of blended sources on a single image,
117  given data from a reference record.
118 
119  @param[in] exposure lsst.afw.image.ExposureF, containing the pixel data to
120  be measured and the associated Psf, Wcs, etc. Sources
121  not in the blended hierarchy to be measured will have
122  been replaced with noise using deblender outputs.
123  @param[in,out] measCat lsst.afw.table.SourceCatalog to be filled with outputs,
124  and from which previously-measured quantities can be
125  retrieved, containing only the sources that should be
126  measured together in this call.
127  @param[in] refCat lsst.afw.table.SimpleCatalog that contains additional
128  parameters to define the fit, as measured elsewhere.
129  Ordered such that zip(sources, references) may be used.
130  @param[in] refWcs The coordinate system for the reference catalog values.
131  An afw.geom.Angle may be passed, indicating that a
132  local tangent Wcs should be created for each object
133  using afw.image.makeLocalWcs and the given Angle as
134  a pixel scale.
135 
136  In the normal mode of operation, the source centroids will be set to the
137  WCS-transformed position of the reference object, so plugins that only
138  require a reference position should not have to access the reference object
139  at all.
140  """
141  raise NotImplementedError()
142 
143 
145  """Config class for forced measurement driver task."""
146 
147  plugins = ForcedPlugin.registry.makeField(
148  multi=True,
149  default=["base_PixelFlags",
150  "base_TransformedCentroid",
151  "base_SdssCentroid",
152  "base_TransformedShape",
153  "base_SdssShape",
154  "base_GaussianFlux",
155  "base_CircularApertureFlux",
156  "base_PsfFlux",
157  "base_LocalBackground",
158  ],
159  doc="Plugins to be run and their configuration"
160  )
161  algorithms = property(lambda self: self.plugins, doc="backwards-compatibility alias for plugins")
162  undeblended = ForcedPlugin.registry.makeField(
163  multi=True,
164  default=[],
165  doc="Plugins to run on undeblended image"
166  )
167 
168  copyColumns = lsst.pex.config.DictField(
169  keytype=str, itemtype=str, doc="Mapping of reference columns to source columns",
170  default={"id": "objectId", "parent": "parentObjectId", "deblend_nChild": "deblend_nChild",
171  "coord_ra": "coord_ra", "coord_dec": "coord_dec"}
172  )
173 
174  checkUnitsParseStrict = lsst.pex.config.Field(
175  doc="Strictness of Astropy unit compatibility check, can be 'raise', 'warn' or 'silent'",
176  dtype=str,
177  default="raise",
178  )
179 
180  def setDefaults(self):
181  self.slots.centroid = "base_TransformedCentroid"
182  self.slots.shape = "base_TransformedShape"
183  self.slots.apFlux = None
184  self.slots.modelFlux = None
185  self.slots.psfFlux = None
186  self.slots.instFlux = None
187  self.slots.calibFlux = None
188 
189 
195 
196 
198  """!
199  \anchor ForcedMeasurementTask_
200 
201  \brief A subtask for measuring the properties of sources on a single
202  exposure, using an existing "reference" catalog to constrain some aspects
203  of the measurement.
204 
205  The task is configured with a list of "plugins": each plugin defines the values it
206  measures (i.e. the columns in a table it will fill) and conducts that measurement
207  on each detected source (see ForcedPlugin). The job of the
208  measurement task is to initialize the set of plugins (which includes setting up the
209  catalog schema) from their configuration, and then invoke each plugin on each
210  source.
211 
212  Most of the time, ForcedMeasurementTask will be used via one of the subclasses of
213  ForcedPhotImageTask, ForcedPhotCcdTask and ForcedPhotCoaddTask. These combine
214  this measurement subtask with a "references" subtask (see BaseReferencesTask and
215  CoaddSrcReferencesTask) to perform forced measurement using measurements performed on
216  another image as the references. There is generally little reason to use
217  ForcedMeasurementTask outside of one of these drivers, unless it is necessary to avoid
218  using the Butler for I/O.
219 
220  ForcedMeasurementTask has only three methods: __init__(), run(), and generateMeasCat().
221  For configuration options, see SingleFrameMeasurementConfig.
222 
223  Notes
224  -----
225 
226  *Forced* measurement means that the plugins are provided with a reference
227  source containing centroid and/or shape measurements that they may use
228  however they see fit. Some plugins can use these to set the location and
229  size of apertures, but others may choose to ignore this information,
230  essentially performing an unforced measurement starting at the position
231  of the reference source (which may nevertheless be useful for certain
232  investigations). Knowing how the plugin uses the reference information is
233  essential to interpreting its resulting measurements. Typically, centroid
234  and shape measurement plugins (e.g., ``SdssCentroid`` and ``SdssShape``)
235  are performing unforced measurements.
236  """
237 
238  ConfigClass = ForcedMeasurementConfig
239 
240  def __init__(self, refSchema, algMetadata=None, **kwds):
241  """!
242  Initialize the task. Set up the execution order of the plugins and initialize
243  the plugins, giving each plugin an opportunity to add its measurement fields to
244  the output schema and to record information in the task metadata.
245 
246  Note that while SingleFrameMeasurementTask is passed an initial Schema that is
247  appended to in order to create the output Schema, ForcedMeasurementTask is
248  initialized with the Schema of the reference catalog, from which a new Schema
249  for the output catalog is created. Fields to be copied directly from the
250  reference Schema are added before Plugin fields are added.
251 
252  @param[in] refSchema Schema of the reference catalog. Must match the catalog
253  later passed to generateMeasCat() and/or run().
254  @param[in,out] algMetadata lsst.daf.base.PropertyList used to record information about
255  each algorithm. An empty PropertyList will be created if None.
256  @param[in] **kwds Keyword arguments passed from lsst.pipe.base.Task.__init__
257  """
258  super(ForcedMeasurementTask, self).__init__(algMetadata=algMetadata, **kwds)
260  self.mapper.addMinimalSchema(lsst.afw.table.SourceTable.makeMinimalSchema(), False)
261  self.config.slots.setupSchema(self.mapper.editOutputSchema())
262  for refName, targetName in self.config.copyColumns.items():
263  refItem = refSchema.find(refName)
264  self.mapper.addMapping(refItem.key, targetName)
265  self.config.slots.setupSchema(self.mapper.editOutputSchema())
266  self.initializePlugins(schemaMapper=self.mapper)
267  self.schema = self.mapper.getOutputSchema()
268  self.schema.checkUnits(parse_strict=self.config.checkUnitsParseStrict)
269 
270  def run(self, measCat, exposure, refCat, refWcs, exposureId=None, beginOrder=None, endOrder=None):
271  """!
272  Perform forced measurement.
273 
274  @param[in] exposure lsst.afw.image.ExposureF to be measured; must have at least a Wcs attached.
275  @param[in] measCat Source catalog for measurement results; must be initialized with empty
276  records already corresponding to those in refCat (via e.g. generateMeasCat).
277  @param[in] refCat A sequence of SourceRecord objects that provide reference information
278  for the measurement. These will be passed to each Plugin in addition
279  to the output SourceRecord.
280  @param[in] refWcs Wcs that defines the X,Y coordinate system of refCat
281  @param[in] exposureId optional unique exposureId used to calculate random number
282  generator seed in the NoiseReplacer.
283  @param[in] beginOrder beginning execution order (inclusive): measurements with
284  executionOrder < beginOrder are not executed. None for no limit.
285  @param[in] endOrder ending execution order (exclusive): measurements with
286  executionOrder >= endOrder are not executed. None for no limit.
287 
288  Fills the initial empty SourceCatalog with forced measurement results. Two steps must occur
289  before run() can be called:
290  - generateMeasCat() must be called to create the output measCat argument.
291  - Footprints appropriate for the forced sources must be attached to the measCat records. The
292  attachTransformedFootprints() method can be used to do this, but this degrades HeavyFootprints
293  to regular Footprints, leading to non-deblended measurement, so most callers should provide
294  Footprints some other way. Typically, calling code will have access to information that will
295  allow them to provide HeavyFootprints - for instance, ForcedPhotCoaddTask uses the HeavyFootprints
296  from deblending run in the same band just before non-forced is run measurement in that band.
297  """
298  # First check that the reference catalog does not contain any children for which
299  # any member of their parent chain is not within the list. This can occur at
300  # boundaries when the parent is outside and one of the children is within.
301  # Currently, the parent chain is always only one deep, but just in case, this
302  # code checks for any case where when the parent chain to a child's topmost
303  # parent is broken and raises an exception if it occurs.
304  #
305  # I.e. this code checks that this precondition is satisfied by whatever reference
306  # catalog provider is being paired with it.
307  refCatIdDict = {ref.getId(): ref.getParent() for ref in refCat}
308  for ref in refCat:
309  refId = ref.getId()
310  topId = refId
311  while(topId > 0):
312  if topId not in refCatIdDict:
313  raise RuntimeError("Reference catalog contains a child for which at least "
314  "one parent in its parent chain is not in the catalog.")
315  topId = refCatIdDict[topId]
316 
317  # Construct a footprints dict which looks like
318  # {ref.getId(): (ref.getParent(), source.getFootprint())}
319  # (i.e. getting the footprint from the transformed source footprint)
320  footprints = {ref.getId(): (ref.getParent(), measRecord.getFootprint())
321  for (ref, measRecord) in zip(refCat, measCat)}
322 
323  self.log.info("Performing forced measurement on %d source%s", len(refCat),
324  "" if len(refCat) == 1 else "s")
325 
326  if self.config.doReplaceWithNoise:
327  noiseReplacer = NoiseReplacer(self.config.noiseReplacer, exposure,
328  footprints, log=self.log, exposureId=exposureId)
329  algMetadata = measCat.getTable().getMetadata()
330  if algMetadata is not None:
331  algMetadata.addInt("NOISE_SEED_MULTIPLIER", self.config.noiseReplacer.noiseSeedMultiplier)
332  algMetadata.addString("NOISE_SOURCE", self.config.noiseReplacer.noiseSource)
333  algMetadata.addDouble("NOISE_OFFSET", self.config.noiseReplacer.noiseOffset)
334  if exposureId is not None:
335  algMetadata.addLong("NOISE_EXPOSURE_ID", exposureId)
336  else:
337  noiseReplacer = DummyNoiseReplacer()
338 
339  # Create parent cat which slices both the refCat and measCat (sources)
340  # first, get the reference and source records which have no parent
341  refParentCat, measParentCat = refCat.getChildren(0, measCat)
342  for parentIdx, (refParentRecord, measParentRecord) in enumerate(zip(refParentCat, measParentCat)):
343 
344  # first process the records which have the current parent as children
345  refChildCat, measChildCat = refCat.getChildren(refParentRecord.getId(), measCat)
346  # TODO: skip this loop if there are no plugins configured for single-object mode
347  for refChildRecord, measChildRecord in zip(refChildCat, measChildCat):
348  noiseReplacer.insertSource(refChildRecord.getId())
349  self.callMeasure(measChildRecord, exposure, refChildRecord, refWcs,
350  beginOrder=beginOrder, endOrder=endOrder)
351  noiseReplacer.removeSource(refChildRecord.getId())
352 
353  # then process the parent record
354  noiseReplacer.insertSource(refParentRecord.getId())
355  self.callMeasure(measParentRecord, exposure, refParentRecord, refWcs,
356  beginOrder=beginOrder, endOrder=endOrder)
357  self.callMeasureN(measParentCat[parentIdx:parentIdx+1], exposure,
358  refParentCat[parentIdx:parentIdx+1],
359  beginOrder=beginOrder, endOrder=endOrder)
360  # measure all the children simultaneously
361  self.callMeasureN(measChildCat, exposure, refChildCat,
362  beginOrder=beginOrder, endOrder=endOrder)
363  noiseReplacer.removeSource(refParentRecord.getId())
364  noiseReplacer.end()
365 
366  # Undeblended plugins only fire if we're running everything
367  if endOrder is None:
368  for measRecord, refRecord in zip(measCat, refCat):
369  for plugin in self.undeblendedPlugins.iter():
370  self.doMeasurement(plugin, measRecord, exposure, refRecord, refWcs)
371 
372  def generateMeasCat(self, exposure, refCat, refWcs, idFactory=None):
373  """!Initialize an output SourceCatalog using information from the reference catalog.
374 
375  This generates a new blank SourceRecord for each record in refCat. Note that this
376  method does not attach any Footprints. Doing so is up to the caller (who may
377  call attachedTransformedFootprints or define their own method - see run() for more
378  information).
379 
380  @param[in] exposure Exposure to be measured
381  @param[in] refCat Sequence (not necessarily a SourceCatalog) of reference SourceRecords.
382  @param[in] refWcs Wcs that defines the X,Y coordinate system of refCat
383  @param[in] idFactory factory for creating IDs for sources
384 
385  @return Source catalog ready for measurement
386  """
387  if idFactory is None:
389  table = lsst.afw.table.SourceTable.make(self.schema, idFactory)
390  measCat = lsst.afw.table.SourceCatalog(table)
391  table = measCat.table
392  table.setMetadata(self.algMetadata)
393  table.preallocate(len(refCat))
394  for ref in refCat:
395  newSource = measCat.addNew()
396  newSource.assign(ref, self.mapper)
397  return measCat
398 
399  def attachTransformedFootprints(self, sources, refCat, exposure, refWcs):
400  """!Default implementation for attaching Footprints to blank sources prior to measurement
401 
402  Footprints for forced photometry must be in the pixel coordinate system of the image being
403  measured, while the actual detections may start out in a different coordinate system.
404  This default implementation transforms the Footprints from the reference catalog from the
405  refWcs to the exposure's Wcs, which downgrades HeavyFootprints into regular Footprints,
406  destroying deblend information.
407 
408  Note that ForcedPhotImageTask delegates to this method in its own attachFootprints method.
409  attachFootprints can then be overridden by its subclasses to define how their Footprints
410  should be generated.
411 
412  See the documentation for run() for information about the relationships between run(),
413  generateMeasCat(), and attachTransformedFootprints().
414  """
415  exposureWcs = exposure.getWcs()
416  region = exposure.getBBox(lsst.afw.image.PARENT)
417  for srcRecord, refRecord in zip(sources, refCat):
418  srcRecord.setFootprint(refRecord.getFootprint().transform(refWcs, exposureWcs, region))
Base config class for all measurement plugins.
def callMeasure(self, measRecord, args, kwds)
Call the measure() method on all plugins, handling exceptions in a consistent way.
def __init__(self, refSchema, algMetadata=None, kwds)
Initialize the task.
A subtask for measuring the properties of sources on a single exposure, using an existing "reference"...
def callMeasureN(self, measCat, args, kwds)
Call the measureN() method on all plugins, handling exceptions in a consistent way.
def run(self, measCat, exposure, refCat, refWcs, exposureId=None, beginOrder=None, endOrder=None)
Perform forced measurement.
def attachTransformedFootprints(self, sources, refCat, exposure, refWcs)
Default implementation for attaching Footprints to blank sources prior to measurement.
static std::shared_ptr< IdFactory > makeSimple()
def __init__(self, config, name, schemaMapper, metadata, logName=None)
def generateMeasCat(self, exposure, refCat, refWcs, idFactory=None)
Initialize an output SourceCatalog using information from the reference catalog.
static Schema makeMinimalSchema()
def measure(self, measRecord, exposure, refRecord, refWcs)
Base class for plugin registries.
A do-nothing standin for NoiseReplacer, used when we want to disable NoiseReplacer.
def measureN(self, measCat, exposure, refCat, refWcs)
static std::shared_ptr< SourceTable > make(Schema const &schema, std::shared_ptr< IdFactory > const &idFactory)
Class that handles replacing sources with noise during measurement.
Ultimate base class for all measurement tasks.
Base config class for all measurement driver tasks.
def doMeasurement(self, plugin, measRecord, args, kwds)
Call the measure() method on the nominated plugin, handling exceptions in a consistent way...