lsst.meas.base  16.0-8-g2ce35ff
forcedMeasurement.py
Go to the documentation of this file.
1 #!/usr/bin/env python
2 #
3 # LSST Data Management System
4 # Copyright 2008-2015 LSST Corporation.
5 #
6 # This product includes software developed by the
7 # LSST Project (http://www.lsst.org/).
8 #
9 # This program is free software: you can redistribute it and/or modify
10 # it under the terms of the GNU General Public License as published by
11 # the Free Software Foundation, either version 3 of the License, or
12 # (at your option) any later version.
13 #
14 # This program is distributed in the hope that it will be useful,
15 # but WITHOUT ANY WARRANTY; without even the implied warranty of
16 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
17 # GNU General Public License for more details.
18 #
19 # You should have received a copy of the LSST License Statement and
20 # the GNU General Public License along with this program. If not,
21 # see <http://www.lsstcorp.org/LegalNotices/>.
22 #
23 """Base classes for forced measurement plugins and the driver task for these.
24 
25 In forced measurement, a reference catalog is used to define restricted measurements (usually just fluxes)
26 on an image. As the reference catalog may be deeper than the detection limit of the measurement image, we
27 do not assume that we can use detection and deblend information from the measurement image. Instead, we
28 assume this information is present in the reference catalog and can be "transformed" in some sense to
29 the measurement frame. At the very least, this means that Footprints from the reference catalog should
30 be transformed and installed as Footprints in the output measurement catalog. If we have a procedure that
31 can transform HeavyFootprints, we can then proceed with measurement as usual, but using the reference
32 catalog's id and parent fields to define deblend families. If this transformation does not preserve
33 HeavyFootprints (this is currently the case, at least for CCD forced photometry), then we will only
34 be able to replace objects with noise one deblend family at a time, and hence measurements run in
35 single-object mode may be contaminated by neighbors when run on objects with parent != 0.
36 
37 Measurements are generally recorded in the coordinate system of the image being measured (and all
38 slot-eligible fields must be), but non-slot fields may be recorded in other coordinate systems if necessary
39 to avoid information loss (this should, of course, be indicated in the field documentation). Note that
40 the reference catalog may be in a different coordinate system; it is the responsibility of plugins
41 to transform the data they need themselves, using the reference WCS provided. However, for plugins
42 that only require a position or shape, they may simply use output SourceCatalog's centroid or shape slots,
43 which will generally be set to the transformed position of the reference object before any other plugins are
44 run, and hence avoid using the reference catalog at all.
45 
46 Command-line driver tasks for forced measurement can be found in forcedPhotImage.py, including
47 ForcedPhotImageTask, ForcedPhotCcdTask, and ForcedPhotCoaddTask.
48 """
49 
50 import lsst.pex.config
51 import lsst.pipe.base
52 
53 from .pluginRegistry import PluginRegistry
54 from .baseMeasurement import (BaseMeasurementPluginConfig, BaseMeasurementPlugin,
55  BaseMeasurementConfig, BaseMeasurementTask)
56 from .noiseReplacer import NoiseReplacer, DummyNoiseReplacer
57 
58 __all__ = ("ForcedPluginConfig", "ForcedPlugin",
59  "ForcedMeasurementConfig", "ForcedMeasurementTask")
60 
61 
63  """Base class for configs of forced measurement plugins."""
64  pass
65 
66 
68 
69  # All subclasses of ForcedPlugin should be registered here
70  registry = PluginRegistry(ForcedPluginConfig)
71 
72  ConfigClass = ForcedPluginConfig
73 
74  def __init__(self, config, name, schemaMapper, metadata, logName=None):
75  """Initialize the measurement object.
76 
77  @param[in] config An instance of this class's ConfigClass.
78  @param[in] name The string the plugin was registered with.
79  @param[in,out] schemaMapper A SchemaMapper that maps reference catalog fields to output
80  catalog fields. Output fields should be added to the
81  output schema. While most plugins will not need to map
82  fields from the reference schema, if they do so, those fields
83  will be transferred before any plugins are run.
84  @param[in] metadata Plugin metadata that will be attached to the output catalog
85  """
86  BaseMeasurementPlugin.__init__(self, config, name, logName=logName)
87 
88  def measure(self, measRecord, exposure, refRecord, refWcs):
89  """Measure the properties of a source on a single image, given data from a
90  reference record.
91 
92  @param[in] exposure lsst.afw.image.ExposureF, containing the pixel data to
93  be measured and the associated Psf, Wcs, etc. All
94  other sources in the image will have been replaced by
95  noise according to deblender outputs.
96  @param[in,out] measRecord lsst.afw.table.SourceRecord to be filled with outputs,
97  and from which previously-measured quantities can be
98  retreived.
99  @param[in] refRecord lsst.afw.table.SimpleRecord that contains additional
100  parameters to define the fit, as measured elsewhere.
101  @param[in] refWcs The coordinate system for the reference catalog values.
102  An lsst.geom.Angle may be passed, indicating that a
103  local tangent Wcs should be created for each object
104  using afw.image.makeLocalWcs and the given angle as
105  a pixel scale.
106 
107  In the normal mode of operation, the source centroid will be set to the
108  WCS-transformed position of the reference object, so plugins that only
109  require a reference position should not have to access the reference object
110  at all.
111  """
112  raise NotImplementedError()
113 
114  def measureN(self, measCat, exposure, refCat, refWcs):
115  """Measure the properties of a group of blended sources on a single image,
116  given data from a reference record.
117 
118  @param[in] exposure lsst.afw.image.ExposureF, containing the pixel data to
119  be measured and the associated Psf, Wcs, etc. Sources
120  not in the blended hierarchy to be measured will have
121  been replaced with noise using deblender outputs.
122  @param[in,out] measCat lsst.afw.table.SourceCatalog to be filled with outputs,
123  and from which previously-measured quantities can be
124  retrieved, containing only the sources that should be
125  measured together in this call.
126  @param[in] refCat lsst.afw.table.SimpleCatalog that contains additional
127  parameters to define the fit, as measured elsewhere.
128  Ordered such that zip(sources, references) may be used.
129  @param[in] refWcs The coordinate system for the reference catalog values.
130  An lsst.geom.Angle may be passed, indicating that a
131  local tangent Wcs should be created for each object
132  using afw.image.makeLocalWcs and the given Angle as
133  a pixel scale.
134 
135  In the normal mode of operation, the source centroids will be set to the
136  WCS-transformed position of the reference object, so plugins that only
137  require a reference position should not have to access the reference object
138  at all.
139  """
140  raise NotImplementedError()
141 
142 
144  """Config class for forced measurement driver task."""
145 
146  plugins = ForcedPlugin.registry.makeField(
147  multi=True,
148  default=["base_PixelFlags",
149  "base_TransformedCentroid",
150  "base_SdssCentroid",
151  "base_TransformedShape",
152  "base_SdssShape",
153  "base_GaussianFlux",
154  "base_CircularApertureFlux",
155  "base_PsfFlux",
156  "base_LocalBackground",
157  ],
158  doc="Plugins to be run and their configuration"
159  )
160  algorithms = property(lambda self: self.plugins, doc="backwards-compatibility alias for plugins")
161  undeblended = ForcedPlugin.registry.makeField(
162  multi=True,
163  default=[],
164  doc="Plugins to run on undeblended image"
165  )
166 
167  copyColumns = lsst.pex.config.DictField(
168  keytype=str, itemtype=str, doc="Mapping of reference columns to source columns",
169  default={"id": "objectId", "parent": "parentObjectId", "deblend_nChild": "deblend_nChild",
170  "coord_ra": "coord_ra", "coord_dec": "coord_dec"}
171  )
172 
173  checkUnitsParseStrict = lsst.pex.config.Field(
174  doc="Strictness of Astropy unit compatibility check, can be 'raise', 'warn' or 'silent'",
175  dtype=str,
176  default="raise",
177  )
178 
179  def setDefaults(self):
180  self.slots.centroid = "base_TransformedCentroid"
181  self.slots.shape = "base_TransformedShape"
182  self.slots.apFlux = None
183  self.slots.modelFlux = None
184  self.slots.psfFlux = None
185  self.slots.instFlux = None
186  self.slots.calibFlux = None
187 
188 
194 
195 
197  """!
198  @anchor ForcedMeasurementTask_
199 
200  @brief A subtask for measuring the properties of sources on a single
201  exposure, using an existing "reference" catalog to constrain some aspects
202  of the measurement.
203 
204  The task is configured with a list of "plugins": each plugin defines the values it
205  measures (i.e. the columns in a table it will fill) and conducts that measurement
206  on each detected source (see ForcedPlugin). The job of the
207  measurement task is to initialize the set of plugins (which includes setting up the
208  catalog schema) from their configuration, and then invoke each plugin on each
209  source.
210 
211  Most of the time, ForcedMeasurementTask will be used via one of the subclasses of
212  ForcedPhotImageTask, ForcedPhotCcdTask and ForcedPhotCoaddTask. These combine
213  this measurement subtask with a "references" subtask (see BaseReferencesTask and
214  CoaddSrcReferencesTask) to perform forced measurement using measurements performed on
215  another image as the references. There is generally little reason to use
216  ForcedMeasurementTask outside of one of these drivers, unless it is necessary to avoid
217  using the Butler for I/O.
218 
219  ForcedMeasurementTask has only three methods: __init__(), run(), and generateMeasCat().
220  For configuration options, see SingleFrameMeasurementConfig.
221 
222  Notes
223  -----
224 
225  *Forced* measurement means that the plugins are provided with a reference
226  source containing centroid and/or shape measurements that they may use
227  however they see fit. Some plugins can use these to set the location and
228  size of apertures, but others may choose to ignore this information,
229  essentially performing an unforced measurement starting at the position
230  of the reference source (which may nevertheless be useful for certain
231  investigations). Knowing how the plugin uses the reference information is
232  essential to interpreting its resulting measurements. Typically, centroid
233  and shape measurement plugins (e.g., ``SdssCentroid`` and ``SdssShape``)
234  are performing unforced measurements.
235  """
236 
237  ConfigClass = ForcedMeasurementConfig
238 
239  def __init__(self, refSchema, algMetadata=None, **kwds):
240  """!
241  Initialize the task. Set up the execution order of the plugins and initialize
242  the plugins, giving each plugin an opportunity to add its measurement fields to
243  the output schema and to record information in the task metadata.
244 
245  Note that while SingleFrameMeasurementTask is passed an initial Schema that is
246  appended to in order to create the output Schema, ForcedMeasurementTask is
247  initialized with the Schema of the reference catalog, from which a new Schema
248  for the output catalog is created. Fields to be copied directly from the
249  reference Schema are added before Plugin fields are added.
250 
251  @param[in] refSchema Schema of the reference catalog. Must match the catalog
252  later passed to generateMeasCat() and/or run().
253  @param[in,out] algMetadata lsst.daf.base.PropertyList used to record information about
254  each algorithm. An empty PropertyList will be created if None.
255  @param[in] **kwds Keyword arguments passed from lsst.pipe.base.Task.__init__
256  """
257  super(ForcedMeasurementTask, self).__init__(algMetadata=algMetadata, **kwds)
259  self.mapper.addMinimalSchema(lsst.afw.table.SourceTable.makeMinimalSchema(), False)
260  self.config.slots.setupSchema(self.mapper.editOutputSchema())
261  for refName, targetName in self.config.copyColumns.items():
262  refItem = refSchema.find(refName)
263  self.mapper.addMapping(refItem.key, targetName)
264  self.config.slots.setupSchema(self.mapper.editOutputSchema())
265  self.initializePlugins(schemaMapper=self.mapper)
266  self.schema = self.mapper.getOutputSchema()
267  self.schema.checkUnits(parse_strict=self.config.checkUnitsParseStrict)
268 
269  def run(self, measCat, exposure, refCat, refWcs, exposureId=None, beginOrder=None, endOrder=None):
270  """!
271  Perform forced measurement.
272 
273  @param[in] exposure lsst.afw.image.ExposureF to be measured; must have at least a Wcs attached.
274  @param[in] measCat Source catalog for measurement results; must be initialized with empty
275  records already corresponding to those in refCat (via e.g. generateMeasCat).
276  @param[in] refCat A sequence of SourceRecord objects that provide reference information
277  for the measurement. These will be passed to each Plugin in addition
278  to the output SourceRecord.
279  @param[in] refWcs Wcs that defines the X,Y coordinate system of refCat
280  @param[in] exposureId optional unique exposureId used to calculate random number
281  generator seed in the NoiseReplacer.
282  @param[in] beginOrder beginning execution order (inclusive): measurements with
283  executionOrder < beginOrder are not executed. None for no limit.
284  @param[in] endOrder ending execution order (exclusive): measurements with
285  executionOrder >= endOrder are not executed. None for no limit.
286 
287  Fills the initial empty SourceCatalog with forced measurement results. Two steps must occur
288  before run() can be called:
289  - generateMeasCat() must be called to create the output measCat argument.
290  - Footprints appropriate for the forced sources must be attached to the measCat records. The
291  attachTransformedFootprints() method can be used to do this, but this degrades HeavyFootprints
292  to regular Footprints, leading to non-deblended measurement, so most callers should provide
293  Footprints some other way. Typically, calling code will have access to information that will
294  allow them to provide HeavyFootprints - for instance, ForcedPhotCoaddTask uses the HeavyFootprints
295  from deblending run in the same band just before non-forced is run measurement in that band.
296  """
297  # First check that the reference catalog does not contain any children for which
298  # any member of their parent chain is not within the list. This can occur at
299  # boundaries when the parent is outside and one of the children is within.
300  # Currently, the parent chain is always only one deep, but just in case, this
301  # code checks for any case where when the parent chain to a child's topmost
302  # parent is broken and raises an exception if it occurs.
303  #
304  # I.e. this code checks that this precondition is satisfied by whatever reference
305  # catalog provider is being paired with it.
306  refCatIdDict = {ref.getId(): ref.getParent() for ref in refCat}
307  for ref in refCat:
308  refId = ref.getId()
309  topId = refId
310  while(topId > 0):
311  if topId not in refCatIdDict:
312  raise RuntimeError("Reference catalog contains a child for which at least "
313  "one parent in its parent chain is not in the catalog.")
314  topId = refCatIdDict[topId]
315 
316  # Construct a footprints dict which looks like
317  # {ref.getId(): (ref.getParent(), source.getFootprint())}
318  # (i.e. getting the footprint from the transformed source footprint)
319  footprints = {ref.getId(): (ref.getParent(), measRecord.getFootprint())
320  for (ref, measRecord) in zip(refCat, measCat)}
321 
322  self.log.info("Performing forced measurement on %d source%s", len(refCat),
323  "" if len(refCat) == 1 else "s")
324 
325  if self.config.doReplaceWithNoise:
326  noiseReplacer = NoiseReplacer(self.config.noiseReplacer, exposure,
327  footprints, log=self.log, exposureId=exposureId)
328  algMetadata = measCat.getTable().getMetadata()
329  if algMetadata is not None:
330  algMetadata.addInt("NOISE_SEED_MULTIPLIER", self.config.noiseReplacer.noiseSeedMultiplier)
331  algMetadata.addString("NOISE_SOURCE", self.config.noiseReplacer.noiseSource)
332  algMetadata.addDouble("NOISE_OFFSET", self.config.noiseReplacer.noiseOffset)
333  if exposureId is not None:
334  algMetadata.addLong("NOISE_EXPOSURE_ID", exposureId)
335  else:
336  noiseReplacer = DummyNoiseReplacer()
337 
338  # Create parent cat which slices both the refCat and measCat (sources)
339  # first, get the reference and source records which have no parent
340  refParentCat, measParentCat = refCat.getChildren(0, measCat)
341  for parentIdx, (refParentRecord, measParentRecord) in enumerate(zip(refParentCat, measParentCat)):
342 
343  # first process the records which have the current parent as children
344  refChildCat, measChildCat = refCat.getChildren(refParentRecord.getId(), measCat)
345  # TODO: skip this loop if there are no plugins configured for single-object mode
346  for refChildRecord, measChildRecord in zip(refChildCat, measChildCat):
347  noiseReplacer.insertSource(refChildRecord.getId())
348  self.callMeasure(measChildRecord, exposure, refChildRecord, refWcs,
349  beginOrder=beginOrder, endOrder=endOrder)
350  noiseReplacer.removeSource(refChildRecord.getId())
351 
352  # then process the parent record
353  noiseReplacer.insertSource(refParentRecord.getId())
354  self.callMeasure(measParentRecord, exposure, refParentRecord, refWcs,
355  beginOrder=beginOrder, endOrder=endOrder)
356  self.callMeasureN(measParentCat[parentIdx:parentIdx+1], exposure,
357  refParentCat[parentIdx:parentIdx+1],
358  beginOrder=beginOrder, endOrder=endOrder)
359  # measure all the children simultaneously
360  self.callMeasureN(measChildCat, exposure, refChildCat,
361  beginOrder=beginOrder, endOrder=endOrder)
362  noiseReplacer.removeSource(refParentRecord.getId())
363  noiseReplacer.end()
364 
365  # Undeblended plugins only fire if we're running everything
366  if endOrder is None:
367  for measRecord, refRecord in zip(measCat, refCat):
368  for plugin in self.undeblendedPlugins.iter():
369  self.doMeasurement(plugin, measRecord, exposure, refRecord, refWcs)
370 
371  def generateMeasCat(self, exposure, refCat, refWcs, idFactory=None):
372  """!Initialize an output SourceCatalog using information from the reference catalog.
373 
374  This generates a new blank SourceRecord for each record in refCat. Note that this
375  method does not attach any Footprints. Doing so is up to the caller (who may
376  call attachedTransformedFootprints or define their own method - see run() for more
377  information).
378 
379  @param[in] exposure Exposure to be measured
380  @param[in] refCat Sequence (not necessarily a SourceCatalog) of reference SourceRecords.
381  @param[in] refWcs Wcs that defines the X,Y coordinate system of refCat
382  @param[in] idFactory factory for creating IDs for sources
383 
384  @return Source catalog ready for measurement
385  """
386  if idFactory is None:
388  table = lsst.afw.table.SourceTable.make(self.schema, idFactory)
389  measCat = lsst.afw.table.SourceCatalog(table)
390  table = measCat.table
391  table.setMetadata(self.algMetadata)
392  table.preallocate(len(refCat))
393  for ref in refCat:
394  newSource = measCat.addNew()
395  newSource.assign(ref, self.mapper)
396  return measCat
397 
398  def attachTransformedFootprints(self, sources, refCat, exposure, refWcs):
399  """!Default implementation for attaching Footprints to blank sources prior to measurement
400 
401  Footprints for forced photometry must be in the pixel coordinate system of the image being
402  measured, while the actual detections may start out in a different coordinate system.
403  This default implementation transforms the Footprints from the reference catalog from the
404  refWcs to the exposure's Wcs, which downgrades HeavyFootprints into regular Footprints,
405  destroying deblend information.
406 
407  Note that ForcedPhotImageTask delegates to this method in its own attachFootprints method.
408  attachFootprints can then be overridden by its subclasses to define how their Footprints
409  should be generated.
410 
411  See the documentation for run() for information about the relationships between run(),
412  generateMeasCat(), and attachTransformedFootprints().
413  """
414  exposureWcs = exposure.getWcs()
415  region = exposure.getBBox(lsst.afw.image.PARENT)
416  for srcRecord, refRecord in zip(sources, refCat):
417  srcRecord.setFootprint(refRecord.getFootprint().transform(refWcs, exposureWcs, region))
Base config class for all measurement plugins.
def callMeasure(self, measRecord, args, kwds)
Call the measure() method on all plugins, handling exceptions in a consistent way.
def __init__(self, refSchema, algMetadata=None, kwds)
Initialize the task.
A subtask for measuring the properties of sources on a single exposure, using an existing "reference"...
def callMeasureN(self, measCat, args, kwds)
Call the measureN() method on all plugins, handling exceptions in a consistent way.
def run(self, measCat, exposure, refCat, refWcs, exposureId=None, beginOrder=None, endOrder=None)
Perform forced measurement.
def attachTransformedFootprints(self, sources, refCat, exposure, refWcs)
Default implementation for attaching Footprints to blank sources prior to measurement.
def __init__(self, config, name, schemaMapper, metadata, logName=None)
def generateMeasCat(self, exposure, refCat, refWcs, idFactory=None)
Initialize an output SourceCatalog using information from the reference catalog.
static Schema makeMinimalSchema()
def measure(self, measRecord, exposure, refRecord, refWcs)
Base class for plugin registries.
static std::shared_ptr< SourceTable > make(Schema const &schema, std::shared_ptr< IdFactory > const &idFactory)
A do-nothing standin for NoiseReplacer, used when we want to disable NoiseReplacer.
def measureN(self, measCat, exposure, refCat, refWcs)
Class that handles replacing sources with noise during measurement.
static std::shared_ptr< IdFactory > makeSimple()
Ultimate base class for all measurement tasks.
Base config class for all measurement driver tasks.
def doMeasurement(self, plugin, measRecord, args, kwds)
Call the measure() method on the nominated plugin, handling exceptions in a consistent way...