wiki:PSPSPhotoZ
PSPS Help Sections Software Components PSPS User Tools Project Wikis Related sites * -- restricted access

Back to Main PSI Help Wiki

The Photometric Classification Server (PCS)

Introduction

The Photometric Classification Server (PCS) is a science server which provides additional data products to the PanSTARRS project, generated from the photometry published in the "Fundamental IPP Data Products" section of PSPS. The PCS data products are stored in the "Derived Data Products" section of PSPS, with meta data stored in the "Observational Metadata and Supporting System MetaData? Tables" section. All tables containing data from PCS are prefixed with "pcs".

The source can be downloaded here

Working with photometric redshift probability distributions

Deriving the photoz probability distributions for selected objects requires selection of data from the PSPS pcsPhotoZProbability table, downloading the data in csv format to your local computer, and processing it with the python code published here.

  1. Getting probability distributions from PSPS

In principle, a query looks something like this:

select pzp.* from pcsPhotoZProbability pzp

In practice, users need to apply some constraints on the query that select the desired probability distributions. Generally, only objects with flags_gtfit=0 in the pcsPhotoZ table should be used at all, you need to apply this constraint to any queries on the pcsPhotoZ or pcsPhotoZProbability table.

Example 1: Select the probability distribution of the best fitting model only

select pzp.* from pcsPhotoZ pz join pcsPhotoZProbability pzp of pz.objID=pzp.objID and pz.modelID_1=pzp.modelID where flags_gtfit=0

Example 2: Apply spatial constraints to the above query, requires the Object table containing ra and dec of the object

 select pzp.* from Object o join pcsPhotoZ pz on o.objID=pz.objID join pcsPhotoZProbability pzp of pz.objID=pzp.objID and pz.modelID_1=pzp.modelID where flags_gtfit=0 and ra between <ramin> and <ramax> and dec between <decmin> and <decmax>

Example 3: Apply constraints to the query used in example 2, to select only objects that are classified as galaxies by the PanZ code

select pzp.* from Object o join pcsPhotoZ pz on o.objID=pz.objID join pcsPhotoZProbability pzp on pz.objID=pzp.objID and pz.modelID_1=pzp.modelID where flags_gtfit=0 and chi2_1<chi2

Example 4: Apply constraints to the query used in example 2, to select only objects that are classified as galaxies by the PanDiSC code

select pzp.* from Object o join pcsPhotoZ pz on o.objID=pz.objID join pcsPhotoZProbability pzp on pz.objID=pzp.objID and pz.modelID_1=pzp.modelID join pcsStarGalQSOSep sgs on o.objID=sgs.objID 
where flags_gtfit=0 and probStar<probGalaxy and probQuasar<probGalaxy and flags=0

Example 5: Select photozs and probability distributions for a set of PSPS objIDs selected somewhere outside PSPS

First, upload your set of objIDs into a MyDB table named e.g. MyObjects? using the upload functionality in PSI's MyDB page. Then, insert the MyObjects? table into your query, e.g. Example 3.

select pzp.* from MyDB.MyTable m join Object o on m.objID=o.objID join pcsPhotoZ pz on o.objID=pz.objID join pcsPhotoZProbability pzp on pz.objID=pzp.objID and pz.modelID_1=pzp.modelID where flags_gtfit=0 and chi2_1<chi2

You are free to choose whatever constraints from whatever PSPS table you need, just apply the appropriate joins and where clauses to the basic queries above.

Once your query is built, run it in the PSPS slow queue, extract the resulting MyDB table to a csv file and download it to your local computer.

  • 2. Expanding the splines

Once your table has been downloaded to you local computer into e.g. MyFile?.csv, replace the commas in the csv file with spaces or tabs, and remove the first line and the quotes from the values.

$ cat MyFile.csv | tr ',' ' ' | sed 's/\"//g' | tail -n +2 > MyFile.ssv

Then, process the file MyFile?.ssv with the python code:

 $ python probabilities.py --input=MyFile.ssv --output=MyFile_Expanded.ssv

This creates a new file MyFile_Expanded.ssv having the following structure

 objID     xmin     xmax     xint     y1     y2 ... y250

The values xmin and xmax give the range, xint the stepwith. The y1 to y250 values are logarithmic to the base 10. Note that the probabilities are not normalized within each model, but within all models used (to be verified...).

The Photometric Classification Server tables in PSPS

The Photometric Classification Server provides photometric redshifts generated by a generic template fitting code developed at MPE, a star/galaxy/quasar separation generated by an empiric support vector machine code developed at MPIA, and a stellar classification (effective temperature only), also generated by support vector machine code from MPIA. See the individual data product table for more details.

pcsPhotoZ

Photometric redshifts are generated by a generic template fitting code developed at the Max-Planck-Institute for Extraterrestrial Physics in Munich. Photometry is taken from the stackApFlx table, column flxR7 (PV1) or flxR5 (PV2), which both correspond to 7.44 arcsec aperture. Prior to fitting, fluxes are dereddened using the galactic extinction E(B-V) derived from Schlegel maps. Results for the best (suffixed with _1) and the second best (suffixed with _2) fitting model are published in this table. Additionally, all objects are fitted to a set of stellar templates, these results are also published in this table. Warning flags are raised whenever there's an issue with galactic extinction or missing photometry, only objects having flags_gtfit=0 (and flags_stfit=0) should be used for science. The Chi2 values of the best fitting galaxy template and the best fitting stellar template can be used as an alternative/additional indicator for star/galaxy separation.

objID ODM object identifier index ippObjID IPP object number (not set so far) pcsPhotoZRecipeID Recipe of the photoz calculation, refers to pcsPhotoZRecipe galExt Galactic extinction E(B-V) used for photoz determination (derived from Schlegel Maps) photoz_1 Photometric redshift derived from best fitting template photozErr_1 Estimated error for this template chi2_1 Chi2 for this template modelID_1 Identifier for the template for this recipe, refers to pcsGalaxyModel photoz_2 Photometric redshift derived from the second best fitting template photozErr_2 Estimated error for this template chi2_2 Chi2 for this template modelID_2 Identifier for the template for this recipe, refers to pcsGalaxyModel flags_gtfit Warning flags from galaxy template fitting

0x0001-0x00FF warning flags from the extinction determination 0x0100 less than 5 band available for this object

av Extinction determined by the template fitting algorithm for the best fitting stellar template chi2 Chi2 for the best fitting stellar template modelID ModelID of the best fitting stellar template, refers to pcsStarModel flags_stfit Warning flags from stellar template fitting

0x0100 less than 5 bands available for this object

starGalQSOSep Alternative star/galaxy separation, determined from the model fitting

0: Object is a star (chi2<chi2_1) 1: Object is a galaxy (chi2>chi2_1)

pcsPhotoZProbability

For each object and galaxy model, the photoz code generates a probability distribution of the photometric redshift. Only the probability distributions of the two best fitting models (best and second best in the pcsPhotoZ table) are published in PSPS. The range of the probability distributions published here is determined to cover 99% of the total probability around the highest peak. To further reduce the required amount of data, a spline fit with 25 nodes in the determined range is applied to the original probability distribution, the resulting spline parameters are stored in the pcsPhotoZProbability table. Please see below for a python code example to expand the spline. Note that the spline may not always reproduce the correct height of sharp peaks in the original probability distribution. As a hint, the highest original probability is always at the corresponding redshift published in the pcsPhotoZ table (photoz_1 or photoz_2, respectively). objID ODM object identifier index ippObjID IPP object number (not set so far) pcsPhotoZRecipeID Recipe for the photoz determination, refers to pcsPhotoZRecipe modelID ModelID of the template, refers to pcsGalaxyModel sp1 Spline parameter 1 ... sp29 Spline parameter 29 sp30 Unused

pcsPhotoZRecipe

This table contains a short description of the recipe used for the photometric redshift determination.

photozRecipeID Recipe index description String describing the recipe

pcsGalaxyModel

This table contains a short description of every galaxy template used for photometric redshift determination, and a list of priors applied to the given template.

galaxyModelID modelID description model description, spectral type etc. priors priors applied to this model

pcsStarModel

This table contains a short description of every stellar template used in photometric redshift determination.

galaxyModelID modelID description model description, spectral type etc.

pcsStarGalQSOSep

This table provides a star/galaxy/quasar separation generated by a support vector machine code called PanDiSC (PanSTARRS Discrete Source Classifier). Ideally, it requires training data covering the whole population observed with PanSTARRS. Which training data is used and how it is derived is described in the pcsSGSepRecipe table. Only objects having flags=0 should be used for science.

objID ODM object identifier index ippObjID IPP object number pcsSGSepRecipeID Recipe for the star/galaxy/quasar separation, refers to pcsSGSepRecipe probStar Probability object is a star probGalaxy Probability object is a galaxy probQuasar Probability object is a quasar flags Warning flags

0x0100 less than 5 bands available for this object

pcsSGSepRecipe

This table contains a short description of the recipe used for star/galaxy/quasar separation.

sgSepRecipeID Star/Galaxy? Separator Recipe ID description Description

pcsStellarParams

This table provides stellar parameters generated by a support vector machine code developed at MPIA. Only Teff is available. Ideally, it requires training data covering the whole population observed with PanSTARRS. Which training data is used and how it is derived is described in the pcsStellarParamsRecipe table. Only objects having flags=0 should be used for science.

objID ODM object identifier index ippObjID IPP object number pcsStellarParamsRecipeID Recipe for the star/galaxy/quasar separation, refers to pcsStellarParamsRecipe Teff Effective temperature A0 Gravity (not set) Fe_H Metallicity (not set) flags Warning flags

0x0100 less than 5 bands available for this object

pcsStellarParamsRecipe

This table contains a short description of the recipe used for stellar parameters determination.

stellarParamsRecipeID RecipeID description Description

Last modified 4 years ago Last modified on 08/12/13 19:04:38

Attachments (1)

Download all attachments as: .zip