wiki:PSPSFAQ
PSPS Help Sections Software Components PSPS User Tools Project Wikis Related sites * -- restricted access

Frequently Asked Questions

Accessing the data

How do I connect to PSPS?

Data is retrieved from PSPS through the DRL. We offer various ways to connect to the DRL:

  • a web interface called PSI
  • a desktop application called PSVO
  • Some examples of client side scripts written in Perl, Java, and Python that accesses the DRL directly

This answer prepared by Roy Henderson.

Is there any way to connect IDL applications directly to the DRL?

The DRL and hence any database to which connects can be accessed using SOAP calls as illustrated in the example shown here. A quick scan of the web doesn't turn up any SOAP library for IDL but that doesn't mean there isn't one, just that it's hard to located. Rather than undertaking a coding exercise to interface directly to IDL it would be faster to simply download data from the PSPS using our Perl script and using IDL's import functions to read the data into that environment. Unless you really need to deal with large data volumes this solution will be far less painful that writing your own SOAP code.

This answer prepared by Jim Heasley.

How do I write a query?

Whichever client software you use, queries to PSPS must be written using SQL. A page of sample queries is maintained here.

This answer prepared by Roy Henderson.

Are there any sample queries?

Yes. A page of sample queries is maintained here.

This answer prepared by Roy Henderson.

Where can I see the database schema?

PSI incorporates a schema browser here. There is also a good high-level outline of the schema here.

This answer prepared by Roy Henderson.

Data

How is PSPS populated?

PSPS tables are populated with data from the (IPP). A full breakdown of each column of each table can be found in the following locations:

detection batches
stack batches
object batches

This answer prepared by Roy Henderson.

What type of data is loaded?

We load two types of data-product from the IPP, single-frame detections and stacks. The former are derived from single exposures on the sky, i.e. one PSPS frame is equivalent to a single PS1 exposure on the sky. The stacks are in the form of skycells, summed from n images. We can consume two different types of stacks: nightly and deep. The deep stacks (large n) include extended-source parameters as well as the usual psf information.

Again, a full breakdown of each column of each table can be found here.

This answer prepared by Roy Henderson.

What surveys can be found in PSPS?

A detailed and up-to-date overview of what surveys are currently available to users can be found here.

This answer prepared by Roy Henderson.

How can I see the current sky coverage in PSPS?

This information is maintained on our news page, here.

This answer prepared by Roy Henderson.

What are the known issues with the data?

  • There are still photometry issues with the re-processed data from the IPP. Details can be found on the DRAVG pages of the PS1SC wiki here.
  • Currently, the skycellIDs in the stacks are not correct.
  • The exposure time for stacks across all surveys is currently wrong. It is usually out by a factor of 4. Sorry.

This answer prepared by Roy Henderson.

What are the flags and how do I use them?

Single-frame detections, stacks and objects all have flags that can be used to filter results obtained from PSPS queries. Flags are attributes that are either on or off for a given detection or object and are stored as single bits within an large integer value called infoFlag in the detection tables (such as Detection and StackDetection), and objInfoFlag in the Object table. The following table hopefully explains how each is defined.

type flags column name definition
single-frame detections infoFlag DVO flags << 32 | smf FLAGS
stack detections infoFlag cmf FLAGS << 32 | cmf FLAGS2
objects objInfoFlag FLAGS from the DVO cpt files

So, for the case of single-frame detections, the first 32-bits of the 64-bit infoFlag value are made up with the 32-bits of the smf 'FLAGS' value. The remainder are made up with flags from DVO. (Note as can be seen above, FLAGS from the smf/cmf file populate different parts of the infoFlag value for single-frame detections and stacks. Ideally, for consistency, they would occupy the first 32-bits of both. This is a known issue.)

PSPS has a table of defintions for the detection flags only (stacks and objects will come in the future). To view the definition of the single-frame detections, run the following query from PSI or PSVO.

SELECT * FROM DetectionFlags ORDER BY value;

will return:

PSFMODEL 1       Source fitted with a psf model (linear or non-linear)
EXTMODEL 2       Source fitted with an extended-source model
FITTED   4       Source fitted with non-linear model (PSF or EXT; good or bad)
FITFAIL  8       Fit (non-linear) failed (non-converge\; off-edge\; run to zero)
POORFIT  16      Fit succeeds\; but low-SN\; high-Chisq\; or large (for PSF -- drop?)
PAIR     32      Source fitted with a double psf
PSFSTAR  64      Source used to define PSF model

etc

The above flags, in binary, are:

PSFMODEL 00000001       Source fitted with a psf model (linear or non-linear)
EXTMODEL 00000010       Source fitted with an extended-source model
FITTED   00000100     Source fitted with non-linear model (PSF or EXT; good or bad)
FITFAIL  00001000       Fit (non-linear) failed (non-converge\; off-edge\; run to zero)
POORFIT  00010000      Fit succeeds\; but low-SN\; high-Chisq\; or large (for PSF -- drop?)
PAIR     00100000      Source fitted with a double psf
PSFSTAR  01000000      Source used to define PSF model

To apply a set of flags to detections or objects, the user must define a mask. A mask is simply an integer where multiple bits have been set either on or off. So, for example, if we were looking for detections with a poor or bad PSF fit from the above set of flags, we would make a mask like this:

FITFAIL - 00001000
POORFIT - 00010000
mask -    00011000

This mask (00011000), which has a decimal value of 24, could then be used to include or exclude detections from a PSPS query. For a good example of how to do this, please see the FAQ regarding false detections, below.

This answer prepared by Roy Henderson.

How can I filter out 'false' detections?

To filter out false detections from query results, we use the detections flags. For background information about the flags see this FAQ.

The current recommendations from the IPP (courtesy of Chris Waters) to reduce the number of false positives are as follows.

To remove poor detections, use:

POORFIT
PAIR
BLEND
MOMENTS_SN
BLEND_FIT
ON_SPIKE
ON_GHOST
OFF_CHIP

This set of flags corresponds to a mask of 3762553136

To remove bad detections, use:

FITFAIL
SATSTAR
BADPSF
DEFECT
SATURATED
CR_LIMIT
MOMENTS_FAILURE
SKY_FAILURE
SKYVAR_FAILURE
SIZE_SKIPPED

This set of flags corresponds to a mask of 268680328

The union of the above poor and bad masks is 4031233464

To filter out detections using either mask, simply add a WHERE (infoFlag & mask) = 0 constraint to any query made against Detection or DetectionFull tables. For masks that use more than 32 bits (like this one), it is necessary to first declare the mask as a variable, for example:

DECLARE @mask BIGINT
SET @mask = 268680328
SELECT TOP 10 ra, dec 
FROM DetectionFull 
WHERE CAST(infoFlag & @mask AS INT) = 0

Certain PSVO plugin queries already have this implemented.

An additional cut can be made using the PSF quality factor. In PSPS, this is the psfCf column in the detection tables. Including only those detections with psfCf > 0.85 can reduce the number of false detections more than just using the masks, above.

Finally, we can make a 5-sigma cut on the magnitude by including only those detections with (instFlux / instFluxErr) >= 5.

This answer prepared by Roy Henderson.

I tried to look at the attributes in the object table and the entries all came back as -999. Does this mean the object isn't found?

This one is a bit tricky, but here goes... IPP reports detections to the PSPS when they're found and they have an object associated with them. That means the source has been seen at least twice with "good" S/N so that IPP has confidence it's real. (High confidence single detections will end up eventually in the orphans table.) So, while IPP knows this is an object we won't have a good handle on its mean properties (which is what will populate the object table) until sometime after it has been observed in all the filters for a given observing season. In the mean time PSPS makes an entry into the object table to show we have an object there and use that for facilitating spatial searches over objects and detections. When IPP gives us the real (final) photometry and astrometry for the year we will update those object records (rolling around the sky) changing the version number from 0 to 1.

In the mean time the best way to deal with this is look at the detection records for the object in your region and compute the average magnitudes and color properties yourself.

This answer prepared by Jim Heasley.

Why are there multiple entries for objID/filterID in the stack tables?

It is possible to have multiple detections in the stack tables for a given pairing of object ID and filter ID due to overlapping skycells. Near skycell corners there may be up to four detections for the same object in the same filter. As each stack is generated from a different set of inputs the photometry can differ and so photometry errors (and possibly the infoFlag field) should be taken into account when deciding which is the most 'trustworthy' detection to use. If all errors are small then an average value across all detections may be preferable.

This answer prepared by Roy Henderson.

Why are the apMag values all negative?

apMag in the Detection and DetectionFull tables is the instrumental aperture magnitude from the IPP. To get a meaningful magnitude you need to use the zero-point and exposure, like this:

apMag + zeroPoint + 2.5*log10(expTime)

Where the zeroPoint is different for each filter, but can be found here and the exposure time can be found in the FrameMeta table. So, an example query for the 'g' filter would be as follows.

SELECT TOP 10
Detection.objID,
Detection.apMag AS instAPMag,
Detection.apMag + 24.61 + 2.5*log10(expTime) AS apMag
FROM Detection
JOIN ImageMeta ON Detection.imageID = ImageMeta.imageID
JOIN FrameMeta ON ImageMeta.frameID = FrameMeta.frameID
JOIN Object ON Detection.objID = Object.objID
WHERE Object.objID IN
(SELECT objID FROM dbo.fGetNearbyObjEq(330, 0.3, 0.8))
AND nDetections > 5
AND grMeanColor != -999
AND riMeanColor != -999
AND psfLikelihood > 0.99
AND Detection.filterID = 1

Which gives the results:

objID instAPMagapMag
107903305495163807 -8.72562 19.9680514795251
108013305234315454 -9.96844 18.7252316934534
108133306970236456 -6.851 21.8426719171472
108173306655618860 -8.56845 20.1252208215662
108233306405939161 -10.985 17.7086720926233
107453299540017145 -11.8973 16.79637198276
107473301542544331 -11.4936 17.2000718576379
107633300938147391 -11.8911 16.802571819491
108113303937173038 -6.15037 22.5433021051355
108143303426662235 -11.5714 17.1222720605798

This answer prepared by Roy Henderson.

What is the best way to make joins?

Joins can take a performance out of a query. However, it depends on whether or not the columns the joins are on are indexed or not. The best practice when possible is to join the ObjID. It is also possible to join on a combination of columns such as ObjID with the StackDetectionID in any of the stack related tables or DetectID in any of the detection tables.

See further explaination and example here.

This answer prepared by Conrad Holmberg

How do I Calculate Magnitudes from Detections or Stack Detections Fluxes

The Object table only contain the mean magnitudes and the Stack or Detection tables only have fluxes which can be converted into magnitudes.

 -2.5 log10(flux) + ZP

Example Query:

SELECT TOP 100 sdf.objID,
               sdf.stackDetectID, 
               sdf.ra, 
               sdf.dec, 
               sdf.psfFlux AS sdf_psfFlux, 
               sdf.psfFluxErr AS sdf_psfFluxErr, 
               (-2.5*log10(sdf.psfFlux) + sdf.zp) as psfMag,
               sdf.zp, 
               sdf.zpErr, 
               sdf.exptime AS sdf_exptime
 FROM StackDetectionFull AS sdf 
 where psfFlux > 0 and psfFluxErr > 0

The query above takes 100 StackDetections? and calculates the pdfMag as a double by having the conversion equation in the select clause of the query.

How to work with Photometric Redshift Data?

A full wiki on this can be read here.

PSVO

How do I increase the Java heap space?

Sometimes large queries can cause PSVO to run out of heap space, which depending on your machine may be set very low. To specify a larger amount of heap space simply run the PSVO.jar file like this:

java -Xmx512m -jar PSVO.jar

where the '512m' is the amount of heap space in megabytes. You can choose any (reasonable) value.

This answer prepared by Roy Henderson.

Last modified 4 years ago Last modified on 07/23/13 21:03:37