wiki:PSPSNews
PSPS Help Sections Software Components PSPS User Tools Project Wikis Related sites * -- restricted access

PSPS News

Update May 6, 2014, Merge 27 of PV1 is now available with a total of 29,383,869,914 detections. The right hand plot below is updated. There is one small remaining hole, this will be filled. For PV1 the Celestial North Pole and the galactic bulge will not be populated. They will be populated in PV2. See the processing status pages on the PS1SC wiki.The Object table has also been updated as well, restoring lost objects and adding new ones as well bring the total to 5,958,153,577 objects.

Update Apr 8, 2014, Merge 26 of PV1 is now available with a total of 28,385,635,859 detections. The right hand plot below is updated.

Update Mar 21, 2014. Merge 25 of PV1 is now available with a total of 26,533,230,159 detections. The right hand plot below is updated.

Update Mar 4, 2014. Merge 24 of PV1 is now available with a total of 25,254,863,383 detections. The right hand plot below is updated.

Update Feb 4, 2014. Merge 23 of PV1 is now available with a total of 24,609,602,861 detections. The right hand plot below is updated accordingly. See PV2 status on PS1SC wiki for current status of PV2.

Update DEC 2, 2013, merge 22 in 3pi is now available with a total of 23,813,486,699 detections. Also, the Photoz tables are fully populated (with what has been provided by the PCS group.) Note: Loading and merging is stopped as of today as we begin to move the load/merge machines to Manoa. PSPS will continue to be fully functional to the user until Dec 12. Then the PSPS and PSI interface will be down as we move the remaining parts of the PSPS. (Machine queries to the postage stamp server will still work, as they go through the IPP.) If all goes well the PSPS should be fully functional again by Dec 24, and when it does we will start the next merge and should finish publishing 3pi PV1 by the end of December. If we run into issues arising from the move this could be delayed until early January. We recommend that you take these dates into account with your planning. MD04 will be published on the sandbox for testing by the PSPS working group - kcc.

Update Nov 20, 2013, merge 21 in 3pi available, total of 22,314,009,320 detections.

Update Nov, 11, 2013. merge 19 in 3pi available, total of 19,100,690,254 detections, not including galactic bulge.

Update Oct 31, 2013. MD09 is available, with 47,784,653 objects and 539,493,106 detections. SAS 30 coming soon.

Update Oct 8, 2013 - merge 17 complete, total of 14,928,110,386 individual detections published, see revised second plot.

Below on the left is a plot of the distribution of the current 5,670,289,421 billion objects loaded in Merge 7.

On the right is the spatial distribution of all the detections, stacks and all other tables loaded. Note if an object is only detected once, it is still promoted to an object. The majority of the ~30 billion detections we expect are largely multiple detections at different times and different filters of the same object, and are associated as such into an object.

Objects: http://web01.psps.ifa.hawaii.edu/psps-trac/raw-attachment/wiki/PSPSNews/merge720130811.png Detections: http://web01.psps.ifa.hawaii.edu/psps-trac/raw-attachment/wiki/PSPSNews/merge27.png

July 9, 2013: Current summary and outlook for expected data products to come.

Note the first digit in the PV number identifies the reprocessing code version of (a) pixels - images, and (b) analysis of those pixels - smf & cmf files. The second digit identifies the version of the code(s) that does the (a) relative photometry, (b) relative astrometry, (c) ubercalibration, (d) ipp - to - psps translation.

Summary of PV1.2 Issues (SAS21, and 3pi_PV1.2)

  1. Detection and Stack table:

The fluxes are in instrumental counts, to get ubercalibrated magnitudes use the zero point (zp) and only the zp for that detection given in Detection Calib and Stack Detection Calib. These are Uber-calibrated zeropoints except as flagged. The Primary F flag is now set in stack Detection, indicating which detection out of a possible maximum of four in regions of overlapping sky cells is the primary one.

  1. Object table photometry:

The Object table photometry for PV1 has correct mean and stack psf, kron, and curve of growth corrected single aperture photometry. Ndetections now indicates the number of times an object was detected in a single epoch image; it is not necessarily the number that went into the robust mean. There is flag in the Detection Table that indicates if it was included in the mean. The Object.xFlags (x = grizy) also has bit 0x4000 set if the stack photometry comes from the primary skycell.

  1. Stack Aperture photometry:

Stack Aperture photometry for PV1 is measured only on convolved stacks (for example: flxR1 is measured on the convolved stack with a common psf per skycell in each filter and then matched to the 6 and 8 pixel FWHM targets for c1flxR1 and c2flxR1, etc.) Both kinds of convolved stacks sometimes have an excessively large target FWHM, leading to large flux losses in the smaller radii (R6 and smaller). In addition, many objects are skipped due to a bug at larger radii. For more on this see the dravg page, go to items for July 1, 2013: radial aperature mags vs Kron mags What you see there will be pervasive in the 3PI_PV1.2 release. As a result, stack aperture photometry for PV1 is not recommended for science. The petrosian parameters are unreliable. The isophotal, C and A parameters are not populated.

  1. Stack Model Fit:

The stack model fit parameters for PV1 have errors and are not recommended for science.

  1. Images:

The PV1 stack images from the postage stamp server are unnecessarily over-masked, where stacks with up to 3 good pixels are still masked. The dynamic masking (e.g. of optical ghosts) is not correctly implemented in PV1, so artifacts from these exist in the images.

Summary of Improvements in PV2 (SAS21, SASxx and 3pi_PV2)

The PV2 release is not a full re-processing of all the raw data, but rather an "update" to get to a scientifically useful 3pi and MD release, albiet without attempts to mitigate the correlated read noise (That will come with a final re-processing and be released as PV3). However the pixels in PV2 will be improved. PV2 makes use of video darks to improve the print-through of corner glows, has dynamic masking of artifacts like optical ghosts, and has improved stacking with proper rejection. The 3pi_PV2 should have all extended source parameters properly computed. Intermediate versions of PV2 processing of SAS will test and verify all parameters. The pixel update for PV2 will start soon (e.g. middle of July), the final static sky analysis (extended source properties) is faster and can begin later and still catch up with PV2 pixel processing. The PSPS will be populated incrementally with PV2 as the static sky analysis proceeds. PV2 will be populated with the new PSPS schema after testing and verification on the sandbox. It will take some weeks of PV2 processing to know exactly how long PV2 will take.

Processing Version Survey Data Product Date Status of PSPS publication Description Issues
PV1.0 SAS(2) SAS13 4/2013 released baseline SAS all known issues as of March 2013 Hawaii meeting, detailed on stsci confluence site, including mean phot problems and stack vs epoch phot problems
PV1.1 SAS(2) SAS19 6/ 2013 released, to be deleted Pixels unchanged from SAS13. Intermediate PV1 release Has fixes to mean vs stack vs epoch calibration issues, but has incorrect ndet and Primary Flag. Other issues as above for PV1 release.
PV1.2 SAS(2) SAS21 7/ 2013 released Pixels unchanged from SAS13. Baseline PV1 release The only ssues should be those issues that are generic to PV1 as listed in the news above.
PV1.2 3pi 3pi_PV1.2 7/2013 in release PV1.2 3pi - will begin with object table, and then fill others The only ssues should be those issues that are generic to PV1.2 as listed in the news above.
PV2.0 SAS(2) SAS18 6/2013 released PV2 precursor. New pixels relative to SAS13 includes video darks, dynamic masking, but not yet stack ap offsets or extended source parameters
PV2.1 SAS(2) SAS22 8/2013 in production PV2 precursor. New pixels relative to SAS18 includes all changes to pixels with full PV2 code, has improved but not final static sky analysis.
PV2.1 SAS(2) SAS24 8/2013 won't be published PSPS Pixels should be close but not exact as 22 includes all changes to pixels done as PV2 update process, same static sky analysis as SAS22
PV2.1 SAS(2) SAS26 9/2013 released - to be deleted PV2 baseline pixels pixels run as PV2 update process, issues with loading
PV2.1 SAS(2) SAS28 10/2013 in production PV2 baseline pixels pixels run as PV2 update process, static sky analysis now includes extended source fits w modest problems
PV2.1 SAS(2) SAS30 11/2013 planned PV2 baseline. pixels run as PV2 update process, final fixes to extended source analysis and radial stack aperture photometry offsets .
PV2.x 3pi 3pi_PV2 11-12/2013 planned PV2 include fix to stack aperture photometry, extended source parameters, other fixes
PV2.x MD MDTemp, Deep 9, 4 10-12/2013 in production PV2 baseline fixes to stacks, dynamic masking, video darks, new relphot, radial aperture fix
PV3.0 SAS(2) TBD 9/2013 planned PV3 PV3 baseline
PV3.x 3pi TBD 10/2013-6/2014 planned PV3 final processing version, earlier version(s) from LANL if possible
PV3.x MD MDTemp, Deep 10/2013-6/2014 planned PV3 final version, earlier version from LANL if possible

  • May 8, 2013: The Postage Stamp Server is now available. Please give it a try. There is substantial documentation, see the ? boxes for help at any stage. This includes postage stamps of SAS 14. Note the PV1 3pi postage stamps will still have over masking until we are able to start releasing Processing Version 2, see the ps1sc wiki or confluence site for schedule details.
  • Associated IPP trunk tags are detailed here
  • Table of IPP Version strings here IPP Version Numbers The IPP version used for a detection is populated in PSPS in the FrameMeta table as analysisVer.

Previous updates of loading PV1.2, see attached images below for rate of progress

UPDATE Sep 22, 2013 - merge 16 complete, total of 12,339,588,017 individual detections published, see revised second plot.

UPDATE Sep 15, 2013 - merge 15 complete, total of 11,368,688,203 individual detections published, see revised second plot.

UPDATE Sep 11, 2013 - merge 14 complete, total of 10,442,592,590 individual detections published, see revised second plot.

UPDATE Sep 7, 2013 - merge 13 complete, total of 9,743,767,217 individual detections published, see revised second plot.

UPDATE Sep 3, 2013 - merge 12 complete, total of 8,755,985,539 individual detections published, see revised second plot.

UPDATE Aug 28, 2013 - merge 11 complete, total of 6,765,887,406 individual detections published, see revised second plot.

UPDATE Aug 23, 2013 - merge 10 complete, total of 4,140,245,610 individual detections published, see second plot. Ingest of detections continues. Recall 5,670,289,421 objects, ~all sky~ already loaded.

UPDATE Aug 21, 2013 - 1.9 billion detections loaded. See updated plot. Merge 9. The region 4 < RA < 6 should be available by Friday, Aug 23. (Merge 10). At this point we think about 4 days to ingest 2 hrs of RA, or 8 days to do 4 hours RA. Adding one new machine to speed up loading.

UPDATE Aug 15, 2013 - 1,118,347,240 detections loaded. Merge 8.

UPDATE: August 12, 2013: ~ We have completed Merge 7 and finished loading 3pi PV1.2 objects for the short term. There are 5,670,289,421 objects available. We will wait till the Detections have finished before returning to the galactic center which is the missing region. F We will keep the following plots up to date after each merge.

*news archive

Last modified 3 years ago Last modified on 05/06/14 13:40:55

Attachments (31)