The WEAVE data are distributed on a regular basis to honour the WEAVE data policy.
Three types of data releases have been defined:
- Internal data release candidate (iDRC)
- Internal data release (iDR)
- Public data release (DR)
The WEAVE archive receives the raw data (from the telescope) and the reduced data from the official pipelines whenever they are available. Under normal circumstances, fresh raw data should be pushed to WAS once a day, while the L1-processed, L2-analysed and Contributed Software (CS) data coming from the Operational Repository (OR) should be transferred on a daily basis with a latency of no more than 5 days. The OR only provides data to the archive once all elements (CPS, APS, CS) are available.
The data products pushed to WAS populate the internal data release candidate. These data are subject to change and only a limited number of people may access the data with strong access restrictions. In particular, the internal data release candidate may be accessed by:
- Open time PIs
- WEAVE Survey Contributed Data Product (CDP) producers
- WEAVE Survey Science Team Leads (STLs)
- WEAVE Survey Quality and Assurance Group (QAG) members
Data products within an iDRC are subject to change without notice through re-processing of the CPS, APS or CS products. Whenever the data products change in an internal data release candidate, it is the responsibility of the user to download the most up-to-date products when they become available . This means that products from the internal data release candidate cannot be used in any published work (as set out in WEAVE-EXE-005, The WEAVE Publication Policy).
CDP producers may access and use the data from an internal data release candidate for the purposes of testing their codes, but they must re-download and re-analyse the data once they are formally released as an iDR.
Although the data continually trickles into the archive on a daily basis, the archive considers the data to be managed in time slots. The minimum time slot is a period of three months; the data publication in the iDR is therefore aligned to trimesters and always refers to a particular trimester. The starting and ending dates of each trimester are defined and available in WASP, although the information will probably be made available through the WAS as well.
WAS considers a trimester closed once it receives all the data from the telescope and the official pipelines (matching the dates which define that trimester for a given iDR).
The WEAVE data transferred to the archive day-by-day are not considered definitive until the trimester closes, as they may change due to the retraction of data . Data retraction may happen for reasons external to the archive. Moreover, CASU will generate supertargets on a monthly basis, which can lead to the pushing of these data products to WAS some days after the closing date of the trimester. This can lead to a delay regarding when the data become available to the community.
Once the trimester closes, the data of an iDRC are considered definitive and relabelled as an iDR . They may now be accessed by a larger group of users through the WAS. In particular, all users that are part of the WEAVE Consortium and those that are External Collaborators as defined in the Publication Policy and approved by the WEAVE Project Executive or Science Executive (or are included through a Memorandum of Understanding with the Project) may access the data. These data constitute an iDR, and they are reliable – only under exceptional circumstances would changes be made to an iDR.
Internal Data Releases have a numbering system that comprise two parts: iDRm.n, where m and n are integers, with m starting at 1 and n starting at 0. The very first iDR with “normal” data (i.e. not from Science Verification) is iDR1.0, and all iDRs up to the point where the first public Data Release (DR) is published has m=1, while n increases sequentially: iDR1.0, iDR1.1, iDR1.2, etc.
Every year starting 24 months after survey start, the WEAVE Consortium publishes a public Data Release. DRs are identified by a single integer as DRn, where n is an integer starting at 1. As an example, DR1 = iDR1.x and DR2 = iDR2.y, where x and y can be different integers , depending on how many iDRs have occurred before the publication of the next public DR. Once DR1 has been published, the next internal Data Release will therefore have the identifier iDR2.0.
Note that an exception to the above system is DR0, through which Science Verification (SV) data will be published. The stages of the internal data releases can be tested using SV data, in which case iDRs will be numbered as iDR0.0, etc., but this must be for testing purposes only, as they are not intended as actual iDRs for scientific purposes ahead of DR0.