The “Campus Spatial de l’Université Paris Diderot” invited me to give a talk entitled “Histoire du traitement des données spatiales” on September 20 2013. This was a personal account, closer to a random walk in the fog than to a true historic perspective on the subject. The text below is the transcription of the third part of this talk, one  last part will appear later on this blog.

Patchy Elements of the Early  Relationship between Ground based and Space Astronomy in High Energy Astrophysics.

X-ray astronomy, which was in the late 1970s and early 1980s the core of high energy astrophysics, was being developed by people who were rather far from the main stream optical astronomy, at least for the part of the scene I was familiar with. The organisation of the EXOSAT observatory on ESOC's premises (ESOC: European Space Operations Center) in Darmstadt was led from ESA's science and technology center (ESTEC) in The Netherlands. There existed only anecdotal contacts with ESO, the European Southern Observatory (ESO) and its headquarters in Garching bei Muenchen. These contacts included buying the complete set of Schmidt photographical plates covering the whole sky and the optical system that was needed to look at them.  The idea was that EXOSAT duty scientists should be able to recognise on these plates the stars that the on-board star tracker would image in case of a loss of attitude. With hindsight, the thought of finding among hundreds of photographic  plates densely packed with stars the region imaged by the on board star tracker seems rather naive.

This contact with ESO, however, gave us an opportunity to meet astronomers and to understand how they organised data. The astronomical community was then already establishing standard formats that were well documented and could serve to transport data between different facilities. The format was called FITS for Flexible Image Transport System. Space data, and in particular the EXOSAT data, had also to be properly structured to be stored on tapes and accessible by a number of computer programs. The software engineers in the EXOSAT team developed the FOTS format for this purpose. This format had a number of specificities that were meant to optimise the use of tapes, but that rendered it rather difficult to use and incompatible with any other format at the time and thereafter. Recognising this incompatibility together with the slowly growing sense that combining data from X-ray and optical observatories might become fruitful and that  individual astronomers might work in both fields led in the late 1980s and in the early 1990s to developments of the FITS format to make it capable of dealing with the particularities of X-ray detectors. This opened the door for a wide (although not universal) use of this format across the whole astronomical community to this date.

Cross fertilisation between space astronomy and optical astronomy also took place in the domain of archives. Space data are, by definition, transmitted by radio waves to the ground where they must be stored one way or another. Ground based data were then obtained on photographic plates (I remember vivid discussions to assess whether the density of information on electronic media would become higher than that on photographic plates), on paper tapes and some early electronic media. The systematic approach to space data than slowly spilled over to optical data that became more and more organised and little by little accessible also to researchers other than those who had performed the observations. This process was made difficult by the sensitivity to meteorological conditions of the optical observations on the ground. The weather information therefore has to be embedded one way or another in the calibration data, a significant problem of which space astronomy is freed. Similarly space observations are pre-planned and often performed without direct link to the operators. This led to a systematic approach to observation procedures that also influenced the way in which large telescopes are being operated. Both movements translate now in a rather common approach to the operation  of major telescopes and their data archival on one side and space observatories on the other side.

In Europe part of this cross fertilisation was done by staff at the ST-ECF, the Space Telescope European Coordinating Facility. The ST-ECF was established jointly by ESA and ESO on ESO's premises in Garching to help European astronomers prepare to use the the Hubble Space Telescope HST.  The ST-ECF began working in 1984, at a time when it was thought that HST would be launched a couple years from then. The Challenger accident moved HST's launch much further in time, thus leaving time for many developments. Among them simulation tools that allowed prospective users to generate the HST data that were expected from what they knew of the source to be observed. Such tools have now become standard for all major instruments to the point that no observation  proposal can be submitted without prior simulation of the results. These tools are also extensively used in the design of instruments and the establishment of science cases needed to convince committees of the necessity to build them.  While this trend is fruitful in that it allows astronomers to avoid making observations of which one could know in advance that signal to noise would be insufficient to obtain the expected results, it can also be be overdone. The dangers are that expected results are often sought and acquired leaving less room for the unforeseen. One also sees now that decisions to fund and  build large instruments  or space missions have become very competitive and that the competition is carried along very far in the design process. One side effect of this evolution is that the scientific community, and in particular the space science community, spends considerable amounts of time simulating  observations with instruments which do not exist and, for most of them, will never be built. Efforts that I cannot help thinking could be spent more fruitfully.

The history of the relation between space and ground base astronomy is much richer than these few elements. It would be worth a complete study, but is, however, far beyond these short personal notes.