The "Campus Spatial de l'Université Paris Diderot"  invited me to give a talk entitled “Histoire du traitement des données spatiales” on September 20 2013. This was a personal account, closer to a random walk in the fog than to a true historic perspective on the subject. The text below is the transcription of the first part of this talk, further parts will appear later on this blog.

I thank A. Parmar, another actor of the EXOSAT observatory, for commenting this account and for mentioning that we did test the occultation method once.

EXOSAT, the first European X-ray observatory

To conceive astrophysical facilities in orbit as "observatories" meaning instruments available to all astronomers irrespective of their contribution to the development of the mission is now widely accepted as a form of operation that maximises the scientific output of the instrument. This was by far not always the case. It took large efforts and the success of EXOSAT for this modus operandi to establish itself.

While COS-B, a European gamma ray detector launched in 1975, had some opening for scientists outside the collaboration that had built the instrument, EXOSAT was the first truly open European high energy astrophysics space observatory.

EXOSAT was launched Mai 26 1983 on an highly inclined orbit with a period of four days (fig. 1). This orbit had been designed so that as many sources as possible could be localised on the celestial sphere using the precise timing of the ingress and egress of X-ray sources behind  the Moon as the latter moved in front of the source as observed from the satellite. One measures the time at which a source disappears behind the Moon and that when it reappears together with a precise location of satellite and Moon to reconstruct the position of the source. For this technique to be efficient one needs a satellite orbit such that as many sources as possible will be occulted by the Moon over the life of the mission and a large detector that gives a high signal to noise  with which the precise occultation times can be measured. This mode of observation was never used  operationally by EXOSAT mainly because of the inclusion of X-ray imaging telescopes on-board EXOSAT, but also because the orbit and attitude control system of the satellite was not up to the task.  However, the combination of a long period orbit with sensitive detectors proved a very efficient combination. X-ray sources could be   observed for long uninterrupted stretches of time. Long uninterrupted observations can indeed not be performed from orbits that are only few hundred kilometers from the Earth's surface. These orbits have periods of about 90 minutes and  the Earth covers half of the sky, as seen from the satellite. Sources are thus behind the Earth for about  45 minutes, half of the duration of a typical revolution, every 90 minutes.  On long period orbits,  the satellite spends most of the time far away from the Earth which then covers only a tiny fraction of the sky. Most sources could therefore be observed by EXOSAT for periods of some three days without interruption.   The variations of the luminosity of the sources were then measured, something that could not be done prior to EXOSAT.  Fig. 2 shows as an example a source observed for 18 hours with no interruption. The source, XB 1254-690, shows a periodical flux decrease and occasional bursts, a discovery that had been almost impossible prior to EXOSAT. The combination of sensitive detectors and long period orbits proved so successful that, since then, many X-ray observatories have been placed on similar orbits.

Figure 1: EXOSAT orbit

Figure 2: The light curve of the X-ray source XB 1254-690 (from Courvoisier T.J.-L., Parmar A.N., Peacock A. and Pakull M., Ap.J. 309, 265, 1986)

Another innovation was  the possibility to design and implement on board different ways of binning scientific data prior to telemetry download. As experience with the instruments was gained, new programs were developed in order to optimise the scientific return of the observations as a function of the type of source and the science expected. The corresponding programs were uploaded for individual observations by the ground staff. The imagination of the scientists was large, but the use of some of these modes was somewhat limited by the need to develop  analysis programmes and calibrations that matched the on board data binning. The most fruitful modes were certainly the standard ones. We learned through this experience  that simplicity in the data and stability in the modes over the months were virtues that should not be underestimated. This message was in part lost between EXOSAT and the infrared mission ISO launched also by ESA in 1995. One of the instruments had such a complex set of observation modes that some  data were taken in modes that had not been calibrated, these data were and remained useless.

Being a first X-ray "observatory", meaning that astronomers from any background could conduct observations, EXOSAT was unchartered territory for the design and operation of the scientific ground segment. Sizing the observatory sector of the ground segment was based on the assumption  that scientists would prepare an automatic analysis of the data for the outside observers, prepare the scheduling of the observations, calibrate the instruments, run all the necessary programs to schedule the observations and analyse the data, prepare the telecommands for the payload including selecting the mode of observing, host the guest observers and perform a few additional tasks. It was thought that two people, postdocs, should be present in the control room in permanence to fulfill these tasks. Since one position occupied 24h/day the whole year around implies that five(1) people are needed, ten  postdocs were hired in the  couple years before launch (of which I as one (2)). The control room where one of the two postdocs on shift was expected to sit permanently had a large panel covering the whole wall to display the main parameters of the payload and spacecraft. The whole thing looked pretty much like Hergé's imagination of flight control in "On a marché sur la lune" ("Explorers of the Moon" in English). But it should be remembered that less time elapsed between the publication of Hergé's comics and the launch of EXOSAT than between the latter and now.

The manpower estimate for EXOSAT's science ground segment proved a major underestimate of the work involved, even though the ten postdocs were immensely enthusiastic (I sometimes  had the impression that uploading all possible programs to optimise the observations was  like  driving  a fiat 500 in the streets of Palermo) and worked until shear exhaustion brought them down around the time of launch.  Hiring postdocs (on stipends rather than proper positions) to be responsible for the  payload was a very cheap way of acquiring hard working and very eager staff.  Since postdocs are not trained to not make errors, it also implied that a number of potentially disastrous mistakes could be done. I remember that early in the mission one of the on board programs had a bug that had to be patched every revolution by uploading a complex bit pattern in a given memory location. Unavoidably the theoretician that I was once exchanged the bit pattern and the address, an error that could have been fatal to the mission (and led to the programme to be permanently fixed).  About a year after launch  ESA had noticed that considerably more staff was needed to operate EXOSAT's observatory. The team was enlarged and working conditions improved. This experience notwithstanding, the habit of underestimating the resources needed to operate space observatories remained. This led again to considerable difficulties for the ISO mission in the mid 1990s.  

The naivety with which the task at hand was approached is illustrated by the fact that we expected the automatic analysis to be the main tool to access data. An interactive analysis was thought to be a second priority, to the point that shortly before launch we noted that we had no tool to look at the data in near real time. A weakness that was corrected at the last moment.

These weaknesses notwithstanding, the exercise made around EXOSAT was epoch making for a lot of high energy astrophysics missions. Orbits similar to that of EXOSAT were used numerous times, the  data analysis software structure was kept for subsequent missions, the data structures were used with appropriate changes on both sides of the Atlantic. This has been going on at least until the INTEGRAL mission  in the early 2000's.

The observatory nature of EXOSAT was difficult to establish. The scientists having built the instruments thought that it would be difficult, if not impossible, for outside observers including the observatory scientists to make sense of the data and therefore to publish meaningful results. These arguments have been used again and again, but EXOSAT and subsequent missions proved them to be wrong. The active participation of outside observers always enriches considerably the output of a given mission. For EXOSAT, but also for ROSAT, XMM-Newton and Chandra or for INTEGRAL, the publications by people outside the community that built the instruments is a large part if not the main contribution to the results of the mission. The same  arguments are now being used in the CTA  project (Cerenkov Telescope Array, a large set of instruments designed to observe the sky at very high energies). There again the science output of the instrument will hugely benefit from the contributions of the community at large.

An observatory approach does, however, dissociate somewhat the instrument development from the scientific exploitation of a given mission. The resulting distance should not be made too large to allow cross fertilization  between the communities. ESA has probably gone too far in this respect by establishing the science ground segment of its astrophysics missions in Spain, while retaining the project scientists in The Netherlands. A further point that must be given proper attention is that the contribution to science in general of the people developing and building instruments should be valued for itself and not be made dependent on  purely astrophysical papers.  The knowledge and skills needed to develop first rate instruments do not necessarily overlap with those needed to understand the physics of the objects to be observed. Promotion of instrument makers should rely on the results brought by the instruments, irrespective of whether they are obtained by members of the instrument building teams or outside scientists.

There are difficulties in the interface between instrument building and development on one side and  instrument exploitation on the other side. Some of these difficulties are inherent to this interface and exist in all missions at various degrees. They are due to the fact that analysis programs and tools must incorporate and encode the knowledge that has been acquired by the instrument developers over long periods of time. This knowledge resides in brains and is organized according to the needs of the developers. It must be transformed in data structures and computer code to be used by people more at ease with the objects observed or studied than with the physics of the detectors and instruments. Sometimes, if not often, the instrument developers have little confidence in the writers of software   expected to capture the details and finesse of their instruments. Experience shows that this lack of confidence is not justified. It is interesting to see that the SIGMA instrument, a complex hard X-ray detector developed by French teams in the 1990s was used very successfully by Soviet astrophysicists, although the latter had had no role in the instrument development. There is also a feeling of loss of control on the side of the developers when their instruments are used and operated by teams with large participation from circles that are not from within. These feelings often lead to tensions and result in instrument teams wanting to keep the authority on the software development. This I feel is a mistake. It is better to approach the real tensions described here by asking the observatory staff to acquire the knowledge of the developers and to embed it in the software to be used by the users. This scheme gives the interface a positive note: Rather than  to force  instrument teams to deliver products they would rather keep for themselves, it gives the responsibility for the analysis programmes to those who use, exploit and distribute them. Whatever the path used to develop the observatory software and the difficulties met in the different projects, the enormous success of these missions in the past decades is a tribute to the ingenuity and work of all those involved.  

(1) There is some doubt as to whether five or six people were needed in Germany at the time  of EXOSAT to cover one position 24h per day over the whole year.

(2) Interestingly a number of these postdocs stayed in high energy astrophysics. Some of them followed highly successful careers.