The Adaptive Gain Integrating Pixel Detector at the European XFEL

A description of the 1 million pixel AGIPD system in use at the SPB beamline of the European XFEL is given.


Introduction
With the start of the European XFEL, a new milestone is set in the field of X-ray research and many related fields due to the high coherence, pulse intensity and repetition rate of the X-ray pulses available at this facility. The superconducting accelerator provides up to 600 ms-long trains with up to 2700 pulses followed by an inter-train gap of 99.4 ms. Inside each train, consecutive pulses of typically less than 100 fs duration are spaced approximately 220 ns apart. This corresponds to an effective repetition rate of 4.5 MHz during a train. Each pulse contains up to 10 12 photons (Altarelli, 2011), which in many cases is sufficient to produce a complete scattering pattern from the sample with a single pulse. This means that the area detectors at the European XFEL not only have to be compatible with the high repetition rate of the source but also need to have a dynamic range from single photons to 10 4 photons pixel À1 pulse À1 . More detailed and complete requirements are given by Graafsma (2009). Dedicated detector development programs for the European XFEL were started more than ten years before its inauguration, as it was clear that no existing detector would be able to meet the ISSN 1600-5775 requirements imposed by the new facility. The Adaptive Gain Integrating Pixel Detector (AGIPD) was designed to fulfil as many of the general requirements as possible with a focus on the requirements most important for scattering experiments in the energy range from 7 to 15 keV, for which it has been successfully used in first user experiments (Grü nbein et al., 2018;Wiedorn et al., 2018). There has been interest to use the system at other bright sources to study fast dynamics down to the microsecond scale.
The AGIPD is not the only camera developed for use at the European XFEL. The LPD system (Hart et al., 2012) is currently in use at the FXE beamline and the DSSC system (Porro et al., 2012) will be available soon. Other FELs are using other custom-developed camera systems like the CSPAD (Philipp et al., 2011) and ePIX detectors (Blaj et al., 2016) at LCLS or the JUNGFRAU detector (Redford et al., 2016) at the SwissFEL.

System layout
The AGIPD camera consists of four individually moveable quadrants, each having four detector tiles with 512 Â 128 pixels per tile, giving a total of 1024 Â 1024, or roughly 1 million pixels. Fig. 1(a) shows a CAD design of the AGIPD 1 million pixel detector with cuts to expose the arrangement of the electronics inside and outside of the vacuum vessel.
Each detector tile consists of a front-end module (FEM), an in-vacuum board that provides power to the FEM and routes signals, two ADC boards and a control and data IO board. Figs. 1(b) and 1(c) show a CAD drawing and a photograph of these boards, respectively.
Being glued to a ceramic board, each FEM consists of a monolithic pixelated silicon sensor responsible for absorbing the X-ray photons and creating an electrical signal per pixel which is proportional to the sum of the energies of all simultaneously absorbed photons in that pixel. The silicon sensor is bump-bonded to 2 Â 8 pixelated Application Specific Integrated Circuits (ASICs), which are responsible for signal integration and intermediate image storage.
The FEMs protrude into the attached sample interaction chamber and can be cooled to stabilize their temperature and improve performance. Although there are many components in the vacuum chamber, vacuum levels better than 10 À5 mbar have been reached in our laboratories. Note that this was without attaching the system to the sample interaction chambers at the European XFEL, so the vacuum levels obtained during XFEL experiments might be different.
The detector logically and electrically divides into two wings. Each wing consists of the ADC boards and control and data IO board of each tile (one set each per tile, eight tiles per wing) as well as a vacuum backplane board, which acts as a vacuum barrier and routes signals into and out of the vacuum vessel, a micro controller board for slow control and a master FPGA board. These boards are located outside the vacuum chamber 1 in a thermally sealed, water-cooled housing.
The two master FPGA boards, one for each side, provide the interface to the clock and control system, and control the detector tiles, including the FEMs. In the following sections the individual components and operational concepts are described in more detail.

Mechanics
Fig. 2(a) shows the front view of the AGIPD 1 million pixel system installed at the SPB station of the European XFEL. The four independently movable quadrants allow the formation of a horizontal slit or a rectangular central hole with userselectable size for the direct beam to pass through. The movement is realized by mounting each cooling block on a motion stage formed by two wedges [ Fig. 2 The cooling blocks were made by a combination of milling and electroforming techniques. In the first step, the basic shape of the cooling block was milled out of a solid block of hexagonal close-packed copper. This basic shape omitted details that would be included in the final milling step, but already included the cooling channels. To enhance the turbulence of the flow of the silicone oil coolant, holes were drilled into the channels and pins were inserted into the holes. Afterwards, the channels were covered by copper using an electroforming process. Initially omitted details, like connector feedthroughs, were defined in a final milling run, which also ensured the overall dimensions and tolerances of the cooling block. Figs. 2(c) and 2(d) show the channels with inserted pins before the cover deposition.

Front-end modules
An AGIPD 1 million pixel detector incorporates 16 FEMs. Each FEM uses a bump-bonded hybrid of a monolithic silicon sensor with 128 Â 512 pixels and 2 Â 8 AGIPD ASICs. The power and signal contacts of the ASICs are wire-bonded to gold-plated pads on an LTCC (low-temperature Co-fired ceramics) board. The bias voltage of the sensor is provided via wire bond connections between sensor and LTCC on all four corners of the assembly. A PT1000 temperature sensor is located on the back side of the LTCC, next to a Samtec 500 pin connector, which connects the FEM to the interface electronics. The hybrid assembly is glued to a silicon heat spreader, which reduces thermal gradients across the tile. For the bonding between heat spreader and LTCC an adhesive with high thermal conductance is used that is able to handle the different coefficients of thermal expansion of silicon and ceramics.
For mounting and handling, the wirebonded assembly is screwed onto an interposer made from a copper alloy which provides sufficient rigidity and heat conductivity. The interposer features an insertion pin for defined mounting to the cooling block. To reduce the thermal resistance between the layers, the interface between LTCC and interposer is filled with vacuumcompatible liquid gap filler and the interface between interposer and cooling block is filled with graphite. Fig. 3(a) shows a macro photograph of the edge of a FEM, detailing sensor, ASICs, heat spreader, LTCC and interposer. Fig. 3(b) shows two fully assembled FEMs and a bare interposer.
In our laboratory the system has been tested at temperatures as low as À20 C on the LTCC, but the system can run at temperatures above that, which was also the case for early user experiments at the SPB beamline. The system is also compatible with operation at ambient pressure and temperature.

The ASIC
The AGIPD 1.1 ASIC (Shi et al., 2010;Mezza et al., 2016a) forms the heart of the system; each ASIC incorporates 64 Â 64 pixels and the necessary readout and control circuitry. It is manufactured in 130 nm CMOS technology, using radiationhardened layout techniques in most parts of the circuitry. Fig. 4 shows a schematic diagram of the pixel circuitry: the input of each pixel is formed by a resettable charge-sensitive preamplifier, built around an inverter core. Its output feeds a discriminator and a correlated double-sampling (CDS) stage with two globally adjustable gain settings. Once the discriminator output exceeds the globally defined threshold, additional feedback capacitors of 3 or 10 pF are added to the 60 fF of the un-switched preamplifier feedback loop. This way the sensitivity of the preamplifier is adaptively decreased and the dynamic range is extended in two steps (Shi et al., 2012), where each pixel automatically and independently adapts its gain to the incoming signal. The CDS stage is used to remove noise from the reset switch and to suppress low-frequency noise components (Buttler et al., 1990).
The pixel response is recorded to an in-pixel memory at high speed, compatible with the 4.5 MHz requirement for the     European XFEL (Becker et al., 2011). Each memory address contains two separate pieces of information: (i) the output voltage of the CDS stage, which is proportional to the detected signal, is stored on a 200 fF capacitor, and (ii) the gain state of the pixel, encoded as a voltage, is stored on a 30 fF capacitor.
The memory matrix occupies about 80% of the pixel area and can store up to 352 images consisting of signal and gain information. The pixel size of 200 mm Â 200 mm is a compromise between resolution, analog performance and number of memory cells.
The memory can be randomly accessed, providing the option of overwriting images or frame-selective readout. At the European XFEL it is used to implement a veto system (Motuk et al., 2012).
During readout of the chip another charge-sensitive amplifier in each pixel is used to read the memory, which happens in parallel for each row of pixels. The further readout uses two interleaved column buses and four multiplexers, each serializing data from a block of 16 Â 64 pixels. This parallelizing onto four outputs reduces the power consumption of the readout circuitry by reducing its speed. Instrumentation amplifiers convert the signal to differential levels, which are driven off-chip for subsequent digitization.
A command-based control circuit provides all the signals for memory access, read and write operations to the pixels. It uses a three-line serial current mode logic interface and also provides slow control tasks, like the programming of internal timings and on-chip biases generated by digital-to-analog converters (Mezza et al., 2016b).

The sensor
Much like the ASIC, the sensor design was driven by the requirements set forth by the expected experiments and performance of the European XFEL (Graafsma, 2009). The main requirements were a thickness of 500 mm in order to reach sufficiently high quantum efficiency and a tolerance of a total dose of 1 GGy during the expected lifetime of the detector. All of this should be accomplished while making large-area sensors that allow minimizing the dead area of the final detector system.
In addition to the challenge of radiation tolerance, the impact of plasma effects due to the high number of instantaneously absorbed photons was investigated (Becker et al., 2010). When many photons are locally absorbed, sufficient electron-hole pairs are created to form a plasma. This plasma shields its core from the electric field required to drift the charges to the readout ASICs.
Diffusion will eventually expand the plasma and lower its density enough for the drift field to take over, but this process takes time. As a result the charge cloud will spread laterally, potentially degrading the spatial resolution, and the total charge collection time will increase, potentially piling up with the next photon pulse.
These studies concluded that the sensor should be constructed using p + electrodes in a highly resistive n-type bulk, thus collecting holes. Also a voltage of at least 500 V should be applied to the AGIPD sensor to suppress the consequences of the plasma effects as much as possible. At this voltage the time to collect at least 95% of the deposited charge is less than 60 ns for tightly focused spots (3 mm r.m.s.) of up to 10 5 photons of 12 keV.
Surface damage is the dominant type of radiation damage for the sensor, as the damage threshold for silicon bulk damage is far above 12 keV, the original design photon energy of the European XFEL.
Therefore, surface damage, namely the creation of positive charges in the SiO 2 , and the introduction of traps at the Si-SiO 2 interface, were studied in detail with numerous irradiation campaigns  to establish the relevant parameters, which in turn were used for sensor optimization studies (Schwandt et al., 2012).
These studies showed that several design parameters, i.e. oxide thickness, pixel implant depth and metal overhang, have significant influence on the ability to operate the sensor at a high voltage. Some of the findings show conflicting results for the situation before and after irradiation, i.e. a thick oxide is preferred before irradiation and a thin oxide after irradiation (Schwandt et al., 2013a). Schematics of the AGIPD 1.1 readout ASIC.
Finally, a compromise layout was chosen that fulfilled the design specifications and showed a breakdown voltage above 800 V in simulation (Schwandt et al., 2013b). This layout was produced and its radiation tolerance was studied , finding sufficient performance for operation at the European XFEL.
The dead area of the final system is minimized for each FEM by using a monolithic sensor bump-bonded to 2 Â 8 individual ASICs. The large sensor guarantees that there are no blind spots within the sensitive surface (0% dead area); however, since the ASICs cannot physically touch each other, the pixels horizontally in between two ASICs are twice as wide (400 mm Â 200 mm), covering the area of two ordinary pixels. Pixels vertically in between two ASICs are of normal size.
The entire sensitive area is surrounded by a guard ring that is 1.2 mm (or six pixels) wide. A detailed description of the sensitive and non-sensitive areas and the implications for coherent diffraction imaging at the SPB instrument of the European XFEL are given by Giewekemeyer et al. (2013). Disregarding the intentional gap between quadrants caused by the moving apparatus, each quadrant has less than 15% insensitive area, which increases to less than 18% for the whole system, still disregarding the intentional gaps.

Signal handling, data preparation and control
As indicated earlier, the AGIPD 1 million pixel detector is made of two electronically independent halves. Each half consists of a master FPGA board, a micro-controller-based slow control unit, an eight-port vacuum barrier board and eight tiles consisting of FEMs and their readout boards.
The eight-port vacuum barrier board, which interconnects between the different boards, is realized using a multi-layer printed circuit board which acts as a vacuum barrier and as a backplane for signal distribution.
The master FPGA acts as an interface device between the detector, the clock and control system of the European XFEL (Motuk et al., 2012) and the control PC. It receives configuration 'start' and 'stop' signals from the control computer, receives bunch synchronized clocks and classification flags, synchronizes the operation of the ASICs and fast ADCs, and triggers the readout FPGAs of the tiles.
The boards constituting the readout electronics of each tile are located either inside the vacuum vessel or in the enclosed air environment of the external housing.
From each FEM, 64 differential analog signal lines are guided via the in-vacuum board and vacuum barrier board to the analog boards. The flexible part of the in-vacuum board [indicated in Figs. 1(a) and 1(b)] allows the movement of the quadrants.
The ADCs (Analog Devices AD9257) sample at a resolution of 14 bit and operate at a frequency of up to 33 MHz, which results in a minimum possible readout time of approximately 22 ms for 352 frames (signal amplitude and gain state information are read separately, no overheads), well within the 99.4 ms inter-train spacing at the European XFEL. At 14 bit resolution the ADC noise and the quantization errors do not contribute significantly to the overall noise of the system.
Each readout FPGA (Zimmer & Sheviakov, 2012;Xia et al., 2014) orders the information of the 64 ADC channels of its tile into a single frame and sends the data via a 10 GbE optical link to the data acquisition system of the European XFEL.
The slow control board monitors the status of the detector by collecting data on supply currents and voltages, as well as temperature, humidity and cooling fan information.
During the detector start-up phase the experimental control computer sends commands to the micro controller to power the electronics sequentially. Collected monitor and status information can be queried by the slow control computer of the experiment which adds time stamps and makes the data available to the XFEL system.
In addition, the slow control board serves as a second-level interlock, transmitting hard-wired flags to the experimental programmable logic controller (PLC) to initiate a shutdown, if operational conditions are outside pre-defined safety margins. The architecture of the electronics is discussed in more detail by Goettlicher et al. (2015).

Calibration
The performance of the AGIPD ASIC has been extensively tested and documented over the years (Mezza et al., 2016a,b). Since every pixel can be in any of its three gain states in any image (i.e. memory cell), the current calibration procedure calibrates each memory cell individually for all three gain settings requiring more than 2.8 billion calibration parameters for a 1 million pixel detector system. In a big system of one million pixels or more, calibrating all relevant parameters using only external sources is not feasible in a time-efficient manner. Therefore the ASIC includes two internal calibration sources -a current source and a pulsed capacitor -to enable dynamic range scans without the need for external stimulus. Both sources combine elements that are global to the chip and local in each pixel. The current source is implemented as a distributed current mirror and each pixel features a calibration capacitor while the voltage step is generated in the periphery.
Each data point from the detector contains two values per pixel per memory cell: the analog signal value and the encoded gain state information. Each value is expressed as analog-todigital converter units (ADU) in the raw data stream of the detector.
The internal sources allow the determination of valuable information. For the analog signal value, that is the relative gain of all three states within each pixel and the relative gain of the pixels with respect to each other. For the encoded gain state used, that is the discrimination threshold that indicates the different states of the gain.
The offsets for each gain state are measured without illumination (dark frames) and the gain of the high-gain state is measured for each pixel using a flood illumination with characteristic X-rays. The characteristic X-rays were generated by illuminating a metal foil with an X-ray tube. The distance research papers 78 Aschkan Allahgholi et al. The AGIPD at the European XFEL between foil and sensor was approximately 10-20 cm, depending on the foil, to ensure sufficient count rate in all pixels without creating a strong gradient in count rate over the module.
In a last step the calibration data are matched and merged, resulting in independent calibration constants for each memory cell in each pixel. Each memory cell is characterized by eight parameters: three offsets (in ADU, one per state), three gain values (in ADU keV À1 , one per state), and two thresholds (in ADU) for state discrimination. This totals more than 2.8 billion calibration parameters in a 1 million pixel detector system. Fig. 5 shows examples of intermediate results during the calibration procedure. Fig. 5(a) shows the histogram of 10 000 analog signal values measured for a single memory cell of a single pixel. The data were acquired using illumination with characteristic X-rays from a molybdenum foil. The absolute gain of this memory cell of this pixel is given by the distance between the peaks in the histogram (noise to single photon or photon to photon). The average gain value of a typical FEM is 7.7 ADU keV À1 AE 3.4%.
The integration time during data taking was increased to 50 ms, a value significantly beyond the 130 ns typically used for experiments at the European XFEL. This was done to increase the number of detected photons per frame to at least 1000 and thereby reduce the statistical uncertainty of the fit. Due to the long integration time the noise of the system (width of the peaks) is much higher than for typical experimental conditions. The noise measurement of the high gain state is extracted from dark frames taken under experimental conditions (not shown), and typical values for the equivalent noise charge are 320 and 240 electrons in CDS gain low or high, respectively. For the other gain states the noise is higher, but still below the Poisson limit (Mezza et al., 2016a).
The internal current sources allow extrapolation of the absolute calibration of the high-gain state to other states. The (a) Histogram of 10 000 frames for a single memory cell of a single pixel illuminated with characteristic X-rays from molybdenum. The gain factor is derived from the mean peak-to-peak distance. (b) Using the internal current source all three gain levels can be sampled. (c) The pulsed capacitor samples only the high-gain state and parts of the medium-gain state. The non-ideal transition between the two gains is caused by the finite bandwidth of the calibration circuit. procedure using the internal current source utilizes injecting a constant current to the input of the pre-amplifier while sweeping the integration time.
An example of a current source scan is shown in Fig. 5(b). From linear fits to the data the ratios of low, medium and high gain are determined. 2 The high-and medium-gain regime can also be scanned with a pulsed capacitor as shown in Fig. 5(c).
The pulsed capacitor scans the dynamic range by gradually increasing the height of the applied voltage step while keeping the integration time constant. The encoded gain state values and the corresponding discrimination thresholds can be extracted from both current source and pulsed capacitor scans, and are shown by the green dots in Figs. 5(b) and 5(c).
Recording the required data to extract the calibration data typically takes 12-14 h and occupies more than 20 TB of disk space. The largest contribution to this is the current source scan, which typically takes 10-11 h and generates roughly half of the data volume. The smallest contribution to this is the X-ray illumination, which is the only time the detector must be illuminated during the calibration data taking process, with about 30 min duration and 0.4 TB of data volume.
Analysis of the data on the DESY MAXWELL cluster using routines that have been optimized for parallel data processing can be performed in less than a day.
The electrical calibration described so far is essential for every experiment. In addition to this, the mechanical calibration, which determines the position of each pixel in all three dimensions and especially in relation to the beam, is of great importance for many experiments as well.
The detector mechanics are designed and built with high precision, but the movability of the four quadrants requires a quick, robust and accurate method of calibration for the position and orientation of the FEMs at any time. On top of the uncertainties introduced by the movements, small displacements and tilts of the FEMs 3 with respect to ideal positions are unavoidable as their manufacture involves gluing steps which are naturally limited in their positioning accuracy.
A commonly used approach to calibrate the absolute position of each FEM in space is to take diffraction data of a well known sample and fit the detector geometry (White et al., 2012).
The detector has just entered routine operation at the European XFEL and first crystal structures have been successfully determined (Grü nbein et al., 2018;Wiedorn et al., 2018). For these experiments the detector was calibrated using the procedures described here. However, the procedures and recommended intervals are likely to evolve as the system becomes used more often.
Currently, we recommend to record dark data for offset determination before and after each practical block of scientific data taking to account for any small drifts, e.g. of the temperature, that might occur during the scientific data taking process. Rechecking of the absolute gain and the gain ratios between the gain states using X-rays and the internal sources should be done periodically, as these might change over longer periods of time, e.g. due to accumulation of radiation damage.
While the system has been designed to be radiation hard, as detailed in earlier sections, forecasting the point at which radiation damage effects do show up is currently not possible for us.
Further, we recommend an electrical recalibration of the detector every time there has been a beam damage incident and if modules were exchanged. Mechanical position calibration is recommended every time the detector position is changed unless knowledge of the exact module positions is not required for the experiment.
Finding a quicker way to reliably determine high-quality calibration constants is an ongoing development effort together with the detector group from the European XFEL.

Performance data and imaging example
In order to test the noise performance over the entire dynamic range a pulsed IR laser was used, where the deposited energy in a pixel was varied between 1 and 10 4 12.4 keV photon equivalents by reducing the intensity of the IR pulses with calibrated filters.
The noise of the system is higher in medium-and low-gain mode (approximately equivalent to 3.5 keV and 18 keV, respectively), but it was shown that it always remained significantly below the Poisson noise of the incoming signal (Mezza et al., 2016a;Allahgholi et al., 2014). The same IR laser was used to scan the dynamic range of the system, which was measured to be 34.4 Â 10 6 electrons, corresponding to approximately 10 4 photons of 12.4 keV. The non-linearity of the low-gain state proved to be better than 0.5% up to 5 Â 10 3 photons (Allahgholi et al., 2014). Fig. 6(a) shows an X-ray image of a printed circuit board taken with the setup shown in Fig. 6(b). The shown X-ray image is the average of 10000 individual images.
Each individual image is corrected for pixel offset and gain on a per pixel basis. These corrections compensate fixed pattern 'zero-level' variations and pixel-to-pixel sensitivity variations that are commonly caused by process variations during the production of the ASIC, but can have many causes.
Some artifacts remain after the correction. Particularly in the medium-intensity region, vertical stripes can be observed. These originate in the double-sized pixels (400 mm Â 200 mm) between ASICs. These pixels collect, on average, twice the amount of photons, hence they appear brighter. Fig. 6(c) shows an X-ray image of a pen drive. For this image, 30 000 individual images were averaged.
Each image was offset and gain corrected as explained above; in addition, a flat-field correction was applied. The flatfield correction accounts for the effective size of the pixels, removing the effect of the double-size pixels, and removes artifacts from non-uniform illumination. Compensating the double-sized pixels with a flat-field correction is neither the research papers 80 Aschkan Allahgholi et al. The AGIPD at the European XFEL only nor necessarily the best approach to correct for these pixels. 4 The image not only shows the structure of the plastic cover of the pen drive but also the sticky tape which was used to hold the pen drive to the acrylic glass in front of the FEM, demonstrating the high sensitivity of the system.
Lastly, the results of first-user experiments show that the detector is capable of determining protein crystal structures for both known and previously unknown proteins (Grü nbein et al., 2018;Wiedorn et al., 2018). Of course, this success is possible in combination with all the other infrastructure of the beamline at the European XFEL.

Summary
The Adaptive Gain Integrating Pixel Detector, AGIPD, is an X-ray camera developed for use at the European XFEL. It was officially inaugurated together with the European XFEL in August 2017. This paper has reviewed the complete system of the AGIPD 1 million pixel camera currently installed at the SPB beamline of the European XFEL.
The system has a complex mechanical mounting that includes an in-vacuum movement system and many electronic boards, some of which are inside the vacuum vessel, some outside in an external housing. Its four independently movable quadrants can be arranged to form a horizontal slit or a rectangular hole with user-selectable size.
The system is built from monolithic blocks of 2 Â 8 ASICs forming a matrix of 128 Â 512 pixels of 200 mm Â 200 mm size 5 . Each pixel automatically adjusts to the incoming signal such that it can detect any number of photons from single photons to 10 4 photons of 12.4 keV above its noise floor. Images are stored in one of 352 memory cells during the XFEL pulse train and read out in between trains.
The detector noise is approximately 1 keV (750 eV for high CDS gain), which is sufficient to detect single photons in many experiments, and most of the early experiments at the SPB beamline have used the AGIPD with great success (Grü nbein et al., 2018;Wiedorn et al., 2018).
The MID beamline of the European XFEL is scheduled to have a similar system installed in 2018.