A pipeline for megahertz X-ray photon correlation spectroscopy on soft matter samples at the MID instrument of European XFEL

In this article, we present the experimental protocol and data-pro­ces­sing framework for megahertz X-ray Photon Correlation Spectroscopy (MHz-XPCS) experiments on soft matter samples implemented at the Materials Imaging and Dynamics (MID) instrument of the European X-ray Free-Electron Laser (EuXFEL). Due to the introduction of a standard con­fig­u­ra­tion and the implementation of a highly automated data-pro­ces­sing pipeline, MHz-XPCS measurements can now be conducted and analyzed with minimal user intervention. A key challenge lies in managing the extremely large data volumes generated by the Adaptive Gain Integrating Pixel Detector (AGIPD) – often reaching several petabytes within a single experiment. We describe the technical implementation, discuss the hardware requirements related to effective parallel data pro­ces­sing and propose strategies to enhance data quality, in particular related to data reduction strategies and an improvement of the signal-to-noise ratio. Finally, we address strategies for making the processed data FAIR (Findable, Accessible, Interoperable, Reusable), in alignment with the goals of the DAPHNE4NFDI project.

1. Introduction

The development of hard X-ray sources with improved coherence properties, such as X-ray free-electron lasers (XFELs) and diffraction-limited storage rings, makes X-ray photon correlation spectroscopy (XPCS) a powerful tool for studying dynamics in hard and soft condensed matter (Grübel et al., 2007; Zhang et al., 2018; Perakis & Gutt, 2020; Lehmkühler et al., 2021). These sources provide beams of high spatial coherence allowing detection of speckle fluctuations in the scattered intensity to reveal the nanoscale dynamics of proteins, polymers and other biological or soft matter structures. As these advanced X-ray sources become more prevalent and accessible, XPCS is increasingly being exploited in the study of dynamic processes in soft matter and biological systems (Möller et al., 2019; Begam et al., 2021; Girelli et al., 2021; Reiser et al., 2022; Anthuparambil et al., 2023; Dallari et al., 2024; Otto et al., 2024). This development opens new avenues for the routine investigation of com­plex time-dependent phenomena in biology and soft matter science.

This progress is in some ways equivalent to the development of biological small-angle X-ray scattering (BioSAXS), which evolved from the early conception as a specialized method into a routine tool in structural biology with the emergence of third-generation synchrotron radiation sources (Blanchet & Svergun, 2013; Gräwert & Svergun, 2020). The widespread adoption of BioSAXS was propelled by the development of dedicated experimental setups (Classen et al., 2013; Acerbo et al., 2015; Martel et al., 2012; Pernot et al., 2013; Thureau et al., 2021; Cowieson et al., 2020; Blanchet et al., 2015; Fujisawa et al., 2000), data-analysis tools (Förster et al., 2010; Manalastas-Cantos et al., 2021; Hopkins, 2024) and a common database for experimental data (Kikhney et al., 2020), facilitating non-expert users to obtain reproducible and high-quality results. Similarly, the establishment of standardized protocols and data-analysis tools will be crucial in making XPCS accessible for a broader scientific community, for instance, to contribute to a better understanding of the role of dynamics in phase transitions and cellular functions.

The European X-ray Free-Electron Laser facility (EuXFEL) is the first XFEL to produce ultrashort hard X-ray pulses in bursts of megahertz (MHz) repetition rate (Decking et al., 2020). This high repetition rate, combined with the exceptional (laser-like) spatial coherence properties, enables MHz X-ray photon correlation spectroscopy (MHz-XPCS) to probe diffusive dynamics with (sub-)microsecond temporal resolution (Lehmkühler et al., 2020; Dallari et al., 2021a; Lehmkühler et al., 2021; Madsen et al., 2021). The resulting time and length scales match typical diffusion processes in dense cellular environments, therefore enabling the study of com­plex many-body interactions between proteins and their solvent at molecular length scales (Girelli et al., 2021; Reiser et al., 2022; Girelli et al., 2025; Anthuparambil et al., 2026). This capability allows MHz-XPCS to access a range of length and time scales that optical techniques, such as dynamic light scattering (probing larger length scales) and neutron spectroscopy techniques (measuring faster time scales), cannot reach.

However, the high repetition rate in combination with the use of pixelated two-dimensional detectors to record the data results in very large data volumes. This requires advanced data-handling solutions, including high-performance com­puting and efficient data storage systems (Malka et al., 2024), as well as specialized analysis software to effectively manage and quickly calculate the desired correlation functions. In this work, we outline the implementation of an experimental con­fig­u­ra­tion, as well as a data-analysis pipeline (EuXFEL, 2026a), that was customized to the burst-mode acquisition scheme of EuXFEL. The described experimental setup and tools are available for all users of the Materials Imaging and Dynamics (MID) instrument of EuXFEL (Madsen et al., 2021).

2. Experimental setup and measurement protocol

The MID instrument of EuXFEL is used in the standard SAXS con­fig­u­ra­tion, described in detail by Madsen et al. (2021). X-rays are generated by the SASE-2 undulator with a typical photon energy of ∼10 keV. The stochastic self-amplified spontaneous emission (SASE) process results in a pulse bandwidth of ∼30 eV. Narrower bandwidth radiation, with correspondingly improved longitudinal coherence, is optionally available via hard X-ray self-seeding (HXRSS) (Liu et al., 2023; Boesenberg et al., 2024) or by use of crystal monochromators (Petrov et al., 2022; Tasca et al., 2023; Zozulya et al., 2025). The use of HXRSS is preferred, as less contrast reduction with increasing scattering vector q is observed due to the improved longitudinal coherence, but also results in a longer setup time prior to the experiment and more experimental com­plexity. The settings and performance of the XPCS analysis pipeline are not affected by this choice.

The X-ray beam is collimated and subsequently focused onto the sample positions using two transfocator units equip­ped with beryllium com­pound refractive lenses (CRLs) located 229 and 931 m from the source. The sample position is located 959 m from the source. The sample chamber hosts different stages for sample manipulation, including a hexapod (H-840, Physik Instrumente) and a fast scanner stage [see Fig. 1(a)]. This scanner stage features an exchangeable frame system, which can be loaded via a load-lock system without venting the sample chamber. Standardized frames, which can hold up to 15 glass capillaries, are available for users [Fig. 1(b)], but customized solutions are also possible. Measuring samples in air is possible as well, enabled by the use of two diamond windows upstream and downstream of the interaction region (Madsen et al., 2021).

Figure 1 (a) Scanner stage and (b) holder for 15 capillaries used at the MID instrument. (c) Schematic of the beam trajectory (dashed lines) during the scan of a single capillary in one measurement run.

XPCS experiments, especially in SAXS geometry on soft matter samples, are often limited not by the coherent flux but rather by beam-induced effects that influence the sample's structure and/or dynamics. Beam-induced dynamics have been observed even for hard condensed matter samples (Ruta et al., 2017; Pintori et al., 2019), however, for absorbed doses in the range of megagray (MGy), and beyond. Protein samples in solution can typically tolerate a dose up to 1–10 kGy (Jeffries et al., 2015) at dose rates not exceeding ∼1 kGy µs⁻¹ in a time window up to several tens of microseconds (Reiser et al., 2022; Timmermann et al., 2023). Therefore, the experimental task is to optimize the signal-to-noise ratio (SNR), while minimizing the dose and dose rate on the sample. Experiments are per­formed with only modestly focused X-ray beams, which, in return, requires large sample–detector distances to retain sufficient speckle contrast despite the smaller speckle size. In the case of the MID instrument, this is realized by a beam size of ∼15 µm and by placing the detector at ∼7 m downstream of the sample. At a typical photon energy of ∼10 keV, this results in a relatively low speckle contrast of only a few percent, since the speckle size of about 58 µm is much smaller than the pixel size of the Adaptive Gain Integrating Pixel Detector (AGIPD) (200 µm) (Allahgholi et al., 2019b; Sztuk-Dambietz et al., 2024). Next-generation detectors with smaller pixel size would improve the situation, possibly matching the pixel and speckle size for optimal SNR (Lumma et al., 2000; Möller et al., 2019; Madsen et al., 2020). Experiments with a significantly longer sample–detector distance are not possible due to space constraints and the reduced scattering angle range covered by the detector, and a smaller X-ray focal spot size is also not favoured due to beam heating/damage effects (Lehmkühler et al., 2020; Xu et al., 2022; Dallari et al., 2024; Tanner et al., 2025).

Another means to optimize the SNR while enforcing the radiation dose constraints is spreading the measurements over a large scattering volume (many sample positions), with only a limited number of frames acquired at each position (Möller et al., 2019; Reiser et al., 2022). This is enabled by the burst-mode operation of EuXFEL (Decking et al., 2020), which delivers a short train of X-ray pulses (up to a few hundred) every 100 ms. A modest translation speed of the sample, typically 0.4 mm s⁻¹, results in the separation by multiple beam spot sizes between two consecutive trains, while the translation is negligible (<100 nm) within a train. Therefore, an independent correlation function is measured with each train at a fresh spot on the sample [Fig. 1(c)]. This acquisition scheme not only distributes the radiation dose evenly on the sample but also allows the application of a necessary data correction scheme, as detailed in the next sections.

3. Data-pro­ces­sing pipeline

A schematic overview of the subsequent data-handling and pro­ces­sing steps is shown in Fig. 2 using representative scattering data from ferritin in solution for illustration. The different steps indicated in the figure will be described in detail in the following sections.

Figure 2 Sketch of the data-acquisition and pro­ces­sing workflow for MHz-XPCS at the EuXFEL. Scattering patterns are recorded by the AGIPD at acquisition rates matching the EuXFEL time structure (X-ray pulses spaced by several hundred nanoseconds are delivered in pulse trains at a 10 Hz repetition rate). Each frame is labelled with a unique combination of pulseId and trainId. The raw detector data are processed by the dedicated EuXFEL calibration pipeline, which produces corrected datasets (`proc' data). Once the processed detector data become available on the offline cluster, the MHz-XPCS pipeline initiates further pro­ces­sing steps, including geometry and direct beam refinement, outlier detection, additional filtering, region-of-interest (ROI) specification, etc. The train-resolved two-time correlation functions and off-correlations are then calculated. Following several post-pro­ces­sing steps (additional threshold filtering, averaging, etc.), the final output is generated and made accessible to the users. An online XPCS pipeline – currently under development – receives and processes data streamed directly from the online calibration pipeline and will be described in a separate publication.

3.1. Data acquisition and EuXFEL calibration pipeline

The coherent small-angle X-ray scattering data are col­lected using the AGIPD in high-CDS mode (Allahgholi et al., 2019b; Sztuk-Dambietz et al., 2024). In this con­fig­u­ra­tion, the adaptive gain feature of the detector is not supported, but the mode offers a slightly better sensitivity for single-photon detection, which helps to mitigate certain detector artifacts, as explained in later sections. The detector com­prises n_mod = 16 individual modules, each having a dimension of n_pix = 512 × 128 pixels.

The AGIPD was specifically developed to capitalize on the opportunities provided by EuXFEL's burst mode and is capable of recording data at frame rates of up to 4.5 MHz. During each pulse train, up to n_p = 352 scattering acquisitions can be stored locally in the pixels and subsequently read out 10 times per second corresponding to the repetition rate of trains (burst mode). For a typical recorded run (the term used at EuXFEL to describe an act of data acquisition, analogous to the meaning of scan at storage rings) containing n_tr ≃ 2600 trains, this results in an overall volume of n_tr × n_p × n_mod × n_pix × 2 bytes × 2 ≃ 3.8 TB for ∼4.5 min of data collection. Although the last factor of 2 is due to simultaneous recording of intensity and gain stage information, which can be omitted here, this data rate still generates several petabytes (PB) of data over the course of a few days of experiment (Sztuk-Dambietz et al., 2024; Sobolev et al., 2024).

To convert the raw detector output into corrected data suitable for analysis, two calibration pipelines have been developed at EuXFEL, with the main concepts and technical details described in Schmidt et al. (2024), Sobolev et al. (2024) and Sztuk-Dambietz et al. (2024). A distinction is made between online and offline analysis, which are implemented in parallel. Details on the general data-analysis software and infrastructure at EuXFEL are available in EuXFEL (2026b). The online pipeline corrects and streams part of the detector data to dedicated analysis nodes, enabling immediate data analysis and visualization. This part of the analysis aims at minimum latency in order to guide the running experiment, but with potentially only a subset of the overall data being analyzed. An XPCS analysis pipeline based on online data is currently being developed and will be described in a later publication.

The offline pipeline (EuXFEL, 2026a) is based on the input from EuXFEL's HDF5 data files recorded, stored and transferred to the DESY com­puting infrastructure (Maxwell cluster) (DESY, 2026a) prior to being calibrated. Here, the objective is to process the full dataset with the highest possible quality, accepting higher latency. These data are further processed by the XPCS pipeline (dashed frames in Fig. 2) described in the present article.

Correction procedures specific to the AGIPD have been developed and refined in the early years of EuXFEL operation (Sztuk-Dambietz et al., 2024). While high dynamic range and gain switching are essential for some applications, the quality of XPCS data depends on the noise characteristics and single-photon sensitivity. To this end, flat field corrections based on Cu fluorescence data, common-mode corrections and conversion to absolute photon numbers (`photonizing') have been shown to significantly improve the quality of the XPCS results (Dallari et al., 2021b). These correction steps are now integrated into the standard AGIPD calibration pipeline used in user experiments (Sztuk-Dambietz et al., 2024). Additionally, pixels are filtered based on a set of criteria, from which a so-called bad pixel mask (calibration mask) is generated. The corrected data is termed `proc' and saved in designated HDF5 files. Due to photonizing (integers) and the sparse nature of XPCS data, these files exhibit com­pression ratios of ∼50 (∼25 when the calibration mask is included) (Sobolev et al., 2024).

While these correction steps already improve the data quality significantly, certain detector artifacts remain visible in the two-time correlation functions (TTCFs) (Dallari et al., 2021b). These artifacts are linked to drifts and jumps of the pedestal, which are not always caught by the common mode correction (termed `jumping' pixels, see Appendix A for details). Due to the storage cell and pixel read-out architecture of the AGIPD such `jumping' pixels frequently affect blocks of 32 storage cells simultaneously, resulting in square-like features in the TTCF (an example is shown later in Fig. 12). The random occurrence of these jumps cannot be assigned to a few damaged pixels, suggesting they are an intrinsic feature of the AGIPD. Therefore, special attention must be given to identifying and correcting these `jumping' pixels in each recorded dataset. Implementing these additional correction steps imposes a com­putational cost: the data set must be loaded twice for outlier detection prior to correlation calculations (Section 3.4) – and the number of calculated correlations is effectively doubled due to the off-correlation corrections (Section 5). Hence, in Section 7, we discuss possible scenarios to further improve the data quality and reduce the data footprint in the future.

3.2. Mean intensity calculation

As a first step of the XPCS pipeline, the mean intensity is calculated for each storage cell of every pixel. Fig. 3 illustrates examples of a single acquisition, along with subsequent averages over an increasing number of frames. This initial step is required to locate detector pixel outliers (see Section 3.4 below), that need to be masked in the subsequent calculation steps, and additionally offers the possibility of refining the scattering geometry. Pixels already marked as bad pixels by the standard calibration pipeline are already masked and excluded at this stage. In addition, users can supply a static mask for excluding shaded regions on the detector and apply filtering with respect to the sample scanner position [to remove data col­lected during illumination of the sample holder frame, see Fig. 1(c)].

Figure 3 2D scattering patterns of ferritin in solution col­lected with the AGIPD shown at different levels of averaging. (a) Single pulse, (b) single train averaged over 200 pulses, (c) average over 10 trains and (d) single run averaged over 2200 trains. The colour scale indicates the number of photons per pixel.

The mean intensity is then calculated as the intensity averaged over trains:

with t_n and p_n designating the trainId and the pulseId, respectively; m = {0; 1;…; 15} enumerating the detector module; and x and y being the pixel coordinates (integers) within each module. Calculating the mean intensity of the pixels using equation (1) benefits from the structure of the AGIPD data, which are saved for each of the n_mod = 16 modules in separate HDF5 files. Furthermore, the data are divided into groups of 250 trains (`sequences') for each sequence file. For a typical run containing approximately 2600 trains, this results in ∼176 module/sequence files (n_mod × 11), which can be processed efficiently in parallel using multipro­ces­sing.

Performing large-scale parallel I/O operations, even on high-performance com­puting resources, such as the Maxwell cluster, can be demanding and saturate the available bandwidth. Hence, we benchmarked the parallel read-out of data required for calculating equation (1) using a varying number of CPU cores. Based on the results shown in Fig. 4, we chose a default value of n_core = 50 cores for this task (with the option to override this value in the specific pipeline call). To avoid running out of memory, the data are loaded train-wise by each core and summed within the existing data container. This results in ∼20 GB of simultaneous memory load when using the default number of cores [n_core × n_p × n_pix × 17 bytes = 19.6 GB with the factor 17 corresponding to the size (byte) of containers to load the image/mask and save the data].

Figure 4 Performance of the parallel read-out of AGIPD data files as a function of the number of CPU cores used on the DESY Maxwell cluster.

3.3. Scattering geometry

A quantitative evaluation of the X-ray scattering data requires calibration of the momentum transfer (scattering vector) q, which depends on the exact geometry of the scattering experiment. To support a high level of automatization, we have implemented automatic retrieval routines into the data pipeline.

The 16 separate modules of AGIPD are grouped in four quadrants (with four modules forming each quadrant). Each quadrant can be translated in both vertical and horizontal directions for flexible coverage of different regions in reciprocal space (Allahgholi et al., 2019a). The respective motor positions are recorded alongside the scattering data and are used to calculate the position of each quadrant of the detector using the `extra-geom' package of EuXFEL (EuXFEL, 2026c).

To automatically determine the beam centre, defined as the position of the incident directly transmitted beam intersecting the detector plane, we exploit the azimuthal symmetry of the scattering signal and the fact that the mean detector image is already held in memory for the XPCS analysis. An initial estimate of the beam-centre position can be provided by the user or set by the pipeline to the geometric centre of the central rectangular gap between the detector quadrants. Sub­se­quent­ly, an iterative optimization algorithm is applied, which slices the mean detector image into eight equal azimuthal ranges [as shown in Fig. 5(a)] and carries out separate azimuthal integrations of the intensity (Ashiotis et al., 2015; Kieffer et al., 2020). This procedure is repeated with different direct beam positions and iterated to maximize the overlap among the resulting azimuthally integrated intensity profiles I(q). The search for a direct beam position is per­formed within the central gap of the AGIPD. An example of the refinement process is shown in Figs. 5(b) and 5(c). In the case of an anisotropic scattering signal, this procedure can be switched off and the direct beam position can be taken from a previous sample.

Figure 5 Refinement of the direct beam position applied to ferritin scattering data. (a) `Pizza slicing' of the mean detector image into eight equal azimuthal sectors. Azimuthally integrated intensity profiles I(q) for each sector are shown (b) before and (c) after optimization. Improved overlap after refinement indicates a more accurate beam-centre determination.

3.4. Outlier pixel detection

In order to mask `jumping' pixels prior to XPCS calculations, outliers are identified by com­paring pixels expected to exhibit similar mean intensities and counting statistics. After the automatic determination of the scattering geometry (as described in Section 3.3), each pixel is com­pared against others within the same narrow q-interval. By default, the AGIPD detector area is split into 300 concentric annular regions of fixed width but increasing radius, such that all pixels within a given annulus correspond to the equivalent q-value. For each pixel, the r.m.s. value is calculated along the pulseId dimension as

Subsequently, the resulting r.m.s. value is normalized by the median of all r.m.s. values of pixels within the same q-region. A pixel is flagged as an outlier if its normalized r.m.s. value lies outside the empirically determined window of 0.75 to 1.75. These bounds are not symmetric around 1, reflecting the skewed nature of the r.m.s. [see Fig. 6(c)]. Both values are user-configurable.

Figure 6 Removal of pixel outliers based on train-averaged ferritin scattering data. (a) Azimuthally integrated intensity I(q) for selected pulses before outlier filtering. (b) I(q) for the same pulses after filtering. (c) Distribution of normalized r.m.s. values used for outlier detection. The dashed lines indicate the lower and upper threshold values 0.75 and 1.75, respectively.

The effect of outlier removal is demonstrated in Fig. 6, which shows the pulse-resolved azimuthally integrated intensity I(q) before and after outlier pixel removal, along with the distribution of normalized r.m.s. values. The results highlight a clear improvement in data quality following this procedure.

3.5. Intensity filtering

In the experiment, sample capillaries can be filled partially and have air bubbles or other inhomogeneities. This leads to SAXS intensity fluctuations in the data. To address this, the analysis pipeline includes an automatic filter for detecting poorly filled or missing sample based on changes in scattering intensity. The mean intensity values calculated in Section 3.2 are first normalized by the train-averaged X-ray gas monitor (XGM) signal (Maltezopoulos et al., 2019), accounting for fluctuations in the incoming X-ray intensity. All illuminated positions where the intensities (trains) deviate by more than ±20% (this value is adjustable) from the median intensity value are excluded from the subsequent XPCS analysis (see Fig. 7). This filtering step must be applied with care to ensure removal of truly com­promised sample regions only, without discarding sample regions with intrinsic variations, e.g. caused by varying optical thicknesses.

Figure 7 Example of applying intensity filtering to AGIPD data containing a signal drop due to an air bubble (red points) inside the capillary. The decrease in scattered intensity was encountered twice due to the back-and-forth scanning motion of the capillary [see Fig. 1(c)].

4. XPCS calculations

A general approach to quantify the dynamics of both equilibrium and out-of-equilibrium systems is via the two-time correlation function (TTCF) (Sutton, 2002; Madsen et al., 2010),

which is defined as the correlation of intensity at two time points τ₁ and τ₂ for a set of detector pixels belonging to the equivalent absolute scattering vector q. denotes averaging over the pixels from this set. For systems in equilibrium, the dynamics only depend on the time lag Δτ = τ₂ − τ₁ (Madsen et al., 2010), reducing the TTCF to the standard autocorrelation function,

A reduction in com­putational time and resources can be achieved by applying the multi-τ algorithm (Schatzel, 1990; Cipelletti & Weitz, 1999), which is very efficient at synchrotron radiation facilities (Khan et al., 2018), where XPCS is typically conducted in a `continuous' mode with uniform time intervals Δτ between successive frames.

However, this approach is not readily applicable to burst-mode XPCS at EuXFEL, where the recorded frames are grouped within trains. In this mode, the acquisition time of each frame is defined by the trainId (t_n) and pulseId (p_n). This makes use of the multi-τ algorithm much less effective. To allow for the detection of potential beam-induced effects on sample dynamics, explicit calculation of the TTCFs is therefore preferred, as given by

Since each X-ray pulse train illuminates a fresh spot on the sample, correlations only need to be com­puted between pulses within the same train ( t_n1 = t_n2). This results in n_tr independent TTCFs, each of dimension (n_p, n_p), with n_tr being the number of trains and n_p the number of pulses per train. The final train-averaged TTCF is obtained as

To enable further correction procedures, correlations between neighbouring trains t_n2 = t_n1+1 (termed off-correlation) are also calculated.

The train-wise com­putation of TTCFs as defined in equation (6) presents a significant com­putational challenge due to the large volume of experimental data involved. This results in substantial demands on both CPU resources and memory usage. To address this, the calculation is divided into several discrete steps, each optimized for parallel execution and resource efficiency: first, the q-bins of interest are defined (i.e. pixel ROIs in the form of annuli around the direct beam position), which will be separated later in the mathematical kernel to calculate the TTCFs for each q-bin. These q-bins can either be specified by the user [for example, based on the data from the mean AGIPD image or azimuthally integrated intensity I(q)] or generated automatically by the pipeline as equidistant q-bins fitting inside the available detector area. Each q-bin is repre­sent­ed by a Boolean mask with shape (16, 512, 128), matching the AGIPD's native data format. These masks are subsequently merged with the static mask, the calibration mask and the outlier mask as discussed above.

In the second step, the TTCF calculation is per­formed independently for each AGIPD module/sequence file using multipro­ces­sing. Each CPU core returns train-resolved data com­prising: (i) the non-normalized TTCF [i.e. the numerator in equation (3)] prior to normalization by the average intensity values given by the denominator in equation (3); (ii) the off-correlation between adjacent trains; (iii) the pulse-resolved total intensity (used for both TTCF and off-correlation data); and (iv) the number of pixels used for the TTCF/off-correlation calculation. A module/sequence-resolved trainId map is also generated to align entries across all AGIPD data files, which is important for multipro­ces­sing and further merging of the data.

To avoid memory overflow during pro­ces­sing, we estimate the maximum memory load per core. For example, storing TTCFs and off-correlation data (volume equals the volume of TTCFs) for a single module/sequence file containing data for n_tr = 250 trains, each with n_p = 352 pulses and n_q-bin = 15 q-bins, results in a data volume of n_tr × n_q-bin × n_p × n_p × 4 bytes × 2 = 3.7 GB, where the factor 2 accounts for storing both TTCF and off-correlation arrays. Scaling this to all 176 module/sequence files yields approximately 176 files × 3.7 GB ≃ 651 GB of data, which exceeds the available memory on most nodes of the Maxwell cluster. To ensure that the pipeline is executable with all nodes of the Maxwell cluster, we set a limit for the number of cores used in multipro­ces­sing (flexible, based on the estimation above) and implement caching of data to disk after finishing the calculation for each individual module/sequence file. Guided by the performance benchmarks discussed in Section 3.2 (see also Fig. 4), we set the default value of cores to be exploited during the calculation to 50. If memory estimates exceed the node capacity, this number is automatically reduced. This results in a memory usage of ∼185 GB, which is available on any node of the Maxwell cluster (DESY, 2026a).

Another important factor influencing the overall runtime of the calculation is the performance of individual CPU cores. This performance can be optimized by considering the data density of the arrays to be processed, i.e. the proportion of non-zero values in the photonized data array relative to the total number of entries. For arrays with low photon density (where most of the pixel/cell values zero), com­putational performance can be improved by converting dense arrays into sparse formats and performing built-in linear algebra operations on the sparsified representation of the data. To evaluate this approach, we per­formed benchmarking tests using the native AGIPD array dimensions; the results are shown in Fig. 8. We find that sparsification offers a com­putational advantage only below a certain data density threshold. Above an array density value of ∼10⁻¹ we observed a loss of performance com­pared to conventional TTCF calculations using dense arrays. Therefore, to ensure optimal performance, the mean image density of each AGIPD module is evaluated, and the data representation — dense or sparse — is selected accordingly for each module/sequence file. The default threshold value for data sparsification is set to 5 × 10⁻².

Figure 8 Benchmarking of TTCF com­putation using an arbitrary data set with array shape (300, 65536) and 10 q-bins, where 300 is the number of time points and 65536 = 512 × 128 is the reshaped dimension of a single AGIPD module. The `dense calc' refers to standard matrix multiplication using dense arrays. The `sparse calc' includes both the time required to convert the data to sparse format (`sparsification') and the subsequent multiplication of sparse matrices (`sparse math').

5. Post-pro­ces­sing and averaging

After all module/sequence files have been processed and the corresponding results stored in the cache directory, the data must be merged. Specifically, for each train, the non-normalized TTCFs and off-correlations are summed across all detector modules, along with the corresponding total signal intensities and the number of contributing pixels. The trainId map described earlier ensures synchronization of data across mod­ule/sequence files — that is, each chunk of data has a consistent length and trainId ordering. This enables efficient use of linear algebra operations to merge data at the array level, avoiding slower element-wise (i.e. train-by-train) operations.

Subsequently, the TTCF and off-correlation matrices are normalized by the corresponding mean intensities, and the average photon count per pixel per pulse () is com­puted for each trainId/pulseId by dividing the mean intensity by the number of used pixels. To further improve data quality, the off-correlation matrix is subtracted from the TTCF to suppress residual artifacts from `jumping' pixels and other memory-cell related instabilities (Madsen et al., 2021; Dallari et al., 2021b) (see Appendix A for details). This correction also shifts the TTCF baseline from 1 to 0, as shown later in Fig. 12. However, if the scattering intensity falls below ≲ 10⁻³, residual detector artifacts are still dominating the signal. Therefore, XPCS measurements at such low scattering intensities are considered impracticable with the AGIPD detector and the corresponding data entries are excluded from further analysis. Finally, the corrected and filtered data are averaged over all trains and the mean TTCFs and g₂ functions for each q-bin are stored for further analysis.

6. Output

Users at the MID instrument have several options for in­specting and obtaining the output of the XPCS pipeline, including the DAMNIT tool developed at EuXFEL (EuXFEL, 2026d). A graphical user interface (GUI) provides an overview of all runs acquired during a given experiment. A representative screenshot of the GUI is displayed in Fig. 9. Each run is repre­sent­ed as a row in the overview table, while columns are customizable to display various types of data and metadata, such as motor positions, monitor readings or sample tem­per­a­ture. New runs are added automatically to the table and the corresponding fields are updated accordingly.

Figure 9 Example view of the DAMNIT tool showing a data table summarizing the results of the XPCS pipeline at the MID instrument. Each row corresponds to a single run, while columns display customizable metadata such as motor positions, monitor signals and sample parameters.

The output of the XPCS pipeline appears as columns in the DAMNIT table. Plots are repre­sent­ed by small icons, which can be opened as new figures. Data processed by the online XPCS pipeline, which is currently being developed and tested, is also transferred to DAMNIT. This allows the users to perform a preliminary analysis of the XPCS data immediately after a run is finished, even before the fully calibrated detector data become available on the Maxwell cluster.

Further inspection and analysis of the XPCS results are enabled through Jupyter Notebooks (Fangohr et al., 2019), provided via the JupyterHub portal on the Maxwell cluster (DESY, 2026b). Data contained in the DAMNIT table and stored in a database using HDF5 files can be accessed using a dedicated Python API (EuXFEL, 2026d). An example usage is shown in Fig. 10, where users retrieve processed data arrays for further analysis using custom or external software. In addition to database access, output data from the XPCS pipeline are currently stored in the experiment directory as both image files and NetCDF4 format data. A future transition to NeXus-format output is planned to improve com­patibility with community standards.

Figure 10 Accessing DAMNIT data through its Python API in a Jupyter Notebook. The API allows users to retrieve, filter and analyze experimental metadata and processed XPCS results directly within an interactive Python environment.

In line with the goals of the DAPHNE4NFDI consortium (Barty et al., 2023; Murphy et al., 2025), efforts are ongoing to make MHz-XPCS data and metadata FAIR (Findable, Accessible, Interoperable, Reusable). Current work focuses on establishing a secure connection between DAMNIT and the external DAPHNE4NFDI database (XPCS platform), which is envisioned as the central repository for high-quality processed and merged XPCS data. Following an initial embargo period, the platform is expected to provide open access to the broader scientific community.

7. Discussion and conclusion

The MHz-XPCS data-analysis pipeline pre­sent­ed here is designed to deliver results within a reasonable timeframe both during the experiment and in the post-beamtime period, em­phasizing minimal user interaction and high data quality suitable for publication. Based on operational experience, several insights and suggestions have emerged that point towards desirable developments in detector capabilities and operation modes.

A reoccurring challenge in high-repetition-rate experiments using two-dimensional pixel detectors is the generation of high data volumes. These set significant demands on data transfer, storage and com­puting infrastructure. The discrepancy between the data volumes read-out from the detector and the actual input to the final calculations can be staggering for sparse data techniques like single-particle imaging or MHz-XPCS pre­sent­ed here.

Typical MHz-XPCS experiments yield fewer than one photon per pixel per frame, meaning the vast majority of pixels carry no intensity information. Nonetheless, full dense data arrays — including these zero-signal pixels — are repeatedly transferred from the detector to the online cluster and subsequently to the offline com­puting environment before any significant data reduction can occur (e.g. after dark pedestal subtraction). This inefficiency delays the identification and exclusion of empty pixels and represents a missed opportunity for early data com­pression.

To address this, implementing zero-value suppression as early as possible in the data acquisition chain — ideally on the detector itself — would significantly improve efficiency. Such functionality could be enabled by integrating real-time pro­ces­sing and filtering at the detector level using field programmable gate arrays (FPGAs). This concept of a `smart detector' capable of onboard data correction and selective readout is especially promising within the burst-mode operation scheme of the European XFEL. In the long term, this approach could offer a path toward more sustainable data-handling and scalable-analysis workflows.

It is also evident that a high detector noise level com­plicates the discrimination of single photons (single-photon sensitivity), which results in artifacts like `jumping' pixels becoming problematic and also requiring more pro­ces­sing time (≃ ×2 in both I/O and correlation operations due to the need for outlier detection and off-correlation). From a data reduction standpoint, high noise levels also hinder efficient com­pression. The conversion of analog signals to integer photon counts (`photonization') becomes unreliable when photon signals cannot be clearly distinguished from noise. As a result, com­pression is no longer lossless, and residual uncertainties propagate through the analysis. Consequently, reducing detector noise not only enhances scientific data quality but also lowers com­putational demands and facilitates more efficient data reduction. Future detector developments that prioritize low-noise operation would enable both higher data quality and more efficient data pro­ces­sing.

Finally, we demonstrated that XPCS calculations in the sparse data regime benefit significantly from performing linear algebra operations on sparsified data representations. How­ever, the step of sparsification (converting the data from a dense array to a sparse array/list of events) can be more time consuming than the actual TTCF calculations. This makes detector concepts like the timepix4 development (Llopart et al., 2022; Correa et al., 2024; Chushkin et al., 2025), which features an option of event-based read-out (apart from traditional frame mode), very promising for future XPCS experiments, since a sparsified list representation of photon hits is directly output by the detector. This not only streamlines pro­ces­sing but also opens the door to novel experimental regimes, all without requiring proportional increases in com­putational resources.

MHz-XPCS experiments are often constrained by the large pixel sizes of current detectors, which limit spatial resolution and reduce speckle contrast, thereby lowering the SNR of the correlation functions. This poses a particular challenge for radiation-sensitive samples, where increasing the beam size to reduce dose and dose-rate effects is typically not an option. Additionally, extending XPCS into the wide-angle X-ray scattering (WAXS) regime – essential for accessing atomic length scales – is difficult with detectors like the AGIPD, due to the inherently lower speckle contrast at high scattering vectors (Lehmkühler et al., 2021). A factor 4 reduction in pixel size (e.g. from 200 to 50 µm) would improve the contrast and SNR significantly, but also increase the number of pixels by a factor of 16 (keeping the detector area constant). Still, the amount of data generated by an event-based detector would not scale proportionally, since the number of registered photons remains constant. Therefore, new detector developments in this direction hold the promise of expanding the range of experiments that are feasible, but also reducing the demands on data transfer, storage and com­putation hardware to a fraction of the current requirements.

Figure 11 (a) Train-averaged pulse-resolved intensity of pixels within a selected q-bin: solid lines represent normal pixel behaviour, whereas the dashed line represents typical behaviour of the `jumping' pixel. (b) Photon counts from the `jumping' pixel shown on the left, as a function of pulseId and trainId.

APPENDIX A

`Jumping' pixels

`Jumping' pixel is a phenomenon that describes the spontaneous drift of the pedestal of 32 consecutive memory cells of a single AGIPD pixel. Jumps can be observed on various time scales (from some trains to several minutes) and amplitudes (below and above a single-photon equivalent), but always for one row of the 11 × 32 = 352 storage cell memory matrix. Most of the jumps are corrected by the common-mode correction or masked by the outlier detection. However, if the drift has an amplitude close to or above a single-photon signal and lasts only for a few consecutive trains, it might occasionally be missed. This can lead to clear false signal, as shown in Fig. 11, but might also result in more subtle higher photon count probabilities. The presence of such jumps results in 32 × 32 square artifacts in the TTCFs along the main diagonal. Fig. 12(a) shows an example of such an artifact arising from the presence of just a single `jumping' pixel from Fig. 11 within the selected q-bin.

Figure 12 Performance of the off-correlation correction for a selected q-bin, containing the `jumping' pixel shown in Fig. 11: (a) train-averaged TTCF without correction; (b) train-averaged off-correlation; (c) train-averaged TTCF after subtraction of the off-correlation (with this step the baseline shifts to 0).

Calculating the pulse–pulse correlations between the adjacent trains (off-correlations) can help to remove such artifacts, as shown in Fig. 12(b). Subtracting the off-correlation from the initial TTCF can significantly improve the TTCF behaviour and remove the existing artifacts, as shown in Fig. 12(c).