ADDENDA AND ERRATA

A correction has been published for this article. To view the correction, click here.

Robotic sample changers for macromolecular X-ray crystallography and biological small-angle X-ray scattering at the National Synchrotron Light Source II

Here we present two robotic sample changers integrated into the experimental stations for the macromolecular crystallography (MX) beamlines AMX and FMX, and the biological small-angle scattering (bioSAXS) beamline LiX. They enable fully automated unattended data collection and remote access to the beamlines. The system designs incorporate high-throughput, versatility, high-capacity, resource sharing and robustness. All systems are centered around a six-axis industrial robotic arm coupled with a force torque sensor and in-house end effectors (grippers). They have the same software architecture and the facility standard EPICS-based BEAST alarm system. The MX system is compatible with SPINE bases and Unipucks. It comprises a liquid nitro­gen dewar holding 384 samples (24 Unipucks) and a stay-cold gripper, and utilizes machine vision software to track the sample during operations and to calculate the final mount position on the goniometer. The bioSAXS system has an in-house engineered sample storage unit that can hold up to 360 samples (20 sample holders) which keeps samples at a user-set temperature (277 K to 300 K). The MX systems were deployed in early 2017 and the bioSAXS system in early 2019.

1. Introduction

Macromolecular X-ray crystallography (MX) and biological small-angle X-ray scattering (bioSAXS) are complementary techniques that can reveal important structural information of biological samples. The Advanced Beamlines for Biological Investigations using X-rays project, which was funded by the National Institutes of Health, constructed two MX beamlines and one bioSAXS beamline: the highly Automated Macromolecular Crystallography beamline (AMX; https://www.bnl.gov/nsls2/beamlines/beamline.php?r=17-ID-1), the Frontier Macromolecular Crystallography beamline (FMX; Schneider et al., 2021) and the Life Science X-ray scattering beamline (LiX; Yang et al., 2020). Due to the brightness of the National Synchrotron Light Source II (NSLS-II), these beamlines can achieve fluxes greater than 10¹² photons s⁻¹ at the sample. The high flux, along with fast data acquisition rates, have reduced data collection to the order of milliseconds per image for MX and seconds for bioSAXS experiments. As a result, robust and reliable robotic sample changers are required for efficient continuous experiments at these beamlines.

High throughput, versatility, high capacity, resource sharing and robustness are important factors that were considered at the time of the design phase of the next automounter systems for NSLS-II. A modular design is desired because it enables implementation at all three beamlines (and beyond) and this would be advantageous because it would allow for rapid development and deployment to the beamlines and ease of maintenance. Another advantage versatility affords is the ability to adapt to the changing needs of each beamline. In order to exploit the fast data rates, high-capacity sample holders along with fast sample exchanges are a must. Finally, the systems need to be sturdy and flexible (rapid software updates) for overnight unattended operations. Based on these criteria we chose the following systems for further study as reference implementations.

The Advanced Light Source (ALS) translation pneumatic based sample mounter (Cork et al., 2006) was considered due to in-house expertise. At the Stanford Synchrotron Radiation Lightsource its four-axis robotic arm-based sample mounter (SAM) piqued interest in its adaptive utilization of the force torque sensor (FTS; Russi et al., 2016). Across the pond, Diamond Light Source achieved one of the fastest sample exchange times with BART, a six-axis robotic arm system (O'Hea et al., 2017), and the European Synchrotron Radiation Facility (ESRF) developed the FlexED8, a six-axis robotic arm-based sample-changer, coupled with a novel ice-filtering dewar designed to accept several sample holders: SPINE, SPINEplus, miniSPINE and NewPin (Cipriani et al., 2006; Papp, Felisaz et al., 2017). In addition, the ESRF, in collaboration with the European Molecular Biology Laboratory (EMBL) Grenoble and Hamburg outstations, developed one of the newest translation-stage-based bioSAXS sample mounters (Round et al., 2015).

Ultimately, we decided to take certain attributes of the systems studied that met our needs such as the ALS-style gripper and design our own sample changer for NSLS-II which share hardware and software and are optimized for integration at an MX or bioSAXS experimental station, centered around a six-axis robotic arm and designed for fully automated, unattended operations.

2. Robotic sample changer

Fig. 1 demonstrates the orientation of the robotic sample changer with respect to the experimental stations. The MX experimental stations (left) are equipped with Eiger detectors, Cryostream sample coolers, on-axis microscopes capable of seeing crystals down to 1 µm (3.4 pixels µm⁻¹ at the highest magnification) and high-precision goniometers (Bhogadi et al., 2016). The robotic arms are bolted to the hutch floors at AMX and FMX and are situated between the dewars and the goniometers where samples are exposed to X-rays. The dewars are fixed to custom tables and the goniometers are mounted onto their own granite blocks. The experimental station at LiX can handle multiple types of experiments, with bioSAXS as the principal emphasis (Yang et al., 2020). To achieve the required flexibility the robotic arm is reversibly bolted to a kinematic plate on the hutch floor, so staff can move the robotic arm easily downstream of the sample storage unit when switching to the bioSAXS configuration as shown in Fig. 1 (right). The robotic arm in this mode is between the sample storage box and the scattering sample handler. For more information on the scattering sample handler see the work by Yang et al. (2020).

Figure 1 Overview of the experimental stations. Left: AMX experimental station shown with a Stäubli TX60-L robotic arm. FMX has a similar setup with the major differences being that it is equipped with an Eiger 16M instead of an Eiger 9M and it has the standard TX60. Right: LiX experimental station in the bioSAXS configuration. It is equipped with a Stäubli TX40, an in-house developed scattering sample handler and a sample storage unit.

The principle of a robotic sample changer is to transfer valuable samples from one place to another with precision and repeatability, while maintaining a special environment (e.g. cold). In the case of MX, the robotic arm moves samples mounted on SPINE bases from the dewar to the goniometer (Fig. 1, left) at a temperature below 100 K. The bioSAXS system, on the other hand, manipulates samples in a custom sample tray from the sample storage unit to the scattering sample handler without rotating the custom sample tray to eliminate sample sloshing (Fig. 1, right).

2.1. Robotic arm

The TX60-L (shown in Fig. 2), TX60 and TX40 (shown in Fig. 1) from Stäubli are the six-axis robotic arms of AMX, FMX and LiX, respectively, controlled by the CS8C. Each robotic arm is equipped with pneumatic lines and solenoid valves, and the number of each depends on the model of the arm. The solenoid valves operate the gripper and release/lock the gripper without external plumbing, except on the TX-40 which only has a single solenoid valve instead of two and thus requires external plumbing to swap grippers. The CS8C is equipped with a pendant for manual control, as well as Ethernet connections, digital input/outputs board, and serial connections for open-architecture system design; the CS8C has the capabilities to handle fieldbus communications as an upgrade option, which is the case here. The main difference between the models is the work-envelope where the TX60-L has the longest reach and the TX40 has the shortest. The models at FMX and LiX were chosen to match the local environments, and at AMX the L version provides additional reach for possible later upgrades for puck-loading automation.

Figure 2 Overview of robotic sample changer at AMX. The system accepts Unipucks with SPINE bases. Pucks are loaded via the sample-loading port which is also where the robotic arm picks up samples.

2.2. Sample storage

The need to store samples is vastly different for MX than bioSAXS. At AMX and FMX the samples are microscopic protein crystals, require a constant temperature of 100 K or lower, supplied by liquid nitro­gen, and are mounted on SPINE bases; at LiX the samples were liquid, require cool temperatures above freezing and are in PCR tubes. Based on these needs, dewars were acquired for the MX beamlines while a custom sample storage unit was developed for LiX.

2.2.1. MX dewar

The dewars purchased from AbsolutSystem (Fig. 2) are based on the ESRF-design developed for the MASSIF-1 beamline (Bowler et al., 2015, 2016; Nurizzo et al., 2016) with eight plates that hold three pucks each. Dewar modifications performed are as follows. The puck plates are re-designed to handle Unipucks. The original design was for SC3 baskets of vials (Unipucks can hold 16 samples and the SC3 can only hold 10). The reduction of the sample and gripper port diameters minimizes ice formation in the dewar. A heater system coupled to the gaseous exhaust line prevents ice formation on the liquid nitro­gen (LN₂) and gas nitro­gen (GN₂) common area. The puck-support plate is rotated via the NSLS-II standard Delta-Tau Geobrick motor controller connected to a high-resolution stepper motor (model PKP244MD15A). The dewar is equipped with WAGO inputs/outputs (I/O) managed by an Ethernet controller (model 750-881). These I/O interfaces allow controlling actions such as opening and closing of the dewar pneumatic ports. They are coupled to the CS8C controller via a Stäubli fieldbus card (model D221.459.66) which enables the LSDC server to control the dewar (see Section 3.1) for remote dewar operations from outside the laboratory. Fill levels are kept between 94% and 97% (monitored by an LN₂ sensor), with a 40 min duty cycle including 15 min re-fill time. A complete dewar warm-up is scheduled twice weekly in summer and once weekly in winter, taking 7.5 h for the dewar to dry and 2.5 h for the refill.

In order to reliably position the motor of the dewar plate, the dewar has the high-resolution – 262 144 counts per revolution – absolute encoder (LIKA model HSCT18/17-14-RL2) coupled with a stepping motor and a backlash-free gearbox (Harmonic Drive CSF-11-100-1u-cc-1). This setup maintains the dewar plate positions at ±1 count; each count represents 5.6 µm at the outermost sample positions [Figs. 3(a) and 3(b)]. In addition, an accurate teaching method is required. A dewar metrology survey conducted for each plate sample position with and without liquid nitro­gen demonstrated that the average position contracted towards the dewar center by 494 µm. Thus, every dewar plate is taught with three designated Unipucks loaded with SPINE bases [Fig. 3(c)]. These pucks have a special cross hair on the center pole. The dewar is partially filled with LN₂ ∼25 mm above the plate surface in order to allow the cross hairs to be above the LN₂ by ∼10 mm. This process defines the puck centers of each plate visually. From there, each sample position within the puck is calculated based off the puck centers. A calibration procedure (application) is then executed in which every sample position, each carrying a SPINE base, is probed with the gripper. The gripper goes over each sample as if to pick it up, closes for 5 s, then opens without actually picking up the sample. The feedback from the force torque sensor (see Section 2.4) allows for refining of the puck centers by adjusting robot parameters such as translation (x, y, z) and orientation (R_x, R_y, R_z).

Figure 3 (a) Top-view of dewar. The height measures 350 mm (excluding the center shaft) and the diameter 650 mm. (b) Plate under liquid nitro­gen with Unipucks loaded with SPINE bases. The samples within the light-blue box represent the samples positions farthest from the dewar center, ∼227 mm. (c) Zoomed-in view of the designated pucks used for teaching. The black `+' represent the cross-hairs used to visually teach the puck centers. During teaching the puck covers are removed and the pucks are loaded with SPINE bases.

2.2.2. bioSAXS sample storage unit

The design of the sample-storage unit was primarily driven by the need to provide temporary housing for 20 PCR tube holders at a temperature between 277 K and 300 K within grasping range of the TX-40 robot. We found that a ten-row-by-two-column array suits the available space in the LiX experimental station [Fig. 4(a)]. The storage unit is bolted onto the granite slab within the upstream reach of the TX-40 via a metal fixation bracket and a horizontal support plate. The primary structural support for the PCR tube holder is a two-part, vertically oriented, water-cooled central plate, which also serves as the heat-sink for the hot-sides of the Peltier array. To maximize the thermal performance, the storage unit is surrounded by a polycarbonate enclosure that helps to minimize convective heat gain from the ambient environment. A pneumatically activated swing door with a soft O-ring gasket allows the TX-40 intermittent access to the PCR tube holders while maintaining the inner user-set temperature. Each of the PCR-tube-holder column-arrays is supported on a large cold-plate that itself is connected thermally to the cold side of the Peltier array carried by the heat-sink plate [Figs. 4(b) and 4(c)]. Water-cooling is delivered by a U-shaped loop of annealed copper tubing pressed into the round aperture formed by two C-shaped profiles in the mating-sides of the heat-sink plate halves [Fig. 4(c)]. Soft graphite foils used as interstitial layers maximize the thermal contacts to the Peltier array on both the hot side and the cold side. A layer of Nomex 993 fibreboard provides low-thermal-conductivity mechanical cushioning between the cold plate and heat-sink plate. The copper tubing is fed into the storage box through two neoprene grommets in the horizontal support plate [Fig. 4(d)].

Figure 4 (a) Overall layout of the sample storage unit. The polycarbonate enclosure of the sample storage unit measures 445 mm high from the horizontal support plate, 140 mm wide and 180 mm deep. (b) Detailed view of the inside of the sample storage unit showing PCR tube holders and shelf details. (c) Detailed cross section of the two-part vertical water-cooled central plate (heat-sink). (d) Detailed view of horizontal support plate. All the plates are made of MIC-6 cast aluminium.

Each PCR tube is situated on a small horizontal shelf that contains the same alignment pin and a permanent magnet array as in the solution scattering experimental chamber (Yang et al., 2020), allowing repeatable and secure docking for loading and unloading by the TX-40. Furthermore, the presence of a PCR tube holder on each shelf is monitored by a normally closed mechanical micro limit switch (Panasonic AV4044). A PT-100 sensor monitors the temperature of a representative PCR tube holder and provides closed-loop temperature control. This feedback signal, the driving current for the Peltier array, and the low-voltage limit switch signals are all fed into and out of the sample storage unit through D-sub miniature connectors embedded in the horizonal support plate [Fig. 4(d)]. A connection for dry GN₂, which is also fed through the horizontal support plate, provides purging capabilities to mitigate condensation build-up on the cold surfaces within the storage box on humid days. The sample storage unit can maintain the inner user-set temperature at ±0.1 K (Fig. 5).

Figure 5 Temperature readings of the LiX sample storage unit. A reading is done every 30 s. The 486 readings shown here represent ∼4 h.

The sample storage unit currently is taught by expert staff. Once the robot is bolted down, three PCR tubes loaded by hand into positions 2, 19 and 20 are probed using the pendant. Feedback from the FTS (see Section 2.4) enable staff to define the position. The remaining PCR positions are calculated based on the frame defined by the first three PCR tubes. Fine tuning of translations (x, y, z) and orientations (R_x, R_y, R_z) with the FTS is carried out for all positions. The scattering sample handler mount position is defined the same way as the first three PCR tubes. An automated teaching method is under development (see Section 4.1).

2.3. Grippers

The main driving force behind our gripper designs is the the form factor of the sample holders. For MX, it is SPINE bases mounted in Unipucks; for bioSAXS, it is custom PCR-tube holders loaded in the sample storage unit. This led to our basing the MX gripper on an existing design and finding a partial commercial solution for the bioSAXS gripper.

2.3.1. MX gripper

The MX gripper (Fig. 6) is a multi-generational evolution of the ALS automounter gripper design (Snell et al., 2004). It is designed to stay cold during extended/continuous operation, thus an insulator (thermal break 1) made from fiberglass (McMaster–Carr part No. 8467K46) lies between the gripper body and the pneumatic actuator. This pneumatic actuator is a custom design for NSLS-II purchased from Bimba, USA (Fig. S1 of the supporting information). It comes with side racks that hold magnetic sensors (Bimba HC), which can be adjusted to read the internal permanent magnet position. The permanent magnet is part of the internal shaft that moves when actuated. The gripper body is machined from a single piece of easy-to-machine 303 stainless steel rod (McMaster–Carr part No. 8984k39) to assure better concentricity with the three-pronged collet and outer tube. Thermal break 2, which also serves as the three-pronged collet holder and is made from fiberglass (McMaster–Carr part No. 8669 K24), further minimizes heat transfer from the gripper body to the collet. A vented screw allows passage of LN₂ into the gripper pocket (LN₂ pocket). A temperature sensor (Honeywell part No. HEL-705-T-0-12-00) in one of the collet prongs delivers constant temperature readout and ensures that samples are mounted at a temperature lower than 100 K and is connected to the ATI ten-pin gold plated connector (part No. 9120-E10T-005). The outer tube walls were increased to 635 µm thick, providing improved strength. The robotic arms are equipped with two electrical connectors, one at the arm base and the other at the forearm, connected by an internal wire harness that allows for a clean installation of the temperature and magnetic sensors, and the built-in solenoid valves provide easy control of gripper pneumatic actions.

Figure 6 MX gripper. Left: CAD-drawing of the gripper sliced in half. The gap between the Teflon plug and thermal break two forms a pocket where liquid nitro­gen collects, we call the LN­2 pocket. The three-pronged collet is made of 510 bronze (McMaster part No. 88555K39). Right: photograph of the gripper with additional components such as the QC-005 tool changer from ATI.

To ensure that samples are safely transferred between the goniometer and the dewar at cryogenic temperatures and since the actual device could not be tested under operating conditions, engineering studies were performed. Here, viewing only the tip of the gripper holding a SPINE base, the overall length changes can be observed between room temperature (RT) and LN₂ (∼77 K; Fig. 7). A comparison between the measured position of the prototype gripper and the theoretical contraction shows a difference of 29 µm, well within experimental error, and close enough to use the theoretical value to define the normal operation gripper offset in the robot low-level programming language (Section 3.1).

Figure 7 Left: image of the tip of the gripper with a SPINE base at room temperature. Right: image of the tip of the gripper at liquid nitro­gen temperature holding a SPINE base. The gripper did not move during this measurement. The difference between points A and B is the measured contraction, which is 260 µm. The theoretical contraction is 231 µm. The scale is in micrometres.

2.3.2. bioSAXS gripper

The bioSAXS gripper (Fig. 8) is based on a Schunk pneumatic actuator (model MPG 40) coupled to an ATI tool changer (model QC-001). The claws enable self-centering of the PCR tube holder: one with a half-sphere and another with a half-cylinder shape. These claws interface with a conical hole and a V-grove on the PCR tube holder to provide a fully constrained support when engaged with the gripper. The pneumatic actuator was purchased with Hall-effect sensors to indicate if the gripper is open or closed.

Figure 8 Photograph of the bioSAXS gripper holding a PCR tube holder. From left to right, adaptor plates, FTS, adaptor plates, tool changer, adaptor, pneumatic actuator with Hall-effect sensors, claws, PCR tube holder.

2.4. Force-torque sensor

All the robotic systems are equipped with ATI force-torque sensors (model Mini45) installed between the robotic arm and the gripper (Fig. 2 MX; Fig. 8 bioSAXS). An FTS was favored over a traditional collision sensor in our early prototype. First, it offers better accuracy because the FTS thresholds can be fine-tuned unlike monitoring of air pressure. Second, the FTS allows for dynamic (on the fly) thresholding. For instance, low thresholds are set when the robotic arm is near the goniometer, this combined with slightly lower approaching speed significantly reduces the risk of equipment damage. Third, the FTS is used as a touch probe to `teach' positions such as the goniometer position and the dewar positions for the MX beamlines (see Section 2.2.1) and the scattering sample handler position and sample storage unit positions for the bioSAX beamline (see Section 2.2.2). The FTS also allows for the incorporation of the auto-recovery feature at AMX and FMX where, on rare occasions gripper pneumatic actuations violate the FTS thresholds, the robot can continue seamlessly by re-checking the FTS F_x, F_y and F_z values. At LiX, the FTS is used to auto-recover if a PCR tube holder is not placed properly by pushing the PCR tube holder into position safely.

2.5. MX-specific components

2.5.1. Machine vision

Machine-vision capabilities are critical to high-throughput and reliability. Here two types of systems have been implemented, a commercial Keyence machine-vision system (Fig. 2) that checks whether the sample has been properly picked up by the gripper, and a custom-made system that determines proper placement of the pin tip at the goniometer using a top-view camera (Fig. 1).

The Keyence system comprises a controller (model CV-X152F) and a monochrome camera (model CV-035M) coupled to a telecentric lens (image magnification is independent of the distance from the object, model CA-LM0307). Tools developed using the Keyence software determine if a sample is present or tilted in the gripper, and to measure the gap between the SPINE base and the gripper collet (Fig. 9). The gap is used to calculate the final robot position at the goniometer (see Section 2.5.4).

Figure 9 Images taken by the Keyence machine vision system of the gripper in the open state with no sample (left), and with the gripper in the closed state, holding a SPINE base (right). The boxes are regions of interest used to determine edges. The gap represents the ideal distance between the collet and the bottom edge of a SPINE base.

Additionally, the custom system using a camera (Clark et al., in preparation) ensures the cap is properly mounted on the goniometer, and locates the pin tip. Based on the pin-tip location, the distances by which the goniometer stages need to move are determined so that the pin tip is in the field of view of the experimental station on-axis microscope (Fig. 1). The software takes two images 90° apart right after a sample is mounted, and employs bash scripts (Free Software Foundation, 2007) running ImageMagick (https://www.imagemagick.org).

2.5.2. Smart magnet

The goniometers at AMX and FMX are equipped with an Arinax `smart magnet', which senses inductive changes to indicate the presence of a sample base, driven by its own controller. It has a boost feature which increases the voltage to the electromagnetic coil to prevent the base from falling during data collection by the increased magnetic force. The controller is equipped with tunable base-detection thresholds; this allows for detection of bases made of different metal alloys. The current controller version SM3 adds tunable voltages for all magnet states (on, off and boost), and has an increased voltage output up to 7.5 V (from 5 V for the SM2), improved sensitivity to bases and an improved graphical user interface.

The original electromagnetic coils operate up to 5 V and have only 100 µm tolerances – the difference between the SPINE base diameter and the electromagnetic coil diameter (Fig. 10). The new electromagnetic coils allow for ∼180 µm tolerance, to reduce the chances of a collision, and can operate at higher output voltages produced by the new SM3 controller. The heat generated from these new electromagnetic coils is exploited to quickly melt the ice on the base when it is mounted and when coupled with the SM3 controller can reach 366 K compared with the SM2 setup which was measured to reach 307 K. At 366 K, a slow drift of the sample is observed and presumed to be from thermal expansion of the pin. This will be monitored closely so that optimal voltage is applied, and sample drift minimized. AMX is now using the SM3 controller with the new electromagnetic coil, and FMX is in the process of upgrading to it.

Figure 10 Drawing of a SPINE base (top) and the electromagnetic coil (bottom). The original electromagnetic coil diameter is 9.6 mm; the new model at AMX has a diameter of 9.52 mm. The poles are shown in red and light gray. The two wires coming from the bottom (gray and red) connect the electromagnetic coil to the controller.

2.5.3. Eddy-current proximity sensors

Eddy-current proximity sensors from Lion Precision (model ECL2020) at the AMX and FMX experimental stations (Fig. 1) ensure continuous peak performance of the robotic arms (TX60 and TX60-L). The robotic arms have a repeatability better than 40 µm peak-to-peak (Fig. 11). The Eddy-current sensors validate performance of new hardware, such as the vetting of the longer robot interconnection cable (power supply cable between the CS8C and the robotic arm) needed to accommodate the new location of the CS8C of the TX60 at FMX or the characterization of the spare grippers. After ∼25 min of continuous operation, the temperature of the robotic arms equilibrates at 306 K, which is a rise of ∼7 K, at the forearm between joints four and five. Using these proximity sensors, measurements showed that the robotic arm expands by ∼45 µm as the robotic arm warms. This information helped us define the gripper offset (see Section 2.5.4) along with the theoretical contraction of a cold gripper (see Section 2.3.1) and the gap measured from the Keyence machine vision system (see Section 2.5.1).

Figure 11 Left: graph of ∼2 h worth of measurements from eddy-current sensors A (top), B (middle) and C (bottom). The intervals on the y axis are 2 µm. The blue vertical lines represent ∼30 min blocks. Right: picture of sensors with respect to the gripper. Sensor C is 90° from Sensor B. We have a similar setup for the TX60 at FMX. The robotic arm was warm when this measurement was conducted.

2.5.4. Gripper offset

To help us manage the approach at the goniometer, the room-temperature gripper is taught to maintain ∼600 µm between the SPINE base and the electromagnetic coil ledge (Fig. 10) and we introduced a gripper offset, which accounts for the mechanical changes the gripper undergoes between extreme temperature changes (see Section 2.3.1) and the variability by which SPINE bases are picked up (see Section 2.5.1). The expansion of the robotic arms can safely be ignored since it is small compared with the other parameters (see Section 2.5.3). Thus, the gripper offset is defined as

where g_off is the gripper offset, MV_gap is the Keyence machine vision gap value and g_t is the gripper theoretical contraction value (290 µm in our case).

3. Controls

3.1. Robot control software architecture

The robot control system architecture consists of a core set of hardware and software components accessible through the Experimental Physics and Industrial Control System (EPICS); Dalesio et al., 1994; Fig. 12). The implementation of the Jlib system, from the robot task layer written in the VAL3 language (Stäubli, Switzerland), up to the included `EMBL' EPICS-based input–output controller (IOC) have been implemented as described by Papp, Felisaz et al. (2017). Both the LiX Bluesky server (https://nsls-ii.github.io/bluesky/) and AMX/FMX Life Science Data Collection (LSDC) server (Schneider et al., 2021) interface with the robot control system via a Python (Van Rossum & Drake, 2009) based robot control library customized for each beamline, which provides both control of and feedback from the robot system.

Figure 12 EPICS (light green) is the central hub for information exchange that encompasses PVs from the EMBL, PPS, FTS and experimental station devices such as the MX goniometer or bioSAXS sample storage unit. The FTS and PPS have a direct connection to the CS8C represented by the red arrow. StaubCom is a VAL3 library installed on the CS8C that allows the integration of Stäubli robotic arms into high-level control systems (https://software.embl-em.de/software/21). Blue boxes are hardware and software components. Communication is represented by black arrows.

3.2. Continuous safe and robust operations

To ensure safe and robust operations, these systems are equipped with sensors that give feedback throughout every step of the workflow, all of them connected through EPICS (Fig. 12). EPICS serves as the central hub where information is exchanged between the robot CS8C controller and the `outside world'. This information includes the status of the FTS, Personnel Protection System (PPS), experimental station devices, such as the smart magnet (sample/no sample) at AMX and FMX, or the presence of PCR tube holders in the sample storage unit at LiX. The PPS and FTS are also connected directly to the CS8C and serve as hardware triggers to shut off power to the robotic arms when the hutch interlock is disengaged or if the thresholds of the FTS are violated. The FTS communicates via EPICS to read out the forces and torques at 10 Hz for other applications, such as the teaching of the dewar positions (see Section 2.2.1). The door hutch secure signal from the PPS is read through EPICS for other uses as well, such as closing the detector shield when the hutch door is opened. The EPICS-based BEAST alarm system (Kasemir et al., 2009) enables these robotic sample changers to detect three conditions: sample safety, equipment safety and an issue with the robot. For example, the liquid-nitro­gen level falling below a certain level in the dewar at the MX beamlines or if the sample storage unit door stays open for 3 min while the hutch is interlocked at the bioSAXS beamline will generate alerts which are sent out to staff via e-mail and a phone call. These automated alarms have proven crucial for robust day and night operations.

4. Current developments

4.1. Automated teaching at LiX

The different experimental modes that LiX supports are possible because the TX40 robotic arm on its stand is movable from the parked position upstream and away from the sample area when users perform experiments other than bioSAXS relying on the robotic arm. This was a requirement and part of the design. However, this introduces the problem of the robot not being returned to the exact position when LiX returns to bioSAXS experiments and as a result the robotic arm needs to be re-calibrated with respect to the sample storage unit and the sample position for data collection. Currently, only expert staff can perform this task; but we are implementing an automated teaching procedure requiring a lower level of expertise to re-calibrate the robotic arm. The method is based on using fiducials, the force torque sensor and a special tool (Fig. 13). The robot will probe the three fiducials on the sample storage unit and, using the force torque sensor as a touch probe, determine the center of each. From that newly refined frame, the 20 sample holder positions are calculated.

Figure 13 TX40 at LiX with the teaching tool coupled with the QC-001 tool changer – for automatic tool exchange – will probe the fiducials on the face of the sample storage unit. The red circles indicate positions of the fiducials. The fiducials are metallic spheres that screw into the face of the sample storage unit.

4.2. Sample visualization under liquid nitro­gen at AMX and FMX

In order to improve reliability of the MX robotic sample changer, a puck visualization setup detecting tilted SPINE bases or bent pins is currently under development. The setup comprises a cylinder with a transparent bottom in contact with liquid nitro­gen, and a translucent top accepting three cameras (Fig. 14). The dewar will rotate to capture images of all eight plates and these images are analyzed by a machine-vision software (still under development) to determine if any SPINE bases are tilted or if pins are bent. Fig. 15 shows two images, with and without liquid nitro­gen. Another benefit will be that the application can unambiguously distinguish between SPINE bases and unsupported types such as ALS bases. The application will also compare the database with what is in the dewar to avoid issues during data collection, detect obstructions that could cause a collision and flag problematic samples automatically.

Figure 14 Puck visualization setup mounted on the dewar view port with three cameras. Each camera views one puck.

Figure 15 Images captured by the puck visualization showing tilted SPINE bases (red), ALS cap (aqua) and a bent pin (green). The rest are SPINE bases that can be safely picked up by the robot. The left side is without liquid nitro­gen and the right is with liquid nitro­gen (single image, no averaging).

4.3. General-purpose puck

A general-purpose puck has been implemented to support our remote operation modes. This application allows beamline staff to mount specialized SPINE bases stored in the room-temperature accessible general-puck (Fig. 2) to the goniometer. Existing and planned specialized SPINE bases include a sharp tip to be used for rotation axis alignment, a base with specific elements to be used for energy calibration, and a YAG thin crystal (see the supporting information for a video of the robot mounting a SPINE base onto the goniometer from the general puck).

5. System performance

The MX systems have been in operation since early 2017 for AMX and FMX. At the time of writing, combined, the systems have seen over 35 000 samples (Fig. 16) and achieved a reliability of 99.95%. We define reliability as

where r is reliability expressed as a percentage, mount_s is the number of successful mounts and mount_a is the number of attempted mounts. A successful mount means the sample is placed on the goniometer, i.e. not lost or damaged.

Figure 16 Number of samples mounted per cycle. Yellow and green represent FMX and AMX, respectively. The blue shows the number of propriety samples mounted. Note: for the year 2020, only COVID-19 related experiments were performed.

The sample exchange time, defined as time necessary to remove the previous sample from the goniometer and mounting the next one on the goniometer, is currently <35 s. See the supporting information for a video of the sample exchange.

The bioSAXS system saw its first users on Valentine's day 2019. The system has not lost a single sample during transfers between the sample-storage unit and the scattering-sample handler. As of April 2021 the system has transferred 550 PCR tube holders (Fig. 17). See the supporting information for the transfer of samples by the bioSAXS system.

Figure 17 Number of sample trays mounted per cycle by LiX. Each sample tray can hold up to 18 samples. Note: for the year 2020, only bioSAXS experiments related to COVID-19 were conducted.

6. Conclusions

We have developed two robotic sample-mounting systems for MX and bioSAXS experiments. By using similar hardware and sharing software components when possible, the bioSAXS system was deployed in six months, after an initial development cycle of two years for the MX system. The resource sharing brings down the overall cost, since one spare part, one engineer and one programmer can serve all three beamlines.

The MX system can be upgraded to accept miniSPINE bases (Papp, Rossi et al., 2017) alongside SPINE bases (Cipriani et al., 2006). The new gripper required for mini­SPINE bases can be characterized and taught using the method developed using FTS. This will allow for a seamless transfer between SPINE and miniSPINE bases. Our current gripper can handle meshes that support very small crystals (1–5 µm). The collet can be modified to grab other types of holders, using the current design, if needed. These new grippers can work seamlessly alongside the standard gripper thanks to the QC-005 tool changer.

The bioSAXS robotic arm can easily move so that both experimental configurations can be accommodated (solution scattering and micro-diffraction imaging) and, due to the QC-001 tool changer, automatically swap between the gripper and teaching tool. In the future, this system will be upgraded to be compatible with LiX's micro-diffraction configuration.

We designed the systems to cope with changing needs, to run fully automated unattended data collections and to fulfill the beamlines' needs. For AMX that would be to work towards mounting 1000 samples per day for specialized experiments, such as ligand screening. For FMX the goal is to mount samples on the precision goniometry required to exploit the 1 µm beam; for LiX one must enable high-throughput data collection in a highly flexible scattering experimental station. All three beamlines can now run without human intervention during the night so that researchers can focus on the challenging experiments during the day when full support is available (beamline scientists, expert crystallographers/solution scatterers, control engineers). Other sensors and software such as the top-view camera, auto-recovery with the FTS, smart magnets and the BEAST alarm, have allowed for the MX beamlines to only lose one sample for every 2000 samples and for the bioSAXS system not to lose a single sample. The high capacity of the dewar (384 samples; with miniSPINE 816) and the sample storage unit (360 samples) reduce the need for entering the hutches. All this combined increases reliability and throughput, makes efficient use of precious beam time, and is a pre-requisite for remote access by users. This was planned for in the design phase but is now an absolute requirement during the pandemic. The MX systems have been in operation for four years and have mounted over 35 000 samples while the bioSAXS has been in operation for two years and has done over 500 transfers.