Performing software maintenance at the end of the Earth

For most of us, updating software requires merely a few taps or clicks on our device before watching a progress bar fly across the screen. For University of Birmingham astronomer George Dransfield, however, it isn’t so easy. 

Last summer she took a plane to Hobart in Australia, then, after a spell in quarantine, flew  to Wilkins Aerodrome on the coast of Antarctica. After an Antarctic bus ride to Casey Station, and one final flight on board a tiny Basler turboprop, she arrived at her final destination: Concordia Station, which sits 680 miles inland. She came armed with a couple of portable hard drives, a laptop, and seven weeks to perform critical software maintenance.

Dransfield’s mission was to update the software that processes data collected by the most remote space telescope on Earth. The epic journey was essential because the station lacked anything resembling decent internet connectivity. And this is a problem when images taken by the telescope, A-STEP, are 4,096 x 4,096 pixels each.

“We spend the time taking images of the sky, and each image is about 150MB,” explains Dransfield. “Like most telescopes, you download the raw images to your computer after a night of observing, and you process them yourself.”

It’s getting the images out of Antarctica that’s the tricky bit. The station itself does have an connection back to the rest of the world, but only via satellite, and it can only reach dial-up era speeds that would have been impressive in, say, 1996. “Our internet connection has recently been upgraded,” says Dransfield. “We can now reach speeds of 80kB/sec.”

The only way to retrieve the raw data is if somebody goes the following year and puts it on a hard drive, she adds. Rather than beaming them back at snails’ pace, then, Concordia uses software to automatically analyse images and return a snapshot of the data – akin to attaching a low-resolution JPEG instead of a fully-fledged RAW image file – over the satellite connection. It means that scientists back at the University of Birmingham, and in other places around the world, at least have some data to work with.

Patching the hard way

The software Dransfield was sent to upgrade manages the data pipeline, taking data from the telescope and processing it. It was originally written in IDL, a language similar to Fortran developed in the 1970s. But before heading south, Dransfield was tasked with rewriting the entire stack in the relatively modern Python.

Updating it remotely was not an option. “If I want to edit maybe five or six lines of code, I need to set aside at least an hour,” says Dransfield. “You’re in a terminal, you type a letter, and then it takes two or three seconds for that letter to actually appear. I cannot begin to explain how tedious it is.”

Getting the new software installed was important, though, so the telescope could actually do some science. “They changed the camera so the images that came in this year were different from last year’s images. So, I had to do a million fixes to adapt everything.” 

Even if Dransfield could put up with the dismally slow connection, the limited connectivity means only one person could access the virtual private network (VPN) at once. So, it was time to head for the Antarctic. “This is such an enormous piece of software, so I couldn’t just upload it,” says Dransfield, “Because I’d written the software, I had to be the one to install it.”

The two towers of the climate station Concordia, based in Antarctica

Before taking off, however, other obstacles presented themselves. “You’d think that with such big important operations, people would be really careful about everything, but no one could tell me for sure what the specs of the new server were that they had shipped down there,” she laughs. “I needed to test installation on a really similar computer, but I didn’t know what that was.”

Although the code was destined to run on a Linux system, she couldn’t be sure exactly which packages and binaries would run on the destination system. Before she set off, she set out to test her code on as many different computer systems as possible, including a Raspberry Pi. This was necessary because the effective lack of internet connection removed a key tool in every coder’s arsenal – access to Stack Overflow. “It was genuinely super challenging to do this job without access to the internet and access to help from other people,” says Dransfield.

Embracing the satellite broadband disruption

Concordia is currently stuck with a torturously slow connection, but things may be about to change. Last September, SpaceX announced that its Starlink satellite broadband system was now available in Antarctica. Satellite broadband is gaining momentum, with many suggesting it could be a definitive solution to rural connectivity woes.

What makes it different to what came before from the likes of Iridium is that Starlink is a mega-constellation of up to 12,000, with almost 3,000 already in space. This means each satellite covers a significantly smaller patch of the Earth – so bandwidth isn’t shared between so many users – and they’re closer to the ground, meaning latency becomes less of a problem for activities such as videoconferencing. 

Given Concordia doesn’t have any neighbours, it’s conceivable that if the system was deployed, it could enjoy up to 200Mbits/sec, roughly equivalent to an excellent home broadband connection, and plenty fast enough to shuffle masses of data around.

Other Antarctic research bases, such as the American McMurdo station, are already trialling the technology. What difference could it make to Corcondia? “That level of bandwidth increase would be transformative,” says Dransfield. “It would save me so much time, I’d be able to SSH in and just make any adjustments that I have to make quickly. It would also mean that we’d be able to, more often, download raw images.”

Dransfield recently attended a conference of Antarctic scientists, and the enthusiasm for Starlink was palpable. “Most people are just like, yes, Starlink, as soon as it’s available, we’re going to go for that because we need the bandwidth.”

Of the Antarctic scientists present, however, only a handful were astronomers – and this gives Dransfield a very different perspective on satellite broadband. It turns out, despite the enormous bandwidth benefits, she’s not a fan of Starlink. “We have a different thing to balance,” says Dransfield, “because we can see Starlink in our images.”

Could satellite broadband scupper astronomy?

“We’re not super keen on the idea,” she adds. “If you look at images from places like other popular observing sites there’s little streaks that you see.”

The streaks are the motion trails of satellites in orbit around the Earth, captured because space telescopes take long exposures to capture objects far away in space. SpaceX will be putting up a lot of satellites, and while Dransfield wants the internet, she doesn’t want the satellites ruining what’s one of the most important and unique astronomical sites we have.

It’s a criticism SpaceX has heard before. “I am confident that we will not cause any impact whatsoever in astronomical discoveries – zero,” said founder Elon Musk in 2020, “That’s my prediction, we will take corrective action if it’s above zero.”

Space telescopes take long exposure shots and are susceptible to picking up streaks

Theoretically, it’s possible for astronomers to correct their data to account for satellites getting in the way of the telescope. But Dransfield ultimately believes there’s a more fundamental problem. “If we know their orbits then we can just correct our images for them, but I think [SpaceX] underestimate how much additional work that is.”

This sort of error correction is already common in astronomy – the satellites and other objects are referred to as “systematics”, and astronomers are used to modifying their data to account for, say, the Hubble Space Telescope moving into view in the skies above. But Starlink is on a different level.

“It’s an additional systematic error in our data,” says Dransfield. “The problem is the more you correct your data for systematics, the more that increases the uncertainty on the actual end result.”

For example, if she were to detect a planet in orbit around a distant star, she might attempt to calculate the size of the planet, but the error bars on the calculation will be much wider – because the “systematic” has introduced more uncertainty. “Sometimes we’re looking for really, really small events, like shallow changes in brightness of a star, and if you can’t trust those changes in brightness because you had a Starlink transit as well, then it could end up just ruining so much otherwise awesome data,” she says.

Weighing the trade-off of the potential benefits of Starlink with the impact on her work is difficult – but it’s a battle she worries she is going to lose. “Because there’s not a lot of astronomy in Antarctica, our voices are not as important as everybody else who is just like, ‘Yeah, but we could have internet.’

“As an astronomer with access to Antarctica, I cannot begin to tell you how unique it is and how special what we can do from there is, in our field of searching for planets,” Dransfield says. “I would prefer it if they didn’t contaminate our skies further, to be honest, because the skies are so, so precious.”