In total, hackers have stolen nearly $2 billion worth of cryptocurrency since the beginning of 2017, mostly from exchanges, and that’s just what has been revealed publicly. These are not just opportunistic lone attackers, either. Sophisticated cybercrime organizations are now doing it too: analytics firm Chainalysis recently said that just two groups, both of which are apparently still active, may have stolen a combined $1 billion from exchanges.

We shouldn’t be surprised. Blockchains are particularly attractive to thieves because fraudulent transactions can’t be reversed as they often can be in the traditional financial system. Besides that, we’ve long known that just as blockchains have unique security features, they have unique vulnerabilities. Marketing slogans and headlines that called the technology “unhackable” were dead wrong.

That’s been understood, at least in theory, since Bitcoin emerged a decade ago. But in the past year, amidst a Cambrian explosion of new cryptocurrency projects, we’ve started to see what this means in practice—and what these inherent weaknesses could mean for the future of blockchains and digital assets.

How do you hack a blockchain?

Before we go any further, let’s get a few terms straight.

A blockchain is a cryptographic database maintained by a network of computers, each of which stores a copy of the most up-to-date version. A blockchain protocol is a set of rules that dictate how the computers in the network, called nodes, should verify new transactions and add them to the database. The protocol employs cryptography, game theory, and economics to create incentives for the nodes to work toward securing the network instead of attacking it for personal gain. If set up correctly, this system can make it extremely difficult and expensive to add false transactions but relatively easy to verify valid ones.

That’s what’s made the technology so appealing to many industries, beginning with finance. Soon-to-launch services from big-name institutions like Fidelity Investments and Intercontinental Exchange, the owner of the New York Stock Exchange, will start to enmesh blockchains in the existing financial system. Even central banks are now looking into using them for new digital forms of national currency.

But the more complex a blockchain system is, the more ways there are to make mistakes while setting it up. Earlier this month, the company in charge of Zcash—a cryptocurrency that uses extremely complicated math to let users transact in private—revealed that it had secretly fixed a “subtle cryptographic flaw” accidentally baked into the protocol. An attacker could have exploited it to make unlimited counterfeit Zcash. Fortunately, no one seems to have actually done that.

The protocol isn’t the only thing that has to be secure. To trade cryptocurrency on your own, or run a node, you have to run a software client, which can also contain vulnerabilities. In September, developers of Bitcoin’s main client, called Bitcoin Core, had to scramble to fix a bug (also in secret) that could have let attackers mint more bitcoins than the system is supposed to allow.

Still, most of the recent headline-grabbing hacks weren’t attacks on the blockchains themselves, but on exchanges, the websites where people can buy, trade, and hold cryptocurrencies. And many of those heists could be blamed on poor basic security practices. That changed in January with the 51% attack against Ethereum Classic.

The 51% rule

Susceptibility to 51% attacks is inherent to most cryptocurrencies. That’s because most are based on blockchains that use proof of work as their protocol for verifying transactions. In this process, also known as mining, nodes spend vast amounts of computing power to prove themselves trustworthy enough to add information about new transactions to the database. A miner who somehow gains control of a majority of the network’s mining power can defraud other users by sending them payments and then creating an alternative version of the blockchain in which the payments never happened. This new version is called a fork. The attacker, who controls most of the mining power, can make the fork the authoritative version of the chain and proceed to spend the same cryptocurrency again.

For popular blockchains, attempting this sort of heist is likely to be extremely expensive. According to the website crypto51.com, renting enough mining power to attack Bitcoin would currently cost more than $260,000 per hour. But it gets much cheaper quickly as you move down the list of the more than 1,500 cryptocurrencies out there. Slumping coin prices make it even less expensive, since they cause miners to turn off their machines, leaving networks with less protection.

Toward the middle of 2018, attackers began springing 51% attacks on a series of relatively small, lightly traded coins including Verge, Monacoin, and Bitcoin Gold, stealing an estimated $20 million in total. In the fall, hackers stole around $100,000 using a series of attacks on a currency called Vertcoin. The hit against Ethereum Classic, which netted more than $1 million, was the first against a top-20 currency.

David Vorick, cofounder of the blockchain-based file storage platform Sia, predicts that 51% attacks will continue to grow in frequency and severity, and that exchanges will take the brunt of the damage caused by double-spends. One thing driving this trend, he says, has been the rise of so-called hashrate marketplaces, which attackers can use to rent computing power for attacks. “Exchanges will ultimately need to be much more restrictive when selecting which cryptocurrencies to support,” Vorick wrote after the Ethereum Classic hack.

A whole new can of worms bugs

Aside from 51% attacks, there is whole new level of blockchain security weaknesses whose implications researchers are just beginning to explore: smart-contract bugs. Coincidentally, Ethereum Classic—specifically, the story behind its origin—is a good starting point for understanding them, too.

A smart contract is a computer program that runs on a blockchain network. It can be used to automate the movement of cryptocurrency according to prescribed rules and conditions. This has many potential uses, such as facilitating real legal contracts or complicated financial transactions. Another use—the case of interest here—is to create a voting mechanism by which all the investors in a venture capital fund can collectively decide how to allocate the money.

Just such a fund, called the Decentralized Autonomous Organization (DAO), was set up in 2016 using the blockchain system called Ethereum. Shortly thereafter, an attacker stole more than $60 million worth of cryptocurrency by exploiting an unforeseen flaw in a smart contract that governed the DAO. In essence, the flaw allowed the hacker to keep requesting money from accounts without the system registering that the money had already been withdrawn.

As the hack illustrated, a bug in a live smart contract can create a unique sort of emergency. In traditional software, a bug can be fixed with a patch. In the blockchain world, it’s not so simple. Because transactions on a blockchain cannot be undone, deploying a smart contract is a bit like launching a rocket, says Petar Tsankov, a research scientist at ETH Zurich and cofounder of a smart-contract security startup called ChainSecurity. “The software cannot make a mistake.”

There are fixes, of a sort. Though they can’t be patched, some contracts can be “upgraded” by deploying additional smart contracts to interact with them. Developers can also build centralized kill switches into a network to stop all activity once a hack is detected. But for users whose money has already been stolen, it will be too late.

The only way to retrieve the money is, effectively, to rewrite history—to go back to the point on the blockchain before the attack happened, create a fork to a new blockchain, and have everyone on the network agree to use that one instead. That’s what Ethereum’s developers chose to do. Most, but not all, of the community switched to the new chain, which we now know as Ethereum. A smaller group of holdouts stuck with the original chain, which became Ethereum Classic.

Last month, Tsankov’s team at ChainSecurity saved Ethereum from a possible repeat of the DAO catastrophe. Just a day before a major planned software upgrade, the company told Ethereum’s lead developers that it would have the unintended consequence of leaving some contracts on the blockchain newly vulnerable to the same kind of bug that led to the DAO hack. The developers promptly postponed the upgrade and will give it another go later this month.

Nevertheless, hundreds of valuable Ethereum smart contracts were already vulnerable to this so-called reentrancy bug, according to Victor Fang, cofounder and CEO of blockchain security firm AnChain.ai. Tens of thousands of contracts may contain some other kind of vulnerability, according to research conducted last year. And the very nature of public blockchains means that if a smart-contract bug exists, hackers will find it, since the source code is often visible on the blockchain. “This is very different than traditional cybersecurity,” says Fang, who previously worked for the cybersecurity firm FireEye.

Buggy contracts, especially those holding thousands or millions of dollars, have attracted hackers just as advanced as the kind who attack banks or governments. In August, AnChain identified five Ethereum addresses behind an extremely sophisticated attack that exploited a contract flaw in a popular gambling game to steal $4 million.

Can the hackers be defeated?

AnChain.ai is one of several recent startups created to address the blockchain hacking threat. It uses artificial intelligence to monitor transactions and detect suspicious activity, and it can scan smart-contract code for known vulnerabilities.

Other companies, including Tsankov’s ChainSecurity, are developing auditing services based on an established computer science technique called formal verification. The goal is to prove mathematically that a contract’s code will actually do what its creators intended. These auditing tools, which have begun to emerge in the past year or so, have allowed smart-contract creators to eliminate many of the bugs that had been “low-hanging fruit,” says Tsankov. But the process can be expensive and time consuming.

It may also be possible to use additional smart contracts to set up blockchain-based “bug bounties.” These would encourage people to report flaws in return for a cryptocurrency reward, says Philip Daian, a researcher at Cornell University’s Initiative for Cryptocurrencies and Contracts.

But making sure code is clean will only go so far. A blockchain, after all, is a complex economic system that depends on the unpredictable behavior of humans, and people will always be angling for new ways to game it. Daian and his colleagues have shown how attackers have already figured out how to profit by gaming popular Ethereum smart contracts, for instance.

In short, while blockchain technology has been long touted for its security, under certain conditions it can  be quite vulnerable. Sometimes shoddy execution can be blamed, or unintentional software bugs. Other times it’s more of a gray area—the complicated result of interactions between the code, the economics of the blockchain, and human greed. That’s been known in theory since the technology’s beginning. Now that so many blockchains are out in the world, we are learning what it actually means—often the hard way.