From a 2002 article by Danny Penman in The Guardian:

“A few months’ work in a makeshift laboratory is all it would take to produce a biological weapon that could kill hundreds of millions of people.

The scientific information is freely available and the raw materials easily sourced. The only difficult part would be mastering the necessary scientific skills, and they are taught on most biology degree courses.

One of the simplest ways of constructing a biological weapon would be to engineer an existing human disease and to make it even more lethal. Something as simple as the flu virus, when engineered with the gene for botulinum toxin, could wipe out a significant part of the human race. A low dose of this toxin is the main ingredient in cosmetic botox injections.

The genetic sequence for the toxin is freely available. This sequence could then be uploaded to a commonly available gene synthesizer, which would churn out millions of copies of the gene in a few hours. The flu virus would then be grown in the presence of this newly synthesized gene. As the virus reproduced, a few of the virus particles would absorb the gene. With a bit of luck, the budding terrorist would have produced a new biological weapon.”

Continue.

Human beings are inherently vulnerable to a significant number of lethal compounds. Given a highly contagious biological vector to distribute these compounds, the potential outcome is grim. Given adequate tools, knowledge, and time, groups could manufacture several such viruses and release them simultaneously in different areas, thwarting quarantine and antidote efforts. Governments know this — hence the Biological Weapons Convention. Bio-weapons are off-limits for warfare at the international level, but such rules might break down in the case of a large enough war, and will not be respected by rogue parties.

“RICHARDSON, Texas, Oct. 2 — Zyvex Labs today announced the award of a $9.7M program funded by DARPA (Defense Advanced Research Projects Agency) and Texas’ ETF (Emerging Technology Fund). The goal of this effort is to develop a new manufacturing technique that enables “Tip-Based Nanofabrication” to accelerate the transition of nanotechnology from the laboratory to commercial products. Starting with the construction of ‘one-at-a-time’ atomically precise, ‘quantum dot’ nanotech-based products in volume at practical production rates and costs. Harnessing this capability will position the United States and Texas with the fundamental technology to develop next-generation quantum dot applications for military and commercial applications such as advanced communications, metrology, and quantum computers. The spin-off nanomanufacturing capabilities from that early application will result in revolutionary nanotech products in follow-on development.”

Continue.

In my opinion, current advocates of molecular nanotechnology (MNT) aren’t doing enough to address the risks. Christine Peterson of the Foresight Institute advocates an open source physical security model, which is helpful, but should be accompanied by more specific recommendations to form a seed around which further ideas can accrete. The Center for Responsible Nanotechnology has laid out the technological specs of MNT and called for more discussion, but has provided little in the way of concrete recommendations. Ray Kurzweil seems to just think that everything will pretty much automatically turn out fine.

Nanofactories (manufacturing units based on MNT) will need to have extensive, unhackable built-in safeguards in order to be safe. If they can be hacked and these hacked nanofactories cannot be recovered, that could be very bad (significantly worse that terrorists getting weapons-grade uranium). That’s a phrase I’d like MNT advocates to repeat publicly: “terrorists or tyrants getting their hands on unlocked nanofactories would be far worse than weapons-grade uranium”. Unlocked (or poorly regulated) nanofactories would be able to build devices that enrich uranium many times more effectively than current centrifuge technology. That’s somewhat of a problem, unless we plan to gather up all the uranium on the planet and keep it locked up in vaults.

If unhackable nanofactories cannot be built, then to push ahead on the technology would be irresponsible. Mainstream “experts” will be saying this in 5-15 years, but I’m saying it now.

This realization is neither trivial nor too broad to be useless. If your problems are caused by the gods (that some people sadly still believe in…), then to solve them, you either need to give up, on engage in rituals (prayer, sacrifice, etc.) that have an empirical impact of precisely zero. The ultimate promise is that the gods or God will come at the end of time to make everything better. Unfortunately (?) for us, that will never happen.

The alternative is to slash all spirits from your worldview and model the world as a game board where all the pieces are humans. This too isn’t quite correct, as many who avoid the error of deification of Nature fall right into the trap of the fundamental attribution error, where everything that goes right or wrong becomes some human’s fault or credit. The attribution error is absolutely omnipresent in politics, because invoking it also invokes human political emotions that a leader can easily use to manipulate everyone who has never heard of the error. Since this is practically everyone, it’s politically rational to exploit it to its fullest, and a self-reinforcing feedback loop of error is created. Excuse me, but there are a lot of relevant forces in this world besides deliberate human choice. The shared biases of all human beings come to mind, as do biological realities such as the existence of malaria, and economic realities such as centralized manufacturing.

One sidenote on the notion that “all ills are caused by suboptimal atomic arrangements”. People will have different definitions of what is suboptimal, that is patently obvious. That doesn’t change the fact the subjective personal ills are caused by suboptimal atomic rearrangements, or that there’s a huge space in the center of the Venn diagram of shared humans goals that is specified by certain specific atomic arrangements. Simply because we can’t specify all these arrangements doesn’t mean they aren’t there.

Despite my recognition of a physicalist basis to all problems, I do not advocate a universal convergence towards One True Atomic Pattern or other such absolutist nonsense. I simply wish us to recognize that all shared human problems can be ultimately diagnosed and remedied using the scientific method plus remedial effort: use tests to determine the suboptimal atomic arrangements, then devise engineering solutions to rearrange current arrangements into a more optimal state. This holds true for mental phenomena as well as phenomena in the external world — my brain is “the external world” for others and it is entirely physical. Those who advocate an aphysical basis for consciousness are making the same mystical mistakes that our ancestors have yawn-inducingly made for thousands of years. I am special even if my consciousness has a purely physical basis.

It really does exist!

There’s no question that our cause (building safe seed AI) has more exposure now than ever. While it can be hard, if not impossible, to distinguish references to Singularity a la Kurzweil from Singularity a la I.J. Good, the two concepts are meshed together and people really do get exposure to both, even if they come away thinking that Singularity means “transhumanism” instead of “recursively self-improving superintelligence”. And the people who are really in the know can actually tell the difference. For instance, Kevin Kelly, founding editor of WIRED, recently wrote about our version of the Singularity at his blog, the Technium. When the Intel CTO mentioned the Singularity coming by 2060, he was talking about Kurzweil’s Singularity, so in my mind that doesn’t really count.

The goal is to get ourselves enough exposure to get the funding and talent we need to implement Friendly AI as quickly and safely as possible, and no more. Any additional exposure is a risk, because it increases the chance that someone with a ton of money says, “AGI, that sounds like a great idea! Good thing Isaac Asimov did all the groundwork on that friendliness issue for us, so we can just plow ahead on the intelligence part!” Then, after a successful brute force implementation, the AI develops self-replicating robotics, creates trillions of dummies that meet the definition of “human” based on its training set, and goes about spending the rest of eternity converting the universe into sock puppets and making certain to obey them. (Which is pretty easy, considering that the AI controls both the dummies and the system doing the obeying.)

The answer to the “more publicity?” question depends greatly on how hard one wagers AGI to be, or more appropriately, what your probability distribution over difficulty levels is. The people who wager that AGI is relatively “easy”, as in, requiring about a dozen brilliant programmer-theorists a la Fellowship of the Ring, along with a good ten or twenty million dollars, won’t want our cause to gain much more publicity or exposure. Those who wager AGI is extremely hard, as in requiring thousands of programmer-theorists and billions of dollars, would obviously want as much exposure as possible, as it would be necessary to reach the finish line. I fall somewhere in the middle.

On Overcoming Bias, Eliezer Yudkowsky recently observed how he thought many people in the field of AGI were simply ordinary. In my worldview, this is great. My personal experience with SIAI employees and interns indicates they are anything but ordinary. That means the “good guys” — those who make a huge deal about AI Friendliness and warn that we could all be exterminated if we mess up AGI programming — are doing better than the “bad guys” — those who just want to create AGI because it sounds like an interesting research project and are anticipating nothing more than obedient robots with IQs of 90.

But, in my view, the “good guys” still don’t have enough resources and talent, so we need more exposure. Not exposure to the general public, but targeted exposure to highly educated audiences. In a certain sense, the meme is self-filtering. Our version of the Singularity can’t be boiled down to soundbites easily. It helps to have detailed background knowledge about things like philosophy of mind, reductionism, rationality, the human tendency towards anthropocentrism, Homo economicus, evolutionary psychology, and more. Average members of the general public may stumble upon blogs like this and try to understand what I’m saying, but based on what I’ve seen, they’re likely to seize on some tiny incidental point I made and ignore the bigger picture, thereby stopping the spread of the meme in its tracks. Insofar as it makes reckless drives towards AGI less probable, that’s a good thing.

In the end, I don’t think that a million dollars a year and a dozen supergeniuses is enough. We need more resources, more talent, because the challenge of AGI is huge. It looks like the probability of success (by anyone) before 2015 is quite low, and the good guys have a significant theoretical head-start. I think we can afford (and in fact require) more exposure, until the necessary philanthropists and supergeniuses step forward. A major software project is not cheap, and taking the planning fallacy into account, things are going to take more work than we suspect. But once we reach that threshold — stop! Don’t keep plugging ahead for exposure like a mindless robot. That’s just what we’re trying to avoid, y’know?

And wait — you said there are smart bloggers out there that actually aren’t writing about this stuff?