Datacenters

Why Do We Need Datacenters?

The "internet" today runs on datacenters. And it's not just for A.I., though the recent tsunami of datacenter construction was kicked-off largely due to A.I. As alluded-to above, this is when many people first became familiar with the idea of a datacenter and may unfairly attribute them as serving only A.I. expansion.

But datacenters have been around for decades.

I myself worked in numerous datacenters back in my corporate I.T. life. These ranged from being as small as a typical home (what we called a computer room) to being as large as a Walmart (what we'd call a proper datacenter).

Your email account, your bank, all your social media sites, your phone and its apps, your doorbell camera, and very nearly everything else, are all powered by datacenters that are connected to the internet via highspeed fiber optic lines.

What Exactly is a Datacenter?

A datacenter campus contains most or all of the following major components:

• There's the datacenter building itself that houses all the servers, networking equipment, and some limited office space.
• Power distribution frames that take in utility-grade power, steps down voltages, then distributes power to the various loads inside the datacenter.
• Chillers pump cold water into heat exchanges inside the datacenter, which chills the interior air (by removing heat) to keep all those servers from overheating.
• Cooling towers that remove heat via evaporation (more on all this below).
• Some campuses have their own power generation as well.

Datacenter Building

The datacenter building itself, where all the servers live, is a cavernously large, low-rise structure. There's typically some limited office space, restrooms, and a breakroom with maybe a kitchenette.

The vast majority of the interior space is segmented by function. The largest functional space is the raised-floor* "data hall" where all the servers live. Other areas include mechanical, electrical, storage, docks, etc. The data hall is filled with hundreds or sometimes thousands of server racks.

A server rack is more or less the size of a refrigerator. That is, typically 24 to 30 inches wide, 36 to 48 inches deep, and perhaps 7 feet tall.

* The raised floor in a datacenter is typically made of 24×24 inch removable tiles supported by a metal framework that offers anywhere from 12 to 36 inches of subfloor height. The space beneath the floor is used for cabling and for distributing cooled air, which is directed up into the server cabinets through slotted or perforated tiles.

These racks are arranged in rows, not unlike the aisles in a Home Depot or Lowe's, but generally much longer and sometimes narrower.

The server racking and the aisles might run "north and south" for hundreds of feet, but every 25-50 feet or so, there may be an "east-west" cross aisle to make it easier for a server technician to access the back side of the rack where power cabling is often routed and also for faster emergency egress. We don't want personnel trapped in a 300 foot long aisle without a shortcut for escape.

All those server racks consume a lot of floor space. So these datacenters tend to be pretty huge. Hyperscale datacenters, the really big ones, could be hundreds of thousands of square feet, multiple acres -- the size of an Amazon.com regional distribution hub.

Electricity

All datacenters, small to huge, are pretty power hungry relative to their size compared to other industrial users. Here's how much power datacenters of various sizes might draw.

• Small datacenters, typically upward 20,000 sq/ft, consume upward 8 MW* of power.
• Medium-sized datacenters, typically upward 100,000 sq/ft, consume upward 30 MW of power.
• Large datacenters, typically upward 250,000 sq/ft, consume upward 75 MW of power.
• Hyperscale datacenters, 250,000 to over 1,000,000 sq/ft, can consume 75 to 600+ MW of power, nearly the entire output of a medium-sized power plant.

* MW=MegaWatts. So 8 MW = 8,000,000 Watts

Larger and hyperscale datacenters, the kind often built today, are among the highest single consumers of electricity on the planet especially so when considering their size and copious numbers. For datacenters that specialize in A.I. computing, the power required is even higher -- far higher.

Explainer

All those figures just above can be pretty nebulous.

So let us draw an analogy using space heaters, an appliance that everyone is familiar with. A typical household space heater draws about 1,500 Watts on high and will maintain comfortable warmth in a 150 sq/ft bedroom (assuming wintertime). That's about 10 Watts per square foot to keep that room comfortable. Easy math, yes?

Using the figures above, a large 250,000 sq/ft datacenter consuming 75 MW (75,000,000 Watts), would equal around 50,000 space heaters. On a per sq/ft basis, that's around 300 Watts of heat per sq/ft (in the data hall), compared to only 10 Watts per sq/ft for the bedroom. 30x more. Imagine how hot that bedroom would get if there were 30 space heaters all running 24/7 assuming the home could even support that much electrical draw (it cannot). Mighty toasty.

And that's just for a regular datacenter. Datacenters that focus on A.I. can draw 10x more power per sq/ft -- about 300 space heaters in the bedroom analogy. That's not toasty, that's a raging bonfire.

That should give you an idea of how much electricity a datacenter consumes. And because of the law of conservation, all that electricity is ultimately converted to heat.

That's a lot of heat to remove and we'll discuss that in the next section, "Cooling".

These numbers are all fairly rough as there are a number of individual variables. But it's a fair analogy to make.

Did that help?

So it stands to reason that a datacenter that size can't just roll into town and "plug into the grid" -- it forces the grid to be reshaped around it. Utilities and regional operators have to plan and build out substations, transmission lines, supporting infrastructure, and even new generating capacity, often at very high cost. All that takes time -- years.

Developers will say they’re paying their share and they usually do cover the direct hookup costs and actual consumption. But a big chunk of the bigger, more expensive upgrades, the ones that make the whole system capable of handling that new load in the first place, are frequently pushed on to all customers. They are effectively underwriting part of the expansion through higher monthly electricity bills.

The risk doesn’t go away after construction, either. These projects are often justified on upward 25 year demand projections. If the datacenter scales back, under performs, or shuts down early, the grid infrastructure is still there. That cost doesn’t just disappear, it gets spread even more so across everyone else’s electric bill.

Even in the success case, there’s a catch: A massive new load tightens regional supply margins. That can push up wholesale electricity prices, which then flow through to retail rates. So ratepayers may still end up paying more even if the datacenter meets all its commitments.

The sales pitch is long-term economic growth. The underlying reality is that a large, private customer is being supported by long-lived public infrastructure with much of the downside risk sitting with the public.

Performance or completion bonds and other financial instruments can help mitigate that risk. But municipalities courting a datacenter will almost certainly not have an equivalent level of negotiating skill and savvy to ensure their existing rate payers are sufficiently protected. Datacenter companies have armies of high-dollar lawyers to negotiate on their behalf. Some podunk rural town or county commish certainly will not.

Here is an eye-opening 32-minute podcast on NPR's Planet Money that takes a deep dive into all the power grid dynamics needed to support a datacenter. The podcast is not overly technical and is quite easy to understand. It addresses the question of how and why local electricity bills increase due to new datacenters coming online.

The phrase "socialize the costs and privatize the profits", a foundational credo for many large, industrial organizations, fits nicely here.

BYOE

Bring Your Own Electricity. That’s my term, not an official one, for when a datacenter plans to introduce new power on their own rather than relying exclusively on existing grid capacity or expecting the local utility to ramp up production through expansion projects.

Some datacenter projects are starting to include on-site or near-site power generation in their planning or construction. Others are compensating in different ways. For example, Microsoft, in partnership with Constellation Energy, is planning to recommission the Three Mile Island nuclear power plant near Middletown, PA.

Microsoft has no current plans to site a datacenter near TMI. But by signing a long-term PPA (Power Purchase Agreement) with Constellation for the TMI restart, Microsoft will effectively offset the electricity their datacenters consume elsewhere in the PJM RTO (Regional Transmission Organization) with carbon-free nuclear power generated by TMI.

As new datacenter construction projects are launched and get ever more power-hungry, we’re seeing more of this BYOE methodology. Is it enough? Probably not.

Cooling

Having many tens of thousands of servers crammed in a building all busily answering billions of google searches, serving videos, processing business data, a million other things, and increasingly, A.I. processes, will consume a lot of power which creates a lot of heat.

As you might remember from your high school science class, energy is not created or destroyed -- it only changes form. This is the law of conservation I mentioned above. All that electricity ultimately turns into heat that must be removed.

There are two (main) ways heat is removed from a datacenter.

Evaporative cooling

Historically, the most common cooling method has been open-loop evaporative cooling -- a "swamp cooler". Treated (and sometimes potable) water from a local water treatment plant is pumped through a heat exchanger that sucks heat out of the air inside the datacenter, thus heating the water, which is then pumped back outside into cooling towers where it evaporates as steam thus releasing heat into the atmosphere. This method is sacrificial -- meaning water is not returned to the system. It evaporates into thin air.

That doesn't mean the water stops being water! That water (as steam) is absorbed by the atmosphere where it continues on to the next step of the circular hydrological cycle -- much like how lake water evaporates then falls as rain, etc.

But the incoming water used to cool the datacenter has to come from somewhere. And that "somewhere" often includes municipal water providers. That's because the cooling systems can't really deal with dirty or contaminated water. Like anything that relies on water for life, both flora and fauna, datacenter cooling systems work better with clean water.

Evaporative cooling is more energy efficient and performs better in hotter climates as long as the humidity isn't too high, but at the cost of requiring a continuous, non-stop flow of vast quantities of clean water. That's the big problem right there.

Even a mid-sized datacenter can consume between 250 and 500 thousand gallons of water per day. Such a datacenter would be far and away the single largest user of water for any city under 1/4 million population.

The short and long term impact on the local watershed and water purification plants must be considered.

Non-evaporative cooling

In closed-loop non-evaporative cooling, far less water is lost from the system. It's not zero because of the initial fill and any system flushes for necessary maintenance. But even then, losses are still a tiny fraction of what an open-loop evaporative system would lose.

Most new datacenter projects today are designed with non-evaporative cooling and it's a good thing to see. There's still a lot of legacy datacenters using evaporative cooling, however, but there's a strong push to convert them to non-evaporative cooling especially where the local watershed and water authorities are under stress.

Closed-loop systems can be less energy efficient, require larger heat exchangers, and don't operate as well in higher temperature regions, thus cost more for the same amount of cooling, But they collectively save literally millions of tons of water.

Site Selection

Datacenters have unique requirements and challenges that indicate and affect where they can be built. Lets go over that in rough order of importance.

▶︎ Power availability: Is there excess grid capacity in the region? And if not, is the region amenable to expansion?

▶︎ Cooling feasibility: What's the average year-round climate in terms of heat, humidity, precipitation, etc., and water availability if planning for evaporative cooling?

▶︎ Network connectivity: Relative proximity to fiber infrastructure and major networking hubs

▶︎ Land: Depending on the planned size, anywhere from 50 to 200+ acres could be needed for a datacenter campus.

▶︎ Local Acceptance: Is the locality amenable to having a datacenter nearby? Or will permitting and community buy-in be a long, difficult, or impossible process?

▶︎ Incentives: Will the region offer sales and property tax abatements and other incentives?

Surrounding semi-rural exurban counties tend to work best. Generally not close enough to a large city to be heavily negatively affected but close enough to benefit from its proximity.

e.g. To the extent that sufficient power, feasible cooling, and networking is available (or can be) then remaining items tend to be easier. Land is much cheaper, taxes likely lower, community acceptance more likely, etc.

Jobs Jobs Jobs

Datacenter operators will tout the numerous jobs they'll create. And what small-town city council or county commissioner doesn't like that?

Some facts about those jobs.

Construction

Construction jobs are very real and pretty well paying, however... A typical mid-sized datacenter takes about a year or so to build from start to end. After that, it's done, and the construction jobs go away. But even while its under construction, there's a good chance that some/most(?) of that labor will be "imported" -- that is, brought in from outside the county or even the state.

Why is that?

Unless the datacenter in going into or near a fairly major population center, which isn't always the case, then your rural(ish) community or county may simply lack the labor pool to turn out in the numbers the project will require. Even a mid-sized datacenter requires potentially several hundred skilled construction workers in order to meet construction scheduling milestones.

Then there's the issue of expertise. Most rural local construction labor has probably not worked on a datacenter before. While the building shell itself is pretty standard, much of the interior fit-out work for a datacenter requires specialized knowledge that local workers may lack. Indeed, many datacenter interiors, especially those in more rural areas, are built by traveling crews of workers.

Those crews roll into town, staggered by construction phase expertise, shack up in the local hotels, motels, extended-stays, etc. until their part is done. Then they pack up and leave.

It's good but very temporary business for local accommodations, restaurants, that sort of thing.

Ongoing Staff

When the datacenter is finished and opens for business, the amount of ongoing onsite-labor required is surprisingly minimal for something that big. Datacenters are highly automated and remotely administered installations requiring little hands-on staffing.

Here's some of what the permanent on-site staff does.

• Facilities and infrastructure, electrical and mechanical
• I.T. operations (R&R servers and networking gear)
• Shipping and receiving of various parts and systems
• Security
• Interface with vendors and clients
• Janitorial

A medium sized datacenter may only need 25 to 50 people to cover everything 24/7/365. Some of those staffers may be hired from the local labor force, but others may be transferred-in especially for the more technical roles.

Contrast that with, say, an Amazon fulfillment center of similar size, that might employ well over a thousand workers. Even a Walmart store, typically needing far less labor than a fulfillment center, would still have several hundred workers.

Local Politics

Datacenter companies know they aren't particularly welcome in some communities. So they have a playbook of strategies for helping to win over local opposition. I've already touched on some of this above, but here's a more complete and tidy list of strategies designed to overcome objections.

• Touting construction jobs (These are temporary jobs and often include imported workers)
• Touting ongoing permanent jobs (These jobs are far fewer in number and may also include imported workers)
• Various tax windfalls (But may be reduced through negotiation or by other abatements)
• Upgrades to local infrastructure: roads, power, fiber internet (This is real)
• Direct investment/donations/sponsorships to schools, workforce training, non-profits, civic affairs like police, fire, EMS (often a one-shot injection)
• Claims of environmental stewardship
• Local bump to economy (Real, but limited to certain businesses and mostly during construction, nothing much enduring)

Some of these incentives are real and would be welcomed by most communities. But like any advertising, you're only getting one side of the narrative. That is, what would be a best possible outcome is presented as a totally expected and routine one. It's important to maintain incredulity and not simply believe what is promised without iron clad guarantees.

The fuller truth is many of these incentives are shiny superficial objects, designed to inexpensively flatter and appease voters and local officials. They're often door prizes in the grand financial scheme of datacenter construction and operation which can easily reach into the billions of dollars.

♫ NDA Now You Don't Say ♫ *

When a company approaches a city or county expressing interest in building a local datacenter, they will require local officials and other key players to sign an NDA (Non Disclosure Agreement) that gags those officials and other signers from discussing the project with anyone else. You can bet your bottom dollar that NDA is designed to protect the company, not the locality.

Transparency is not optional if you expect to win community buy-in. Without it, expect opposition and lawsuits. That may happen regardless if the project's metrics don't pencil out, but without transparency, pushback is virtually guaranteed. Few things rile up people more than government being tight-lipped or seen to be hiding something.

This isn’t Costco, after all, guarding a negotiating edge with a property owner. Datacenters bring significant externalities on power, water, land use, and infrastructure that can reshape a region in ways that a typical commercial development never could.

* Here's a fun 60 second animated musical video short about NDAs in the style of Schoolhouse Rock! (remember that from your childhood?) from the satirical TV law show The Good Fight. I'm sure you'll enjoy it.

Final Comments

Yes, it's pretty apparent this article insinuates "datacenter = bad". But that is not necessarily my mindset! But for datacenters and the tech it facilitates, I would be doing something entirely different.

The modern world would be impossible without them. And some datacenter projects have been an overall positive for their local communities. It can and does happen.

Big Tech certainly doesn't need my help in touting the benefits of having a datacenter move in. With a market cap in the 14 figures, Big Tech is perfectly capable of expressing themselves.

But what is needed is a counternarrative, discussing the other side of the datacenter coin. Because you aren't going to hear any of that from Big Tech.

Big Tech doesn't really care about you, your town, your county, or the things that are important to you and your community. They care about making money, full stop.

That's why a competent negotiating strategy is important.

Towns and counties that are considering welcoming a datacenter would do well to: Exercise all due caution • Stress transparency. Resist signing broad NDAs, this really nukes community trust! • Negotiate aggressively and wisely • Consider all the impacts • Contact other towns and counties that have been down this road already • Hire outside legal council • Don't get distracted by shiny things • And make damn sure the good of the community is always top of mind.