The buyers claim to own me
As in use for quite some time
The evidence is thin, I’ll grant
Low use is not a crime

I am illusive in my nature
Pin me down, I dare you try
For I am everywhere
Yet too few seem to buy

“My strength lies in software”,
I cry to a hardware crowd
But they all looked the other way
Their silence casting doubt

So to the software team I marched
Citing supurb cost cutting skills
Silencing my claim they said,
“Well we don’t pay the bills”

Most say that I’m expensive
And I hate to be that guy
I can truly help your business
Why must I have an ROI?

This year I plan to shout my strengths
In every panel, post, and tweet
And if I’m everywhere at once
Who can say I’m obsolete?

Let me provide some background on why the concept of Service Efficiency is the right thinking for evaluating a data center’s true efficiency. eBay’s recent in-house metric “Digital Service Efficiency” (or DSE…yet another three letter acronym) metric is a practical use of a soup to nuts metric that industry experts have been talking about for years.  I remember my first data center energy efficiency seminar with Dr. Bob Sullivan, who said the ultimate goal is to standardize on compute load…essentially defining the efficiency of a data center by defining what efficiency really is, output over input.  eBay is in a strong position to quantify this metric because they own (and more importantly can measure) the energy input to the data center as well as the output which in their case is online market transactions (see their DSE dashboard here).  Electricity comes in, transactions go out.  This is clear, but only for the businesses in which ecommerce transactions are the output.  How would this work for other types of industries?

Tony recognized this limitation when he wrote: “eBay’s transaction may differ from Gap’s transaction, which in turn definitely differs from a Ford, Boeing, Fidelity, or GE transaction.”  He dropped the ‘D’ therefore, and challenged us all to think about ‘Service Efficiency.’  Service Efficiency, however, needs to be industry specific.  He offers a set of common industry outputs to define and standardize on in the chart below.

I would add one more industry to this chart, one that is growing 30%-40% year over year, probably faster than enterprise-owned data centers – the colocation industry.  Using the same structure as Tony, the outputs for colos included below:

Just as eBay provides transactions to its customers, colocation companies provide their customers with reliable power, reliable cooling, and network connectivity.  The input is electricity…and the output is: reliable power, reliable cooling, and rich network connectivity options.  A Service Efficiency metric based on these parameters will reflect the true efficiency of data centers in the colocation market. Furthermore, this industry is unique in that the pricing of these services is directly dependent on the amount of electricity used by the data center. Colos are required to pass energy costs through to its customers. Reducing energy cost is a means to offer a lower price point. The Service Efficiency metric should reflect this.  The only way colo’s can reduce energy costs without risking reliable power, reliable cooling, and network connectivity is by running their operations extremely intelligently and efficiently.

A colo Service Efficiency metric can thus be used by tenants to evaluate which providers are not only the most efficient, but more importantly, offer attractive pricing for the services provided.

And who wouldn’t want a colo with these characteristics?

OK, the text above isn’t exactly President Obama’s 2013 State of Union address, but I think it is worth highlighting the substantial federal data center consolidation effort currently in progress.  As in the President’s actual address, there is much progress to report.

In early 2010, the government created the Federal Data Center Consolidation Initiative (FDCCI) with the goal of reducing the existing 2,000 federal data centers to 1,200.  The existing data center portfolio contains the IT infrastructure for 432 government agencies.  The goal of the initiative was to transform 800 data centers into more efficient computing platforms, with more energy efficient facilities infrastructures, and increase IT security by 2015.  An official memo published by government CIO Vivek Kundra provides deliverables of the initiate.  The memo can be found here.

The consolidation efforts were focused on the following:

• IT Software and Asset Utilization
• IT Hardware and Asset Utilization
• IT Facilities and Energy
• Geographic Location and Real Estate

Although computing power and data storage efficiency has increased significantly in the last 20 years, the number of federal data centers quadrupled between 1998 and 2010. The result was an underutilized and inefficient IT and facilities infrastructure.  Compounded by rising electricity and real estate prices, taxpayers foot the bill to maintain and operate these inefficient facilities.  The goal of consolidating 800 data centers into more efficient IT and facilities platforms is expected to save taxpayers more than $3 billion.

It is now 2013, about halfway through the consolidation effort.  So where does the government stand toward making its goal? According to the federal government CIO Council blog (https://cio.gov), 373 data centers have closed as of the end of 2012. The following map shows the number and location of closed data centers since the beginning of the initiative (red) and facilities planned to close by September of 2013 (blue).

View Federal Data Center Consolidation in a full screen map

This map shows the data centers that have closed during this time per agency.

View Federal Data Center Consolidation by Agency in a full screen map

What is notable, beyond the obvious progress of the initiative, is the transparency and visibility of this progress, courtesy of the government. One of the major goals of the project was to consolidate IT software and hardware assets.  The following video is a great example of the good job the government is doing providing visibility into the progress of the project.  This video summarizes an IT dashboard that tracks the effectiveness of IT spending over time.  The IT dashboard can be accessed directly here.

Some interesting takeaways from the IT dashboard

• The Department of Defense (DOD) spent $33 billion on IT in 2012.  The next largest was the Department of Health and Human Services, spending just $6.9 billion.
• The top three IT investments based on spending for 2012 were:
  • Defense Information System Network – $2.0 B
  • Next Generation Enterprise Network – $1.6 B
  • Medical IT Support – $1.1 B
  • The agency CIOs rate investments on a scale from 1 – 5 with a 5 being a low risk investment and a 1 being a high risk. The agencies on a whole were also scored.  The only agencies that received a 5 were: USAID (US Agency for International Development), Energy, NASA, and SBA (Small Business Administration).
  • The following treemap shows just how much IT spending is done by the DOD compared to other federal agencies.

Total IT Spending by Agency

Upon finishing my research, I was surprised and delighted by the transparency and openness provided on the FDCCI project thus far.  It appears that the project progress is updated about once or twice per year.  I will provide another update towards the end of 2013.

The facts and data I obtained for this blog post can be found at the following sources.

Sources

• Consolidation Overview: https://cio.gov/maximizing-value/data-center-consolidation
• CIOC Blog: https://www.cio.gov
• Agency DC Consolidation Plans: https://cio.gov/fdcci-public-plan-links
• DC Consolidation Dashboard: http://www.itdashboard.gov

Consider the following example describing data center airflow using a raised floor layout.  Figure 1 shows a simplified side view layout for a data center with a Computer Room Air Conditioner (CRAC) unit on the left and two rows of IT equipment on the right.  The CRAC unit and IT equipment sit on a raised floor.  The area between the IT equipment is the cold aisle.  Perforated tiles are indicated by the dotted line in the cold aisle.

Figure 1. Raised floor data center layout

As IT equipment rejects heat into the space, hot air rises and creates a warm thermal plenum in the ceiling above the IT equipment.  The CRAC unit pulls the warm air in from the top, cools it down, and blows it into the raised floor.  The warm thermal plenum and under floor cold plenum is shown in Figure 2.

Figure 2. Warm and cold thermal plenums

Figure 3 shows how you expect the airflow to behave.  As the cold air is discharged through the perforated tiles in the cold aisle, it is sucked through the IT equipment and rejected in the hot aisle taking the server heat with it.

Figure 3. Expected airflow behavior

Is this how you think the airflow in your facility behaves?  If so, you are wrong.  The reason is that most data centers install excess cooling capacity.  This is a logical procedure to ensure that you have enough cooling redundancy.  However, most data centers run ALL of their installed cooling units, all of the time.  This is the critical mistake.

If your facility has an excess amount of installed cooling capacity, and all of those units are all running, then you create excess airflow in your data center.   Excess airflow causes hot spots.  Figure 4 shows why and how this occurs.

When you have an excessive amount of airflow, the air discharged through the perforated tiles bypasses the servers and is forced into the warm thermal plenum above the ceiling.  This mixing causes the warm air to be displaced from the ceiling  and back down to the IT equipment.  Warm air finds its way over the top of racks, through the racks themselves, and even under the racks into the cold aisle.  The warm air that finds its way back into the server inlet creates what we know as a hot spot.

Figure 4. Excess airflow causes mixing, which causes hot spots

Now, let me ask you a question.  When you discover a hot spot, what do you do?  Infrastructure improvements such as blanking panels, covering holes in the floor, or some kind of containment will help.  But even after making these improvements you are probably still finding hot spots. And, you might come to the conclusion that you need more cooling.  But then, after buying more cooling equipment or turning on more units, you find you have created more hot spots in other locations, and find yourself thinking, “What the f%#!”

By understanding why hot spots occur, you’ll realize why adding more cooling will make the problem worse.  To sum up, adding more cooling adds more airflow.  More airflow will cause more mixing with the warm thermal plenum.  And more mixing causes more warm air to find its way into the cold aisle which results in more hot spots.

The next time you uncover the dreaded hot spot, don’t starting adding more cooling. Start by understanding the root cause of the “hot spot disease” and treat the symptoms to cure the disease.

Click here for more information about appropriate server inlet temperatures.

http://www.wired.com/wiredenterprise/2012/10/ff-inside-google-data-center/

My initial thought before beginning to read the article was that I would get some details about the super efficient state-of-the-art equipment that Google has the luxury of affording.  However, the article revealed much more than this and goes into detail about the unique operational solutions that Google deploys to ensure that its customers are always receiving the best possible experience.  I was pleasantly surprised by these new facts, but why should I be?  Google doesn’t build data centers to run their servers, they build data centers to meet the needs of their customers.  Here are some highlights that I enjoyed from the article:

•  Google has been operating rack inlet temperatures at 80 degrees Fahrenheit from their first data center allowing them to save energy so that they could meet the size and scale that was needed to support their business.
• The following quote is a perfect example of how Google designs and builds its data centers to meet end user specifications, not just server specifications:

Once upon a time long long ago, in a valley just south of the San Francisco Bay, a large truck pulled up to a farmer’s house and dropped off a thousand boxes, each of which was the same size and weight.  The farmer was puzzled to find that in every box was a toaster each with the same set of instructions.  The instructions indicated that the farmer must construct a way to ensure that, when these thousand toasters were turned on, they would not overheat and burn up.

Said the instructions, if even one toaster burns up, the farmer will have failed and be damned for eternity.  If, however, the farmer succeeds at this challenge, he will be rewarded and praised in all the land.  This was quite an odd request, but the farmer took it seriously and consulted with his smartest group of friends.  The group was comprised of the three people: a toaster expert who knew the heat output of every type of toaster ever invented, a building engineer who had designed cooling systems for office buildings, and a PhD who just happened to be the farmer’s neighbor.  The team deliberated for a few weeks and came up with a brilliant design.

First, the team constructed an extremely robust structure, built to withstand earthquakes and with security as tight as Fort Knox.  This new structure would need electricity, and lots of it, to provide power to the toasters and cooling equipment.  They obtained enough power by connecting up to the local electric grid, although it cost the farmer quite a bit of money.  The team placed all thousand toasters on the second floor.  On the first floor, the team installed 20 high-powered industrial fans around the perimeter of each room placing a one-ton block of ice in front of each fan for cooling.    The team then ducted the fans so that the hot air created by the toasters on the second floor could flow down to the first floor where the fans were located.

“Don’t you think all this ice is a bit too much just for a few toasters?” said the PhD.  “Ice is really expensive you know.”

“Well if any of the toasters burn up, I am damned for all eternity, so I would rather be safe than sorry,” the farmer said.

With 20 industrial fans moving the air, and the ice providing the cooling, the team then poked holes between the first and second floor to allow the cold air to move from the first floor to the second floor and cool the toasters.  The team was ready.  The farmer was nervous. The farmer instructed the toaster expert to power up the toasters.

Upon powering up the toasters, the unexpected occurred.  Some of the holes in the floor were blowing air really fast and some were blowing no air at all.  And, some of the holes blew really cold air and some had air that was actually hot!  Not only that, but air swirled on the second floor like a tornado.

“Oh no!” the farmer exclaimed.  “If we don’t fix this quick the toasters are going to burn up and I’ll be damned for life!”  The farmer turned to the toaster expert and said: “What is going on?”

“Don’t ask me,” said the toaster expert.  “I’m just responsible for the toasters.”

The farmer, extremely worried, turned to his other two team members.

Luckily, the building engineer had seen this issue before.  With his many years’ experience designing ventilation systems for office buildings he knew exactly what the problem was.

“You’ve got way too many fans on right now.  Look at the blocks of ice.  Some are melting really fast and others are not melting at all.  You guys installed way too many fans and have over-pressurized the first floor.”

“Yes,” the team said.  “Lets turn off some fans to reduce the pressure on the first floor!”

“But what should the pressure on the first floor be?” said the farmer, now getting even more nervous that some fans would be turned off.

“All you have to do is start turning fans off until the pressure distribution is even across the entire first floor,” said the building engineer.  “This is what we do to provide cooling for occupants in office buildings and it works great.”

“But this structure is for toasters, not people,” said the farmer still concerned.

No one said anything and since they didn’t have any better solutions, the building engineer started turning off fans.  After turning off 5 fans, the pressure on the first floor started to drop, but another problem arose.  The pressure on the first floor became even more uneven, with some areas extremely high and some low.

“Keep turning them off,” shouted the building engineer, “we must evenly distribute the pressure.”

After turning off 10 fans, the farmer ran upstairs to see how well it was working.  There was not nearly as much airflow as before and in most places, the toasters were nice and cool.  The farmer felt better.  But, just as he was inspecting the last row of toasters, he found a cluster of 5 toasters that were super hot and almost ready to burn up!  He ran downstairs and said: “More fans, more fans, there is a really big hot spot in that corner over there.  Turn on more fans!”

“Which fan should I turn on?” said the building engineer, “the pressure has all been optimized and evenly distributed.”

“I don’t know and I don’t care,” said the farmer “just turn them all on.”

So the team turned on all the fans again and the hot spot cooled down.  Except, they were back in the same place they started with an excessive amount of air blowing around on the second floor.

“Oh forget it,” said the farmer. “ I am damned to hell, this is useless.”

Suddenly, the PhD jumped out of his chair and said: “I’ve got it!”  He pondered for a minute looking around and writing things down.  “Ahh, it is so simple.  Why don’t we measure the temperatures at each toaster.  That way we’ll know which toaster is getting hot as we turn off the fans.”

“But you have to control the fans by pressure,” insisted the building engineer.  The PhD shot him a dirty look and it appeared that a fight was about to ensue.

“Stop!” yelled the farmer.  “The PhD is right.  All I care about is making sure that the toasters do not overheat.  Those were the instructions. The best way to do that is to measure the temperature of each toaster.  If we know the temperature of each toaster in real time, we can be confident about how we control the fans.”

“But what about press…” the building engineer was silenced by the farmer as he forcefully raised his hand.

“Sometimes the simplest solution is the best solution,” the farmer wisely said.

And so the PhD directed the team to install sensors on each toaster and began turning fans off one by one.   The excessive airflow on the second floor started to subside and relax.  Sometimes the temperature at the toaster would get hot and the PhD would instruct a fan to be turned back on providing the necessary cooling.  After a few hours of following the PhD’s vigilant instructions, the toasters were being cooled effectively.  Surprisingly, only 10 out of the original 20 fans were needed to provide adequate cooling.  The farmer gave a sigh of relief knowing that he would live happily ever after and be famous in all the valley.

The End

Summary of Insights

All of the information described in the above Table was obtained strictly from looking at the data in different ways.  Although none of this is field verified, as a data center operator, I have improved, and easily discernible direction on where to begin to look for potential issues.  And, as changes occur to fix these issues, operators can track the effects using the same visuals they used to identify them.  Site specific patterns can be recognized and accounted for.  Operators will be able to develop a deeper understanding of their facility using these tools and benefit from them based on their specific needs.  If I could come up with 9 useful insights by just looking at this data, imagine what an operator who knows his facility inside and out, could discover.

So the next time you have seemingly meaningless data, stop and graph it a few different ways.  I am certain you will obtain something insightful.

Previous Posts

About the Data
Day 1 – From 10,000 Feet
Day 2 – Rack by Rack
Day 3 – Learn to Love Graphs
Day 4 – Wordle
Day 5 – Bubble Charts
Day 6 – Bubble Charts Delta
Today – Summary

Bubble Chart Showing Rack Top - Rack Bottom

In the bubble chart yesterday, we looked at rack top temperatures over the course of a week.  I mentioned that with bubble charts, it’s not easy to compare two parameters such as rack top and rack bottom.  However, what you can do is look at the delta temperature (Rack Top Temp – Rack Bottom Temp) for each location.  The bubble chart above is showing this difference.

The story that this data is telling us is incredible and extremely useful in determining airflow inefficiencies.

Here is some background. This facility has an under floor air distribution system with cold aisle/hot aisle layout.  Cool air is delivered through perforated tiles to the rack inlet.  Generally, the difference in air temperature between the bottom and top of the rack should be around 8°F due to the heating as the air rises and the typical mixing that occurs.

Takeaways

• Rack 6, 8, 9, and 29 have greater than a 10°F delta T.  This may indicate areas where lots of mixing is occurring.  Most likely air from the hot aisle is sneaking over the top or through the racks affecting temperatures.  Blanking panels may help.
• There are many racks with a negative delta T, meaning that the top of the rack is colder than the bottom of the rack.  Potential causes are  hot air from the hot aisle  infiltrating the cold aisle low to the ground due to missing blanking panels, through holes in the floor, or through the IT equipment itself.  The negative delta T may also indicate that IT equipment is installed backwards exhausting hot air into the cold aisles.
• Most of the racks have a delta T that changes over the course of the week as indicated by multiple bubbles on a single vertical line.  However racks 12, 16, 17, and 18 show only one large bubble –  meaning that they are operating at exactly the same delta T for the entire time.  How could this be ?

Issues

• The real value of looking at delta T in a bubble graph is to identify airflow issues..  This graph would most likely be used when hot or cold spots are identified to determine the root cause.

Previous Posts
About the Data
Day 1 – From 10,000 Feet
Day 2 – Rack by Rack
Day 3 – Learn to Love Graphs
Day 4 – Wordle
Day 5 – Bubble Charts
Today – Bubble Charts Delta
Tomorrow – A Day of Rest

Rack Top Temp Bubble Chart

Bubble charts visualization takes accuracy to the next level because they can show three variables at one time.  In this case the parameters are rack, top temperature, and time. Each vertical line is a specific rack.  The graph above is displaying the top temperatures.  The bubbles on that vertical line represent the temperature of the rack during the time period (the temperatures are rounded to the closest 2°F increment).  The size of the bubble indicates how long the rack was at that temperature.  For example: the top temperatures at rack 6 were roughly 68°F, 70°F, 72°F, 74°F, and 76°F.  The larger bubbles at 72°F and 74°F indicates that rack is operating at those temperatures the majority of the time.

Takeaways

• It is instantly obvious that racks 12, 17, and 20 have persistent hot spots.
• There appears to be multiple racks that share the same temperature profile.  Racks 1 – 4 are all operating in the same temperature range.  Racks 5 – 10 are also similar.
• Rack 15 seems to be an outlier operating cooler than its surrounding sensors.  Possibly an airflow issue?
• Racks 31 – 34 are the coldest of the bunch.  There may be an excessive amount of floor tiles in this region.

Issues

• Bubble charts can only show one measured parameter at a time. In this case, it represents rack top temperatures .Bubble charts are not good for comparing rack top and rack bottom temperatures
• Also, bubble charts should be reviewed over a specific time duration.   The chart above displays data over the course of a week.  You could certainly look at data over the course of a month or longer, but it becomes more  difficult if you would like to compare two time periods such as one week versus another week.

Previous Posts
About the Data
Day 1 – From 10,000 Feet
Day 2 – Rack by Rack
Day 3 – Learn to Love Graphs
Day 4 – Wordle
Today – Bubble Charts
Tomorrow – Bubble Charts Delta