The Power of Security, The Security of Power

Have things improved?

2012-01-31T08:30:00.000-05:00

Following my rant last night about the suppression of security tools and vulnerability information, one of my readers brought up a good point.

sbromberger
@rformer nice article! but perhaps bad assumption that security is better than 3 years ago, esp. for utils who have deployed those meters.
1/30/12 11:06 PM

At first I thought, "Well, YEAH! It HAS improved!"

But the more I think about it, the more I am aware that I am seeing a fairly narrow slice of the industry as a whole. Because I work for a meter manufacturer, I know what we, our customers, and our suppliers do very well but I have no insight into what our competitors are up to on security. (Because I am a good boy and don't engage in Industrial Espionage)

So I put it to you; how do YOU see the state of security in Smart Meters? I already know that security in Smart GRID is in sad shape, so don't lump them together.

What do you KNOW as facts about it?

What can you INFER about it?

Please post comments or, if you want to make a longer statement, link to your own blog. I'd also be happy to put your post here with appropriate attribution.

Security researchers: Spawn of Satan, Necessary Evil, or Security Salvation?

2012-01-30T22:27:00.000-05:00

In the power industry, few topics will elicit more passionate opinions than security research. Even among security professionals opinions range from, "They help us make a better product" to, "They should all be thrown in Guantanamo as a threat to National Security". (People who think like that tend to capitalize National Security. Listen to them, you can hear the caps!)

Recently, a damn good team of researchers developed a toolset to test ANSI c12.18 implementations on smart meters. 18+ months of hard work wrapped up in a nice presentation and ready to go at a major security conference. Naturally, there was some buzz about this. Smart meters are still a hot topic among security people. Word found it's way around to an unnamed vendor, and after they had a chance to think about it, they politely asked our intrepid team of researchers to retract their talk and not release the tool kit. Being decent, professional folks, the research team did as they were asked and as a consequence, the free world has not yet collapsed.

So what was accomplished here? This team was actually assisted by some major players in the smart meter universe. Meter manufacturers even. Clearly having this virtual nuclear weapon handed out for free to all the script kiddies and malicious actors in the world doesn't trouble them at all.

There is a reason why it doesn't trouble most of them.

When smart meters first began to hit the streets, security measures were virtually non-existent. Many meters didn't even require passwords to configure. This is because the manufacturers were focused on getting product out the door. Deliver meters, make money, please the stockholders, keep their jobs. The utilities didn't exactly push for reasonable security measures because they had never needed to look at meters as being remotely accessible. After a short time, the utilities actually started to consider the implications af unsecured meters. As the meters were installed, more and more technologically savvy consumers became aware of them. They started asking some pointed questions, like, "What's to keep someone from turning my meter off", and, "How could someone use this meter to steal power?"

Consumer questions inevitably led to some basic research into the safety and security of smart meters, and these became serious concerns. The security research (read as "hacker") community brought these concerns to the utility, who then started asking the meter vendors the same questions. The utilities began to make security a requirement and Voila'! Meters became more and more secure. The market solved the problem.

The point here is that without security researchers ASKING these pesky questions, and raising some very public concerns, no one would have thought to make security such a priority. The market wouldn't have demanded it.

Meter vendors have learned some valuable lessons from all of this, and now many of them actually incorporate security research teams as part of their development process. The value of these teams, both internal and 3rd party has become a part of the marketing story, and hence vital to sales. Supporting security research teams like the folks who TRIED to give us the c12.18 SMACK toolkit is in the best interests of everyone. It is shortsighted at best to try and surpress the results of this work and makes the industry appear backward and secretive.

If your product is incapable of standing up to tools like this, then you need to pull your product OFF the market and rework your design. The rest of us learned this lesson a long time back. Time to get with the program.

Security researchers and tools are not the problem, bad design IS!

If you can't see the value in good design, "someone" is going to show you. Suppressing tools like this all but guarantees that the "someone" will be a truly malicious actor and not looking out for your, or our best interests.

Free the SMACK toolkit!

Safe Door on a Screen Tent

2011-06-17T15:23:00.000-04:00

So… you built the world’s most impregnable Widget. Good for you. Sadly, that means bupkis.

In todays installment of “Ranting ‘bout the Grid”, we will discuss the pitfalls of poor implementation. SmartGrid components are no different than enterprise components in many ways. It is completely within the realm of possibility to take and incredible architecture and design and totally FUBAR the implementation.

I call it “Putting a safe door on a screen tent”.

The first stumbling block is a having a security policy. Most utilities have one, it’s just that the line folks don’t know it exists because they have never had to deal with it. Until recent events brought to light the sheer inanity of using a single password for every meter, re-closer, synchrophaser, etc., most meter shops and dispatch centers have not had reason to be concerned with simple security measures. The simple fact is that a majority of utility security policies I have read are geared toward the enterprise and lack specifics for the field. That puts the policy into the realm of “Speculative Fiction”.

So, step one; update the utility security policy to address SCADA systems. That done, it follows logically that process, procedure, SoP, etc. need to be updated as well. When you do this, please engage a security professional who understands control system concerns. All too often, I see polices written with the best intentions by the uninitiated, and it leads to mis-interpretation, confusion, and ultimately fails. Remember a security policy is supposed to be like the law. It requires legislative (Board of Directors) approval. It informs and directs the process and procedure.

Great, you have a policy and you have a process to use it. Now lets look at the landscape for a bit.

Working for a manufacturer, I always try to emphasize the concept of having as many points where you can exercise security controls as is practical. Separate the control network from the enterprise network from the outset. The isn’t much data that needs to flow directly from a control system to an enterprise desktop and what data DOES need to move into the enterprise, like billing data, can be moved through a proxy or bastion host. The weakest point in any network is inside the perimeter where you have to count more on administrative controls rather than technical ones. All it takes is one Stupid User Trick ^® (SUT) to bring a botnet to the office.

Beyond segregating the control network from the enterprise network, it’s a good idea to develop trust zones and control movement between them. One way to look at it is, the Head End (HE) is one zone, the WWAN another, the control components networks a third, and finally the control components themselves. By doing this, you can contain a compromise to a trust zone and act appropriately. Control components represent the largest attack surface most security professionals have or will ever deal with. Ask yourself, how much do you trust the traffic from an area like this? Not so much. With that in mind, you need to assure that the data flowing into the system has appropriate confidentiality and integrity. Crypto controls like DSA and AES help with this, but only if applied appropriately. If you have a good design, a hacker compromising one control component has accomplished just that. One component. Granted, if that one component can dump several hundred mega-watts, it’s still a Very Bad Thing, but at least it’s not ALL of that type of component that is now compromised. The WWAN represents the next zone, and while it has a smaller attack surface, it is a higher value target. Adding application and packet firewalls as well as IDS/IPS at the entry/exit points of this zone will help ensure that if someone effects a compromise at the carrier or on a Metro Wireless network, you still have a barrier. In theory, a well-designed system will have all the control data encrypted and signed, so a compromise of the backhaul SHOULD only allow DoS type attacks. There is a lot of “SHOULD” in this world.

In the next installment, I will discuss security at the Head End, Personnel Vetting, and Responsible Patching.

Security Testing and the SmartGrid (or why we need to pull the industry's head out of it's butt)

2011-06-16T18:50:00.000-04:00

*DISCLAIMER*
I work as the Head of Security Research and Testing for one of the major SmartMeter/AMI vendors, so that will naturally slant my views. Deal with it.

I just love going over security assessments with vendors in the SmartGrid and SmartMeter space. I have been doing this for a few years now and it never fails to amuse me when I get the response, "But, you are not supposed to do that with that interface!" Golly Mr. (or Ms.) Developer Person, you are indeed correct, however security testing is about seeing what I CAN do with an interface, not what I SHOULD do with it. This seems to come as a shock to some folks.

Here is a news flash for hardware, firmware, and software developers in the SG/SM space. Security assessors are coming to mess your stuff up. We will take the fruits of your labors and we will rip them open, probe, prod, shock, dissect, analyze, under-volt, over-volt, glitch, x-ray, de-solder, re-solder, JTAG, ADA-Pro, and generally do rude things to them. We will then write smug reports about how easy it was to break into your stuff.
There are two things you can do here;

You can either whine about how hard it is to secure this or that feature and how no one would EVER try that!
You can step up and find ways to plug the many, many holes (sieve like in some cases) so we can retest it prior to going to production installs.

Security, when approached properly, is a matter of balancing risk and mitigation. The power industry is a tricky place to play games with assessing risk however. When a utility make a risk decision, it can impact it's neighbors. When a manufacturer makes a risk decision, it can impact the entire industry. Think about it this way; Widget, Inc decides that incorporating security features in a meaningful fashion into its product has a significantly negative impact on the sales margin for that component (because in a competitive market like this, security never adds to the price, only the cost). Widget's component is installed in a small part of the distribution or transmission grid. M@D_sk11ls_Skr1p7_M@s73r decides, just for LULZ, to attack this component and publish results. Next day the headline reads,

"SMARTGRID HACKED!!! ARMAGEDDON AT HAND!!!"

Of course all vendors, manufacturers, utilities, regulatory bodies, and consultants are now considered Satan's lackeys. All because security features don't increase shareholder value.

There is good news in all this. Equipment manufacturers are stepping up and being responsible (my own employer being one that has been doing this for some time) and implementing Secure Development Lifecycle programs. This is bringing quite a bit of gear under the scrutiny of security researchers and assessors and that is a Very Good Thing (r) in my opinion.

The bottom line? Test your product. Hire an in-house team to do this, and then send it to one of the reputable labs for MORE testing. Find the flaws before you ship because you can be assured that if you don't, someone will. AFTER you ship.

The new rules... umm, sort of... but...not really...

2010-09-17T13:57:00.000-04:00

NIST-IR 7628

So big it needs three PDFs to contain it...

Lots of words, any value?

Let's hear your thoughts!

http://csrc.nist.gov/publications/nistir/ir7628/nistir-7628_vol1.pdf
http://csrc.nist.gov/publications/nistir/ir7628/nistir-7628_vol2.pdf
http://csrc.nist.gov/publications/nistir/ir7628/nistir-7628_vol3.pdf

New rules in California

2010-07-01T10:38:00.000-04:00

http://info.sen.ca.gov/pub/09-10/bill/sen/sb_0801-0850/sb_837_bill_20100622_amended_asm_v93.pdf

(scroll to page 7 and read section 8364.5)

It looks like California is beginning to pay attention to SmartMeters and security. Starting in 2012, power and gas utilities that use SmartMeter/AMI systems will have to post the results of their security audits so the public can review them. There are some additional rules that apply to the vendors that state they have to provide the details of any encryption methods employed as well as provide the results of their own security audit of the system to be deployed.

While these provisions will add burden to the utilities and vendors, I am firmly of the opinion that they are long overdue. One of the major problems developing for the industry is trust. The public has an inherent distrust of large corporations and utilities, and the introduction of AMI has added fuel to the fire. Conspiracy theorists see Big Brother looking out at them from every meter, some (not all) independent security researchers have hyped vulnerabilities as precursors to the Apocalypse without any risk context. The advent of Time of Use and more granular billing structures has had a significant impact on some of the more economically challenged customers, and the press has taken all of this and sensationalized it to sell copy. By requiring these audits and other provisions, the CPUC has mandated that measurable results be published in such a way that they can be used to refute many of the claims that SmartMeters are inherently unsecure.

I am sure the utilities and vendors will raise a ruckus about having additional regulation and about how this will cost them pile of money to comply. I am also certain that there are members of my profession who will cry foul about posting an audit that exposes the weaknesses of a system. I am going to take a stand against the conventional wisdom in the information security industry and say that this is a GOOD thing. The best way to make sure that your weaknesses are not published to the world? Deal with them! Fix the holes, tighten up the procedures, improve the process. That way when you have to publish an audit, you can publish something that has nothing to give you away.

Sadly, security isn't even a second thought in many cases, it is less than an afterthought. Because it is very difficult to quantify a return on investment from security, it is most often only given attention when is is breached, or someone points out that the emperor has a pretty transparent wardrobe. These audits make a business case to improve security.

Time will tell how these rules impact the deployment of AMI...

Wake up!!

2010-06-18T17:41:00.000-04:00

Time to reactivate this blog. the only thing worse than no blog is a dead one.

Ask yourself a question;

How would you feel about your electric meter being controlled from the Internet? How about getting your Internet from your power meter? Sounds silly, right? Maybe not.

Over the last few conferences, I have gotten the impression that some utilities are looking for a way to reduce infrastructure costs and possibly monetize their proposed AMI networks. There has been a strong push toward higher bandwidth, IP to the meter, and even WiFi on the meter. This all smells like Internet via the meter to me.

I CAN see a reasonable commercial/business reason for this. Internet connectivity reaches most places people live these days so why not use this existing network and save the capital costs? Alternatively, if you are going to all the trouble to install this huge network, why not find a way to make it pay for itself? The Telcos and Cable operators did it, why not the utilities?

Why indeed...

There are a few darn good reasons not to. Consider the risks of combining a major control network with a public access medium. The obvious is that it allows remote access to the meter. I don't care how good you think your built-in firewall is, it isn't good enough to mitigate this risk. To an accomplished hacker a firewall is nothing more than a logging router that does NAT. Remote access to a large block of meters is a primary target. With this you can hold an entire city for ransom, wreak havoc, or even weaponize it for a nation/state actor. Next is the Denial of Service (DOS) attack. it doesn't even need to be directed or deliberate. Once the network used to control meters and collect data is overrun with the latest worm traffic, your expensive new AMI system has reverted to the functional equivalent of the old style electro-mechanical meters.

In coming days and week, I will offer fact, opinion, news, and trivia on the new power grid initiatives from an insiders perspective.

Stay tuned!

Robert

Travis Goodspeed's Blog

2010-03-25T10:42:00.000-04:00

Another plug for a security blog

Travis is the guy who found and published the PRNG weakness in the TI ZigBee chipset.

Travis Goodspeed's Blog

Cassandra Security

2010-03-25T10:34:00.000-04:00

Just a plug for the guys at Cassandra

Check em out, well reasoned and written fact and opinion

Cassandra Security

Smart Meters - An introduction

2010-03-23T18:35:00.004-04:00

Smart Meters - An introduction
INTRODUCTION
Like any commodity, energy, gas, water and sewage disposal must be measured to be sold. Over time a number of technologies have evolved to accomplish this goal. Naturally as long as there has been a way to meter a service, there have been people trying to figure out how to get around the metering systems and get those services for free. This paper will examine some of the new metering technologies and compare them to the older traditional methods in the light of a secure implementation.

TRADITIONAL METERING (Electro Mechanical)
The way services and utilities have traditionally been monitored has been with a mechanical system. Gas or water runs past some sort of vane which spins, driving gears and turning dials. Similarly a mechanical electric meter uses induction loops to spin an aluminum disk which then turns gears that spin dials. Of course this data needs to be collected to be billed. With this type of metering, a utility will typically get an actual read through a meter reader and use sophisticated statistical modeling to estimate the rest of the time. How often the actual read occurs is really a function of state regulation and utility policy. Now an interesting thing that you can do (among many) is that if you understand the cycle that your utility uses, you can invert the meter. That’s right, pull it out of the socket, turn it over, put it back in and you still get electricity, but the meter runs BACKWARDS! Cool, free electricity as long as you don’t let the meter reader see the meter installed upside down. Some meters had mechanisms to prevent them from running backwards, but not all. A way the utilities have come up with to combat this and other methods of fraud goes back to the sophisticated statistical modeling. If the system sees a significant change in your usage pattern, you get flagged for a live read.

AUTOMATIC METER READING (AMR)
Automated Meter Reading was introduced to accomplish two primary goals; improve the accuracy of metering by getting more live reads, and reduce the work force required to gather meter data. AMR is set up to send the data back to the billing system via some communication method, be it fixed service radio, data over power line, Meter reader with a handheld device, or by a truck driving around and gathering the data over a low power radio link to the meter. AMR introduced another interesting feature: the ability to detect if someone had tampered with the meter. As the technology has advanced, and traditional mechanical meters are replaced, AMR systems have become increasingly prevalent. The major downside to an AMR system is that it is mostly a one way communication system. The meters “bubble up” the data on a periodic basis to the utility. Newer Advanced AMR systems have integrated some limited two way functionality adding features like reading meter data on demand and remote disconnect switches. “Demand Reset” is also a function sometimes provided in two-way AMR. When you track a customer’s peak demand, the meter records the maximum peak instantaneous usage over a period. At the end of that period, you need to reset the demand back to zero so you can record during the next period. In earlier days, there was a locked button or lever the meter reader would press (they had the key) when they did the readings. More modern devices can do it based on a clock. In two-way AMR you can send a demand reset command after you’ve successfully retrieved the demand value. This inevitable march of progress has led us to the latest in service metering technology, AMI.

ADVANCED METER INFRASTRUCTURE (AMI)
Advanced Meter Infrastructure is where it gets cool (and spooky to some). AMI is the natural evolution of AMR technology. A full two way communication system that allows on demand reading, Time of Use (TOU) billing, remote disconnect operation, load limiting, demand response, and more. Where things get really interesting is when a Home Area Network (HAN)is integrated into the meter. Now the meter can talk to properly equipped furnaces, air conditioners, water heaters, micro generation systems, thermostats and In Home Display (IHD) devices that tell you just how much running the dishwasher will cost you to run at 3 in the afternoon vs. 3 in the morning. Past that it enables portable billing and usage of Personal Electric Vehicles (PEV) which can also be leveraged as local storage devices to help manage load spikes on the distribution system. Another interesting feature is the ability to subscribe to a service from the utility that allows them to reduce your power consumption by managing large appliances in return for a lower billing rate.

Notice I said SUBSCRIBE. You have to sign up for it. This is important because for all the really interesting things AMI can do, it has also stirred up some serious political debate. While the technical concerns around security are addressed further on, let us examine the paranoia that AMI has raised. The first fear is that The Government (you know who I mean, the socialist/fascist (sic) fat cats in and around DC) is going to take control of your appliances and dictate when you can use them, and how much. This is followed rapidly by the fear that The Government will collect your usage information and somehow use it against you. None of this is helped by political pundits and demagogues who have decided to whip the masses into a frenzy using this as one of the egg beaters. Politics aside, there ARE some genuine security concerns to look at, and that is one of the things we will look at starting with the next installment, “The Risks”

There are some intermediate steps in the evolution from electro mechanical metering to AMI, but they are outside the scope of this discussion. If you would like more information about the details, Wikipedia has some good articles.

The Power of Security

2010-03-23T10:23:00.000-04:00

Pay any attention to the security press and you can't help but notice all the discussion about securing the power grid. My goal with this site is to document events and progress on this topic as well as to offer the opinions of myself and as many experienced professionals in the industry as I can goad into writing.

My opening post will be a sort of quick primer on the history of residential power metering...