Mmmmm... soft and chewy? How effective is your 'Critical Infrastructure Supplier' at managing information security?

Living on the inside of corporate InfoSec has been an enlightening experience for me. I have had the privilege of seeing some of the best, and sadly, the WORST corporate InfoSec in my career. How is that an industry responsible for the health and safety of the majority of the developed world can simultaneously demonstrate the best and worst?


"Right, what set him off NOW??", you may ask.


The genesis of this post actually comes from outside the power industry.  By now, if you spend any energy on reading the news, you know that Stratfor lost a massive pile of e-mail to the Anonymous collective. Right and wrong aside, this action points out a major failing by a company that is supposed to embody the essence of InfoSec. If these guys can't get it right, how do we know that the companies in charge of our energy infrastructure are doing any better?


Sadly, we DON'T know... yet.


In my dealing with major utilities deploying AMI projects, I have worked with some very bright and knowledgeable individuals who have had the task of ensuring the security of power and gas distribution systems. Without exception, they have been what I call "True Believers". The problem, it seems, is not with the people who manage the security programs. It is with the people who manage the people who manage the InfoSec programs. Security is a funny thing when it comes to calculating business risk. Utilities have generally had to be concerned with the forces of the market and nature when building a risk model. AMI and the Internet have introduced that most fickle and insidious of actors to the risk equation, Humans, with Malicious Intent.


You see, natural disasters have been occurring with predictable regularity through out human history. There are nifty formulae developed by these brilliant folks called actuaries. Market forces are a bit less predictable, but they also have a longer term effect, and thus can be accommodated over time. People though, people are less predictable. People with a touch of the naughty in their soul even less so. In the connected distribution systems being deployed today, one intelligent malcontent, one internet literate criminal with business model based on extortion, one smart assed kids with a recent copy of Metasploit and a war dialer can bring an entire city to it's knees.


"FUD!", you say. Ahhh, but we now have an object lesson to examine!


By targeting and compromising Stratfor, the Anonymous Collective (or some subset of said same) brought the unpredictability of people who are pissed off AND technologically capable to bear. Was there any warning? Were there any indicators that were ignored? I doubt it. What I believe happened (Key words here kiddies!! "I Believe" This means I have nothing but what I have seen and heard in the media to base my opinion upon) is that Stratfor thought themselves impregnable. After all, they ARE one of the foremost organizations in security and information, right? Hubris is a bitch, isn't it?


The key mistake that Stratfor appear to have made is that they had a somewhat, although apparently not completely, adequate perimeter in place. If my guess is right, this led them to believe that not storing and transmitting all this hyper sensitive and embarrassing data in an encrypted fashion was not really a risk.


 Ooooops!


The lesson here, as I see it, is that security isn't a value proposition until it fails. Too many companies, utilities included, view security as an unnecessary expense and a barrier to profit. "We have firewalls, That's what the compliance guys say we need. What heck else do you really need?" 


In the utility space, we need to make sure that some effort is expended on securing more than just the meter. This will only come if systems come under scrutiny. Controlled, professional scrutiny. The only way that will happen is if the customer DEMANDS it! The utilities have started to demand it of the vendors, it's time the consumer demanded it of the utilities.


Don't think of it in terms of securing a technology. 


Think of it in terms of securing the data. 


Think of it in terms of securing the control of the system. 


When you secure the billing information, the privacy data, and the safety and control of the entire system, you secure profit.


 And isn't that why you are in business?


What are your thoughts?

Have things improved?

Following my rant last night about the suppression of security tools and vulnerability information, one of my readers brought up a good point. 

sbromberger @rformer nice article! but perhaps bad assumption that security is better than 3 years ago, esp. for utils who have deployed those meters. 1/30/12 11:06 PM

At first I thought, "Well, YEAH! It HAS improved!" 


But the more I think about it, the more I am aware that I am seeing a fairly narrow slice of the industry as a whole. Because I work for a meter manufacturer, I know what we, our customers, and our suppliers do very well but I have no insight into what our competitors are up to on security. (Because I am a good boy and don't engage in Industrial Espionage)


So I put it to you; how do YOU see the state of security in Smart Meters? I already know that security in Smart GRID is in sad shape, so don't lump them together. 


What do you KNOW as facts about it? 


What can you INFER about it? 


Please post comments or, if you want to make a longer statement, link to your own blog. I'd also be happy to put your post here with appropriate attribution. 

Security researchers: Spawn of Satan, Necessary Evil, or Security Salvation?

In the power industry, few topics will elicit more passionate opinions than security research. Even among security professionals opinions range from, "They help us make a better product" to, "They should all be thrown in Guantanamo as a threat to National Security". (People who think like that tend to capitalize National Security. Listen to them, you can hear the caps!)


Recently, a damn good team of researchers developed a toolset to test ANSI c12.18 implementations on smart meters. 18+ months of hard work wrapped up in a nice presentation and ready to go at a major security conference. Naturally, there was some buzz about this. Smart meters are still a hot topic among security people. Word found it's way around to an unnamed vendor, and after they had a chance to think about it, they politely asked our intrepid team of researchers to retract their talk and not release the tool kit. Being decent, professional folks, the research team did as they were asked and as a consequence, the free world has not yet collapsed.


So what was accomplished here? This team was actually assisted by some major players in the smart meter universe. Meter manufacturers even. Clearly having this virtual nuclear weapon handed out for free to all the script kiddies and malicious actors in the world doesn't trouble them at all. 


There is a reason why it doesn't trouble most of them.


When smart meters first began to hit the streets, security measures were virtually non-existent. Many meters didn't even require passwords to configure. This is because the manufacturers were focused on getting product out the door. Deliver meters, make money, please the stockholders, keep their jobs. The utilities didn't exactly push for reasonable security measures because they had never needed to look at meters as being remotely accessible. After a short time, the utilities actually started to consider the implications af unsecured meters. As the meters were installed, more and more technologically savvy consumers became aware of them. They started asking some pointed questions, like, "What's to keep someone from turning my meter off", and, "How could someone use this meter to steal power?"


Consumer questions inevitably led to some basic research into the safety and security of smart meters, and these became serious concerns. The security research (read as "hacker") community brought these concerns to the utility, who then started asking the meter vendors the same questions. The utilities began to make security a requirement and Voila'! Meters became more and more secure. The market solved the problem.


The point here is that without security researchers ASKING these pesky questions, and raising some very public concerns, no one would have thought to make security such a priority. The market wouldn't have demanded it.


Meter vendors have learned some valuable lessons from all of this, and now many of them actually incorporate security research teams as part of their development process. The value of these teams, both internal and 3rd party has become a part of the marketing story, and hence vital to sales. Supporting security research teams like the folks who TRIED to give us the c12.18 SMACK toolkit is in the best interests of everyone. It is shortsighted at best to try and surpress the results of this work and makes the industry appear backward and secretive.


If your product is incapable of standing up to tools like this, then you need to pull your product OFF the market and rework your design. The rest of us learned this lesson a long time back. Time to get with the program.


Security researchers and tools are not the problem, bad design IS! 


If you can't see the value in good design, "someone" is going to show you. Suppressing tools like this all but guarantees that the "someone" will be a truly malicious actor and not looking out for your, or our best interests.


Free the SMACK toolkit!

Safe Door on a Screen Tent

So… you built the world’s most impregnable Widget. Good for you. Sadly, that means bupkis.

In todays installment of “Ranting ‘bout the Grid”, we will discuss the pitfalls of poor implementation. SmartGrid components are no different than enterprise components in many ways. It is completely within the realm of possibility to take and incredible architecture and design and totally FUBAR the implementation.

I call it “Putting a safe door on a screen tent”.

The first stumbling block is a having a security policy. Most utilities have one, it’s just that the line folks don’t know it exists because they have never had to deal with it. Until recent events brought to light the sheer inanity of using a single password for every meter, re-closer, synchrophaser, etc., most meter shops and dispatch centers have not had reason to be concerned with simple security measures. The simple fact is that a majority of utility security policies I have read are geared toward the enterprise and lack specifics for the field. That puts the policy into the realm of “Speculative Fiction”.

So, step one; update the utility security policy to address SCADA systems. That done, it follows logically that process, procedure, SoP, etc. need to be updated as well. When you do this, please engage a security professional who understands control system concerns. All too often, I see polices written with the best intentions by the uninitiated, and it leads to mis-interpretation, confusion, and ultimately fails. Remember a security policy is supposed to be like the law. It requires legislative (Board of Directors) approval. It informs and directs the process and procedure.

Great, you have a policy and you have a process to use it. Now lets look at the landscape for a bit.

Working for a manufacturer, I always try to emphasize the concept of having as many points where you can exercise security controls as is practical. Separate the control network from the enterprise network from the outset. The isn’t much data that needs to flow directly from a control system to an enterprise desktop and what data DOES need to move into the enterprise, like billing data, can be moved through a proxy or bastion host. The weakest point in any network is inside the perimeter where you have to count more on administrative controls rather than technical ones. All it takes is one Stupid User Trick ^® (SUT) to bring a botnet to the office.

Beyond segregating the control network from the enterprise network, it’s a good idea to develop trust zones and control movement between them. One way to look at it is, the Head End (HE) is one zone, the WWAN another, the control components networks a third, and finally the control components themselves. By doing this, you can contain a compromise to a trust zone and act appropriately. Control components represent the largest attack surface most security professionals have or will ever deal with. Ask yourself, how much do you trust the traffic from an area like this? Not so much. With that in mind, you need to assure that the data flowing into the system has appropriate confidentiality and integrity. Crypto controls like DSA and AES help with this, but only if applied appropriately. If you have a good design, a hacker compromising one control component has accomplished just that. One component. Granted, if that one component can dump several hundred mega-watts, it’s still a Very Bad Thing, but at least it’s not ALL of that type of component that is now compromised. The WWAN represents the next zone, and while it has a smaller attack surface, it is a higher value target. Adding application and packet firewalls as well as IDS/IPS at the entry/exit points of this zone will help ensure that if someone effects a compromise at the carrier or on a Metro Wireless network, you still have a barrier. In theory, a well-designed system will have all the control data encrypted and signed, so a compromise of the backhaul SHOULD only allow DoS type attacks. There is a lot of “SHOULD” in this world.

In the next installment, I will discuss security at the Head End, Personnel Vetting, and Responsible Patching. 

Security Testing and the SmartGrid (or why we need to pull the industry's head out of it's butt)

*DISCLAIMER*
I work as the Head of Security Research and Testing for one of the major SmartMeter/AMI vendors, so that will naturally slant my views. Deal with it.


I just love going over security assessments with vendors in the SmartGrid and SmartMeter space. I have been doing this for a few years now and it never fails to amuse me when I get the response, "But, you are not supposed to do that with that interface!" Golly Mr. (or Ms.) Developer Person, you are indeed correct, however security testing is about seeing what I CAN do with an interface, not what I SHOULD do with it. This seems to come as a shock to some folks.

Here is a news flash for hardware, firmware, and software developers in the SG/SM space. Security assessors are coming to mess your stuff up. We will take the fruits of your labors and we will rip them open, probe, prod, shock, dissect, analyze, under-volt, over-volt, glitch, x-ray, de-solder, re-solder, JTAG, ADA-Pro, and generally do rude things to them. We will then write smug reports about how easy it was to break into your stuff.
There are two things you can do here;

• You can either whine about how hard it is to secure this or that feature and how no one would EVER try that! 
• You can step up and find ways to plug the many, many holes (sieve like in some cases) so we can retest it prior to going to production installs.

Security, when approached properly, is a matter of balancing risk and mitigation. The power industry is a tricky place to play games with assessing risk however. When a utility make a risk decision, it can impact it's neighbors. When a manufacturer makes a risk decision, it can impact the entire industry. Think about it this way; Widget, Inc decides that incorporating security features in a meaningful fashion into its product has a significantly negative impact on the sales margin for that component (because in a competitive market like this, security never adds to the price, only the cost). Widget's component is installed in a small part of the distribution or transmission grid. M@D_sk11ls_Skr1p7_M@s73r decides, just for LULZ, to attack this component and publish results. Next day the headline reads,

"SMARTGRID HACKED!!! ARMAGEDDON AT HAND!!!"


Of course all vendors, manufacturers, utilities, regulatory bodies, and consultants are now considered Satan's lackeys. All because security features don't increase shareholder value.

There is good news in all this. Equipment manufacturers are stepping up and being responsible (my own employer being one that has been doing this for some time) and implementing Secure Development Lifecycle programs. This is bringing quite a bit of gear under the scrutiny of security researchers and assessors and that is a Very Good Thing (r) in my opinion.

The bottom line? Test your product. Hire an in-house team to do this, and then send it to one of the reputable labs for MORE testing. Find the flaws before you ship because you can be assured that if you don't, someone will. AFTER you ship.

The new rules... umm, sort of... but...not really...

NIST-IR 7628

So big it needs three PDFs to contain it...

Lots of words, any value?

Let's hear your thoughts!

http://csrc.nist.gov/publications/nistir/ir7628/nistir-7628_vol1.pdf
http://csrc.nist.gov/publications/nistir/ir7628/nistir-7628_vol2.pdf
http://csrc.nist.gov/publications/nistir/ir7628/nistir-7628_vol3.pdf

New rules in California

http://info.sen.ca.gov/pub/09-10/bill/sen/sb_0801-0850/sb_837_bill_20100622_amended_asm_v93.pdf

(scroll to page 7 and read section 8364.5)

It looks like California is beginning to pay attention to SmartMeters and security. Starting in 2012, power and gas utilities that use SmartMeter/AMI systems will have to post the results of their security audits so the public can review them. There are some additional rules that apply to the vendors that state they have to provide the details of any encryption methods employed as well as provide the results of their own security audit of the system to be deployed.

While these provisions will add burden to the utilities and vendors, I am firmly of the opinion that they are long overdue. One of the major problems developing for the industry is trust. The public has an inherent distrust of large corporations and utilities, and the introduction of AMI has added fuel to the fire. Conspiracy theorists see Big Brother looking out at them from every meter, some (not all) independent security researchers have hyped vulnerabilities as precursors to the Apocalypse without any risk context. The advent of Time of Use and more granular billing structures has had a significant impact on some of the more economically challenged customers, and the press has taken all of this and sensationalized it to sell copy. By requiring these audits and other provisions, the CPUC has mandated that measurable results be published in such a way that they can be used to refute many of the claims that SmartMeters are inherently unsecure.

 I am sure the utilities and vendors will raise a ruckus about having additional regulation and about how this will cost them pile of money to comply. I am also certain that there are members of my profession who will cry foul about posting an audit that exposes the weaknesses of a system. I am going to take a stand against the conventional wisdom in the information security industry and say that this is a GOOD thing. The best way to make sure that your weaknesses are not published to the world? Deal with them! Fix the holes, tighten up the procedures, improve the process. That way when you have to publish an audit, you can publish something that has nothing to give you away.

Sadly, security isn't even a second thought in many cases, it is less than an afterthought. Because it is very difficult to quantify a return on investment from security, it is most often only given attention when is is breached, or someone points out that the emperor has a pretty transparent wardrobe. These audits make a business case to improve security.

Time will tell how these rules impact the deployment of AMI...