Ummmm... What??.

Today I decided to indulge in a bit of Narcisurfing to see if I was being quoted or reposted. (Don't snicker, you've done it too!) I found a few cross references to the article Brian Kerebs did as well as some of the talks I have done in the past two years. What I was NOT expecting was to be included as an authoritative source on how easy it is to hack smart meters on an anti-smart meter site.


Wait.. What? 


The kind folks over at Stop OC Smartmeters posted a link to the YouTube entry with my talk at GRRCon smart meter hacking.

Now, as I recall, I was talking about how difficult it actually is to hack a properly configured meter. No where in that talk to I recall saying that smart meters are easy to hack and therefore evil. Either this person only listened to the parts where I say that with enough equipment and time you can hack about anything, OR they completely missed the point of the talk.

I am not sure if I should be offended that my material was used by someone I consider a adversary in my professional role with out asking me, or if I should be amused at the irony of my words being used to defend the indefensible by the Pakleds of the power industry.

"He is smart, he will make our point. He will make it go."

I think this may actually qualify for posting in the Security Facepalm blog! So now I need to decide if I should send a cease and desist letter (backed by a team of bloodthirsty corporate lawyers fresh off a patent frenzy) or expose the irony of the whole thing and laugh my ass off...
 Feel free to post your thoughts in the comments.
Troll away if you like.
Could be fun!

Aaaaaaand we're back!

The last few weeks have been an interesting adventure. Earlier last month, I was quoted in a security blog entry that discussed a security breach in Puerto Rico. In the quote, I think I came across a bit more critical of vendors and utilities than I intended.

For any who have taken exception to my statement, I humbly apologize. 

I DO however, stand by the facts in the statement. I just wish I had phrased it better. Like all controversial statements, it has a back story...

I got into the power side of the security business about 3 years ago. In late 2008, I was contracted to assess the security controls of a major gas and power utility's new Smart Meter program. At that time, I hadn't even heard of Smart Meters, but I think that played a part in why I was chosen for the project. Because I had no preconceived notions or opinion on the matter, I could provide an unbiased assessment. 

What I found in the course of my investigations was instructive, to say the least. 

Smart Meters obviously represent a huge new attack surface for electric and gas distribution utilities. The utility I was working for recognized this and wisely, in my opinion, decided to engage a third party to look things over. They had already deployed a pilot program that was showing promise from an operational perspective. The utility has a strong security program in place and a seriously competent team to manage it. Thus, having sorted out that they could successfully collect consumption data, they now wanted to be able to collect it securely. 

To make a long story short, the system they were deploying was not up to their stringent standards for security. I wrote a report to that effect and, like a good security consultant, went along my merry way to perform more PCI-DSS assessments. The impression I had been left with was that the utility was approaching the project with more emphasis on revenue than safety and security. Being the cynic that I am, I just assumed that my report would get filed under "Due Diligence" and ignored until a lawyer needed it for regulatory or litigation support.

Boy, was I in for a surprise.

The security team at this utility took the report seriously. Very seriously. Seriously enough to halt deployment and suspend orders for meters. The meter vendor was consulted and some agreement reached as far as resolving the aforementioned issues. The program was put back on track and today, this utility has one of the largest operational, billing AMI systems in the world. 

Okay, let's review this real quick:

• The utility has, and has had for a long time, a good, solid security program in place.
• The utility followed a fundamental best practice and had an unbiased third party assessment performed. 
• Gaps were identified in the assessment. 
• The utility performed a risk analysis on the gaps. 
• The utility found that some gaps were outside their acceptable risk. 
• The utility halted the deployment and engaged the vendor. 
• The vendor addressed the utility concerns by making improvements to the meter. (It is worth noting that the vendor was already in the process of addressing the issues raised by the utility at the time they were raised)
• The utility resumed deployment once their concerns were properly addressed.

Holy cow! Just like it's supposed to work!

A few months later, that same meter vendor hired me and a couple of REALLY sharp people to be part of its security team. 

That was two and a half years ago. In that time I have watched something unprecedented in my experience. My employer actually took a serious approach to building an iterative and sustainable product security program. This did not take place overnight, mind you. It has taken time and no small effort on the part of the product security folks to align the organization with solid security principles. Two and a half years later, I am witness to what I consider "Miracles and Wonders" as far as InfoSec is concerned. 

Today, my employer has a good approach for product development. We have instituted a Secure Development Lifecycle (SDL) across the organization. We have added security testing, both internal and third party, to the release cycle. 

• The developers get it. 
• The product managers get it. 
• Corporate management gets it. 
• And perhaps the most important and most difficult group to convince... 
• The executives get it.

The way I know just how far we have come is by the actions of a few key people in my organization. In the last few weeks, a VP consulted with me about how we incorporate security into our product line, two hardware development teams have engaged me in the early conceptual phase of design, and another senior executive has been inquiring about testing the corporate network. This demonstrates that the drive for security is coming from the top. That is one of the core principles in a successful security program.  The sad part is that I have been so focused on what has been wrong, I forgot to look for what we are doing right.

I have been hearing similar stories around the industry of utilities demanding usable, functional, and effective security controls. This, in turn, has been driving the vendors and manufacturers to develop and sell solutions that meet these customer requirements. This is a case where the market is doing what would be done by regulation otherwise.

There seems to be a rush to judgement in the security industry against anything to do with Smart Meters.

"Smart Meters are BAD!" 

"Smart Meters invade your privacy!!" 

"Smart Meters give you cancer!!!" 

...and so on...

Of course, none of this is helped by an inherent distrust of corporations and utilities these days. A traditional and conservative approach to security is opacity and utilities and their vendors tend to be pretty conservative and traditional when it comes to security. An unfortunate side effect of this is that by not discussing security, organizations are giving the appearance that there IS no security. Transparency can help, however it is a delicate balancing act. We want to be able to reassure the public that we are doing this, and making progress, BUT we need to be able to do it without tipping our hands to the bad guys. Independent reviews of designs and systems by experienced groups help immensely. The Public Utility Commissions (PUCs) who are charged with representing the public interests have been holding the industry as a whole accountable for security, and this has driven progress as well.

Sadly, there will always be organizations and vendors that will chose the bottom line over the safety and security of their customers and systems. This is not unique to the utility industry, it is just a fact of life. The vast majority, however, are working to do the right thing. It is critical to remember that security is a process, not a project. While it is not an easy process, it is coming along.

Rather than judging the entire industry based on the unfortunate operational security mis-steps a utility took several years ago, look at what has been accomplished to date. I know that when I stopped for a moment this morning to reflect on it, I was astounded.

Things move slowly in the utility industry, and this is a long journey we are on. We cannot afford to lose sight of the goal of a safe and secure energy delivery infrastructure. We must continue to hold all the parties involved to a better standard. It doesn’t hurt, however, to stop and admire our progress occasionally. Then we can get back to work with the confidence that our efforts actually do make a difference.

Having bitten the hand that feeds me once too often, (or more accurately, the hands that feed the hand that feeds me) I have chosen to close the Power of Security blog.

I suggest following the Cutaway Security "Security Ripcord" blog at:

http://www.cutawaysecurity.com/blog/cutaway-security

Thanks for reading my ramblings.

Cheers!

Robert Former

It's time to change professions...

Smart Grid Hacking is now officially lame.


http://www.bing.com/videos/watch/video/the-electric-suitcase/17up8ge0b


It has mainstream cred...


Maybe I'll turn my focus to having the UAVs that many police departments want to deploy!

Mmmmm... soft and chewy? How effective is your 'Critical Infrastructure Supplier' at managing information security?

Living on the inside of corporate InfoSec has been an enlightening experience for me. I have had the privilege of seeing some of the best, and sadly, the WORST corporate InfoSec in my career. How is that an industry responsible for the health and safety of the majority of the developed world can simultaneously demonstrate the best and worst?


"Right, what set him off NOW??", you may ask.


The genesis of this post actually comes from outside the power industry.  By now, if you spend any energy on reading the news, you know that Stratfor lost a massive pile of e-mail to the Anonymous collective. Right and wrong aside, this action points out a major failing by a company that is supposed to embody the essence of InfoSec. If these guys can't get it right, how do we know that the companies in charge of our energy infrastructure are doing any better?


Sadly, we DON'T know... yet.


In my dealing with major utilities deploying AMI projects, I have worked with some very bright and knowledgeable individuals who have had the task of ensuring the security of power and gas distribution systems. Without exception, they have been what I call "True Believers". The problem, it seems, is not with the people who manage the security programs. It is with the people who manage the people who manage the InfoSec programs. Security is a funny thing when it comes to calculating business risk. Utilities have generally had to be concerned with the forces of the market and nature when building a risk model. AMI and the Internet have introduced that most fickle and insidious of actors to the risk equation, Humans, with Malicious Intent.


You see, natural disasters have been occurring with predictable regularity through out human history. There are nifty formulae developed by these brilliant folks called actuaries. Market forces are a bit less predictable, but they also have a longer term effect, and thus can be accommodated over time. People though, people are less predictable. People with a touch of the naughty in their soul even less so. In the connected distribution systems being deployed today, one intelligent malcontent, one internet literate criminal with business model based on extortion, one smart assed kids with a recent copy of Metasploit and a war dialer can bring an entire city to it's knees.


"FUD!", you say. Ahhh, but we now have an object lesson to examine!


By targeting and compromising Stratfor, the Anonymous Collective (or some subset of said same) brought the unpredictability of people who are pissed off AND technologically capable to bear. Was there any warning? Were there any indicators that were ignored? I doubt it. What I believe happened (Key words here kiddies!! "I Believe" This means I have nothing but what I have seen and heard in the media to base my opinion upon) is that Stratfor thought themselves impregnable. After all, they ARE one of the foremost organizations in security and information, right? Hubris is a bitch, isn't it?


The key mistake that Stratfor appear to have made is that they had a somewhat, although apparently not completely, adequate perimeter in place. If my guess is right, this led them to believe that not storing and transmitting all this hyper sensitive and embarrassing data in an encrypted fashion was not really a risk.


 Ooooops!


The lesson here, as I see it, is that security isn't a value proposition until it fails. Too many companies, utilities included, view security as an unnecessary expense and a barrier to profit. "We have firewalls, That's what the compliance guys say we need. What heck else do you really need?" 


In the utility space, we need to make sure that some effort is expended on securing more than just the meter. This will only come if systems come under scrutiny. Controlled, professional scrutiny. The only way that will happen is if the customer DEMANDS it! The utilities have started to demand it of the vendors, it's time the consumer demanded it of the utilities.


Don't think of it in terms of securing a technology. 


Think of it in terms of securing the data. 


Think of it in terms of securing the control of the system. 


When you secure the billing information, the privacy data, and the safety and control of the entire system, you secure profit.


 And isn't that why you are in business?


What are your thoughts?

Have things improved?

Following my rant last night about the suppression of security tools and vulnerability information, one of my readers brought up a good point. 

sbromberger @rformer nice article! but perhaps bad assumption that security is better than 3 years ago, esp. for utils who have deployed those meters. 1/30/12 11:06 PM

At first I thought, "Well, YEAH! It HAS improved!" 


But the more I think about it, the more I am aware that I am seeing a fairly narrow slice of the industry as a whole. Because I work for a meter manufacturer, I know what we, our customers, and our suppliers do very well but I have no insight into what our competitors are up to on security. (Because I am a good boy and don't engage in Industrial Espionage)


So I put it to you; how do YOU see the state of security in Smart Meters? I already know that security in Smart GRID is in sad shape, so don't lump them together. 


What do you KNOW as facts about it? 


What can you INFER about it? 


Please post comments or, if you want to make a longer statement, link to your own blog. I'd also be happy to put your post here with appropriate attribution. 

Security researchers: Spawn of Satan, Necessary Evil, or Security Salvation?

In the power industry, few topics will elicit more passionate opinions than security research. Even among security professionals opinions range from, "They help us make a better product" to, "They should all be thrown in Guantanamo as a threat to National Security". (People who think like that tend to capitalize National Security. Listen to them, you can hear the caps!)


Recently, a damn good team of researchers developed a toolset to test ANSI c12.18 implementations on smart meters. 18+ months of hard work wrapped up in a nice presentation and ready to go at a major security conference. Naturally, there was some buzz about this. Smart meters are still a hot topic among security people. Word found it's way around to an unnamed vendor, and after they had a chance to think about it, they politely asked our intrepid team of researchers to retract their talk and not release the tool kit. Being decent, professional folks, the research team did as they were asked and as a consequence, the free world has not yet collapsed.


So what was accomplished here? This team was actually assisted by some major players in the smart meter universe. Meter manufacturers even. Clearly having this virtual nuclear weapon handed out for free to all the script kiddies and malicious actors in the world doesn't trouble them at all. 


There is a reason why it doesn't trouble most of them.


When smart meters first began to hit the streets, security measures were virtually non-existent. Many meters didn't even require passwords to configure. This is because the manufacturers were focused on getting product out the door. Deliver meters, make money, please the stockholders, keep their jobs. The utilities didn't exactly push for reasonable security measures because they had never needed to look at meters as being remotely accessible. After a short time, the utilities actually started to consider the implications af unsecured meters. As the meters were installed, more and more technologically savvy consumers became aware of them. They started asking some pointed questions, like, "What's to keep someone from turning my meter off", and, "How could someone use this meter to steal power?"


Consumer questions inevitably led to some basic research into the safety and security of smart meters, and these became serious concerns. The security research (read as "hacker") community brought these concerns to the utility, who then started asking the meter vendors the same questions. The utilities began to make security a requirement and Voila'! Meters became more and more secure. The market solved the problem.


The point here is that without security researchers ASKING these pesky questions, and raising some very public concerns, no one would have thought to make security such a priority. The market wouldn't have demanded it.


Meter vendors have learned some valuable lessons from all of this, and now many of them actually incorporate security research teams as part of their development process. The value of these teams, both internal and 3rd party has become a part of the marketing story, and hence vital to sales. Supporting security research teams like the folks who TRIED to give us the c12.18 SMACK toolkit is in the best interests of everyone. It is shortsighted at best to try and surpress the results of this work and makes the industry appear backward and secretive.


If your product is incapable of standing up to tools like this, then you need to pull your product OFF the market and rework your design. The rest of us learned this lesson a long time back. Time to get with the program.


Security researchers and tools are not the problem, bad design IS! 


If you can't see the value in good design, "someone" is going to show you. Suppressing tools like this all but guarantees that the "someone" will be a truly malicious actor and not looking out for your, or our best interests.


Free the SMACK toolkit!