The last few weeks have been an interesting adventure. Earlier last month, I was quoted in a security blog entry that discussed a security breach in Puerto Rico. In the quote, I think I came across a bit more critical of vendors and utilities than I intended.
For any who have taken exception to my statement, I humbly apologize.
I DO however, stand by the facts in the statement. I just wish I had phrased it better. Like all controversial statements, it has a back story...
I got into the power side of the security business about 3 years ago. In late 2008, I was contracted to assess the security controls of a major gas and power utility's new Smart Meter program. At that time, I hadn't even heard of Smart Meters, but I think that played a part in why I was chosen for the project. Because I had no preconceived notions or opinion on the matter, I could provide an unbiased assessment.
What I found in the course of my investigations was instructive, to say the least.
Smart Meters obviously represent a huge new attack surface for electric and gas distribution utilities. The utility I was working for recognized this and wisely, in my opinion, decided to engage a third party to look things over. They had already deployed a pilot program that was showing promise from an operational perspective. The utility has a strong security program in place and a seriously competent team to manage it. Thus, having sorted out that they could successfully collect consumption data, they now wanted to be able to collect it securely.
To make a long story short, the system they were deploying was not up to their stringent standards for security. I wrote a report to that effect and, like a good security consultant, went along my merry way to perform more PCI-DSS assessments. The impression I had been left with was that the utility was approaching the project with more emphasis on revenue than safety and security. Being the cynic that I am, I just assumed that my report would get filed under "Due Diligence" and ignored until a lawyer needed it for regulatory or litigation support.
Boy, was I in for a surprise.
The security team at this utility took the report seriously. Very seriously. Seriously enough to halt deployment and suspend orders for meters. The meter vendor was consulted and some agreement reached as far as resolving the aforementioned issues. The program was put back on track and today, this utility has one of the largest operational, billing AMI systems in the world.
Okay, let's review this real quick:
- The utility has, and has had for a long time, a good, solid security program in place.
- The utility followed a fundamental best practice and had an unbiased third party assessment performed.
- Gaps were identified in the assessment.
- The utility performed a risk analysis on the gaps.
- The utility found that some gaps were outside their acceptable risk.
- The utility halted the deployment and engaged the vendor.
- The vendor addressed the utility concerns by making improvements to the meter. (It is worth noting that the vendor was already in the process of addressing the issues raised by the utility at the time they were raised)
- The utility resumed deployment once their concerns were properly addressed.
Holy cow! Just like it's supposed to work!
A few months later, that same meter vendor hired me and a couple of REALLY sharp people to be part of its security team.
That was two and a half years ago. In that time I have watched something unprecedented in my experience. My employer actually took a serious approach to building an iterative and sustainable product security program. This did not take place overnight, mind you. It has taken time and no small effort on the part of the product security folks to align the organization with solid security principles. Two and a half years later, I am witness to what I consider "Miracles and Wonders" as far as InfoSec is concerned.
Today, my employer has a good approach for product development. We have instituted a Secure Development Lifecycle (SDL) across the organization. We have added security testing, both internal and third party, to the release cycle.
- The developers get it.
- The product managers get it.
- Corporate management gets it.
- And perhaps the most important and most difficult group to convince...
- The executives get it.
The way I know just how far we have come is by the actions of a few key people in my organization. In the last few weeks, a VP consulted with me about how we incorporate security into our product line, two hardware development teams have engaged me in the early conceptual phase of design, and another senior executive has been inquiring about testing the corporate network. This demonstrates that the drive for security is coming from the top. That is one of the core principles in a successful security program. The sad part is that I have been so focused on what has been wrong, I forgot to look for what we are doing right.
I have been hearing similar stories around the industry of utilities demanding usable, functional, and effective security controls. This, in turn, has been driving the vendors and manufacturers to develop and sell solutions that meet these customer requirements. This is a case where the market is doing what would be done by regulation otherwise.
There seems to be a rush to judgement in the security industry against anything to do with Smart Meters.
"Smart Meters are BAD!"
"Smart Meters invade your privacy!!"
"Smart Meters give you cancer!!!"
...and so on...
Of course, none of this is helped by an inherent distrust of corporations and utilities these days. A traditional and conservative approach to security is opacity and utilities and their vendors tend to be pretty conservative and traditional when it comes to security. An unfortunate side effect of this is that by not discussing security, organizations are giving the appearance that there IS no security. Transparency can help, however it is a delicate balancing act. We want to be able to reassure the public that we are doing this, and making progress, BUT we need to be able to do it without tipping our hands to the bad guys. Independent reviews of designs and systems by experienced groups help immensely. The Public Utility Commissions (PUCs) who are charged with representing the public interests have been holding the industry as a whole accountable for security, and this has driven progress as well.
Sadly, there will always be organizations and vendors that will chose the bottom line over the safety and security of their customers and systems. This is not unique to the utility industry, it is just a fact of life. The vast majority, however, are working to do the right thing. It is critical to remember that security is a process, not a project. While it is not an easy process, it is coming along.
Rather than judging the entire industry based on the unfortunate operational security mis-steps a utility took several years ago, look at what has been accomplished to date. I know that when I stopped for a moment this morning to reflect on it, I was astounded.
Things move slowly in the utility industry, and this is a long journey we are on. We cannot afford to lose sight of the goal of a safe and secure energy delivery infrastructure. We must continue to hold all the parties involved to a better standard. It doesn’t hurt, however, to stop and admire our progress occasionally. Then we can get back to work with the confidence that our efforts actually do make a difference.
Posts
