A Common Pattern – Using Insecure JavaScript Libraries

I will be up front and state that I am no JavaScript fan. I REALLY liked it back in the day when it was readable! I much prefer TypeScript as it fosters more robust, readable and maintainable code, especially for large projects. But that’s just my opinion.

With all that said, I understand JS’s importance in today’s modern toolset, and I am fine with that.

But there’s an alarming trend I have noticed with many customers who rely on common JS frameworks such as jQuery, Angular, Bootstrap, React and so on. Seems there’s a new framework every week!

The trend is devs using insecure JS libraries. 100% of the customers I have worked with over the last year or so have had at least one insecure JS library.

My old boss, Steve Lipner, coined a term about 20 years ago to describe these kinds of dependencies: he used the term ‘giblets’.

A giblet is code you depend on but you don’t control and JS frameworks are a prime giblet. If a JS framework you use has a security vulnerability, then you have a security vuln, too! Here’s a little more info on giblets.

So how do you remedy this issue of using insecure JS libraries?

First, you need to understand which libraries you use, and keep abreast of any security vulnerabilites therein. Someone in your org needs to keep track of this; it’s part of their job!

Next, you need to make a hard decision. Does your application host a copy of the libraries and possibly be behind on security fixes? Or, does your application pull the latest-n-greatest from the source and possibly run the risk of regressions? I cannot answer that question for you, you need to determine the policy and live with the tradeoffs offered by each scenario.

As a side note, there’s a great add-on for Chrome named Retire.js that will flag potentially vulnerable JS libraries from the browser. It produces output like the image below when you navigate to a web page that references a vulnerable library.

I want to wrap this up with a story.

One customer told me they had to use an old and insecure version of jQuery for compatibility with Internet Explorer on Windows XP! After looking at their analytics, they realized that some customers were still using Windows XP, but it was only three customers out of thousands!

So we added landing-page text that said, “We see you’re still using Windows XP! Windows XP is no longer supported by Microsoft, and we will retire support for Windows XP on September 30th 2019. After this date you might encounter issues using our system. Please upgrade before then.”

I think that’s reasonable!

-Michael

Azure Policy – A Love Story

In a previous post I briefly mentioned the policy-enforcement capability built into Azure. If you have not seen Azure Policy, used it or simply not aware of it, you really need to know about it!

Before I start, this post is merely to make you aware Azure Policy exists, it’s not a complete tutorial by any stretch and it’s policy from my perspective.

So what is Azure Policy? It’s simply a way to make sure specific conditions exist and if the conditions don’t exist then either raise an audit event or block the action. There’re other options as well, but audit and deny are, by far, the two most important outcomes.

For example, some conditions might be:

  • Require TLS to blob store
  • Secrets in Key Vault must be in hardware
  • Don’t allow VMs to have a public end point
  • Require Threat Detection on Azure SQL Server
  • Require HTTP/2 for Azure Functions

I could keep going, but I am sure you get the picture. The beauty of Azure Policy is once it’s set, enforcement is performed automatically.

So let’s look at an example. Many customers want to, or need to control where their data resides. Data sovereignty is a big deal for some customers. For example, a customer might not want data in specific countries because of legal or regulatory reasons.

A customer wants their CosmosDB data available only in specific regions. One of the nice things about CosmosDB is there’s a simple UI that makes it easy to replicate data geographical. Of course, with that ease and power comes responsibility!

The customer wants to put a requirement in place that limits data replication to only East US, East US 2, South Central US and North Central US. That way ‘something’ comes to the rescue if someone fat-fingers the CosmosDB replication UI, or ‘accidently’ runs a PowerShell script that configures data replication all over the world.

Enter Azure Policy.

I won’t lie, Azure Policy files are a little obtuse. They are declared in JSON. The following file enforces a rule that only allows CosmosDB data in the four regions mentioned above.

{
     "properties": {
         "displayName": "Limit CosmosDB to specific locations",
         "policyRule": {
             "if": {
                 "not": {
                     "field": "Microsoft.DocumentDB/databaseAccounts/Locations[*].locationName",
                     "in": [
                         "East US",
                         "East US 2",
                         "South Central US",
                         "North Central US"
                     ]
                 }
             },
             "then": {
                 "effect": "deny"
             }
         }
     }
 }

Here’s how to read it. It all starts at the policyRule block. Before I start, remember that ComsosDB was known as DocumentDB.

“if the database account locations is not in East US, East US 2, South Central or North Central US, then deny the action. “

In other words, if someone attempts, by any means, to set a data replication site other than one of the four, the request is blocked.

If you change the “deny” to “audit” the action will go through, but an audit event is raised. This is useful if you want to roll out policy in an existing environment and you want to get a feel for how bad things are without stopping the business from running 🙂

Some customers think in terms of preventative and detective controls. “Deny” is preventatve and “Audit” is detective.

This is just the tip of Azure Policy. If you want to experiment, take a look at some of the sample policies available on GitHub.

One final thought. The title says “A Love Story” and clearly that’s a little extreme, maybe even clickbait, but Azure Policy is one of the most important security compliance features built in to Azure.


Security and Compiler Optimizations

A previous post I wrote about password comparison led to much discussion between some colleagues and I; most notably long-time friend and co-author David LeBlanc and Reid Borsuk in the security engineering team.

Reid mentioned an issue that I have not considered in a long time; and that is the potential for compiler optimizations to introduce security vulnerabilities.

The first time I learned of this issue was during the development of Windows Server 2003. Here is the link to my initial message to bugtraq in Nov 2002 on the topic.

Fast forward almost 17 years.

The issue with my initial code is the compiler might generate code that could short-circuit the string comparison. Here’s the code:

static bool TimeConstantCompare(string hashName, string s1, string s2)
{
      var h1 = HashAlgorithm.Create(hashName).ComputeHash(Encoding.UTF8.GetBytes(s1));
      var h2 = HashAlgorithm.Create(hashName).ComputeHash(Encoding.UTF8.GetBytes(s2));

      int accum = 0;
      for (int i = 0; i < h1.Length; i++)
          accum |= (h1[i] ^ h2[i]);

      return accum == 0;
}

The code walks over the bytes of each hash; bytes 0..N from hash 1 and bytes 0..N for hash 2. Each byte is XOR’d and if two bytes are different the result is non-zero (it’s a property of XOR) and this result is OR’d with an accumulator. This means, and this is the important part, that if the accumulator ever gets to non-zero it will always be non-zero from that point on.

Now here’s the fun part.

The code returns a bool (true or false) based on the accumulator being zero or not, but the compiler might emit optimized code that no longer performs the rest of the loop if at ANY point the accumulator is not zero.

For example, let’s say the hashes are 32-bytes in size. In theory, the loop will ALWAYS iterate from 0-31. However, let’s say when i==17, the two bytes are 0xDE and 0xAD, this will yield a value of 0xDE XOR 0xAD == 0x73 and if the accumulator is 0x00, then 0x00 OR 0x73 == 0x73. The compiler knows that from this point forward (because of how OR works), the accumulator will always be >= 0x73. The last line of the function returns a value based on the accumulator == 0, the compiler knows that the accumulator can NEVER be 0x00 from this point on, so skip the rest of the loop!

I hope that makes sense!

This isn’t a huge issue because we’re comparing a hash and not the actual password. But it’s still valuable to know compilers can do this.

There are two fixes, here’s the updated method:

[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)]        
static int TimeConstantCompare(string hashName, string s1, string s2)
{
     var h1 = HashAlgorithm.Create(hashName).ComputeHash(GetRawBytes(s1));
     var h2 = HashAlgorithm.Create(hashName).ComputeHash(GetRawBytes(s2));

     int accum = 0;
     for (int i = 0; i < h1.Length; i++)
          accum |= (h1[i] ^ h2[i]);

     return accum;
}

First, the method returns an int rather than a bool, so the compiler has no clue what you’re doing with an int. In theory, the compiler could look at how the code calling this function uses the return value to optimize the code, but inter-procedural optimizations are harder and less common than intra-procedural optimizations. It would be very hard for the compiler to ascertain how to optimize this method based on every call in the code to the method.

The real fix is to add the NoOptimization directive before the method implementation. This prevents the optimizer from doing little tricks like this. The obvious downside is a potential performance hit. If this is code that’s called constantly, in a tight loop, then you may see degradation. But like everything in life, measure!

Reid provided me with some more in-depth detail about inter- vs intra-procedural optimizations. You probably don’t need to know this, but if you’re nosey, read on.

This statement is true for traditional compilers, but the .NET JIT is prohibited from inter-procedural optimizations only in the case where both NoInlining and NoOptimization is enforced, not just NoOptimization. That’s because the .NET runtime requires method signatures to be carried with the objects they represent. This means every instance of Program (including the static one) carries the pointer to the compiled method (or JIT stub pre-compile), so every caller needs to call the function in the same way. The caller would then be restricted to calling the “standard” implementation of the method without this cheat.
The way .NET implements the scenario you’re talking about is to force an inline to occur. After inlining occurs, the NoOptimization attribute gets stripped since the method disappeared. Then they force optimization on the new parent method with code already inlined. Preventing both inlining and optimization prevents the specific optimization you refer to in your blog post.

My updated code for string comparison is on GitHub

On a side note, you may notice the code to convert a string into a series of bytes is different. That’s a topic for another post 🙂

CosmosDB and SQL Injection

Over the last few months I have started digging deep into CosmosDB. I have to say I really like it, and coming from someone who has used SQL Server since v4.2 on OS/2, that’s saying something!

To be honest, I have only used the SQL API, but eventually I will take a look at the Gremlin graph API.

Being a security guy, my first area of interest is SQL Injection – is it possible to build SQL statements that are subject to SQLi when using CosmosDB?

The answer is YES! (but read on)

In fact, querying CosmosDB using the SQL API is as vulnerable to SQLi as, say, SQL Server or Oracle or MySQL.

IMPORTANT This is not a vulnerability in ComsosDB; it’s an issue with any system that uses strings to build programming constucts. A similar example is Cross-site scripting (XSS) in web sites using
JavaScript (or Java or C# or PHP or Perl or Ruby or Python!) at the back-end. Remember, CosmosDB will give you back whatever you ask for (permissions aside) and this may be more than you bargained for if the query is built incorrectly from untrusted data.

Even if you don’t have CosmosDB setup in your Azure subscription (remember, you can get a free subscription) you can test this out.

  • Go to the CosmosDB query test site at https://www.documentdb.com/sql/demo
  • Run the default query and note you only get one document returned.
  • Now add ‘or 1=1’ at the end of the query (without the quotes) and re-run the query. You get 100 documents returned.

The fact that the ‘or 1=1’ clause changed the query and made it return all documents indicates a potential SQLi issue. If any arguments in the query came from an attacker, we’d be in a world of hurt. For those not aware, the ‘or 1=1’ clause at the end of query is true for every document in the database, so the query returns all documents.

Remedy If you’re programmatically querying CosmosDB, and handle untrusted input that is used to build queries, you must use parameterized queries. This is the same guidance that’s been around in other SQL APIs.

There’s a nice write-up about CosmosDB and parameterized queries here.
The methods you’ll use the most are SqlParameter and SqlParameterCollection.

TL;DR CosmosDB queries are subject to SQLi if you build queries incorrectly from untrusted data. Make sure your queries use parameterized queries, especially in the face of untrusted input.

The Dangers of String-Comparing Passwords

Or, a gentle introduction to timing attacks

One of my favorite things about software security is making people aware of what they didn’t KNOW they didn’t know. That’s not a typo, by the way. And this topic is one such example. Enjoy!

When I see a system that compares passwords, my first response is “Why are you handling passwords?” No modern system should handle passwords directly unless there’s an ancient legacy reason. Even then I’d consider ways to get away from handling the passwords.

With that said, I have seen more than one system over the years that DID need to compare secret data such as passwords.

And every single one of them got it wrong!

I think we have all seen code like the code below at some point.

static bool DoPasswordsMatch(string pwd1, string pwd2)
{
    if (pwd1.Length != pwd2.Length)
        return false;

    for (int i = 0; i < pwd1.Length; i++)
        if (pwd1[i] != pwd2[i])
            return false;

    return true;
}

The code checks if the passwords are the same length, and if they are not, then clearly the passwords are not the same so the function exits.

If the passwords are same length then the code does a byte-by-byte compare of the strings and if there is a mismatch at any point the function returns.

If all is well, and the passwords match the function returns true.

This code is broken because of timing attacks. In the example code, if an attacker provides one password and the password is not the same length as the real password, then the function exits quickly. In other words the attacker can now know the passwords are not the same length.

It’s this ‘short-circuiting’ that causes the problem.

Note this issue is not restricted to passwords, your code could compare AppID secrets or any other authentication secret.

Plenty has been written on this topic over the years:

https://blog.ircmaxell.com/2014/11/its-all-about-time.html 

https://wiki.php.net/rfc/timing_attack

https://stackoverflow.com/questions/25844354/timing-attack-in-php

https://security.stackexchange.com/questions/111040/should-i-worry-about-remote-timing-attacks-on-string-comparison

https://security.stackexchange.com/questions/97798/feasibility-of-time-based-database-brute-force-attacks-on-websites

https://stackoverflow.com/questions/37633688/constant-time-equals

So how do you fix this? For systems that MUST perform some form of comparison of secret strings, using code like the following will mitigate the risk substantially. It does so by removing all forms of short-circuit logic.

bool TimeConstantCompare(string hashName, string s1, string s2)
{
var h1 = HashAlgorithm.Create(hashName).ComputeHash(Encoding.UTF8.GetBytes(s1));
var h2 = HashAlgorithm.Create(hashName).ComputeHash(Encoding.UTF8.GetBytes(s2));
int accum = 0;

for (int i = 0; i < h1.Length; i++)
accum |= (h1[i] ^ h2[i]);

return accum == 0;
}

The way this code works is it first hashes the two strings; for example, hashName is SHA256 and this yields two series of bytes of equal length. Now the fun starts – the code walks over the two series of bytes XOR-ing each byte, if the bytes are the same the result is zero. If the bytes are not the same, then the result is non-zero. The result is then OR-ed with an accumulator, this means that if ANY bytes are not the same, the non-zero result is carried forward.

At the end of the function, if the accumulator is zero, then the bytes are the same which means the two strings match. All in constant time!

I posted some C# code up on GitHub.

tl;dr: Don’t compare passwords. If you MUST compare passwords don’t do a naive string compare. Use a time-constant compare instead.

2018-A Personal Security Retrospective

For so many people, myself included, 2018 was a frenetic year; I worked with customers in many fields, from national and local government to defense and from healthcare to finance and everything in between, so I thought I’d share some observations.

This list below is in no order and I somewhat apologize for the broad generalizations!

Everyone’s moving to the cloud

Every customer I worked with last year is either deployed on Azure or going through the planning process. This is great to see, but it does create some pain. Engineering staff must learn new skills (but that’s a good thing) and old assumptions and ideas need casting aside in favor of new technologies and deployment scenarios.

If you have no experience with cloud technology, then do yourself a professional favor and get a free Azure subscription and build something simple. If you have even an ounce of intellectual curiosity you will be wildly excited at what you see and the possibilities cloud-based solutions offer. You will also learn very quickly that there’s a lot you do not know and even more you don’t know you don’t know!

The “WE-CANNOT-DEPLOY-TO-THE-CLOUD-BECAUSE-IT-IS-NOT-SECURE” days are behind us. The emotion is gone, and people look to the cloud with a more level-headed approach.

As a side note and example case-in-point; I was onsite with a customer recently and on one of their sprint planning boards was a proposed story about “making sure we have the Kubernetes fix for CVE-2018-1002105 is deployed.” I asked the woman who wrote the story what was behind this story because we had fixed the problem within 24 hours in AKS (Azure Kubernetes Services) and all customers needed to do was roll their clusters. She told me she knew that Azure and AWS had both fixed the issue quickly, but that’s wasn’t the problem. The problem was they were grappling with their on-prem Kubernetes instances!

Q.E.D!

Crypto-agility

No-one is doing it. Ok, one customer I worked with is doing it. In fact, this one customer is actively pursuing crypto-agility and embracing it whole-heartedly. Said customer and I have regular meetings to go over their current ideas for crypto-agility to make sure they have no missed something.

Crypto-agility is the ability to change crypto (algorithms, key sizes, padding mode etc.) in an app without having to change the application code and still provide compatibility with data protected using older crypto. In other words, the crypto algorithm policy is separate from the code. Products like Microsoft SQL Server and Microsoft Office are great examples of different approaches to crypto-agility.

You can read more on the topic here.

Injection vulns are dead! Long live injection vulns!

I think I saw one genuine SQL injection vulnerability last year in customer code. I saw a few examples of string concatenation to build SQL statements, but the source data was from a trusted source. I still made the customer change the code! So that’s good, right? I mean, a few years ago I’d see dozens of SQLi bugs. The problem I saw was an uptick in other injection vulns, such as XML External Entity (XXE) issues and people treating JSON data like it’s trusted!

The adage that “incoming data is evil until proven otherwise” is as true today as it was in 2002 when David LeBlanc and I wrote Writing Secure Code (feeling old, yet?)

Folks are logging into Azure as Owner

You know that idiom of not logging on as an admin or as root? Same applies to logging into Azure. Once you have configured the subscription(s) you should not logon as an Owner, rather you should logon with lower privilege accounts or use RBAC roles to restrict who can do what to what. That’s a discussion for another day, but for the most part you really don’t need to use the Azure Portal as Owner once the system’s running, rather you use scripts; any and all Owner access is for “Break Glass” scenarios only.

Compliance is Huge

We all know that Compliance != Security. News outlets are replete with stories of compromised HIPAA and HiTRUST compliant health care institutions. Ditto PCI DSS and merchants. But there is an overlap between Security, Privacy and Compliance and compliance dollars can be used to help drive security programs. What I am trying to say is if you need to drive security programs consider dropping the word “compliance” in your financial requests and co-opt the compliance dollars. Just sayin’! It’s worked for me and I will leave it at that!

By the way, find some time to look at the new public preview for ISO27001, SOC and PCI compliance reporting in Azure Security Center.

Customers are finally building Threat Models

There are no words to explain how happy I am to say this, but every customer I worked with in 2018 understood the value of threat models and have built threat models. Most importantly, the threat models helped drive changes in their architectures and that’s fantastic!

Talking of compliance, there’s a relationship between threat models and compliance programs, but I have a blog post already in the works on that topic!

Devs and IT finally started talking to each other

A few years ago, I made a comment that “In a Cloud-First world, devs have to learn basic networking and, networking folks need to learn basic coding.” I stand behind that comment to this day. I saw this become a reality last year. Devs now write code and deployment scripts that configure CIDR blocks and VNets with help from the Networking team. And the Networking guys and gals are doing likewise. All of sudden a networking person who does not understand, say, JSON, Python and PowerShell will be irrelevant quickly! Same holds for devs, they need to know what it means to put their app in VNet and wrap a Network Security Group (NSG) around it and so much more.

Again, “In a Cloud-First world, devs have to learn basic networking and, networking folks need to learn basic coding.” Ignore this advice at your peril!

Azure Policy is fantastic and many people don’t know it’s there

I plan to cover this topic in much more detail in future, but Azure Policy is fantastic and let me explain by way of example.

A financial customer I work with is interested in using CosmosDB and one of the nice features of CosmosDB is how easy it is to replicate data to other data centers around the globe. You can define geography, consistency rules and other transactional semantics at the click of a button. Want data in Australia so it’s closer to your antipodean clients? Just click on the data center(s) in Australia and that’s it!

One of the architects at the customer said, “That’s cool, but problematic because of data sovereignty issues and let’s not get started with GDPR!” I told him to not worry because we can enforce the allowable data centers through Azure Policy. The policy is defined in JSON (the syntax does take a little getting used to, however) and you can apply the policy in the Policy blade in the Azure Portal if you wish, or push it using any of the various scripting interfaces.

I created a quick-n-dirty policy that only allowed the US South Central and US East data centers.

I waited a few minutes for the policy to deploy and then tried to replicate data to Germany in the Portal. No dice! I then tried to replicate to Germany from a Python script. Nope! Finally, I tried Az CLI. Negative. In all examples, the policy intervened and blocked the attempt to replicate data to Germany.

The really nice thing about policy is there are numerous outcomes of a policy violation, but the most common are “Audit” and “Deny” – I won’t insult your intelligence by explaining what they do.

Azure Policy can apply to just about anything in Azure:

  • Do you require only HSM-backed keys in Azure Key Vault? Policy can do that.
  • What about VMs never have a public endpoint? Got that too!
  • TLS required to blob storage? Got you covered!

I could keep going, but I think you get the picture.

You can learn more about Azure Policy here and there’s a GitHub repo of sample policies here.

Everyone has stickers on their laptops

Ok, this is not security, it’s just an observation. Says he with a bunch of Ninja Cat and Azure stickers on his Surface Pro!

Wrap Up

To wrap up, 2019 is shaping up to be another amazing and busy Azure year for my colleagues and I. I might even do a “Halfway through 2019 Retrospective” because there is so much going on in security and in Azure!