In this episode of the podcast (#216), sponsored by Digicert, we talk with Brian Trzupek, Digicert’s Vice President of Product, about the growing urgency of securing software supply chains, and how digital code signing can help prevent compromises like the recent hack of the firm SolarWinds.
We spend a lot of time talking about software supply chain security these days? But what does that mean. At the 10,000 foot level it means “don’t be the next Solar Winds” – don’t let a nation state actor infiltrate your build process and insert a backdoor that gets distributed to thousands of customers – including technology firms three letter government agencies.
OK. Sure. But speaking practically, what are we talking about when we talk about securing the software supply chain? Well, for one thing: we’re talking about securing the software code itself. We’re talking about taking steps to insure that what is written by our developers is actually what goes into a build and then gets distributed to users.
Digital code signing – using digital certificates to sign submitted code – is one way to do that. And use of code signing is on the rise. But is that alone enough? In this episode of the podcast, we’re joined by Brian Trzupek the SVP of Product at Digicert to talk about the growing role of digital code signing in preventing supply chain compromises and providing an audit trail for developed code.
Brian is the author of this recent Executive Insight on Security Ledger where he notes that code signing certificates are a highly effective way to ensure that software is not compromised -but only as effective as the strategy and best practices that support it. When poorly implemented, Brian notes, code signing loses its effectiveness in mitigating risk for software publishers and users.
In this conversation we talk about the changes to tooling, process and staff that DEVOPS organizations need to embrace to shore up the security of their software supply chain.
“It boils down to do you have something in place to ensure code quality, fix vulnerabilities and make sure that code isn’t incurring tech debt,” Brian says. Ensuring those things involves both process, new products and tools as well as the right mix of staff and talent to assess new code for security issues.
One idea that is gaining currency within DEVOPS organizations is “quorum based deployment” in which multiple staff members review and sign off on important code changes before they are deployed. Check out our full conversation using the player (above) or download the MP3 using the button below.
The recent SolarWinds attack highlights an Achilles heel for enterprises: software updates for critical enterprise applications. Digital signing of code is one solution, but organizations need to modernize their code signing processes to prioritize security and integrity and align with DevOps best practices, writes Brian Trzupek the Senior Vice President of Products at DigiCert in this thought leadership article.
Even in today’s security-charged world, the SolarWinds breach was a wakeup call for cybersecurity professionals. It was distinguished by its sophistication and the fact that it was carried out as part of legitimate software updates. The incident was quickly all over the news and has brought renewed focus on need for secure DevOps.
Automating your way out of PKI chaos.”
Code Signing Alone Is Not Enough
The security incident at SolarWinds was especially unsettling because could easily happen at any large software organization that delivers regular updates. Code signing certificates are a highly effective way to ensure that software is not compromised. However, it is only as effective as the strategy and best practices behind it. When poorly implemented, code signing loses its effectiveness in mitigating risk for software publishers and users. Issues often include:
Using the same signing key to sign all files, across multiple product lines and businesses
Lack of mechanisms in place to control who can sign specific files
Insufficient reporting capabilities for insights into who signed what and when
Failure to sign code at every stage of development, as part of an overall security by design process
Lack of signing and verifying code from third parties
Poor processes for securing keys and updating them to new key size or algorithm requirements
Failure to test code integrity before signing
Inadequate visibility into where certificates are, and how they are managed
Common pitfalls might include using the same signing key to sign all files, across multiple product lines and businesses. Some organizations might have no mechanisms in place to control who can sign specific files. They may also lack reporting capabilities, which can provide insights into who signed what—and when.
What have we learned from the SolarWinds attack? For organizations where DevOps is fundamental, applying best practices to signing processes is more essential than ever. According to some studies, more than half of IT security professionals are concerned about bad actors forging or stealing certificates to sign code—but fewer than a third enforce code signing policies on a consistent basis. It’s time for organizations to do better and enforce zero-trust across all their systems, signing everything, at every stage after verifying it is secure.
Simplifying and standardizing
Traditional code signing processes can be complex and difficult to enforce. They are often based on storing keys on desktops as well as sharing them. Visibility into activities is often limited, making mismanagement or flawed processes difficult to discover and track. To mitigate these issues, many organizations are simplifying their processes using code-signing- as-a-service approaches. Code-signing-as-a-service can accelerate the steps required to get code signed, while making it easier to keep code secure. A robust solution can empower organizations with automation, enabling teams to minimize manual steps and accelerate signing processes. APIs can enable it to integrate seamlessly with development workflows and automated scheduling capabilities enable organizations to proactively and approve signature windows to support new releases and updates.
To strengthen accountability throughout the process, administrators can apply permission- based access. Strictly controlling access helps improve visibility into which users are allowed to sign code and which certificates and private keys they are allowed to utilize.
Standardizing workflows
Standardizing code signing workflows can also help reduce risk to an organization. Instead of allowing everyone in an organization to use the same key for signing, many organizations are using separate code signing keys for different DevOps teams, while granting administrators visibility over key usage. This best practice helps minimize the risk of mistakes that can occur across a company by limiting the ability of breaches to propagate. For example, if a key is used to sign a release that has been compromised, only one team’s code will be impacted.
Maximum flexibility to minimize risk
Key management flexibility is another helpful best practice, reducing risks by enabling administrators to specify shorter certificate lifetimes, rotate keys and control keypairs. For example, Microsoft recognizes publishers that rotate keys with higher reputation levels. With the right key management approach, an administrator could specify a specific number of days or months exclusively for files designed for use in Microsoft operating systems.
Secure key storage offline except during signing events
Taking keys offline is another measure that can secure code signing. With the right code signing administrative tool, administrators can place keys in a “offline mode,” making it impossible to use them to sign releases without the proper level of permission in advance. Release planning is a fundamental to software development, so most developers are comfortable scheduling signatures for specific keys directly into their processes.
Taking keys offline is a strong step to ensure that keys will not be used in situations where they should not be. It also adds a layer of organizational security by splitting responsibilities between signers and those who approve them—while providing improved visibility into which keys are signed by whom.
Freeing up developers to do what they do best
It’s clear that safeguarding DevOps environments correctly remains challenging, but fortunately the right management tools can minimize hassles—and maximize protection. As we’ve discussed, automation is essential for applying security across CI/CD pipelines. Seek out a solution that can fit smoothly within workflows and free up engineers from individual steps required for cryptographic asset protection. The tool should make signing keys easily accessible when pushing code and automate signing of packages, binaries and containers on every merge to master when authorized. Organizations also need a process for testing code integrity before they sign. A centralized, effective signing management tool can handle the signing tasks, while integrating with other systems that perform necessary integrity tests. For key security, the solution should provide the option of storing the keys offline in virtual HSMs. During a signing event, it should enable developers to access the keys to sign with one click, then return them back to secure offline storage
DevOps pros work within a variety of environments, so the signing solution should support portable, flexible deployment models via SaaS or on a public or private data center. Businesses in every industry are becoming increasingly software-driven and the challenges to DevOps organizations won’t disappear anytime soon. However, with the right approach to code signing, organizations can dramatically strengthen their security posture, minimize their chances of becoming the next victim and ensure customer confidence in their solutions.
It already seems like a lifetime ago that the hack of the Orion network management software by SolarWinds consumed the attention of the media, lawmakers and the for-profit information security industry. In reality, it was barely six months ago that the intrusion first came to light.
In the space of a few months, however, there have already been successive, disruptive events of at least the same seriousness. Among them: the attacks on vulnerabilities in Microsoft’s Exchange Server and – in the last week – the ransomware attack that brought down the Colonial Pipeline in the Eastern United States.
Lessons Still Being Learned
The “party” may have moved on from SolarWinds, the impact and import of the SolarWinds Orion hack are still being worked out. What were the “lessons” from SolarWinds? Beyond the 200 + organizations that were impacted by the supply chain compromise, how can private and public sector firms learn from it and become more resilient and less susceptible to the kind of attacks that hit the likes of FireEye, Qualys, Equifax and more?
To help answer those questions, the security firm ForAllSecure assembled* an all-star roundtable to tackle the question not so much of “what happened” with SolarWinds, but “how to prevent it from happening again.” I had the pleasure of moderating that discussion and I’m excited to share the video of our talk below.
I’ve included a link to our discussion below. I urge you to check it out – there are a lot of valuable insights and take aways from our panelists. In the meantime, here are a couple of my high-level take aways:
Supply Chain’s Fuzzy Borders
One of the biggest questions we wrestled with as a panel was how to define “supply chain risk” and “supply chain hacks.” It’s a harder question than you might think. Our panelists were in agreement that there was a lot of confusion in the popular and technology media about what is- and isn’t supply chain risk. Our panelists weren’t even in agreement on whether SolarWinds, itself, constituted a supply chain hack, given that the company wasn’t a supply chain partner to the victims of the attack, so much as a service provider. Regardless, there was broad agreement that the risk is real and pronounced.
“We rely on third party software and services a lot more – APIs and everything,” said Chenxi Wang. “With that, the risk from software supply chain is really more pronounced.”
“What your definition of supply chain risk depends on where you sit,” said Vincent Liu. That starts with third party or open source libraries. “If you’re a software developer your supply chain looks very different from an end user who does no software development but needs to ensure that the software you bring into an organization is secure.”
At the end of the day: getting the exact definition of “supply chain risk” is less important than understanding what the term means for your organization and – of course – making sure you’re accounting for that risk in your security planning and investment.
Third Party Code: Buggy, or Bugged?
Another big question our panel considered was who was behind supply chain attacks and who the targets were. Vincent Liu of Bishop Fox. Supply chain insecurities are deliberately introduced into software development. A different approach needs to be taken when securing against that versus traditional secure software development tools, which are about identifying flaws in the (legitimate) software development process.
The impact of a flawed open source component and a compromised one matters, even if the impact is the same. The distinction between inadvertent vulnerabilities and attacks matters, said Brumley. “You really have to tease out the question of a vulnerability and whether inserted by the attacker or just latent there,” he said. The media is more interested in the concept of a malicious actor working behind the scenes to undermine software, whereas shoddy software may be less dramatic. But both need to be addressed, he said.
Developers in the Crosshairs
Another big takeaway concerns the need for more attention within organizations to an overlooked target in supply chain attacks: the developer. “It used to be that attacks were focused on company repositories and company source code. Now they’re very focused on developer machines instead,” said Moore of Rumble. “There are a lot more phishing attacks on developers, a lot of folks scraping individual developer accounts on GitHub looking for secrets. So it’s really shifted from attacks on large corporations with centralized resources to individual developer resources.”
The days of sensitive information like credentials and API keys hiding in plain site are over. “Especially for developers, your mean time to compromise now is so fast,” said Moore.
Unfortunately, too few organizations are investing to solve that very real problem. Too many firms are continuing to direct the lion’s share of their security investment into legacy tools, technologies and capabilities. ““Tons of tooling and money has been spent in terms of things that find vulnerabilities because they can find them,” notes Liu. “But most of them don’t matter. Really spending your time on the right things rather than just doing things because you can do them is so critically important.”
Security researchers analyzing a widely used open source component have discovered security vulnerabilities that leave hundreds of thousands of software applications vulnerable to attack, according to a report released Monday.
The group of five researchers found the security vulnerabilities in netmask, an open source library used in a staggering 270,000 software projects. According to the report, the flaws open the door to a wide range of malicious attacks that could enable attackers to ferry malicious code into a protected network, or siphon sensitive data out of one.
Among the attacks enabled by the flaw are so-called server-side request forgeries (SSRF), as well as remote file inclusion, local file inclusion and more, the researcher called Sick Codes told The Security Ledger. Work to discover the extent of the flaws continues. The researchers have received a preliminary vulnerability ID for the flaw, CVE-2021-28918.
Even worse, the flaws appear the stretch far beyond a single open source module, affecting a wide range of open source development languages, researchers say.
Work on one vulnerable open source module uncovers another
In addition to Sick Codes, the researchers conducting the investigation are Victor Viale (@koroeskohr), Kelly Kaoudis (@kaoudis), John Jackson (@johnjhacking) and Nick Sahler (@tensor_bodega).
According to Sick Codes, the vulnerability was discovered while doing work to fix another vulnerability in a widely used NPM library known as Private IP. That module, which was also widely used by open source developers, enables applications to block request forgery attacks by filtering out attempts to access private IP4 addresses and other restricted IP4 address ranges, as defined by ARIN. In a report published in November, researchers revealed that the Private IP module didn’t work very well and was susceptible to being bypassed using SSRF attacks against top tier applications.
SSRF attacks allow malicious actors to abuse functionality on a server: reading data from internal resources or modifying the code running on the server. Private-ip was created to help application developers spot and block such attacks. SSRF is one of the most common forms of attack on web applications according to OWASP.
The researchers working to fix the Private IP flaw turned to netmask, a widely used package that allows developers to evaluate whether a IP address attempting to access an application was inside or outside of a given IPv4 range. Based on an IP address submitted to netmask, the module will return true or false about whether or not the submitted IP address is in the defined “block.”
The module has a range of other useful features, as well, such as reporting back on how many IPs are inside a given block. And, with no other “dependencies,” netmask seemed like the perfect fit to fix Private IP’s problems.
There’s only one problem: netmask itself was flawed, as the researchers soon discovered. Specifically: the module evaluates certain IP addresses incorrectly: improperly validating so-called “octal strings” rendering IPv4 addresses that contain certain octal strings as integers.
For example, the IP4 address 0177.0.0.1 should be evaluated by netmask as the private IP address 127.0.0.1, as the octal string “0177” translates to the integer “127.” However, netmask evaluates it as a public IPv4 address: 177.0.0.1, simply stripping off the leading zero and reading the remaining parts of the octal string as an integer.
Sample output from the netmask NPM module showing incorrect parsing of the IPv4 address. The flaw could enable a range of attacks on applications using the popular open source module. (Image courtesy of Sick Codes.)
And the flaw works both ways. The IP4 address 0127.0.0.01 should be evaluated as the public IP address 87.0.0.1 as the octal string “0127” is the same as the integer “87.” However, netmask reads the address as 127.0.0.1, a trusted, localhost address. Treating an untrusted public IP address as a trusted private IP address opens the door to local- and remote file inclusion (LFI/RFI) attacks, in which a remote authenticated or unauthenticated attacker can bypass packages that rely on netmask to filter IP address blocks.
A Popular Module with a Sterling Pedigree
Netmask is the creation of Olivier Poitrey, an accomplished developer who is the Director of Engineering at Netflix and a co-founder of the firm NextDNS. The module was first released nine years ago and gained prominence around 5 years ago, with version 1.06, which has been downloaded more than 3 million times.
The netmask module currently sees around 3.1 million downloads weekly, though development on netmask appears to have ceased during the last five years, up until the release of version 2.0.0 around 10 days ago, after discovery of the security holes.
Throwing open the doors to hackers
The implications for modules that are using the vulnerable version of netmask are serious. According to Sick Codes, remote attackers can use SSRF attacks to upload malicious files from the public Internet without setting off alarms, because applications relying on netmask would treat a properly configured external IP address as an internal address.
Similarly, attackers could also disguise remote IP addresses local addresses, enabling remote file inclusion (RFI) attacks that could permit web shells or malicious programs to be placed on target networks.
Parsing IPv4 Addresses: Black Magic
Updates to address the flaws in netmask began appearing on March 19 with the release of version 2.0.0. A subsequent update, 2.0.1, was released on Monday. To date, only 6,641 downloads of the updated netmask module have completed, meaning the vast majority of open source projects using it remain vulnerable. Contacted via chat, Poitrey advised netmask users to upgrade to the latest version of the module.
But researchers say much more is to come. The problems identified in netmask are not unique to that module. Researchers have noted previously that textual representation of IPv4 addresses weren’t ever standardized, leading to disparities in how different but equivalent versions of IPv4 addresses (for example: octal strings) are rendered and interpreted by different applications and platforms.
In the case of the netmask flaw, the true culprit ends up being “some unexpected behaviors of the parseInt function in Javascript combined with an oversight on those odd IP notations nobody (uses),” Poitrey told Security Ledger. “Parsing IPs correctly is black magic,” he said. “Those weird notations should just be deprecated, so nobody can exploit them like this.”
More to come
The impact of the flaw will become more clear in the weeks ahead, according to Sick Codes, as more examples of the IPv4 parsing problem are identified and patched in other popular open interpreted languages. “The implications are infinite,” he said.
Independent security researchers testing the security of the United Nations were able to compromise public-facing servers and a cloud-based development account for the U.N. and lift data on more than 100,000 staff and employees, according to a report released Monday.
Researchers affiliated with Sakura Samurai, a newly formed collective of independent security experts, exploited an exposed Github repository belonging to the International Labour Organization and the U.N.’s Environment Programme (UNEP) to obtain “multiple sets of database and application credentials” for UNEP applications, according to a blog post by one of the Sakura Samurai researchers, John Jackson, explaining the group’s work.
Specifically, the group was able to obtain access to database backups for private UNEP projects that exposed a wealth of information on staff and operations. That includes a document with more than 1,000 U.N. employee names, emails; more than 100,000 employee travel records including destination, length of stay and employee ID numbers; more than 1,000 U.N. employee records and so on.
The researchers stopped their search once they were able to obtain personally identifying information. However, they speculated that more data was likely accessible.
Looking for Vulnerabilities
The researchers were scanning the U.N.’s network as part of the organization’s Vulnerability Disclosure Program. That program, started in 2016, has resulted in a number of vulnerabilities being reported to the U.N., many of them common cross-site scripting (XSS) and SQL injection flaws in the U.N.’s main website, un.org.
For their work, Sakura Samurai took a different approach, according to Jackson, in an interview with The Security Ledger. The group started by enumerating UN subdomains and scanning them for exposed assets and data. One of those, an ILO.org Apache web server, was misconfigured and exposing files linked to a Github account. By downloading that file, the researchers were able to recover the credentials for a UN survey management panel, part of a little used, but public facing survey feature on the UN site. While the survey tool didn’t expose a tremendous amount of data, the researchers continued scanning the site and eventually discovered a subdomain that exposed a file containing the credentials for a UN Github account containing 10 more private GitHub repositories encompassing databases and database credentials, backups and files containing personally identifying information.
Much more to be found
Jackson said that the breach is extensive, but that much more was likely exposed prior to his group’s discovery.
“Honestly, there’s way more to be found. We were looking for big fish to fry.” Among other things, a Sakura Samurai researcher discovered APIs for the Twilio cloud platform exposed – those also could have been abused to extract data and personally identifying information from UN systems, he said.
In an email response to The Security Ledger, Farhan Haq, a Deputy Spokesman for the U.N. Secretary-General said that the U.N.’s “technical staff in Nairobi … acknowledged the threat and … took ‘immediate steps’ to remedy the problem.”
“The flaw was remedied in less than a week, but whether or not someone accessed the database remains to be seen,” Haq said in the statement.
A disclosure notice from the U.N. on the matter is “still in the works,” Haq said. According to Jackson, data on EU residents was among the data exposed in the incident. Under the terms of the European Union’s Genderal Data Privacy Rule (GDPR), the U.N. has 72 hours to notify regulators about the incident.
Nation State Exposure?
Unfortunately, Jackson said that there is no way of knowing whether his group was the first to discover the exposed data. It is very possible, he said, that they were not.
“It’s likely that nation state threat actors already have this,” he said, noting that data like travel records could pose physical risks, while U.N. employee email and ID numbers could be useful in tracking and impersonating employees online and offline.
Another danger is that malicious actors with access to the source code of U.N. applications could plant back doors or otherwise manipulate the functioning of those applications to suit their needs. The recent compromise of software updates from the firm Solar Winds has been traced to attacks on hundreds of government agencies and private sector firms. That incident has been tied to hacking groups associated with the government of Russia.
Asked whether the U.N. had conducted an audit of the affected applications, Haq, the spokesperson for the U.N. Secretary General said that the agency was “still looking into the matter.”
A Spotty Record on Cybersecurity
This is not the first cybersecurity lapse at the U.N. In January, 2020 the website the New Humanitarian reported that the U.N. discovered but did not disclose a major hack into its IT systems in Europe in 2019 that involved the compromise of UN domains and the theft of administrator credentials.
This podcast is the latest in a series of interviews we’re doing on “left-shifted security” that explores how information security is transforming to embrace agile development methodologies and DEVOPS. If you like this, check out some of the other podcasts in this series!
Information security is “shifting left”: moving closer to the development process and becoming part and parcel of agile “DEVOPS” organizations. But while building security into development may be a familiar idea, what does it mean to build compliance into development?
Galen Emery is the Lead Compliance & Security Architect at Chef Software.
To find out, we invited Galen Emery the Lead Compliance & Security Architect at Chef Software, in to the Security Ledger studios to talk about the job of blending both security and compliance into agile development processes. We also talk about Chef’s increasing investments in security testing and compliance and how the “shift left” is impacting other security investments including access control, auditing and more.
To start out, I asked Galen to tell us a bit about Chef and how the company’s technology has evolved from configuration management to security testing and compliance as well as areas like endpoint protection.
The pandemic isn’t the only thing shaking up development organizations. Application security is a top concern and security work is “shifting left” and becoming more intertwined with development. In this podcast, Security Ledger Editor in Chief Paul Roberts talks about it with Jonathan Hunt, Vice President of Security at the firm GitLab.
Even before the COVID pandemic set upon us, the information security industry was being transformed. Security was long a matter of hardening organizations to threats and attacks. The goal was “layered defenses” starting with firewalls and gateway security servers and access control lists to provide hardened network perimeter and intrusion detection and endpoint protection software to protect IT assets within the perimeter.
Jonathan Hunt is the Vice President of Security at GitLab
These days, however, security is “shifting left” – becoming part and parcel of the development process. “DEVSECOPS” marries security processes like code analysis and vulnerability scanning to agile application development in a way that results in more secure products.
That shift is giving rise to a whole new type of security firm, including the likes of GitLab, a web-based DevOps lifecycle tool and Git-repository manager that is steadily building its roster of security capabilities. What does it mean to be a security provider in the age of DEVSECOPS and left-shifted security?
Application Development and COVID
To answer these questions, we invited Jonathan Hunt, the Vice President of Security at GitLab into the Security Ledger studio to talk about it. In this conversation, Jonathan and I talk about what it means to shift security left and marry security processes like vulnerability scanning and fuzzing with development in a seamless way.
We also discuss how the COVID pandemic has shaken up development organizations – including GitLab itself – and how the changes wrought by COVID may remain long after the virus itself has been beaten back.
 100vw, 300px”><figcaption>Brian Trzupek is Senior Vice President of Products at DigiCert</figcaption></figure>
</div>
<p>The SolarWinds attack took advantage of what is usually a best practice: regular software updates. It was based on malware inserted into the development process for network monitoring software, which created a back door. When the software was updated, the software, including the back door, was distributed into customer environments, compromising a broad array of prominent U.S. government systems. Once in place, it enabled the hackers to quietly spy on them.</p>
<p><!-- Either there are no banners, they are disabled or none qualified for this location! --> </p>
<p>Also read: “<a href=)
Automating your way out of PKI chaos.”
