> For the complete documentation index, see [llms.txt](https://fg0x0.gitbook.io/fg0x0s-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fg0x0.gitbook.io/fg0x0s-notes/bug-bounty/how-i-found-a-credentials-leak-that-aws-patched-4-times-but-still-missed.md). # How I Found a Credentials Leak That AWS Patched 4 Times But Still Missed ### The Idea I've been doing bug bounty for a while now, and one thing I've learned the hard way: **when a vendor patches the same bug class in multiple places, they almost always miss one.** That's incomplete-fix hunting, and it's probably my highest-ROI strategy. So when I saw AWS ship four separate permission fixes in early 2026 - all for the same root cause - I thought: "Did they get every instance?" Spoiler: they didn't. ### Background - What AWS Already Fixed In February 2026, AWS published [GHSA-747p-wmpv-9c78](https://github.com/advisories/GHSA-747p-wmpv-9c78) (Medium severity) for their `cli_history` database file being created with insecure permissions. Then in April, three more PRs landed fixing the exact same pattern in different files: * **PR #10191** - `codeartifact/login.py` (netrc + pypirc files) * **PR #10194** - `iamvirtmfa.py` (virtual MFA seed) * **PR #10206** - CodeDeploy on-host agent config The root cause in all four cases was identical: `os.open(..., O_CREAT, 0o600)` only sets permissions on **newly created** files. If the file already exists, it keeps whatever permissions it had before. The fix was always the same - add `os.chmod` or `os.fchmod` after opening. Four fixes. Same bug. Same pattern. That's a strong signal that there might be more. ### Quick Primer - Unix File Permissions & umask Before diving in, let's make sure we're on the same page about how Unix file permissions work. If you already know this, skip ahead. Every file on a Unix system has three permission groups: **owner**, **group**, and **others**. Each group can have **read (4)**, **write (2)**, and **execute (1)** permissions. These are represented as an octal number: ``` -rw-r--r-- = 0o644 ||| ||| ||| ||| ||| ||+-- others: read (4) ||| ||| |+--- others: no write ||| ||| +---- others: no execute ||| ||+------- group: read (4) ||| |+-------- group: no write ||| +--------- group: no execute ||+------------ owner: read (4) |+------------- owner: write (2) +-------------- owner: no execute ``` * `0o600` = owner can read/write, nobody else can see it - **this is what you want for secrets** * `0o644` = owner can read/write, everyone else can read - **this is the problem** **What's umask?** When you create a file, the OS applies a mask called `umask` to determine the default permissions. Most Linux systems ship with `umask 022`, which means: ```bash # Default permission for new files: 0o666 (no execute) # Minus umask: - 0o022 # Result: = 0o644 (world-readable) touch secret.txt stat -c '%a' secret.txt # 644 ``` So any file created with `touch`, `cp`, `echo >`, or `open()` in Python without explicit permissions will be `0o644` by default. That's fine for most files, but terrible for credentials. The secure way to create a file with restricted permissions: ```python import os # This ONLY works for NEW files: fd = os.open('secret.txt', os.O_WRONLY | os.O_CREAT, 0o600) # For EXISTING files, you need an explicit chmod: os.chmod('secret.txt', 0o600) ``` This distinction - `O_CREAT` mode only applies at creation time - is the entire root cause of this bug class. ### How I Hunted It - The Methodology Here's my actual hunting process. If you want to find similar bugs in other projects, this is the playbook. #### Step 1 - Read the existing patches I started by reading all four PRs carefully. Not just the diff, but the commit message, the discussion, and the test cases. The pattern was clear: ``` Before: open(sensitive_file, 'w') # inherits existing permissions After: open(sensitive_file, 'w') os.chmod(sensitive_file, 0o600) # force restrictive permissions ``` #### Step 2 - Identify the target I asked myself: "What's the most sensitive file the AWS CLI writes to?" The answer is obvious - `~/.aws/credentials`. If the four patched files were worth fixing, this one definitely is. #### Step 3 - Grep for the pattern ```bash # Find every open() call that writes to a file in the aws-cli codebase grep -rn "open(.*'[wa]')" awscli/ --include="*.py" # Find file write operations that DON'T have a chmod nearby grep -rn "open(.*'[wa]')" awscli/ --include="*.py" -l | while read f; do if ! grep -q "chmod" "$f"; then echo "[NO CHMOD] $f" fi done ``` `writer.py` showed up immediately - it has `open()` calls for both write and append modes, with zero `chmod` anywhere in the file. #### Step 4 - Trace the code path I confirmed that `ConfigFileWriter.update_config` is the function behind `aws configure` and `aws configure set`: ``` aws configure set aws_access_key_id AKIA... → awscli/customizations/configure/set.py → ConfigFileWriter().update_config(...) → writer.py line 110: open(config_filename, 'w') ← no chmod ``` #### Step 5 - Write the PoC, run it, confirm Always validate before reporting. I wrote a Python PoC that runs directly against the `ConfigFileWriter` class and confirmed the behavior on both v1 and v2. **This grep-and-trace process took about 30 minutes.** The incomplete-fix pattern gave me a precise signature to search for - that's why this strategy is so efficient. ### The Vulnerable Code The writer lives in `awscli/customizations/configure/writer.py`. Here's the core function: ```python def update_config(self, new_values, config_filename): ... if not os.path.isfile(config_filename): self._create_file(config_filename) # creates new file at 0o600 - safe ✔ self._write_new_section(...) return # File already exists - just open and write with open(config_filename, 'r') as f: contents = f.readlines() try: self._update_section_contents(contents, section_name, new_values) with open(config_filename, 'w') as f: # no chmod here ✘ f.write(''.join(contents)) except SectionNotFoundError: self._write_new_section(...) # also no chmod ✘ ``` The new-file path is fine - `_create_file` uses `os.open(..., O_CREAT, 0o600)`. But the existing-file path just calls `open()`, which inherits whatever permissions the file already has. If that's `0o644`, your secrets are world-readable. ### But Who Creates the File at 0o644? This was the first question HackerOne triage asked me, and it's a fair one. `aws configure` itself creates the file at `0o600`. So how does a `0o644` file get there in the first place? Turns out, it happens all the time: **Docker/Container images:** ```dockerfile COPY aws-credentials /home/app/.aws/credentials # Docker COPY = 0o644 by default ``` **Setup scripts and provisioning:** ```bash mkdir -p ~/.aws && touch ~/.aws/credentials # touch + umask 022 = 0o644 ``` **Configuration management (Ansible, Chef, Puppet):** ```yaml - name: Create AWS credentials copy: content: "[default]\naws_access_key_id = placeholder\n" dest: ~/.aws/credentials # Creates at 0o644 unless you explicitly set mode ``` **Manual creation:** ```bash cp /etc/skel/.aws/credentials ~/.aws/credentials # Inherits source perms or applies umask → 0o644 ``` And here's the thing - boto3, the AWS SDK for Python, **doesn't create** `~/.aws/credentials` at all. It only reads it. So developers who start with boto3 usually create the file manually before running `aws configure set` to populate it. ### Handling Triage - How I Responded to "Needs More Info" This part is worth talking about, because how you handle triage questions can make or break a report. Within 10 hours of submitting, I got a "Needs more info" status with two questions from the triager: > 1. Can you provide a POC showing that other AWS tools (like boto3 or aws-sdk-java) create the initial credential file with full read access during a normal usage workflow? > 2. The logic difference may be intentional to respect an existing file's permissions which a user has deliberately changed for a specific reason. Thoughts? Both are valid questions. The wrong move here is to get defensive or dismissive. Instead, I addressed each one directly: **For question 1** - I listed six concrete scenarios where `0o644` files appear in practice (Docker COPY, touch, Ansible, cp, boto3 not creating the file at all, SDK credential providers). Each with actual code examples showing the exact commands. **For question 2** - I pointed to AWS's own behavior as the counter-argument. If "respecting existing permissions" was the intended design, why did they explicitly override permissions in all four previous fixes? I also cited the principle of least privilege: the secure default should be restrictive, and users who deliberately want wider permissions can `chmod` back after. The report went from "Needs more info" back to "New" in **25 minutes**. Then to "Pending program review" the next morning. **The lesson:** anticipate the objections, prepare your evidence, and let the vendor's own actions speak for you. Don't argue - demonstrate. ### Proof of Concept The full PoC is straightforward: ```bash # Step 1 - Simulate a pre-existing credentials file (common in real deployments) mkdir -p ~/.aws touch ~/.aws/credentials ls -la ~/.aws/credentials # -rw-r--r-- 1 alice alice 0 ... /home/alice/.aws/credentials # That's 0o644 - world-readable # Step 2 - Write real credentials using the CLI aws configure set aws_access_key_id AKIAIOSFODNN7EXAMPLE aws configure set aws_secret_access_key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY # Step 3 - Check permissions stat -c '%a %n' ~/.aws/credentials # 644 /home/alice/.aws/credentials # Still world-readable. Secrets exposed. # Step 4 - As any other local user on the same machine cat /home/alice/.aws/credentials # [default] # aws_access_key_id = AKIAIOSFODNN7EXAMPLE # aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY # Full access. Game over. ``` I also wrote a Python PoC that runs directly against the `ConfigFileWriter` class to confirm the behavior programmatically: ```python import os, sys, tempfile sys.path.insert(0, '') from awscli.customizations.configure.writer import ConfigFileWriter tmp = tempfile.mkdtemp(prefix='cfg-poc-') p = os.path.join(tmp, 'credentials') # Simulate pre-existing file at 0o644 with open(p, 'w') as f: f.write('[default]\naws_access_key_id = OLD\naws_secret_access_key = OLD\n') os.chmod(p, 0o644) print('BEFORE:', oct(os.stat(p).st_mode & 0o777)) # Run the same code path as `aws configure set` ConfigFileWriter().update_config( {'aws_access_key_id': 'AKIANEW', 'aws_secret_access_key': 'NEWSECRET'}, p) print('AFTER :', oct(os.stat(p).st_mode & 0o777)) # Result: # BEFORE: 0o644 # AFTER : 0o644 # [VULNERABLE] credentials file remains world-readable after update ``` ### The SSH Comparison - How It Should Work Compare this to how OpenSSH handles the same problem. `ssh-keygen` always creates private keys at `0o600`, and - here's the important part - **`ssh` refuses to use a private key if the permissions are too open**: ```bash $ chmod 644 ~/.ssh/id_rsa $ ssh user@host @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0644 for '/home/user/.ssh/id_rsa' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. ``` SSH has two layers of defense: 1. Create files with correct permissions 2. **Refuse to use files with bad permissions** AWS CLI (pre-fix) had neither for existing files. It silently wrote secrets to a world-readable file and carried on. AWS credentials are equivalent in sensitivity to SSH private keys - they both grant full access to infrastructure. ### The Impact On any multi-user Unix environment - shared workstations, CI/CD runners, bastion hosts, containers with multiple service users - any local user can passively read the victim's long-term AWS credentials. No exploitation needed. Just `cat` the file. Long-term AWS keys = full IAM-principal access to the victim's AWS account(s). That's S3 buckets, EC2 instances, Lambda functions, databases - everything that principal can touch. The irony? AWS already rated the **less sensitive** version of this bug (command history leaking via `cli_history`) as Medium severity. The credentials file contains the keys to the kingdom, and it was the one they missed. ### The Severity Discussion HackerOne triage rated this **Low**. I originally submitted it as Medium (6.5). Here's the reasoning on both sides: **Why triage downgraded to Low:** * The pre-condition (file already exists at `0o644`) is created by external tooling, not by the AWS CLI itself * Exploitation requires local access to the same machine - no remote attack vector * The attacker needs a valid local user account on the same system **Why I submitted as Medium:** * AWS rated the **less sensitive** sibling bug (GHSA-747p-wmpv-9c78, `cli_history`) as Medium, and command history is far less sensitive than access keys * The pre-condition (`0o644` file) is extremely common in real-world deployments (Docker, Ansible, manual setup) * The impact is severe - full IAM-principal access, not just information disclosure * The attack is completely passive (read a file) with no detection **My honest take:** I understand the Low rating given the "external tool creates the file" pre-condition. But I think there's an inconsistency - if `cli_history` at `0o644` is Medium, then credentials at `0o644` should be at least Medium too. The pre-condition is the same; the impact is worse. Either both are Low, or both are Medium. That said, severity debates are part of the game. The bug is real, the fix shipped, and the finding is on my profile. That matters more than the severity label. ### The Fix AWS deployed a fix in **v1.45.14** (v1) and **v2.34.53** (v2). The CLI now warns users when credential-writing commands target a file with overly permissive permissions. > **Note:** The code example below is an illustrative representation of the fix pattern. Refer to the actual commit in [aws/aws-cli](https://github.com/aws/aws-cli) for the deployed implementation. **Before (vulnerable):** ```python def update_config(self, new_values, config_filename): if not os.path.isfile(config_filename): self._create_file(config_filename) self._write_new_section(...) return with open(config_filename, 'r') as f: contents = f.readlines() self._update_section_contents(contents, section_name, new_values) with open(config_filename, 'w') as f: f.write(''.join(contents)) # permissions unchanged ✘ ``` **After (fixed - illustrative):** ```python def update_config(self, new_values, config_filename): if not os.path.isfile(config_filename): self._create_file(config_filename) self._write_new_section(...) return # Warn about overly permissive permissions current_perms = os.stat(config_filename).st_mode & 0o777 if current_perms & 0o044: warnings.warn( f'{config_filename} has permissions {oct(current_perms)} ' f'which allow other users to read it.' ) with open(config_filename, 'r') as f: contents = f.readlines() self._update_section_contents(contents, section_name, new_values) with open(config_filename, 'w') as f: f.write(''.join(contents)) ``` If you're writing similar code in your own projects, the most secure approach is to force permissions rather than just warn: ```python # The gold standard - force 0o600 on every write with open(config_filename, 'w') as f: f.write(''.join(contents)) os.chmod(config_filename, 0o600) # always tighten ``` ### The Response & Timeline AWS's handling was professional throughout. Here's how it went: | Date | Event | | -------- | -------------------------------------------------------------------- | | April 15 | Report submitted (Medium 6.5) | | April 15 | Triage asks for more info (pre-existing file scenarios) | | April 15 | Responded with 6 real-world scenarios - status back to New in 25 min | | April 16 | Severity adjusted to Low, sent to remediation team | | April 17 | AWS VDP team confirms review in progress | | May 13 | AWS confirms fix in development | | June 12 | Fix deployed in v1.45.14 and v2.34.53, report resolved | AWS's CNA team decided this doesn't meet CVE threshold. Their reasoning: the insecure permissions originate from external tooling, not the CLI itself - unlike the prior GHSA where the CLI was creating the files with bad permissions. It's a fair distinction. The CLI never created the insecure state; it just failed to fix it when writing secrets. They did offer public disclosure, a testimonial on my HackerOne profile, and a $40 AWS Gear Shop gift card. They also invited me to send a blog post draft for technical review before publishing - a nice collaborative touch. ### Am I Affected? - How to Check and Fix #### Check your systems right now ```bash # Check permissions on your credentials file stat -c '%a %n' ~/.aws/credentials 2>/dev/null # If it shows 644, 664, or anything other than 600 - you're exposed # Also check the config file while you're at it stat -c '%a %n' ~/.aws/config 2>/dev/null ``` #### Quick scan across multiple users (sysadmins) ```bash # Find all world-readable AWS credentials files on the system find /home -name "credentials" -path "*/.aws/*" -perm /o=r 2>/dev/null # Same for CI/CD service accounts find /var/lib /srv /opt -name "credentials" -path "*/.aws/*" -perm /o=r 2>/dev/null # Check containers docker exec stat -c '%a' /home/*/.aws/credentials 2>/dev/null ``` #### Fix it ```bash # Fix permissions immediately chmod 600 ~/.aws/credentials chmod 600 ~/.aws/config # Verify stat -c '%a %n' ~/.aws/credentials # 600 /home/user/.aws/credentials ✔ ``` #### Rotate if exposed If your credentials file was world-readable and you're on a shared system, **assume compromise and rotate your keys**: ```bash # Generate new access keys in AWS Console or via CLI aws iam create-access-key --user-name # Update your credentials aws configure set aws_access_key_id aws configure set aws_secret_access_key # Delete the old key aws iam delete-access-key --user-name --access-key-id ``` #### Upgrade the CLI ```bash # v1 pip install awscli>=1.45.14 --upgrade # v2 curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip && sudo ./aws/install --update aws --version # aws-cli/2.34.53 or later ✔ ``` #### Prevent in your infrastructure ```yaml # Ansible - always set mode explicitly - name: Create AWS credentials copy: content: "{{ aws_credentials }}" dest: "{{ ansible_env.HOME }}/.aws/credentials" mode: '0600' # ← don't forget this # Dockerfile - fix permissions after COPY COPY aws-credentials /home/app/.aws/credentials RUN chmod 600 /home/app/.aws/credentials ``` ### Where Else to Look - Applying This Pattern Beyond AWS The incomplete-fix / pre-existing file permission bug class is not unique to `aws-cli`. Any CLI tool that writes secrets to disk is a potential target. Here's where I'd look next (and where you should too): #### The grep template For any CLI tool that writes credentials or tokens to a file, the pattern is the same: ```bash # Clone the repo, then: # 1. Find all file write operations grep -rn "open(.*'[wa]')" src/ --include="*.py" grep -rn "os\.open\|ioutil\.WriteFile\|fs\.writeFile" src/ # 2. Check which ones DON'T have chmod nearby grep -rn "open(.*'[wa]')" src/ --include="*.py" -l | while read f; do if ! grep -q "chmod\|Chmod\|perm" "$f"; then echo "[NO CHMOD] $f" fi done # 3. Cross-reference with files that contain sensitive data grep -rn "token\|secret\|password\|credential\|api_key" src/ --include="*.py" -l ``` #### High-value targets to check | Tool | Credentials file | Language | What to grep | | ----------------- | -------------------------------------- | -------- | --------------------------------- | | `gcloud` CLI | `~/.config/gcloud/credentials.db` | Python | `open(`, `os.open` | | `kubectl` | `~/.kube/config` | Go | `os.OpenFile`, `ioutil.WriteFile` | | `terraform` CLI | `~/.terraform.d/credentials.tfrc.json` | Go | `os.OpenFile`, `os.WriteFile` | | `docker` CLI | `~/.docker/config.json` | Go | `os.OpenFile` | | `gh` (GitHub CLI) | `~/.config/gh/hosts.yml` | Go | `os.OpenFile`, `os.WriteFile` | | `az` (Azure CLI) | `~/.azure/accessTokens.json` | Python | `open(`, `os.open` | | `npm` | `~/.npmrc` (auth tokens) | JS | `fs.writeFile`, `fs.openSync` | | `helm` | `~/.config/helm/registry/config.json` | Go | `os.OpenFile` | #### What to look for The vulnerable pattern is always the same: 1. Tool creates new files with secure permissions (`0o600`) ✔ 2. Tool **writes to existing files without checking or fixing permissions** ✘ 3. Sensitive data (tokens, keys, passwords) is written to the file If step 1 is secure but step 2 is missing, you have a finding. The vendor will have a harder time dismissing it if they've already fixed the same pattern in the same codebase (like AWS did four times). #### Go-specific gotcha In Go, `os.OpenFile` takes a permission mode parameter, but - just like Python's `os.open` - **it only applies when creating a new file**: ```go // This does NOT fix permissions on an existing file: f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE, 0600) // You still need: os.Chmod(path, 0600) ``` This is the same trap across every language. The function signature makes it look like you're setting permissions, but you're only setting them on creation. ### Lessons Learned 🧠 **Incomplete-fix hunting works.** When you see a vendor patch the same bug class in multiple files, grep the entire codebase for every other instance. They will miss at least one. In this case, four fixes shipped and the most critical file was left behind. **The "pre-existing file" pattern is everywhere.** Any code that writes secrets to a file needs to handle the case where that file already exists with bad permissions. `os.open` with `O_CREAT` only sets mode on creation - you need an explicit `chmod` for existing files. This applies way beyond AWS - go check your own projects. **Compare with established standards.** SSH's behavior (refuse to use lax-permission keys) is the gold standard. When you find a tool that doesn't do this, you've probably found a bug. Other tools to compare against: GnuPG (`~/.gnupg`, enforces `0o700`), Docker (`~/.docker/config.json`, stores registry auth tokens). **Handle triage questions professionally.** When triage asks "does this really happen in practice?" - don't get defensive. List concrete, reproducible scenarios with code. Let the evidence speak. Anticipate the "this is by design" objection and counter it with the vendor's own prior behavior. **Severity isn't everything.** I could've argued the Low rating, but the bug is real, the fix shipped, and the finding is on my profile. Spend your energy finding the next bug, not arguing about CVSS scores. **Grep is your best friend.** The actual hunting took 30 minutes - read the existing patches, understand the pattern, grep for the same anti-pattern, trace the code path, write a PoC. You don't need fancy tools for this. `grep -rn "open(.*'[wa]')" + "no chmod nearby"` is all it takes. #### Acknowledgments 🙏 Huge thanks to the AWS Security team - especially @vulnguard - for the professional and transparent handling throughout the entire process. From validation to fix deployment, the communication was clear and timely. It's always a great experience when a vendor treats researchers as collaborators rather than adversaries. Looking forward to future contributions.