A while ago, I stumbled upon Airflow instances being vulnerable to stateless session issues, which allowed me to log into any Airflow instance which had a misconfigured secret key. Since then, I’ve been hunting for similar issues, hoping to find more applications that would have made a similar error.
For a while, I ran the excellent Flask-Unsign tool on the cookies I found with my bug bounty scanning. It found a good amount of issues at first, but I ended up disabling it as it took up too many resources. Having been a while though, I decided to re-enable Flask-Unsign in October, and as I was on a plane I got a curious Slack alert that it had found a vulnerable cookie:
Visiting the site, I was greeted with a login page for Redash, a self-hosted product acquired by Databricks. I had never heard of it, but it seemed to involve connecting itself to internal data sources like SQL databases — certainly a good indicator for a security researcher to look into it. At this point, I took a look at the source code for Redash, and saw that it silently fell back to a static secret key if one was not explicitly configured. The default Flask-Unsign wordlist had picked up this key, and this bug bounty program was running a vulnerable instance!
At this point, it would be nice to hope that we have an easy, critical vulnerability. And while we might, it takes a bit more work than Airflow did. Because we know this session signing key, we can tamper with any of the session data and the server will be none the wiser. In Airflow, the server stores a numeric user_id which is auto-incrementing, so on most Airflow instances, we can send {user_id: 1} as the session data and be logged into the first user (typically the administrator).
However, Redash is a bit clever, and stores a random user UUID in the session. For our purposes, this is likely insurmountable — we have no access to the instance at all, and I couldn’t find an easy way to leak any user UUIDs. We might be able to escalate our privileges if we are a lowly-privileged user and can see other UUIDs, but this is not good enough for bug bounties!
I was a bit defeated at this point, all while suffering with the quality of in-flight WiFi, until I stumbled upon a very intriguing piece of the Redash source code, which generates the password reset links for users:
from itsdangerous import URLSafeTimedSerializer
serializer = URLSafeTimedSerializer(settings.SECRET_KEY)
def invite_token(user):
return serializer.dumps(str(user.id))This is exactly what we need! The password reset links are also stateless and are based on the same secret key that we have access to. Best of all, they use an auto-incrementing user ID instead of a UUID. As a result, we can generate this invite token ourselves, for user #1:
>>> from itsdangerous import URLSafeTimedSerializer, SignatureExpired, BadSignature
>>> serializer = URLSafeTimedSerializer("c292a0a3aa32397cdb050e233733900f")
>>> serializer.dumps(str("1"))
'IjEi.YdEknw.htZVzELmilJCgsSYu1oMXXEVhWY'I visited /reset/IjEi.YdEknw.htZVzELmilJCgsSYu1oMXXEVhWY and was prompted to set a new password! I was able to login to this instance as an administrator and was able to query quite a bit of data from this company.
On the plane, I immediately reported it via HackerOne to the program and it was impressively fixed in minutes, although sadly the program has not yet approved the disclosure. I was awarded their highest bounty of $1,500. The in-flight WiFi was worth it after all!
At this point, I also contacted Redash and Databricks about the issue, who were great to work with on resolving and preventing this misconfiguration. Redash now has comprehensive documentation on configuring its secret keys, and will refuse to start if you do not explicitly configure a secret key.
Scanning for CVE-2021-41192
One bounty is great, but many bounties are even better. Strangely, I queried my database of page titles for bug bounty programs, and could see plenty of Redash instances but no other findings. Why did I only find this one instance with Flask-Unsign? It appears that I found a very old version of Redash, and later versions do not send session cookies until the user is successfully logged in. As a result, we have no way to detect this with cookies for most instances — I got really lucky finding this in the first place.
Luckily, we can easily take the reset link we generated in the previous step and send a simple HTTP request to see if the instance is vulnerable. Instances which are vulnerable return a new password field, but those that are not just say our invite is invalid. I wrote a quick rule for this in my automation framework, which flags the server if it returns Enter your new password:
{Path: `/reset/IjEi.YdSlKg.79E2f3PIKsGC0UzV_X5oP8Gs3fk`}: {
{
Handler: processors.SimpleString(`Enter your new password`),
Name: "redash_confirmed_vulnerable",
Severity: types.SeverityTypeCritical,
},
},This method of detection has an unfortunate side effect of the URLSafeTimedSerializer enforcing an expiry date on the token, and you cannot forge the token to get around this. As a result, we must regenerate our payload every week.
Scan results
I became concerned after seeing some of the results from the scanning! While default installations of Redash via their official installation methods were not impacted, many companies chose to deploy Redash on their own and fell into this misconfiguration. I was also lucky to work with a couple other researchers, haxor1337 and nagli, to scan for even more of these instances.
Ultimately, we earned over $90,000 in bounties from this issue. Two notable instances stood out in the findings:
- A large American corporation had numerous vulnerable instances on the internet, for nearly every one of their business units. They paid $75,000 in bounties for these reports.
- A small company had a vulnerable instance on the internet, who paid us $5,000 in bounties and said they “need to [disclose] it to some legal institutions”. 😳
I also sent several good-faith notifications to various organizations which were plainly vulnerable and on the internet, including domain registrars and universities. Strangely or perhaps concerningly, there seems to still be a population of ad-tech companies who are still vulnerable to this issue but have no viable security contact.
Escalating to SSRF with CVE-2021-43780
While reporting this issue to some companies, we encountered some instances which did not have a ton of data inside of them. If we cannot query data from databases, what else can we do with our administrator access? It turns out that Redash implements a URL data source, which loads data from arbitrary URLs. However, it attempts to prevent local network requests, in order to prevent server-side request forgery (SSRF).
Unfortunately, this security check was flawed, as we can see in the source code:
def is_private_address(url):
hostname = urlparse(url).hostname
ip_address = socket.gethostbyname(hostname)
return ipaddress.ip_address(text_type(ip_address)).is_privateA URL such as http://169.254.169.254\\@8.8.8.8 returns a hostname of 8.8.8.8 when parsed by urllib, but libraries like requests will parse it as a request to the AWS metadata service. This allows it to pass as a public IP and allows full SSRF on the Redash instance.
I reported this to Redash and they promptly fixed this issue as well with CVE-2021-43780, moving to the Advocate library to safely perform external HTTP requests.
Other thoughts
This issue is what lead me to develop CookieMonster, so that I could scan for these cookies without the performance overhead. It still relies on the excellent Flask-Unsign wordlists, though, which amazingly had picked up this secret key, presumably from GitHub.
Nearly every bug bounty program we reported this issue to had to ignore their scope restrictions for our report, as it was often on other parts of their infrastructure but was too severe to ignore. This is a good reminder that more things are connected to your products (and their databases) than just themselves! Strict scope harms both researchers and the programs, and this potentially could have been found earlier if programs were more permissive about their research.
It is also still time to stop letting applications implement their own authentication. It does not matter if Redash implements SAML, OAuth, or passwords if their authentication implementation is broken. If you have things like this on the internet, put it behind oauth2-proxy or your favorite auth proxy and stop worrying about this type of thing.