I stumbled upon the web’s equivalent of a secret passage in a castle, and what I learned has enabled me to automate processes I never thought possible. 

Let me explain.

Most of us think of an API as something a web app either has or doesn’t have. But that’s not really true. Every web app that stores data has some connection to a backend, and essentially anything you can do in the interface can be done with code as well. 

Just because a tool you use doesn’t have a public API, doesn’t mean you can’t automate it. You don’t always have to resort to difficult and tedious web scraping techniques.

In this post, I'm going to show you how to find “hidden” APIs, why you should know about them, and what you can do with them. I'll also share some use-cases, like getting a transcript from a Loom video and automating tasks on a website that would otherwise require a lot of clicking.

What Are Hidden APIs?

“Hidden” is a useful misnomer. They’re not so much hidden as they are undocumented and not really intended for public use. 

Developers use them to send information back and forth between the website you see (the frontend) and the server that holds all the data (the backend). For example, when you load a Loom video, the page sends several requests to Loom’s backend GraphQL API to load the information that displays on the page. When you click to view the transcript, your browser sends another API request to load this information without having to reload the page. 

Many times, these APIs require a user to be logged in to use them. Their authentication is via the user’s browser cookies. Other times they use traditional API authentication, such as a Bearer token. Sometimes they require no authentication at all. As a rule of thumb, if you don’t need to log in to view the page, the API does not require authentication. 

Many of the things you can do with these APIs could also be done with web scraping techniques. But building a bot with puppeteer or selenium comes with extra complexity and unreliability. In fact, most people who are seasoned web scrapers will advise you to use the API directly if it’s available!

Finding Hidden APIs

Discovering hidden APIs is surprisingly easy once you know how. Go to the app you’re curious about—we’ll open up a sample loom video. 

Open Inspect Element: Right-click on the webpage and select "Inspect" or press Ctrl+Shift+I (Cmd+Option+I for Mac users). This opens the developer tools.

Open the Network tab: Click on the "Network" tab to see all the data requests the webpage makes. Once you have this open, try pressing F5 to refresh the page. It will only record network requests when you have it open.

Check “Preserve log” and “Disable cache”:

This allows you to catch requests that happen right before you change pages, which is common with form submits. Disable cache avoids a scenario where the browser keeps the data stored and doesn’t do the request again.

Filter to Fetch/XHR and Doc: Requests to get data will always be Fetch/XHR requests, and clicking this button will help narrow down the list. Hold CTRL to select multiple options. “Doc” requests are usually loading .html pages, and occasionally have something interesting.

Click on things and see what requests are made (if any): do whatever you’re interested in discovering how to do automatically. For example, let’s click on the “Transcript” button to see the transcript and see if it does anything. 

In this case, several requests occur but none of them have data about the transcript. This is because it was already loaded when the page loaded. 

How about creating a comment?

Type a comment and click create, and voila 🤩

Found you!

With any request, you’ll be interested in a few details:

• The request URL and method (GET, POST, etc.). Available on the headers tab.
• Headers for the request (especially any authentication headers or cookie headers). Also available on the headers tab. 
• The Payload tab which shows what data was sent with the request body.
• The Response tab which shows the response from the server. 

There are two ways to search network requests to find what you’re looking for. 

The first is to use this Filter box:

But this will only filter by the name of the request. 

Hit this magnifying glass or CTRL/CMD + F to do a powerful all-encompassing search.

By using this search feature, I was able to quickly find the request that fetches the transcript.

If you submitted a form, you can try searching for the text you submitted to see how it was sent to the server. 

Ok but how do I use a hidden API?

Once you've found a hidden API, here's how to make it work for you:

Copy the Request: Right-click on the network request and copy it as cURL or JavaScript. This gives you a starting point.

Trim the Fat: The copied request will have a lot of extra stuff you don't need, like headers sent by your browser. 99% of the time you can get away with removing all but authentication or cookie headers. 

Ask ChatGPT for Help: You can paste the cURL request into ChatGPT and ask questions about it. For example, “Format this cURL request and remove headers that aren’t for authentication/cookies.”

Test with Postman or your preferred API tool. You can import cURL requests directly into Postman by pasting the cURL directly into the URL bar. (It’s a good idea to clean it up with ChatGPT first, and I found that bash cURL syntax works better than cmd.)

Tips and tricks for testing API calls

• Start by copying the API call verbatim and get that to work before testing variations like removing headers. 
• Sometimes Postman does not process cURL syntax as expected. For example, I had to manually change the format of the body to JSON from Text. 
• If the API is a GraphQL endpoint, I recommend reading this primer and learning about introspection, which in some cases will allow you to discover every API method available. (It’s like self-generating documentation for the API).
• When working with GraphQL in Postman with undocumented APIs, it’s better to switch the Body type to GraphQL. It will give you separate windows for variables and query. The query should be plain text, and the variables should be valid JSON. To get these values, simply open up the request payload in your browser and right-click to copy each value separately.
• API calls that require cookies for authentication will eventually fail when the cookie expires. Sometimes it’s worth it to go in and update the cookies every once in a while. If you’re particularly keen on it, you could even create a chrome extension that grabs the cookie and sends it to a database for your workflow to access. 

Let’s use what we learned

Now for the fun part—let’s apply what we learned to a couple of use-cases. The possibilities are endless, but here are a couple of processes I’ve automated using these methods. 

Bulk replaying failed N8N workflows

Unlike Zapier, n8n does not have a way to select multiple failed executions and replay them. This is likely because Zapier’s infrastructure makes it able to handle hundreds of simultaneous requests, while each n8n instance runs on its own separate machine that would crash if you started 100 workflows all at once. 

Regardless, I had over 300 failed executions to replay and I certainly didn’t want to do it manually. 

N8N has a “meta” API that allows you to pull execution and workflow information, but this API does not have a method for replaying failed workflows. 

So I used browser inspection to figure out the API call to replay the workflow, and combined the official API with this hidden API method to replay the workflows (I used a delay to replay them in batches to avoid overwhelming the server). 

Here’s how:

I watched the network tab while clicking a button replay the workflow.

‍

‍

This one was nice and obvious:

‍

‍

And the workflow was simple:

‍

‍

Extracting the transcript based on Loom video URL

I created an automation that writes documentation for my workflows based on the workflow’s JSON, but I also wanted to be able to record a loom video of the workflow and use what I say to inform the documentation. 

So I created a form that allows me to add the Loom video link as well as some other details. 

When I click submit, it kicks off a workflow which grabs the transcript and uses AI to summarize the workflow based on what I said. 

Here’s how:

When I loaded my Loom video in the browser, I noticed a series of GraphQL requests. Clicking the transcript button didn’t do anything, so I figured it must have been pulled already when the page loaded. 

Sure enough, there it was:

When I checked the payload, I saw that it had several URLs.

After testing them, I found that you could access the captions_source_url which showed the plain text: 

So with two requests, we can get the transcript of any publicly-available loom video into a workflow!

All we need is the video ID, which happens to be the same ID that’s in the URL of the video. 

This workflow pulls the transcript and summarizes it:

For the first step, I extract the video ID by splitting out the rest of the URL with Javascript and make a GraphQL request to fetch the video transcript.

From that result, the second step grabs the caption URL and does a GET request to retrieve the contents.

That’s all there is to it!

Wrapping up

While the majority of automations will be better off using officially-supported APIs if they’re available, this method is a great option to add to your toolbox if you encounter a pesky workflow that should be possible to automate but just doesn’t have the right API.

This is now the second post in a row where I take you under the hood of a popular online tool and show you the possibilities that can be found with the browser inspect feature. 

Start snooping around—you’ll be amazed what you find. 

‍