Building a Tweet Screenshot API with Dart Frog 🐸, & Puppeteer 🐾

I have been writing dart for over 5 years now, and in that time written a lot of dart code for flutter/packages and web but server-side dart has always been one of the areas I’ve wanted to do a deep dive into.

A few years ago I dabbled in https://aqueduct.io/ and https://angel-dart.dev/. They were very early products in the dart ecosystem, both projects have been archived for a long time and mostly forgotten along with our hopes for a robust dart backend framework (yes I know shelf and others still exist 🌚).

That was until the fantastic folks at https://verygood.ventures/ gave us Dart Frog which although in its early development stages is pretty awesome and straightforward to use.

Creating our Twitter Screenshot API [Tweety]

First, we have to install the dart_frog_cli the tool that does all the magic🪄

# 📦 Install the dart_frog cli from source
dart pub global activate dart_frog_cli

➜ dart_frog              
A fast, minimalistic backend framework for Dart.

Usage: dart_frog <command> [arguments]

Global options:
-h, --help       Print this usage information.
    --version    Print the current version.
    --verbose    Output additional logs.

Available commands:
  build    Create a production build.
  create   Creates a new Dart Frog app.
  dev      Run a local development server.
  update   Update the Dart Frog CLI.

Run "dart_frog help <command>" for more information about a command.

Next, we create the project by running $dart_frog create

# 🐥 Create project using dart_frog_cli

➜ dart_frog create tweety                  
✓ Creating tweety (59ms)
✓ Installing dependencies (2.8s)

Created tweety at /projects/tweety.

Get started by typing:

cd /projects/tweety
dart_frog dev

Now we add puppeteer (or the dart port of the JavaScript version at least) an amazing tool that helps in web scraping and browser automation,

➜ flutter pub add puppeteer
Resolving dependencies...
  intl 0.17.0 (0.18.0 available)
  js 0.6.5 (0.6.6 available)
Got dependencies!

Now to the cool part:

The beauty of dart frog is that it handles almost everything that has to do with routing and server creation and all you need to do is create files and folders with the correct names. More on that here.

Here’s our project’s tree structure:

.
├── lib
│   ├── request_handler.dart
│   └── tweety_core.dart
|
└── routes
    ├── _middleware.dart
    ├── index.dart
    └── twitter.dart

This is the intended URI structure of our requests:

• ${host_url}/twitter?text=${tweet_url} (Fetch a tweet’s text content)
• ${host_url}/twitter?screenshot=${tweet_url} (Fetch a screenshot of a tweet)

We create a simple request handler that is supposed to handle and return errors using a try/catch block.

Next, we build the TweetyCore which handles all our puppeteer implementation and code including creating a puppeteer instance, and setting a userAgent for our headless browser (this is important because Twitter only allows recognized user agents, else you will get an error screen and no tweet data).

Using puppeteer we can create the getTweetText and screenshotTweet methods by query selecting the article tag that comes in the HTML of every tweet.

Next, we create a middleware that can also be used for dependency injection as seen below, we’ll use this to provide a TweetyCore instance to our routes.

Lastly, we create and handle query parameters, read the TweetyCore instance from our provider, initialize it and proceed to handle the different query types (‘screenshot’ and ‘text’)

Now we can run our project and voila:

➜ dart_frog dev
  ✓ Running on http://localhost:8080 (0.1s)
  The Dart VM service is listening on http://127.0.0.1:8181/LfGXl5iFqBw=/
  The Dart DevTools debugger and profiler is available at: http://127.0.0.1:8181/LfGXl5iFqBw=/devtools/#/?uri=ws%3A%2F%2F127.0.0.1%3A8181%2FLfGXl5iFqBw%3D%2Fws
  [hotreload] Hot reload is enabled.

In my next article, I’d be demonstrating how we can host this on Google Cloud Run via docker. But here’s the github repo.