Automating Screen Reader Testing On macOS Using Auto VO

If you’re an accessibility nerd like me, or just curious about assistive technology, you’ll dig Auto-VO. Auto-VO is a node module and CLI for automated testing of web content with the VoiceOver screen reader on macOS. I created Auto VO to make it easier for developers, PMs, and QA to more quickly perform repeatable, automated…

Working with the File System in Deno

Working with the File System in Deno – SitePointSkip to main contentFree JavaScript Book!Write powerful, clean and maintainable JavaScript.RRP $11.95 In this article, we’ll build on our introduction to Deno by creating a command-line tool that can search for text within files and folders. We’ll use a range of API methods that Deno provides to read and write to the file system.
In our last installment, we used Deno to build a command-line tool to make requests to a third-party API. In this article, we’re going to leave the network to one side and build a tool that lets you search the file system for text in files and folders within your current directory — similar to tools like grep.
Note: we’re not building a tool that will be as optimized and efficient as grep, nor are we aiming to replace it! The aim of building a tool like this is to get familiar with Deno’s file system APIs.
Installing Deno
We’re going to assume that you’ve got Deno up and running on your machine locally. You can check the Deno website or the previous article for more detailed installation instructions and also to get information on how to add Deno support to your editor of choice.
At the time of writing, the latest stable version of Deno is 1.10.2, so that’s what I’m using in this article.
For reference, you can find the complete code from this article on GitHub.
Setting Up Our New Command with Yargs
As in the previous article, we’ll use Yargs to build the interface that our users can use to execute our tool. Let’s create index.ts and populate it with the following:
import yargs from “https://deno.land/x/yargs@v17.0.1-deno/deno.ts”;

interface Yargs {
describe: (param: string, description: string) = > Yargs;
demandOption: (required: string[]) = > Yargs;
argv: ArgvReturnType;
}

interface UserArguments {
text: string;
}

const userArguments: UserArguments =
(yargs(Deno.args) as unknown as Yargs)
.describe(“text”, “the text to search for within the current directory”)
.demandOption([“text”])
.argv;

console.log(userArguments);

There’s a fair bit going on here that’s worth pointing out:
We install Yargs by pointing to its path on the Deno repository. I explicitly use a precise version number to make sure we always get that version, so that we don’t end up using whatever happens to be the latest version when the script runs.
At the time of writing, the Deno + TypeScript experience for Yargs isn’t great, so I’ve created my own interface and used that to provide some type safety.
UserArguments contains all the inputs we’ll ask the user for. For now, we’re only going to ask for text, but in future we could expand this to provide a list of files to search for, rather than assuming the current directory.
We can run this with deno run index.ts and see our Yargs output:
$ deno run index.ts
Check file:///home/jack/git/deno-file-search/index.ts
Options:
–help Show help [boolean]
–version Show version number [boolean]
–text the text to search for within the current directory [required]

Missing required argument: text

Now it’s time to get implementing!
Listing Files
Before we can start searching for text in a given file, we need to generate a list of directories and files to search within. Deno provides Deno.readdir, which is part of the “built-ins” library, meaning you don’t have to import it. It’s available for you on the global namespace.
Deno.readdir is asynchronous and returns a list of files and folders in the current directory. It returns these items as an AsyncIterator, which means we have to use the for await … of loop to get at the results:
for await (const fileOrFolder of Deno.readDir(Deno.cwd())) {
console.log(fileOrFolder);
}

This code will read from the current working directory (which Deno.cwd() gives us) and log each result. However, if you try to run the script now, you’ll get an error:
$ deno run index.ts –text=’foo’
error: Uncaught PermissionDenied: Requires read access to , run again with the –allow-read flag
for await (const fileOrFolder of Deno.readDir(Deno.cwd())) {
^
at deno:core/core.js:86:46
at unwrapOpResult (deno:core/core.js:106:13)
at Object.opSync (deno:core/core.js:120:12)
at Object.cwd (deno:runtime/js/30_fs.js:57:17)
at file:///home/jack/git/deno-file-search/index.ts:19:52

Remember that Deno requires all scripts to be explicitly given permissions to read from the file system. In our case, the –allow-read flag will enable our code to run:
~/$ deno run –allow-read index.ts –text=’foo’
{ name: “.git”, isFile: false, isDirectory: true, isSymlink: false }
{ name: “.vscode”, isFile: false, isDirectory: true, isSymlink: false }
{ name: “index.ts”, isFile: true, isDirectory: false, isSymlink: false }

In this case, I’m running the script in the directory where I’m building our tool, so it finds the TS source code, the .git repository and the .vscode folder. Let’s start writing some functions to recursively navigate this structure, as we need to find all the files within the directory, not just the top level ones. Additionally, we can add some common ignores. I don’t think anyone will want the script to search the entire .git folder!
In the code below, we’ve created the getFilesList function, which takes a directory and returns all files in that directory. If it encounters a directory, it will recursively call itself to find any nested files, and return the result:
const IGNORED_DIRECTORIES = new Set([“.git”]);

async function getFilesList(
directory: string,
): Promise {
const foundFiles: string[] = [];
for await (const fileOrFolder of Deno.readDir(directory)) {
if (fileOrFolder.isDirectory) {
if (IGNORED_DIRECTORIES.has(fileOrFolder.name)) {

continue;
}

const nestedFiles = await getFilesList(
`${directory}/${fileOrFolder.name}`,
);
foundFiles.push(…nestedFiles);
} else {

foundFiles.push(`${directory}/${fileOrFolder.name}`);
}
}
return foundFiles;
}

We can then use this like so:
const files = await getFilesList(Deno.cwd());
console.log(files);

We also get some output that looks good:
$ deno run –allow-read index.ts –text=’foo’
[
“/home/jack/git/deno-file-search/.vscode/settings.json”,
“/home/jack/git/deno-file-search/index.ts”
]

Using the path Module
We’re could now combine file paths with template strings like so:
`${directory}/${fileOrFolder.name}`,

But it would be nicer to do this using Deno’s path module. This module is one of the modules that Deno provides as part of its standard library (much like Node does with its path module), and if you’ve used Node’s path module the code will look very similar. At the time of writing, the latest version of the std library Deno provides is 0.97.0, and we import the path module from the mod.ts file:
import * as path from “https://deno.land/std@0.97.0/path/mod.ts”;

mod.ts is always the entrypoint when importing Deno’s standard modules. The documentation for this module lives on the Deno site and lists path.join, which will take multiple paths and join them into one path. Let’s import and use that function rather than manually combining them:

import yargs from “https://deno.land/x/yargs@v17.0.1-deno/deno.ts”;
import * as path from “https://deno.land/std@0.97.0/path/mod.ts”;

async function getFilesList(
directory: string,
): Promise {
const foundFiles: string[] = [];
for await (const fileOrFolder of Deno.readDir(directory)) {
if (fileOrFolder.isDirectory) {
if (IGNORED_DIRECTORIES.has(fileOrFolder.name)) {

continue;
}

const nestedFiles = await getFilesList(
path.join(directory, fileOrFolder.name),
);
foundFiles.push(…nestedFiles);
} else {

foundFiles.push(path.join(directory, fileOrFolder.name));
}
}
return foundFiles;
}

When using the standard library, it’s vital that you remember to pin to a specific version. Without doing so, your code will always load the latest version, even if that contains changes that will break your code. The Deno docs on the standard library go into this further, and I recommend giving that page a read.
Reading the Contents of a File
Unlike Node, which lets you read contents of files via the fs module and the readFile method, Deno provides readTextFile out of the box as part of its core, meaning that in this case we don’t need to import any additional modules. readTextFile does assume that the file is encoded as UTF-8 — which, for text files, is normally what you want. If you’re working with a different file encoding, you can use the more generic readFile, which doesn’t assume anything about the encoding and lets you pass in a specific decoder.
Once we’ve got the list of files, we can loop over them and read their contents as text:
const files = await getFilesList(Deno.cwd());

files.forEach(async (file) = > {
const contents = await Deno.readTextFile(file);
console.log(contents);
});

Because we want to know the line number when we find a match, we can split the contents on a new line character (n) and search each line in turn to see if there’s a match. That way, if there is, we’ll know the index of the line number so we can report it back to the user:
files.forEach(async (file) = > {
const contents = await Deno.readTextFile(file);
const lines = contents.split(“n”);
lines.forEach((line, index) = > {
if (line.includes(userArguments.text)) {
console.log(“MATCH”, line);
}
});
});

To store our matches, we can create an interface that represents a Match, and push matches onto an array when we find them:
interface Match {
file: string;
line: number;
}
const matches: Match[] = [];
files.forEach(async (file) = > {
const contents = await Deno.readTextFile(file);
const lines = contents.split(“n”);
lines.forEach((line, index) = > {
if (line.includes(userArguments.text)) {
matches.push({
file,
line: index + 1,
});
}
});
});

Then we can log out the matches:
matches.forEach((match) = > {
console.log(match.file, “line:”, match.line);
});

However, if you run the script now, and provide it with some text that will definitely match, you’ll still see no matches logged to the console. This is a common mistake people make with async and await within a forEach call; the forEach won’t wait for the callback to be complete before considering itself done. Take this code:
files.forEach(file = > {
new Promise(resolve = > {

})
})

The JavaScript engine is going to execute the forEach that runs on each file — generating a new promise — and then continue executing the rest of the code. It’s not going to automatically wait for those promises to resolve, and it’s exactly the same when we use await.
The good news is that this will work as expected in a for … of loop, so rather than:
files.forEach(file = > {…})

We can swap to:
for (const file of files) {

}

The for … of loop will execute the code for each file in series, and upon seeing use of the await keyword it will pause execution until that promise has resolved. This means that after, the loop is executed, we know that all the promises have resolved, and now we do get matches logged onto the screen:
$ deno run –allow-read index.ts –text=’readTextFile’
Check file:///home/jack/git/deno-file-search/index.ts
/home/jack/git/deno-file-search/index.ts line: 54

Let’s make some improvements to our output to make it easier to read. Rather than store matches as an array, let’s make it a Map where the keys are the filenames and the value is a Set of all the matches. That way, we can clarify our output by listing matches grouped by file, and have a data structure that lets us explore the data more easily.
First, we can create the data structure:
const matches = new Map();

Then we can store matches by adding them to a Set for that given file. This is a bit more work than before. We can’t just push items onto an array now. We firstly have to find any existing matches (or create a new Set) and then store them:
for (const file of files) {
const contents = await Deno.readTextFile(file);
const lines = contents.split(“n”);
lines.forEach((line, index) = > {
if (line.includes(userArguments.text)) {
const matchesForFile = matches.get(file) || new Set();
matchesForFile.add({
file,
line: index + 1,
});
matches.set(file, matchesForFile);
}
});
}

Then we can log the matches by iterating over the Map. When you use for … of on a Map, each iteration gives you an array of two items, where the first is the key in the map and the second is the value:
for (const match of matches) {
const fileName = match[0];
const fileMatches = match[1];
console.log(fileName);
fileMatches.forEach((m) = > {
console.log(“= >”, m.line);
});
}

We can do some destructuring to make this a little neater:
for (const match of matches) {
const [fileName, fileMatches] = match;

Or even:
for (const [fileName, fileMatches] of matches) {

Now when we run the script we can see all the matches in a given file:
$ deno run –allow-read index.ts –text=’Deno’
/home/jack/git/deno-file-search/index.ts
= > 15
= > 26
= > 45
= > 54

Finally, to make the output a bit clearer, let’s store the actual line that matched too. First, I’ll update my Match interface:
interface Match {
file: string;
lineNumber: number;
lineText: string;
}

Then update the code that stores the matches. One really nice thing about TypeScript here is that you can update the Match interface and then have the compiler tell you the code you need to update. I’ll often update a type, and then wait for VS Code to highlight any problems. It’s a really productive way to work if you can’t quite remember all the places where the code needs an update:
if (line.includes(userArguments.text)) {
const matchesForFile = matches.get(file) || new Set();
matchesForFile.add({
file,
lineNumber: index + 1,
lineText: line,
});
matches.set(file, matchesForFile);
}

The code that outputs the matches also needs an update:
for (const [fileName, fileMatches] of matches) {
console.log(fileName);
fileMatches.forEach((m) = > {
console.log(“= >”, m.lineNumber, m.lineText.trim());
});
}

I decided to call trim() on our lineText so that, if the matched line is heavily indented, we don’t show it like that in the results. We’ll strip any leading (and trailing) whitespace in our output.
And with that, I’d say our first version is done!
$ deno run –allow-read index.ts –text=’Deno’
Check file:///home/jack/git/deno-file-search/index.ts
/home/jack/git/deno-file-search/index.ts
= > 15 (yargs(Deno.args) as unknown as Yargs)
= > 26 for await (const fileOrFolder of Deno.readDir(directory)) {
= > 45 const files = await getFilesList(Deno.cwd());
= > 55 const contents = await Deno.readTextFile(file);

Filtering by File Extension
Let’s extend the functionality so that users can filter the file extensions we match via an extension flag, which the user can pass an extension to (such as –extension js to only match .js files). First let’s update the Yargs code and the types to tell the compiler that we’re accepting an (optional) extension flag:
interface UserArguments {
text: string;
extension?: string;
}

const userArguments: UserArguments =
(yargs(Deno.args) as unknown as Yargs)
.describe(“text”, “the text to search for within the current directory”)
.describe(“extension”, “a file extension to match against”)
.demandOption([“text”])
.argv;

We can then update getFilesList so that it takes an optional second argument, which can be an object of configuration properties we can pass into the function. I often like functions to take an object of configuration items, as adding more items to that object is much easier than updating the function to require more parameters are passed in:
interface FilterOptions {
extension?: string;
}

async function getFilesList(
directory: string,
options: FilterOptions = {},
): Promise {}

Now in the body of the function, once we’ve found a file, we now check that either:
The user didn’t provide an extension to filter by.
The user did provide an extension to filter by, and the extension of the file matches what they provided. We can use path.extname, which returns the file extension for a given path (for foo.ts, it will return .ts, so we take the extension the user passed in and prepend a . to it).
async function getFilesList(
directory: string,
options: FilterOptions = {},
): Promise {
const foundFiles: string[] = [];
for await (const fileOrFolder of Deno.readDir(directory)) {
if (fileOrFolder.isDirectory) {
if (IGNORED_DIRECTORIES.has(fileOrFolder.name)) {

continue;
}

const nestedFiles = await getFilesList(
path.join(directory, fileOrFolder.name),
options,
);
foundFiles.push(…nestedFiles);
} else {

const shouldStoreFile = !options.extension ||
path.extname(fileOrFolder.name) === `.${options.extension}`;

if (shouldStoreFile) {
foundFiles.push(path.join(directory, fileOrFolder.name));
}
}
}
return foundFiles;
}

Finally, we need to update our call to the getFilesList function, to pass it any parameters the user entered:
const files = await getFilesList(Deno.cwd(), userArguments);

Find and Replace
To finish off, let’s extend our tool to allow for basic replacement. If the user passes –replace=foo, we’ll take any matches we found from their search, and replace them with the provided word — in this case, foo, before writing that file to disk. We can use Deno.writeTextFile to do this. (Just like with readTextFile, you can also use writeFile if you need more control over the encoding.)
Once again, we’ll first update our Yargs code to allow the argument to be provided:
interface UserArguments {
text: string;
extension?: string;
replace?: string;
}

const userArguments: UserArguments =
(yargs(Deno.args) as unknown as Yargs)
.describe(“text”, “the text to search for within the current directory”)
.describe(“extension”, “a file extension to match against”)
.describe(“replace”, “the text to replace any matches with”)
.demandOption([“text”])
.argv;

What we can now do is update our code that loops over each individual file to search for any matches. Once we’ve checked each line for a match, we can then use the replaceAll method (this is a relatively new method built into JavaScript) to take the contents of the file and swap each match out for the replacement text provided by the user:
for (const file of files) {
const contents = await Deno.readTextFile(file);
const lines = contents.split(“n”);
lines.forEach((line, index) = > {
if (line.includes(userArguments.text)) {
const matchesForFile = matches.get(file) || new Set();
matchesForFile.add({
file,
lineNumber: index + 1,
lineText: line,
});
matches.set(file, matchesForFile);
}
});

if (userArguments.replace) {
const newContents = contents.replaceAll(
userArguments.text,
userArguments.replace,
);

}
}

Writing to disk is a case of calling writeTextFile, providing the file path and the new contents:
if (userArguments.replace) {
const newContents = contents.replaceAll(
userArguments.text,
userArguments.replace,
);
await Deno.writeTextFile(file, newContents);
}

When running this, however, we’ll now get a permissions error. Deno splits file reading and file writing into separate permissions, so you’ll need to pass the –allow-write flag to avoid an error:
$ deno run –allow-read index.ts –text=’readTextFile’ –extension=ts –replace=’jackWasHere’
Check file:///home/jack/git/deno-file-search/index.ts
error: Uncaught (in promise) PermissionDenied: Requires write access to “/home/jack/git/deno-file-search/index.ts”, run again with the –allow-write flag
await Deno.writeTextFile(file, newContents);

You can pass –allow-write or be a little more specific with –allow-write=., which means the tool only has permission to write files within the current directory:
$ deno run –allow-read –allow-write=. index.ts –text=’readTextFile’ –extension=ts –replace=’jackWasHere’
/home/jack/git/deno-file-search/index.ts
= > 74 const contents = await Deno.readTextFile(file);

Compiling to an Executable
Now that we have our script and we’re ready to share it, let’s ask Deno to bundle our tool into a single executable. This way, our end users won’t have to have Deno running and won’t have to pass in all the relevant permission flags every time; we can do that when bundling. deno compile lets us do this:
$ deno compile –allow-read –allow-write=. index.ts
Check file:///home/jack/git/deno-file-search/index.ts
Bundle file:///home/jack/git/deno-file-search/index.ts
Compile file:///home/jack/git/deno-file-search/index.ts
Emit deno-file-search

And then we can call the executable:
$ ./deno-file-search index.ts –text=readTextFile –extension=ts
/home/jack/git/deno-file-search/index.ts
= > 74 const contents = await Deno.readTextFile(file);

I really like this approach. We’re able to bundle the tool so our users don’t have to compile anything, and by providing the permissions up front we mean that users don’t have to. Of course, this is a trade-off. Some users might want to provide permissions such that they have full knowledge of what our script can and can’t do, but I think more often than not it’s good to feed the permissions into the executable.
Conclusion
I really do have a lot of fun working in Deno. Compared to Node, I love the fact that TypeScript, Deno Format, and other tools just come out the box. I don’t have to set up my Node project, then Prettier, and then figure out the best way to add TypeScript into that.
Deno is (unsurprisingly) not as polished or fleshed out as Node. Many third-party packages that exist in Node don’t have a good Deno equivalent (although I expect this will change in time), and at times the docs, whilst thorough, can be quite hard to find. But these are all small problems that you’d expect of any relatively new programming environment and language. I highly recommend exploring Deno and giving it a go. It’s definitely here to stay.
SitePoint has a growing list of articles on Deno. Check them out here if you’d like to explore Deno further.
Related ArticlesUsing Redis with Node.jsJavaScriptBy

Are we in a new era of web design? What do we call it?

Una is calling it the new responsive. A nod to the era we were most certainly in, the era of responsive design. Where responsive design was fluid grids, flexible media, and media queries, the new responsive is those things too, but slotted into a wider scope: user preference queries, viewport and form factor, macro layouts,…

The Ingredients of a Great WordPress Plugin

There are currently over 58,000 listings in the WordPress Plugin Repository. Beyond that, there are likely thousands of free and commercial offerings available elsewhere on the web. Together, they cover a staggering number of functionalities and use cases. But not all plugins are created equally. Only a relative few are labeled as “great” by their…

15 Beautiful Resume and CV Web Templates

Do you want to create an impressive online resume to highlight your coding career and catch the attention of recruiters, hiring managers or clients? On Envato Elements you will find high quality resumes and CVs optimized for the web. These templates professionally formatted and are easy to customize. You can replace the placeholder content with your own credentials…

Face Detection on the Web with Face-api.js

Face Detection on the Web with Face-api.js – SitePointSkip to main contentFree JavaScript Book!Write powerful, clean and maintainable JavaScript.RRP $11.95 Web browsers get more powerful by the day. Websites and web applications are also increasing in complexity. Operations that required a supercomputer some decades ago now runs on a smartphone. One of those things is face detection.
The ability to detect and analyze a face is super useful, as it enables us to add clever features. Think of automatically blurring faces (like Google Maps does), panning and scaling a webcam feed to focus on people (like Microsoft Teams), validating a passport, adding silly filters (like Instagram and Snapchat), and much more. But before we can do all that, we first need to find the face!
Face-api.js is a library that enables developers to use face detection in their apps without requiring a background in machine learning.
The code for this tutorial is available on GitHub.
Face Detection with Machine Learning
Detecting objects, like a face, is quite complex. Think about it: perhaps we could write a program that scans pixels to find the eyes, nose, and mouth. It can be done, but to make it totally reliable is practically unachievable, given the many factors to account for. Think of lighting conditions, facial hair, the vast variety of shapes and colors, makeup, angles, face masks, and so much more.
Neural networks, however, excel at these kinds of problems and can be generalized to account for most (if not all) conditions. We can create, train, and use neural networks in the browser with TensorFlow.js, a popular JavaScript machine learning library. However, even if we use an off-the-shelf, pre-trained model, we’d still get a little bit into the nitty-gritty of supplying the information to TensorFlow and interpreting the output. If you’re interested in the technical details of machine learning, check out “A Primer on Machine Learning with Python”.
Enter face-api.js. It wraps all of this into an intuitive API. We can pass an img, canvas, or video DOM element and the library will return one or a set of results. Face-api.js can detect faces, but also estimate various things in them, as listed below.
Face detection: get the boundaries of one or multiple faces. This is useful for determining where and how big the faces are in a picture.
Face landmark detection: get the position and shape of the eyebrows, eyes, nose, mouth and lips, and chin. This can be used to determine facing direction or to project graphics on specific regions, like a mustache between the nose and lips.
Face recognition: determine who’s in the picture.
Face expression detection: get the expression from a face. Note that the mileage may vary for different cultures.
Age and gender detection: get the age and gender from a face. Note that for “gender” classification, it classifies a face as feminine or masculine, which doesn’t necessarily reveal their gender.
Before you use any of this beyond experiments, please take note that artificial intelligence excels at amplifying biases. Gender classification works well for cisgendered people, but it can’t detect the gender of my nonbinary friends. It will identify white people most of the time but frequently fails to detect people of color.
Be very thoughtful about using this technology and test thoroughly with a diverse testing group.
Installation
We can install face-api.js via npm:
npm install face-api.js

However, to skip setting up build tools, I’ll include the UMD bundle via unpkg.org:

import ‘https://unpkg.com/face-api.js@0.22.2/dist/face-api.min.js’;

After that, we’ll need to download the correct pre-trained model(s) from the library’s repository. Determine what we want to know from faces, and use the Available Models section to determine which models are required. Some features work with multiple models. In that case, we have to choose between bandwidth/performance and accuracy. Compare the file size of the various available models and choose whichever you think is best for your project.
Unsure which models you need for your use? You can return to this step later. When we use the API without loading the required models, an error will be thrown, stating which model the library expects.

We’re now ready to use the face-api.js API.
Examples
Let’s build some stuff!
For the examples below, I’ll load a random image from Unsplash Source with this function:
function loadRandomImage() {
const image = new Image();

image.crossOrigin = true;

return new Promise((resolve, reject) = > {
image.addEventListener(‘error’, (error) = > reject(error));
image.addEventListener(‘load’, () = > resolve(image));
image.src = ‘https://source.unsplash.com/512×512/?face,friends’;
});
}

Cropping a picture
You can find the code for this demo in the accompanying GitHub repo.
First, we have to choose and load the model. To crop an image, we only need to know the boundary box of a face, so face detection is enough. We can use two models to do that: SSD Mobilenet v1 model (just under 6MB) and the Tiny Face Detector model (under 200KB). Let’s say accuracy is extraneous because users also have the option to crop manually. Additionally, let’s assume visitors use this feature on a slow internet connection. Because our focus is on bandwidth and performance, we’ll choose the smaller Tiny Face Detector model.
After downloading the model, we can load it:
await faceapi.nets.tinyFaceDetector.loadFromUri(‘/models’);

We can now load an image and pass it to face-api.js. faceapi.detectAllFaces uses the SSD Mobilenet v1 model by default, so we’ll have to explicitly pass new faceapi.TinyFaceDetectorOptions() to force it to use the Tiny Face Detector model.
const image = await loadRandomImage();
const faces = await faceapi.detectAllFaces(image, new faceapi.TinyFaceDetectorOptions());

The variable faces now contains an array of results. Each result has a box and score property. The score indicates how confident the neural net is that the result is indeed a face. The box property contains an object with the coordinates of the face. We could select the first result (or we could use faceapi.detectSingleFace()), but if the user submits a group photo, we want to see all of them in the cropped picture. To do that, we can compute a custom boundary box:
const box = {

bottom: -Infinity,
left: Infinity,
right: -Infinity,
top: Infinity,

get height() {
return this.bottom – this.top;
},

get width() {
return this.right – this.left;
},
};

for (const face of faces) {
box.bottom = Math.max(box.bottom, face.box.bottom);
box.left = Math.min(box.left, face.box.left);
box.right = Math.max(box.right, face.box.right);
box.top = Math.min(box.top, face.box.top);
}

Finally, we can create a canvas and show the result:
const canvas = document.createElement(‘canvas’);
const context = canvas.getContext(‘2d’);

canvas.height = box.height;
canvas.width = box.width;

context.drawImage(
image,
box.left,
box.top,
box.width,
box.height,
0,
0,
canvas.width,
canvas.height
);

Placing Emojis
You can find the code for this demo in the accompanying GitHub repo.
Why not have a little bit of fun? We can make a filter that puts a mouth emoji (👄) on all eyes. To find the eye landmarks, we need another model. This time, we care about accuracy, so we use the SSD Mobilenet v1 and 68 Point Face Landmark Detection models.
Again, we need to load the models and image first:
await faceapi.nets.faceLandmark68Net.loadFromUri(‘/models’);
await faceapi.nets.ssdMobilenetv1.loadFromUri(‘/models’);

const image = await loadRandomImage();

To get the landmarks, we must append the withFaceLandmarks() function call to detectAllFaces() to get the landmark data:
const faces = await faceapi
.detectAllFaces(image)
.withlandmarks();

Like last time, faces contains a list of results. In addition to where the face is, each result also contains a raw list of points for the landmarks. To get the right landmarks per feature, we need to slice the list of points. Because the number of points is fixed, I chose to hardcode the indices:
for (const face of faces) {
const features = {
jaw: face.landmarks.positions.slice(0, 17),
eyebrowLeft: face.landmarks.positions.slice(17, 22),
eyebrowRight: face.landmarks.positions.slice(22, 27),
noseBridge: face.landmarks.positions.slice(27, 31),
nose: face.landmarks.positions.slice(31, 36),
eyeLeft: face.landmarks.positions.slice(36, 42),
eyeRight: face.landmarks.positions.slice(42, 48),
lipOuter: face.landmarks.positions.slice(48, 60),
lipInner: face.landmarks.positions.slice(60),
};

}

Now we can finally have a little bit of fun. There are so many options, but let’s cover eyes with the mouth emoji (👄).
First we have to determine where to place the emoji and how big it should be drawn. To do that, let’s write a helper function that creates a box from an arbitrary set of points. The box holds all the information we need:
function getBoxFromPoints(points) {
const box = {
bottom: -Infinity,
left: Infinity,
right: -Infinity,
top: Infinity,

get center() {
return {
x: this.left + this.width / 2,
y: this.top + this.height / 2,
};
},

get height() {
return this.bottom – this.top;
},

get width() {
return this.right – this.left;
},
};

for (const point of points) {
box.left = Math.min(box.left, point.x);
box.right = Math.max(box.right, point.x);

box.bottom = Math.max(box.bottom, point.y);
box.top = Math.min(box.top, point.y);
}

return box;
}

Now we can start drawing emojis over the picture. Because we have to do this for both eyes, we can put feature.eyeLeft and feature.eyeRight in an array and iterate over them to execute the same code for each eye. All that remains is to draw the emojis on the canvas!
for (const eye of [features.eyeLeft, features.eyeRight]) {
const eyeBox = getBoxFromPoints(eye);
const fontSize = 6 * eyeBox.height;

context.font = `${fontSize}px/${fontSize}px serif`;
context.textAlign = ‘center’;
context.textBaseline = ‘bottom’;

context.fillStyle = ‘#000’;
context.fillText(‘👄’, eyeBox.center.x, eyeBox.center.y + 0.6 * fontSize);
}

Note that I used some magic numbers to tweak the font size and the exact text position. Because emojis are unicode and typography on the Web is weird (to me, at least), I just tweak the numbers until they appear about right. A more robust alternative would be to use an image as an overlay.

Concluding
Face-api.js is a great library that makes face detection and recognition really accessible. Familiarity with machine learning and neural networks isn’t required. I love tools that are enabling, and this is definitely one of them.
In my experience, face recognition on the Web takes a toll on performance. We’ll have to choose between bandwidth and performance or accuracy. The smaller models are definitely less accurate and would miss a face in some of the factors I mentioned before, like poor lighting or when faces are covered with a mask.
Microsoft Azure, Google Cloud, and probably other businesses offer face detection in the cloud. Because we avoid downloading big models, cloud-based detection avoids heavy page loads, tends to be more accurate as it’s frequently improved, and may even be faster because of optimized hardware. If you need high accuracy, you may want to look into a plan that you’re comfortable with.
I definitely recommend playing with face-api.js for hobby projects, experiments, and maybe for an MVP.
Related ArticlesUsing Redis with Node.jsJavaScriptBy