Introducing SafeTest: A Novel Method to Entrance Finish Testing | by Netflix Expertise Weblog | Feb, 2024
On this submit, we’re excited to introduce SafeTest, a revolutionary library that gives a recent perspective on Finish-To-Finish (E2E) checks for web-based Consumer Interface (UI) functions.
Historically, UI checks have been performed by both unit testing or integration testing (additionally known as Finish-To-Finish (E2E) testing). Nevertheless, every of those strategies presents a novel trade-off: you must select between controlling the take a look at fixture and setup, or controlling the take a look at driver.
As an illustration, when utilizing react-testing-library, a unit testing resolution, you keep full management over what to render and the way the underlying companies and imports ought to behave. Nevertheless, you lose the power to work together with an precise web page, which may result in a myriad of ache factors:
- Issue in interacting with complicated UI components like <Dropdown /> parts.
- Lack of ability to check CORS setup or GraphQL calls.
- Lack of visibility into z-index points affecting click-ability of buttons.
- Complicated and unintuitive authoring and debugging of checks.
Conversely, utilizing integration testing instruments like Cypress or Playwright supplies management over the web page, however sacrifices the power to instrument the bootstrapping code for the app. These instruments function by remotely controlling a browser to go to a URL and work together with the web page. This method has its personal set of challenges:
- Issue in making calls to an alternate API endpoint with out implementing customized community layer API rewrite guidelines.
- Lack of ability to make assertions on spies/mocks or execute code inside the app.
- Testing one thing like darkish mode entails clicking the theme switcher or realizing the localStorage mechanism to override.
- Lack of ability to check segments of the app, for instance if a part is just seen after clicking a button and ready for a 60 second timer to countdown, the take a look at might want to run these actions and will probably be at the very least a minute lengthy.
Recognizing these challenges, options like E2E Element Testing have emerged, with choices from Cypress and Playwright. Whereas these instruments try to rectify the shortcomings of conventional integration testing strategies, they produce other limitations on account of their structure. They begin a dev server with bootstrapping code to load the part and/or setup code you need, which limits their capacity to deal with complicated enterprise functions which may have OAuth or a fancy construct pipeline. Furthermore, updating TypeScript utilization might break your checks till the Cypress/Playwright group updates their runner.
SafeTest goals to deal with these points with a novel method to UI testing. The principle thought is to have a snippet of code in our application bootstrapping stage that injects hooks to run our tests (see the How Safetest Works sections for more information on what that is doing). Be aware that how this works has no measurable influence on the common utilization of your app since SafeTest leverages lazy loading to dynamically load the checks solely when operating the checks (within the README instance, the checks aren’t within the manufacturing bundle in any respect). As soon as that’s in place, we will use Playwright to run common checks, thereby reaching the best browser management we would like for our checks.
This method additionally unlocks some thrilling options:
- Deep linking to a particular take a look at without having to run a node take a look at server.
- Two-way communication between the browser and take a look at (node) context.
- Entry to all of the DX options that include Playwright (excluding those that include @playwright/take a look at).
- Video recording of checks, hint viewing, and pause web page performance for making an attempt out totally different web page selectors/actions.
- Skill to make assertions on spies within the browser in node, matching snapshot of the decision inside the browser.
SafeTest is designed to really feel acquainted to anybody who has performed UI checks earlier than, because it leverages the very best elements of current options. Right here’s an instance of methods to take a look at a whole software:
import { describe, it, anticipate } from 'safetest/jest';
import { render } from 'safetest/react';describe('my app', () => {
it('hundreds the principle web page', async () => {
const { web page } = await render();
await anticipate(web page.getByText('Welcome to the app')).toBeVisible();
anticipate(await web page.screenshot()).toMatchImageSnapshot();
});
});
We will simply as simply take a look at a particular part
import { describe, it, anticipate, browserMock } from 'safetest/jest';
import { render } from 'safetest/react';describe('Header part', () => {
it('has a standard mode', async () => {
const { web page } = await render(<Header />);
await anticipate(web page.getByText('Admin')).not.toBeVisible();
});
it('has an admin mode', async () => {
const { web page } = await render(<Header admin={true} />);
await anticipate(web page.getByText('Admin')).toBeVisible();
});
it('calls the logout handler when signing out', async () => {
const spy = browserMock.fn();
const { web page } = await render(<Header handleLogout={fn} />);
await web page.getByText('logout').click on();
anticipate(await spy).toHaveBeenCalledWith();
});
});
SafeTest makes use of React Context to permit for worth overrides throughout checks. For an instance of how this works, let’s assume we now have a fetchPeople operate utilized in a part:
import { useAsync } from 'react-use';
import { fetchPerson } from './api/individual';export const Individuals: React.FC = () => {
const { information: folks, loading, error } = useAsync(fetchPeople);
if (loading) return <Loader />;
if (error) return <ErrorPage error={error} />;
return <Desk information={information} rows=[...] />;
}
We will modify the Individuals part to make use of an Override:
import { fetchPerson } from './api/individual';
+import { createOverride } from 'safetest/react';+const FetchPerson = createOverride(fetchPerson);
export const Individuals: React.FC = () => {
+ const fetchPeople = FetchPerson.useValue();
const { information: folks, loading, error } = useAsync(fetchPeople);
if (loading) return <Loader />;
if (error) return <ErrorPage error={error} />;
return <Desk information={information} rows=[...] />;
}
Now, in our take a look at, we will override the response for this name:
const pending = new Promise(r => { /* Do nothing */ });
const resolved = [{name: 'Foo', age: 23], {title: 'Bar', age: 32]}];
const error = new Error('Whoops');describe('Individuals', () => {
it('has a loading state', async () => {
const { web page } = await render(
<FetchPerson.Override with={() => () => pending}>
<Individuals />
</FetchPerson.Override>
);
await anticipate(web page.getByText('Loading')).toBeVisible();
});
it('has a loaded state', async () => {
const { web page } = await render(
<FetchPerson.Override with={() => async () => resolved}>
<Individuals />
</FetchPerson.Override>
);
await anticipate(web page.getByText('Consumer: Foo, title: 23')).toBeVisible();
});
it('has an error state', async () => {
const { web page } = await render(
<FetchPerson.Override with={() => async () => { throw error }}>
<Individuals />
</FetchPerson.Override>
);
await anticipate(web page.getByText('Error getting customers: "Whoops"')).toBeVisible();
});
});
The render operate additionally accepts a operate that will probably be handed the preliminary app part, permitting for the injection of any desired components anyplace within the app:
it('has a folks loaded state', async () => {
const { web page } = await render(app =>
<FetchPerson.Override with={() => async () => resolved}>
{app}
</FetchPerson.Override>
);
await anticipate(web page.getByText('Consumer: Foo, title: 23')).toBeVisible();
});
With overrides, we will write complicated take a look at instances equivalent to guaranteeing a service methodology which mixes API requests from /foo, /bar, and /baz, has the proper retry mechanism for simply the failed API requests and nonetheless maps the return worth accurately. So if /bar takes 3 makes an attempt to resolve the strategy will make a complete of 5 API calls.
Overrides aren’t restricted to simply API calls (since we will use additionally use page.route), we will additionally override particular app degree values like characteristic flags or altering some static worth:
+const UseFlags = createOverride(useFlags);
export const Admin = () => {
+ const useFlags = UseFlags.useValue();
const { isAdmin } = useFlags();
if (!isAdmin) return <div>Permission error</div>;
// ...
}+const Language = createOverride(navigator.language);
export const LanguageChanger = () => {
- const language = navigator.language;
+ const language = Language.useValue();
return <div>Present language is { language } </div>;
}
describe('Admin', () => {
it('works with admin flag', async () => {
const { web page } = await render(
<UseIsAdmin.Override with={oldHook => {
const oldFlags = oldHook();
return { ...oldFlags, isAdmin: true };
}}>
<MyComponent />
</UseIsAdmin.Override>
);
await anticipate(web page.getByText('Permission error')).not.toBeVisible();
});
});
describe('Language', () => {
it('shows', async () => {
const { web page } = await render(
<Language.Override with={outdated => 'abc'}>
<MyComponent />
</Language.Override>
);
await anticipate(web page.getByText('Present language is abc')).toBeVisible();
});
});
Overrides are a robust characteristic of SafeTest and the examples right here solely scratch the floor. For extra info and examples, discuss with the Overrides section on the README.
SafeTest comes out of the field with highly effective reporting capabilities, equivalent to computerized linking of video replays, Playwright hint viewer, and even deep link directly to the mounted tested component. The SafeTest repo README hyperlinks to all of the example apps in addition to the reports
Many massive companies want a type of authentication to make use of the app. Usually, navigating to localhost:3000 simply ends in a perpetually loading web page. You should go to a distinct port, like localhost:8000, which has a proxy server to verify and/or inject auth credentials into underlying service calls. This limitation is likely one of the essential causes that Cypress/Playwright Element Assessments aren’t appropriate to be used at Netflix.
Nevertheless, there’s normally a service that may generate take a look at customers whose credentials we will use to log in and work together with the appliance. This facilitates creating a light-weight wrapper round SafeTest to mechanically generate and assume that take a look at person. As an illustration, right here’s mainly how we do it at Netflix:
import { setup } from 'safetest/setup';
import { createTestUser, addCookies } from 'netflix-test-helper';sort Setup = Parameters<typeof setup>[0] & {
extraUserOptions?: UserOptions;
};
export const setupNetflix = (choices: Setup) => {
setup({
...choices,
hooks: { beforeNavigate: [async page => addCookies(page)] },
});
beforeAll(async () => {
createTestUser(choices.extraUserOptions)
});
};
After setting this up, we merely import the above bundle instead of the place we might have used safetest/setup.
Whereas this submit targeted on how SafeTest works with React, it’s not restricted to simply React. SafeTest additionally works with Vue, Svelte, Angular, and even can run on NextJS or Gatsby. It additionally runs utilizing both Jest or Vitest based mostly on which take a look at runner your scaffolding began you off with. The examples folder demonstrates methods to use SafeTest with totally different tooling mixtures, and we encourage contributions so as to add extra instances.
At its core, SafeTest is an clever glue for a take a look at runner, a UI library, and a browser runner. Although the commonest utilization at Netflix employs Jest/React/Playwright, it’s straightforward so as to add extra adapters for different choices.
SafeTest is a robust testing framework that’s being adopted inside Netflix. It permits for straightforward authoring of checks and supplies complete experiences when and the way any failures occurred, full with hyperlinks to view a playback video or manually run the take a look at steps to see what broke. We’re excited to see the way it will revolutionize UI testing and stay up for your suggestions and contributions.