Testing
editTesting
editTo ensure that your changes will not break other functionality, please run the test suite and build (Building a Kibana distributable) before submitting your Pull Request.
Running specific Kibana tests
editThe following table outlines possible test file locations and how to invoke them:
Test runner | Test location | Runner command (working directory is Kibana root) |
---|---|---|
Jest |
|
|
Jest (integration) |
|
|
Mocha |
|
|
Functional |
|
|
Karma |
|
|
For X-Pack tests located in x-pack/
see
X-Pack Testing
Test runner arguments: - Where applicable, the optional arguments
-t=regexp
or --grep=regexp
will only run tests or test suites
whose descriptions matches the regular expression. - [test path]
is
the relative path to the test file.
Examples: - Run the entire elasticsearch_service test suite:
yarn test:jest src/core/server/elasticsearch/elasticsearch_service.test.ts
- Run the jest test case whose description matches
stops both admin and data clients
:
yarn test:jest -t 'stops both admin and data clients' src/core/server/elasticsearch/elasticsearch_service.test.ts
- Run the api integration test case whose description matches the given
string: ``` yarn test:ftr:server –config test/api_integration/config.js
yarn test:ftr:runner –config test/api_integration/config
Cross-browser compatibility
editTesting IE on OS X
- Download VMWare Fusion.
- Download IE virtual machines for VMWare.
- Open VMWare and go to Window > Virtual Machine Library. Unzip the virtual machine and drag the .vmx file into your Virtual Machine Library.
-
Right-click on the virtual machine you just added to your library and
select "
Snapshots…
", and then click the "Take
" button in the modal that opens. You can roll back to this snapshot when the VM expires in 90 days. -
In System Preferences > Sharing, change your computer name to be
something simple, e.g. "
computer
". -
Run Kibana with
yarn start --host=computer.local
(substituting your computer name). -
Now you can run your VM, open the browser, and navigate to
http://computer.local:5601
to test Kibana. - Alternatively you can use browserstack
Running browser automation tests
editCheck out Functional Testing to learn more about how you can run and develop functional tests for Kibana core and plugins.
You can also look into the Scripts README.md to learn more about using the node scripts we provide for building Kibana, running integration tests, and starting up Kibana and Elasticsearch while you develop.
More testing information:
editFunctional Testing
editWe use functional tests to make sure the Kibana UI works as expected. It replaces hours of manual testing by automating user interaction. To have better control over our functional test environment, and to make it more accessible to plugin authors, Kibana uses a tool called the FunctionalTestRunner
.
Running functional tests
editThe FunctionalTestRunner
is very bare bones and gets most of its functionality from its config file, located at {kib-repo}blob/7.9/test/functional/config.js[test/functional/config.js]. If you’re writing a plugin outside the Kibana repo, you will have your own config file.
See Functional Tests for Plugins outside the Kibana repo for more info.
There are three ways to run the tests depending on your goals:
-
Easiest option:
- Description: Starts up Kibana & Elasticsearch servers, followed by running tests. This is much slower when running the tests multiple times because slow startup time for the servers. Recommended for single-runs.
-
node scripts/functional_tests
- does everything in a single command, including running Elasticsearch and Kibana locally
- tears down everything after the tests run
- exit code reports success/failure of the tests
-
Best for development:
- Description: Two commands, run in separate terminals, separate the components that are long-running and slow from those that are ephemeral and fast. Tests can be re-run much faster, and this still runs Elasticsearch & Kibana locally.
-
node scripts/functional_tests_server
- starts Elasticsearch and Kibana servers
- slow to start
- can be reused for multiple executions of the tests, thereby saving some time when re-running tests
- automatically restarts the Kibana server when relevant changes are detected
-
node scripts/functional_test_runner
-
runs the tests against Kibana & Elasticsearch servers that were started by
node scripts/functional_tests_server
- exit code reports success or failure of the tests
-
runs the tests against Kibana & Elasticsearch servers that were started by
-
Custom option:
- Description: Runs tests against instances of Elasticsearch & Kibana started some other way (like Elastic Cloud, or an instance you are managing in some other way).
- just executes the functional tests
- url, credentials, etc. for Elasticsearch and Kibana are specified via environment variables
-
Here’s an example that runs against an Elastic Cloud instance. Note that you must run the same branch of tests as the version of Kibana you’re testing.
export TEST_KIBANA_URL=https://kibana:password@my-kibana-instance.internal.net:443 export TEST_ES_URL=https://elastic:password@my-es-cluster.internal.net:9200 node scripts/functional_test_runner
-
Or you can override any or all of these individual parts of the URL and leave the others to the default values.
export TEST_KIBANA_PROTOCOL=https export TEST_KIBANA_HOSTNAME=my-kibana-instance.internal.net export TEST_KIBANA_PORT=443 export TEST_KIBANA_USER=kibana export TEST_KIBANA_PASS=<password> export TEST_ES_PROTOCOL=http export TEST_ES_HOSTNAME=my-es-cluster.internal.net export TEST_ES_PORT=9200 export TEST_ES_USER=elastic export TEST_ES_PASS=<password> node scripts/functional_test_runner
-
If you are running x-pack functional tests, start server and runner from {kib-repo}blob/7.9/xpack[x-pack] folder:
node scripts/functional_tests_server.js node ../scripts/functional_test_runner.js
-
Selenium tests are run in headless mode on CI. Locally the same tests will be executed in a real browser. You can activate headless mode by setting the environment variable:
export TEST_BROWSER_HEADLESS=1
-
If you are using Google Chrome, you can slow down the local network connection to verify test stability:
export TEST_THROTTLE_NETWORK=1
-
When running against a Cloud deployment, some tests are not applicable. To skip tests that do not apply, use --exclude-tag. An example shell file can be found at: {kib-repo}blob/7.9/test/scripts/jenkins_cloud.sh[test/scripts/jenkins_cloud.sh]
node scripts/functional_test_runner --exclude-tag skipCloud
More about node scripts/functional_test_runner
editWhen run without any arguments the FunctionalTestRunner
automatically loads the configuration in the standard location, but you can override that behavior with the --config
flag. List configs with multiple --config arguments.
-
--config test/functional/config.js
starts Elasticsearch and Kibana servers with the WebDriver tests configured to run in Chrome. -
--config test/functional/config.firefox.js
starts Elasticsearch and Kibana servers with the WebDriver tests configured to run in Firefox. -
--config test/api_integration/config.js
starts Elasticsearch and Kibana servers with the api integration tests configuration. -
--config test/accessibility/config.ts
starts Elasticsearch and Kibana servers with the WebDriver tests configured to run an accessibility audit using axe.
There are also command line flags for --bail
and --grep
, which behave just like their mocha counterparts. For instance, use --grep=foo
to run only tests that match a regular expression.
Logging can also be customized with --quiet
, --debug
, or --verbose
flags.
Use the --help
flag for more options.
Writing functional tests
editEnvironment
editThe tests are written in mocha using @kbn/expect for assertions.
We use WebDriver Protocol to run tests in both Chrome and Firefox with the help of chromedriver and geckodriver. When the FunctionalTestRunner
launches, remote service creates a new webdriver session, which starts the driver and a stripped-down browser instance. We use browser
service and webElementWrapper
class to wrap up Webdriver API.
The FunctionalTestRunner
automatically transpiles functional tests using babel, so that tests can use the same ECMAScript features that Kibana source code uses. See {kib-repo}blob/7.9/style_guides/js_style_guide.md[style_guides/js_style_guide.md].
Definitions
editProvider:
Code run by the FunctionalTestRunner
is wrapped in a function so it can be passed around via config files and be parameterized. Any of these Provider functions may be asynchronous and should return/resolve-to the value they are meant to provide. Provider functions will always be called with a single argument: a provider API (see the Provider API Section).
A config provider:
// config and test files use `export default` export default function (/* { providerAPI } */) { return { // ... } }
- Services
- Services are named singleton values produced by a Service Provider. Tests and other services can retrieve service instances by asking for them by name. All functionality except the mocha API is exposed via services.\
- Page objects
- Page objects are a special type of service that encapsulate behaviors common to a particular page or plugin. When you write your own plugin, you’ll likely want to add a page object (or several) that describes the common interactions your tests need to execute.
- Test Files
-
The
FunctionalTestRunner
's primary purpose is to execute test files. These files export a Test Provider that is called with a Provider API but is not expected to return a value. Instead Test Providers define a suite using mocha’s BDD interface. - Test Suite
-
A test suite is a collection of tests defined by calling
describe()
, and then populated with tests and setup/teardown hooks by callingit()
,before()
,beforeEach()
, etc. Every test file must define only one top level test suite, and test suites can have as many nested test suites as they like. - Tags
-
Use tags in
describe()
function to group functional tests. Tags include:-
ciGroup{id}
- Assigns test suite to a specific CI worker -
skipCloud
andskipFirefox
- Excludes test suite from running on Cloud or Firefox -
includeFirefox
- Groups tests that run on Chrome and Firefox
-
- Cross-browser testing
-
On CI, all the functional tests are executed in Chrome by default. To also run a suite against Firefox, assign the
includeFirefox
tag:
// on CI test suite will be run twice: in Chrome and Firefox describe('My Cross-browser Test Suite', function () { this.tags('includeFirefox'); it('My First Test'); }
If the tests do not apply to Firefox, assign the skipFirefox
tag.
To run tests on Firefox locally, use config.firefox.js
:
node scripts/functional_test_runner --config test/functional/config.firefox.js
Using the test_user service
editTests should run at the positive security boundry condition, meaning that they should be run with the mimimum privileges required (and documented) and not as the superuser. This prevents the type of regression where additional privleges accidentally become required to perform the same action.
The functional UI tests now default to logging in with a user named test_user
and the roles of this user can be changed dynamically without logging in and out.
In order to achieve this a new service was introduced called createTestUserService
(see test/common/services/security/test_user.ts
). The purpose of this test user service is to create roles defined in the test config files and setRoles() or restoreDefaults().
An example of how to set the role like how its defined below:
await security.testUser.setRoles(['kibana_user', 'kibana_date_nanos']);
Here we are setting the test_user
to have the kibana_user
role and also role access to a specific data index (kibana_date_nanos
).
Tests should normally setRoles() in the before() and restoreDefaults() in the after().
Anatomy of a test file
editThis annotated example file shows the basic structure every test suite uses. It starts by importing @kbn/expect
and defining its default export: an anonymous Test Provider. The test provider then destructures the Provider API for the getService()
and getPageObjects()
functions. It uses these functions to collect the dependencies of this suite. The rest of the test file will look pretty normal to mocha.js users. describe()
, it()
, before()
and the lot are used to define suites that happen to automate a browser via services and objects of type PageObject
.
import expect from '@kbn/expect'; // test files must `export default` a function that defines a test suite export default function ({ getService, getPageObject }) { // most test files will start off by loading some services const retry = getService('retry'); const testSubjects = getService('testSubjects'); const esArchiver = getService('esArchiver'); // for historical reasons, PageObjects are loaded in a single API call // and returned on an object with a key/value for each requested PageObject const PageObjects = getPageObjects(['common', 'visualize']); // every file must define a top-level suite before defining hooks/tests describe('My Test Suite', () => { // most suites start with a before hook that navigates to a specific // app/page and restores some archives into {es} with esArchiver before(async () => { await Promise.all([ // start with an empty .kibana index esArchiver.load('empty_kibana'), // load some basic log data only if the index doesn't exist esArchiver.loadIfNeeded('makelogs') ]); // go to the page described by `apps.visualize` in the config await PageObjects.common.navigateTo('visualize'); }); // right after the before() hook definition, add the teardown steps // that will tidy up {es} for other test suites after(async () => { // we unload the empty_kibana archive but not the makelogs // archive because we don't make any changes to it, and subsequent // suites could use it if they call `.loadIfNeeded()`. await esArchiver.unload('empty_kibana'); }); // This series of tests illustrate how tests generally verify // one step of a larger process and then move on to the next in // a new test, each step building on top of the previous it('Vis Listing Page is empty'); it('Create a new vis'); it('Shows new vis in listing page'); it('Opens the saved vis'); it('Respects time filter changes'); it(... }); }
Provider API
editThe first and only argument to all providers is a Provider API Object. This object can be used to load service/page objects and config/test files.
Within config files the API has the following properties
|
An instance of the {kib-repo}blob/7.9/packages/kbn-dev-utils/src/tooling_log/tooling_log.js[ |
|
Returns a promise that will resolve to a Config instance that provides the values from the config file at |
Within service and PageObject Providers the API is:
|
Load and return the singleton instance of a service by name |
|
Load the singleton instances of `PageObject`s and collect them on an object where each name is the key to the singleton instance of that PageObject |
Within a test Provider the API is exactly the same as the service providers API but with an additional method:
|
Load the test file at path in place. Use this method to nest suites from other files into a higher-level suite |
Service Index
editBuilt-in Services
editThe FunctionalTestRunner
comes with three built-in services:
- config:
-
- Source: {kib-repo}blob/7.9/src/functional_test_runner/lib/config/config.ts[src/functional_test_runner/lib/config/config.ts]
- Schema: {kib-repo}blob/7.9/src/functional_test_runner/lib/config/schema.ts[src/functional_test_runner/lib/config/schema.ts]
-
Use
config.get(path)
to read any value from the config file
- log:
-
- Source: {kib-repo}blob/7.9/packages/kbn-dev-utils/src/tooling_log/tooling_log.js[packages/kbn-dev-utils/src/tooling_log/tooling_log.js]
-
ToolingLog
instances are readable streams. The instance provided by this service is automatically piped to stdout by theFunctionalTestRunner
CLI -
log.verbose()
,log.debug()
,log.info()
,log.warning()
all work just like console.log but produce more organized output
- lifecycle:
-
- Source: {kib-repo}blob/7.9/src/functional_test_runner/lib/lifecycle.ts[src/functional_test_runner/lib/lifecycle.ts]
- Designed primary for use in services
- Exposes lifecycle events for basic coordination. Handlers can return a promise and resolve/fail asynchronously
-
Phases include:
beforeLoadTests
,beforeTests
,beforeEachTest
,cleanup
Kibana Services
editThe Kibana functional tests define the vast majority of the actual functionality used by tests.
- browser
-
- Source: {kib-repo}blob/7.9/test/functional/services/browser.ts[test/functional/services/browser.ts]
-
Higher level wrapper for
remote
service, which exposes available browser actions -
Popular methods:
-
browser.getWindowSize()
-
browser.refresh()
-
- testSubjects:
-
- Source: {kib-repo}blob/7.9/test/functional/services/test_subjects.ts[test/functional/services/test_subjects.ts]
- Test subjects are elements that are tagged specifically for selecting from tests
-
Use
testSubjects
over CSS selectors when possible -
Usage:
-
Tag your test subject with a
data-test-subj
attribute:<div id="container”> <button id="clickMe” data-test-subj=”containerButton” /> </div>
-
Click this button using the
testSubjects
helper:await testSubjects.click(‘containerButton’);
-
-
Popular methods:
-
testSubjects.find(testSubjectSelector)
- Find a test subject in the page; throw if it can’t be found after some time -
testSubjects.click(testSubjectSelector)
- Click a test subject in the page; throw if it can’t be found after some time
-
- find:
-
- Source: {kib-repo}blob/7.9/test/functional/services/find.ts[test/functional/services/find.ts]
-
Helpers for
remote.findBy*
methods that log and manage timeouts -
Popular methods:
-
find.byCssSelector()
-
find.allByCssSelector()
-
- retry:
-
- Source: {kib-repo}blob/7.9/test/common/services/retry/retry.ts[test/common/services/retry/retry.ts]
- Helpers for retrying operations
-
Popular methods:
-
retry.try(fn, onFailureBlock)
- Executefn
in a loop until it succeeds or the default timeout elapses. The optionalonFailureBlock
is executed before each retry attempt. -
retry.tryForTime(ms, fn, onFailureBlock)
- Executefn
in a loop until it succeeds orms
milliseconds elapses. The optionalonFailureBlock
is executed before each retry attempt.
-
- kibanaServer:
-
- Source: {kib-repo}blob/7.9/test/common/services/kibana_server/kibana_server.js[test/common/services/kibana_server/kibana_server.js]
- Helpers for interacting with Kibana’s server
-
Commonly used methods:
-
kibanaServer.uiSettings.update()
-
kibanaServer.version.get()
-
kibanaServer.status.getOverallState()
-
- esArchiver:
-
- Source: {kib-repo}blob/7.9/test/common/services/es_archiver.ts[test/common/services/es_archiver.ts]
-
Load/unload archives created with the
esArchiver
-
Popular methods:
-
esArchiver.load(name)
-
esArchiver.loadIfNeeded(name)
-
esArchiver.unload(name)
-
Full list of services that are used in functional tests can be found here: {kib-repo}blob/7.9/test/functional/services[test/functional/services]
- Low-level utilities:
-
-
es
- Source: {kib-repo}blob/7.9/test/common/services/es.ts[test/common/services/es.ts]
- Elasticsearch client
-
Higher level options:
kibanaServer.uiSettings
oresArchiver
-
remote
- Source: {kib-repo}blob/7.9/test/functional/services/remote/remote.ts[test/functional/services/remote/remote.ts]
- Instance of WebDriver class
- Responsible for all communication with the browser
-
To perform browser actions, use
remote
service -
For searching and manipulating with DOM elements, use
testSubjects
andfind
services - See the selenium-webdriver docs for the full API.
-
Custom Services
editServices are intentionally generic. They can be literally anything (even nothing). Some services have helpers for interacting with a specific types of UI elements, like pointSeriesVis
, and others are more foundational, like log
or config
. Whenever you want to provide some functionality in a reusable package, consider making a custom service.
To create a custom service somethingUseful
:
-
Create a
test/functional/services/something_useful.js
file that looks like this:// Services are defined by Provider functions that receive the ServiceProviderAPI export function SomethingUsefulProvider({ getService }) { const log = getService('log'); class SomethingUseful { doSomething() { } } return new SomethingUseful(); }
-
Re-export your provider from
services/index.js
-
Import it into
src/functional/config.js
and add it to the services config:import { SomethingUsefulProvider } from './services'; export default function () { return { // … truncated ... services: { somethingUseful: SomethingUsefulProvider } } }
PageObjects
editThe purpose for each PageObject is pretty self-explanatory. The visualize PageObject provides helpers for interacting with the visualize app, dashboard is the same for the dashboard app, and so on.
One exception is the "common" PageObject. A holdover from the intern implementation, the common PageObject is a collection of helpers useful across pages. Now that we have shareable services, and those services can be shared with other FunctionalTestRunner
configurations, we will continue to move functionality out of the common PageObject and into services.
Please add new methods to existing or new services rather than further expanding the CommonPage class.
Gotchas
editRemember that you can’t run an individual test in the file (it
block) because the whole describe
needs to be run in order. There should only be one top level describe
in a file.
Functional Test Timing
editAnother important gotcha is writing stable tests by being mindful of timing. All methods on remote
run asynchronously. It’s better to write interactions that wait for changes on the UI to appear before moving onto the next step.
For example, rather than writing an interaction that simply clicks a button, write an interaction with the a higher-level purpose in mind:
Bad example: PageObjects.app.clickButton()
class AppPage { // what can people who call this method expect from the // UI after the promise resolves? Since the reaction to most // clicks is asynchronous the behavior is dependant on timing // and likely to cause test that fail unexpectedly async clickButton () { await testSubjects.click(‘menuButton’); } }
Good example: PageObjects.app.openMenu()
class AppPage { // unlike `clickButton()`, callers of `openMenu()` know // the state that the UI will be in before they move on to // the next step async openMenu () { await testSubjects.click(‘menuButton’); await testSubjects.exists(‘menu’); } }
Writing in this way will ensure your test timings are not flaky or based on assumptions about UI updates after interactions.
Debugging
editFrom the command line run:
node --debug-brk --inspect scripts/functional_test_runner
This prints out a URL that you can visit in Chrome and debug your functional tests in the browser.
You can also see additional logs in the terminal by running the FunctionalTestRunner
with the --debug
or --verbose
flag. Add more logs with statements in your tests like
// load the log service const log = getService(‘log’); // log.debug only writes when using the `--debug` or `--verbose` flag. log.debug(‘done clicking menu’);
MacOS testing performance tip
editmacOS users on a machine with a discrete graphics card may see significant speedups (up to 2x) when running tests by changing your terminal emulator’s GPU settings. In iTerm2: * Open Preferences (Command + ,) * In the General tab, under the "Magic" section, ensure "GPU rendering" is checked * Open "Advanced GPU Settings…" * Uncheck the "Prefer integrated to discrete GPU" option * Restart iTerm
Unit testing frameworks
editKibana is migrating unit testing from Mocha
to Jest
. Legacy unit tests
still exist in Mocha but all new unit tests should be written in Jest.
Mocha (legacy)
editMocha tests are contained in __tests__
directories.
Running Mocha Unit Tests
yarn test:mocha
Jest
editJest tests are stored in the same directory as source code files with the .test.{js,mjs,ts,tsx}
suffix.
Running Jest Unit Tests
yarn test:jest
Writing Jest Unit Tests
editIn order to write those tests there are two main things you need to be aware of.
The first one is the different between jest.mock
and jest.doMock
and the second one our jest mocks file pattern
. As we are running js
and ts
test files with babel-jest
both techniques are needed
specially for the tests implemented on Typescript in order to benefit from the
auto-inference types feature.
Jest.mock vs Jest.doMock
editBoth methods are essentially the same on their roots however the jest.mock
calls will get hoisted to the top of the file and can only reference variables
prefixed with mock
. On the other hand, jest.doMock
won’t be hoisted and can
reference pretty much any variable we want, however we have to assure those referenced
variables are instantiated at the time we need them which lead us to the next
section where we’ll talk about our jest mock files pattern.
Jest Mock Files Pattern
editSpecially on typescript it is pretty common to have in unit tests
jest.doMock
calls which reference for example imported types. Any error
will thrown from doing that however the test will fail. The reason behind that
is because despite the jest.doMock
isn’t being hoisted by babel-jest
the
import with the types we are referencing will be hoisted to the top and at the
time we’ll call the function that variable would not be defined.
In order to prevent that we develop a protocol that should be followed:
-
Each module could provide a standard mock in
mymodule.mock.ts
in case there are other tests that could benefit from using definitions here. This file would not have anyjest.mock
calls, just dummy objects. -
Each test defines its mocks in
mymodule.test.mocks.ts
. This file could import relevant mocks from the generalised module’s mocks file(*.mock.ts)
and calljest.mock
for each of them. If there is any relevant dummy mock objects to generalise (and to be used by other tests), the dummy objects could be defined directly on this file. -
Each test would import its mocks from the test mocks
file mymodule.test.mocks.ts.
mymodule.test.ts
has an import like:import * as Mocks from './mymodule.test.mocks'
,import { mockX } from './mymodule.test.mocks'
or justimport './mymodule.test.mocks'
if there isn’t anything exported to be used.
Debugging Unit Tests
editThe standard yarn test
task runs several sub tasks and can take
several minutes to complete, making debugging failures pretty painful.
In order to ease the pain specialized tasks provide alternate methods
for running the tests.
You could also add the --debug
option so that node
is run using
the --debug-brk
flag. You’ll need to connect a remote debugger such
as node-inspector
to proceed in this mode.
node scripts/mocha --debug <file>
With yarn test:karma
, you can run only the browser tests. Coverage
reports are available for browser tests by running
yarn test:coverage
. You can find the results under the coverage/
directory that will be created upon completion.
yarn test:karma
Using yarn test:karma:debug
initializes an environment for debugging
the browser tests. Includes an dedicated instance of the Kibana server
for building the test bundle, and a karma server. When running this task
the build is optimized for the first time and then a karma-owned
instance of the browser is opened. Click the "debug
" button to open a
new tab that executes the unit tests.
yarn test:karma:debug
In the screenshot below, you’ll notice the URL is
localhost:9876/debug.html
. You can append a grep
query parameter
to this URL and set it to a string value which will be used to exclude
tests which don’t match. For example, if you changed the URL to
localhost:9876/debug.html?query=my test
and then refreshed the
browser, you’d only see tests run which contain "my test
" in the test
description.
Unit Testing Plugins
editThis should work super if you’re using the Kibana plugin generator. If you’re not using the generator, well, you’re on your own. We suggest you look at how the generator works.
To run the tests for just your particular plugin run the following command from your plugin:
yarn test:mocha yarn test:karma:debug # remove the debug flag to run them once and close
Automated Accessibility Testing
editTo run the tests locally:
-
In one terminal window run
node scripts/functional_tests_server --config test/accessibility/config.ts
-
In another terminal window run
node scripts/functional_test_runner.js --config test/accessibility/config.ts
To run the x-pack tests, swap the config file out for
x-pack/test/accessibility/config.ts
.
After the server is up, you can go to this instance of Kibana at
localhost:5620
.
The testing is done using axe. The same thing that runs in CI, can be run locally using their browser plugins: