# Running tests

> How to run tests with codemagic.yaml



Test scripts are added under `scripts` in the [overall architecture](../getting-started/yaml#template), before the build commands.

You can display test results visually in the build overview if you use an expanded form of the script in `codemagic.yaml`. Just include the `test_report` field with a glob pattern matching the test result file location. We support every test runner, including GoTest, RSpec, PHPUnit, Karma, PyTest, ESLint, Cucumber, ExUnit, Mocha, CargoTest, and JUnit. If your test runner can export [Junit XML](https://llg.cubic.org/docs/junit/) and `.JSON` for Flutter's `--machine` report test results, Codemagic can use it.

For instructions on testing your app on real devices in Firebase Test Lab, refer [here](./firebase-test-lab).


## Flutter unit tests

To run Flutter unit tests, simply add the `flutter test` command to your scripts section.



```yaml

scripts:
  - name: Unit tests
    script: | 
      mkdir -p test-results
      flutter test --machine > test-results/flutter.json
    test_report: test-results/flutter.json

```


**Tip:** you can display Flutter test results visually in the build overview if you use the expanded form of the script in codemagic.yaml.
Just include the `test_report` field with a glob pattern matching the test result file location:


## Flutter integration tests

The `integration_test` dependency allows you to run integration tests on a real device or emulator. Android application tests can be run on an Android emulator, iOS application tests can be run on an iOS simulator, and web application tests can be run on a web browser driver.

**Tip:** It is possible to generate machine readable output for integration tests using the `--machine` flag; hence the results can be displayed in the UI. Just include the `test_report` field with a glob pattern matching the test result file location:


```yaml

scripts:
  - name: Integration tests
    script: | 
      mkdir -p test-results 
      flutter -d emulator-5554 test --machine > test-results/flutter.json integration_test 
      # for iOS use: -d iPhone
    test_report: test-results/flutter.json

```



To run integration tests for web, it is possible to use `chromedriver`. Take note that for running tests on web, it is necessary to provide the `--driver` and `--target` arguments, and machine-readable output is unavailable.


```yaml

scripts:
  - name: Integration tests
    script: | 
      flutter config --enable-web
      chromedriver --port=4444 &
      flutter -d chrome drive --driver=test_driver/integration_driver.dart --target=integration_test/app_test.dart

```



### Running application tests on a mobile simulator/emulator





### Option: Android





> 
> Due to limitations by Apple silicon, the Apple Virtualization API doesn’t support nested virtualization required for Android emulators. To use Android emulators in macOS workflows, please use third-party service integrations such as emulator.wtf, Katalon, Firebase, or AWS Device Farm.
> 




For the Android emulator you can launch and run your tests as follows:


```yaml

scripts:
  - name: Emulator tests
    script: | 
      flutter config --enable-web
      # The ampersand is used to run the emulator in the background without blocking the next command:
      flutter emulators --launch emulator &
      # adb wait-for-device is used to wait for the emulator to finish loading:
      adb wait-for-device
      flutter -d emulator-5554 test integration_test 

```






### Option: iOS




You can launch the iOS simulator and run tests as follows:


```yaml

scripts:
  - name: Emulator tests
    script: | 
      flutter emulators --launch apple_ios_simulator
      flutter -d iPhone test integration_test

```



You can launch a specific iOS simulator and run tests on the simulator using ‘simctl’ which is a binary to interact with iOS simulators from the command line, as follows:


```yaml

scripts:
  - name: Emulator tests
    script: | 
      # this command will will shutdown the existing simulators to save on resources.
      xcrun simctl shutdown all
      #  
      # create new simulator with specified configuration, you can run 
      # 'xcrun simctl list' - to check the list of available simulator configurations
      TEST_DEVICE=$(xcrun simctl create test-device com.apple.CoreSimulator.SimDeviceType.iPhone-11 com.apple.CoreSimulator.SimRuntime.iOS-15-0)
      #
      # boot the newly created simulator
      xcrun simctl boot $TEST_DEVICE
      #
      # run the actual test command
      flutter -d $TEST_DEVICE test integration_test

```







### Running web application tests on a web browser driver





### Option: Chrome





```yaml

scripts:
  - name: 'Flutter integration test for web'
    script: | 
      chromedriver --port=4444 &
      flutter config --enable-web
      flutter drive --driver=test_driver/integration_driver.dart --target=integration_test/app_test.dart -d web-server --release --browser-name chrome

```






### Option: Safari





```yaml

scripts:
  - name: 'Flutter integration test for web'
    script: | 
      sudo safaridriver --enable
      safaridriver --port 4444 &
      flutter config --enable-web
      flutter drive --driver=test_driver/integration_driver.dart --target=integration_test/app_test.dart -d web-server --release --browser-name safari

```








## React Native Unit Tests using Jest

This basic example illustrates how to use Jest tests defined in the `package.json` file as follows:


```json

// package.json
"scripts": {
  "test": "jest"
},
"jest": {
  "preset": "jest-expo",
  "setupFiles": ["<rootDir>/testing/jest-setup.js"]
}

```



In the root directory of the project, create a new file named `jest.config.js` with the following content:


```javascript

module.exports = {
  preset: 'react-native',
  setupFilesAfterEnv: ['@testing-library/jest-native/extend-expect'],
}

```



The `preset` is used as a base for Jest’s configuration and should point to an npm module that has a `jest-preset.json` or `jest-preset.js` file at the root.

The `setUpFilesAfterEnv` specifies a list of paths to modules that run some code to configure or set up the testing framework before each test file in the suite is executed.

To execute the tests, use the following script in your `codemagic.yaml` file:


```yaml

scripts:
  - name: 'Flutter integration test for web'
    script: | 
      npm test
      # or: yarn test

```



In React Native, 3rd party modules are oftentimes published as **untranspiled**. Since all files inside `node_modules` are not transformed by default, Jest will not understand the code in these modules, resulting in syntax errors. To overcome this, you need to use `transformIgnorePatterns` to allow transpiling such modules.

In such cases, modify your `package.json` as follows:


```json

"jest": {
  "preset": "jest-expo",
  "transformIgnorePatterns": [
    "node_modules/(?!(jest-)?react-native|react-clone-referenced-element|@react-native-community|expo(nent)?|@expo(nent)?/.*|react-navigation|@react-navigation/.*|@unimodules/.*|unimodules|sentry-expo|native-base|@sentry/.*)"
  ]
}

```



## React Native Integration Tests using Appium and Emulator

[Appium](https://appium.io/) is an open-source testing automation framework for testing cross-platform and mobile applications. You can use **Appium** and **WebDriverIO** with React Native thanks to its out-of-the-box support.

Before running tests in Codemagic, you need to install and setup **WebDriverIO** in your project root directory. Run the following command **on your local machine**: provide input to a series of questions:


```bash

npx wdio config

```


After answering a series of questions, a file named `wdio.conf` will be generated inside the `tests` directory. Edit the content of that file as follows to enable WebDriverIO to work with Appium and run tests on Android Emulator:


```javascript

exports.config = {
  services: ['appium'],
  port: 4723,
  runner: 'local',
  specs: [
    './tests/specs/**/*.js'
  ],
  capabilities: [{
     maxInstances: 1,
     browserName: '',
     appiumVersion: '1.13.0',
     platformName: 'Android',
     platformVersion: '<emulator platform version>',    // Specify your emulator details
     deviceName: '<emulator name>',
     app: '<path to APK>',
     automationName: 'UiAutomator2'
  }],

  logLevel: 'trace',
  bail: 0,
  waitforTimeout: 10000,
  connectionRetryTimeout: 90000,
  connectionRetryCount: 3,
  framework: 'mocha',
  reporters: ['spec'],
  mochaOpts: {
    ui: 'bdd'
    timeout: 60000
  }
}

```


To execute the tests, use the following scripts in your `codemagic.yaml` file:



```yaml

    scripts:
      - name: Install npm dependencies    # Add Appium and WebDriverIO dependencies
        script: npm install && npm install -g appium && npm install --save webdriverio @wdio/cli

      - name: Launch emulator             # Insert before the build command
        script: | 
          react-native run-android  &
          adb wait-for-device
      - name: Launch Appium
        script: appium
      - name: Run WebDriver test suite
        script: npx wdio ./wdio.conf.js


  ...

```



## Native iOS

To execute the tests, use the following scripts in your `codemagic.yaml` file:

```yaml

  scripts:
    - name: iOS test
      script: | 
        xcode-project run-tests \
          --workspace MyAwesomeApp.xcworkspace \
          --scheme MyAwesomeApp \
          --device "iPhone 11"
    test_report: build/ios/test/*.xml

```



Please check [Codemagic CLI tools documentation](https://github.com/codemagic-ci-cd/cli-tools/blob/master/docs/xcode-project/run-tests.md#run-tests) to learn more about more optional arguments to `xcode-project run-tests`.

## Native macOS



> 
> **Note:** macOS UI Testing is only supported on Xcode 13 images and above as it requires System Integrity Protection (SIP) to be disabled in order to access the accessibility permissions. Older images with Xcode 12 and below do not have SIP disabled and are unsuitable for UI testing macOS apps.
> 



To execute the tests, use the following scripts in your `codemagic.yaml` file:


```yaml

  scripts:
    - name: macOS test
      script: | 
        xcode-project run-tests \
          --project MyAwesomeApp.xcodeproj \
          --scheme MyAwesomeApp \
          --sdk macosx \
          --test-xcargs "CODE_SIGNING_ALLOWED='no'" \
          --output-dir build/macos/test
      test_report: build/macos/test/*.xml

```



For macOS tests, no destination is specified. Please check [Codemagic CLI tools documentation](https://github.com/codemagic-ci-cd/cli-tools/blob/master/docs/xcode-project/run-tests.md#run-tests) to learn more about optional arguments to `xcode-project run-tests`.

## Native Android

For non-UI tests or unit tests:


```yaml

  scripts:
    - name: Test
      script: ./gradlew test
      test_report: app/build/test-results/**/*.xml

```



For UI tests (also known as instrumented tests):


```yaml

  scripts:
    - name: Launch emulator
      script: | 
        cd $ANDROID_HOME/tools
        emulator -avd emulator &
        adb wait-for-device
    - name: Test
      script: | 
        set -e
        ./gradlew connectedAndroidTest
        adb logcat -d > emulator.log
    test_report: app/build/outputs/androidTest-results/connected/*.xml

```



**Tip**: you can save the emulator log with the `adb logcat -d > emulator.log` command