Testing on Firebase Test Lab

How to run tests on Firebase Test Lab

Firebase Test Lab provides a cloud-based infrastructure for testing Android and iOS apps on various devices and configurations. You can test your apps on Firebase Test Lab as part of the Codemagic build workflow provided you have set up a Firebase project.

To set up testing on Firebase Test Lab in Codemagic, follow these steps.

Firebase Test Lab prerequisites

You will need to set up the following in the Firebase console.

  1. Create a Firebase project.
  2. Create a service account with Editor permissions and download the JSON key file so you can authenticate with Firebase Test Lab during the build. See how to create a service account here.
  3. Enable the Cloud Tools Results API.

Please follow the guides in the Firebase Test Lab documentation to set up a project.

Configure testing in Firebase Test Lab in codemagic.yaml

To access the Firebase project from Codemagic, add the service account JSON key file and the Firebase project name as environment variables in the Environment variables section in Codemagic UI (either in Application or Team variables). Note that you should copy-paste the contents of the JSON key file. Save the variables to the same group. Select secure to encrypt the values.

Then import the group to codemagic.yaml this way:

environment:
  groups:
    - firebase_credentials # <-- (Includes: GCLOUD_KEY_FILE - service account JSON key file, FIREBASE_PROJECT - your Firebase Project ID)

Then, add the scripts to run tests on the preferred platform and device. The testing step should come after build scripts.

Note that Codemagic machines come with gcloud CLI tools preinstalled. Refer to CLI documentation for Android and CLI documentation for iOS for more details.

 - name: Run Firebase Test Lab tests
   script: |
     set -ex
     echo $GCLOUD_KEY_FILE > ./gcloud_key_file.json
     gcloud auth activate-service-account --key-file=gcloud_key_file.json

     gcloud --quiet config set project $FIREBASE_PROJECT     

Android

gcloud firebase test android run \
  --type instrumentation \
  --app your-app.apk \
  --test your-app-test.apk \
  --device model=TestDevice1,version=AndroidVersion1  \
  --device model=TestDevice2,version=AndroidVersion2  \
  --environment-variables coverage=true,coverageFile="/sdcard/coverage.ec" \
  --directories-to-pull /sdcard
  --timeout 3m

iOS

Package your application and prepare it for upload to Firebase Test Lab as described here.

Use the generated MyTests.zip to start testing:

  gcloud firebase test ios run --test PATH/TO/MyTests.zip \
    --device model=MODEL_ID_1,version=VERSION_ID_1,locale=LOCALE_1,orientation=ORIENTATION_1 \
    --device model=MODEL_ID_2,version=VERSION_ID_2,locale=LOCALE_2,orientation=ORIENTATION_2
    --timeout 3m

Check out this sample Flutter project and the relevant codemagic.yaml file for setting up integration tests in Firebase.