Compose Preview Screenshot Testing test drive

In one of our projects we converted Jetpack Compose Previews into screenshot tests to try out this essential part of UI testing strategy at Pale Blue using the Compose Preview Screenshot Testing tool.

While this approach offers great benefits for catching unintended UI changes, our journey with it has revealed both promising features and challenging limitations. Here's what we've learned along the way.

Setup

The setup process for screenshot testing follows the well-written documentation. However, what comes after the basic setup is where things get interesting.

Converting @Preview to Screenshot Tests

Our approach was to leverage our existing @Preview composables for screenshot tests. The idea seemed perfect - we already had carefully crafted previews that showcased our components in various states. Here's an example of one of our original previews:

@Preview
@Composable
fun PreviewSuccessPaymentComponent() {
    OurTheme() {
        ReadyScreen( // The composable of one of the screen in our app
          isSuccess = True,
          otherParameter = False,
        )
    }
}

We converted this into a dedicated screenshot test class:

class SuccessPaymentComponentPreviewsScreenshots {
    @Preview
    @Composable
    fun SuccessPaymentComponentPreview() = PreviewSuccessPaymentComponent()
}

The Cross-Platform Challenge

While this approach seemed promising initially, we quickly encountered a significant challenge: cross-platform consistency. The same tests generated different screenshots depending on the operating system they were run on. This discrepancy posed a major obstacle for our continuous integration pipeline. In the bug tracking, there are other people as well mentioning these cross-platform/color-rendering issues.

The threshold setting and color rendering differences

The screenshot testing tool does provide a threshold setting to accommodate minor pixel differences. However, we discovered that this feature wasn't sufficient for our needs.

The color rendering differences between operating systems were more substantial than what could be handled by reasonable threshold values. The color rendering inconsistency is particularly problematic because:

1. Setting too high a threshold risks missing actual UI regressions
2. Setting too low a threshold leads to false positives due to OS-specific rendering

Test Results Visualization

One helpful feature of the screenshot testing framework is its built-in results page. After running the tests, you can easily visualize everything in one place: the golden (reference) images alongside the newly generated screenshots, complete with a visual diff highlighting the differences and detailed pixel-by-pixel comparison metrics. This comprehensive view makes it much easier to understand why tests are failing and whether the differences are acceptable or indicate real issues.

CI/CD Integration

Another pleasant surprise, is how easy is to integrate screenshot testing with CI/CD. We've integrated screenshot testing into our GitHub Actions workflow. Here's our configuration:

on:
  push:
    branches: [ main ]
  pull_request:
concurrency:
  group: build-${{ github.ref }}
  cancel-in-progress: true

jobs:
  build:
    name: Build apk and run tests
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set Up JDK 17
        uses: actions/setup-java@v4
        with:
          java-version: 17
          distribution: temurin

      - name: Setup Gradle
        uses: gradle/actions/setup-gradle@v4
        with:
          validate-wrappers: true
          cache-cleanup: always

      - name: Run Screenshot Tests
        run: ./gradlew validateDebugScreenshotTest

Looking Forward

While we continue to use screenshot testing, we will probably have to adapt our approach. We're considering moving to server-side rendering for screenshots to ensure consistency. We've also might have to become more selective about which components we subject to screenshot testing, focusing on critical UI elements where visual regression would have the most impact.

We have found these official guidelines on screenshot testing to be valuable, although not giving us ready-made solutions. As the tooling continues to evolve, we hope to see better solutions for handling these platform-specific rendering differences.

Happy coding!

Source: View source