When developing new features or fixing issues for our applications, we want to integrate them along with changes from every other developer, and test them as quickly as possible. We also want to make sure these changes conform to specifications and standards, avoid any known security issues and prevent them from producing unexpected results, such as (unintentionally) changing existing functionality. As developers we love automation and so we created a system of Continuous Integration and Continuous Delivery/Continuous Deployment for most of our clients’ websites and web applications. Tailored to different types of projects we now have several automated tests that test certain types of functionality, analyse our code, check for standards, etcetera.
For apps, however, we were still testing and deploying manually. With an increasing number of apps and increasing complexity, it was time for us to change this.
Our situation
Even though mobile app development only constitutes about 10% of our development, we develop and maintain several apps. Some small (10-100 monthly users), some large (10.000+ monthly users), but in all cases our clients expect a stable app without regression after each update. Some of these apps are native apps, most of them are developed using React Native.
As a digital agency we have a bunch of developers working simultaneously on multiple projects for a range of clients. Generally testing their work on multiple devices, usually it’s another person that then tests and reviews the actual functionality before it is delivered to the client. Most of our clients run tests of their own as part of their acceptance procedure. Once accepted, the app will then be updated and pushed to end-users.
Nearly all of our apps we develop have an API connection. Our clients have some form of separation of systems for testing/accepting and production, generally following DTAP standards. Strict separation of data is of course very important: data from an acceptance environment should never influence data on a live environment. And before deploying new functionality to production (i.e. live users) we want to be able to test and accept it.
Existing systems
Build tools
Both Xcode and Android Studio come with built-in command-line tools to build apps. Xcode has ‘xcodebuild’, Android Studio has ‘gradle’. Xcode also has a tool called ‘altool’ for uploading apps to the store, Android Studio has no such tool, and only offers an API.
Fastlane
Fastlane is a ready-to-use system to build and test Android and iOS apps. It works by reading from a specific ‘Fastlane’ file that contains information on how to build the app.
The issue we ran into with Fastlane, however, is that there is no easy support for ‘build flavours’ with different application identifiers. This means for instance, there’s no simple way to separate Alpha, Beta and Production apps to properly test changes.
Xcode cloud (beta)
Apple recently came with a solution for building apps in the cloud, with a focus on automation for CI/CD systems: Xcode cloud. Even though we signed up for early access, unfortunately it wasn’t yet available to us when we were creating our solution. Perhaps in the future it might grant some of our wishes though, on the Apple-side of things.
Our setup
After extensive research and testing, we came up with an extension of our Gitlab CI/CD system. Now each and every code change (merge) starts a pipeline that checks, builds, tests and deploys our apps to our customer’s App Store / Play Store accounts. Here they utilize Apple Testflight and Google Play ‘testing tracks’ in which several versions of the app can reside at the same time.
Let’s discuss our solution in detail.
Separate apps for separate environments (Alpha, Beta and Live)
First, we need to be able to test according to the DTAP principle. We do our development locally, using the build systems from both Xcode and Android Studio, and Docker for running the API side of things.
For Testing, Acceptance and Production our app needs to talk to separate environments. Both the App Store and Play Store utilize the above mentioned test tracks. This enables the release of updates of the app to a certain test group before releasing it to actual live users. The app is generally built once and is then ‘promoted’ to the next track, eventually ending up as the production version. Since the environments are ‘baked’ into the build, we can not simply make an app that connects to one environment and then push that to Live. We see no other way than having to build three separate versions of the app which we call Alpha (Testing), Beta (Acceptance) and Live (Production).
Git integration
In order to keep our workflow somewhat manageable, we make use of the Gitflow method. This means we make use of the git branching and tagging system to separate versions of an application: feature branches for (local) development, develop branch for testing, master branch for acceptance, and tagging for production releases.
Conversely, we use the same branching/tagging method for our app releases: develop merges build an Alpha version for the app, master merges a Beta version, and tagging releases a Live app to production.
Versioning
Every change in git (push or merge to certain branches) triggers a new pipeline, every pipeline triggers a new build and results in a new release. The release will be versioned with the pipeline number. Switching between build versions is very easy, so detecting whenever a bug has been introduced or pinpointing a specific change in code is similarly easy.
Testing groups
Leveraging the testing tracks in the stores we can create separate groups of users to test separate app versions. At Unc Inc for instance, we generally have our developers test the ‘internal track’, meaning they get an updated version of every build. We then employ a separate internal group that tests the app after initial testing has been completed. Our clients usually test the Beta version of the app, in some cases also with separate groups for internal and external testing. This setup, however, depends on our client’s wishes and the type of app, but is very flexible and can be used in many different ways.
Xcode project setup
Xcode supports separate configuration sets per project. By default this is used for differentiating between a debug and a release build but we can also add our own. Each project setting can in turn be differentiated per configuration. Wecan for example set an app Bundle ID per configuration, meaning we can build a completely different app by switching configurations using the `-configuration` flag of `xcodebuild`.
Android Gradle plugin
Android Studio also supports separate configurations using Build Variants.
In order to make it easier to build from CI we created a Gradle library as a helper tool. This library makes it possible to build with version numbers, application identifier and key signing from the command line as Gradle Properties. By using this we wouldn’t need to replace values in the build.gradle file during the CI pipeline. This also allows us to store the keys in a secure location accessible by CI. The repository with the library is public and can be found at: https://github.com/uncinc/android-ci
Gitlab CI/CD
All of our client’s projects are hosted on our self-hosted Gitlab instance. Given its very good CI/CD system, we use this for most of our integrations and deployments, so in order to integrate with our workflow, we needed to make it work for app deployments as well.
Our pipeline looks a bit like this:
- Start with linting, code checks, etc.
- Build the correct app (Alpha, Beta, Live)
- Test the app
- Verify and upload the app
iOS build step:
build_ios_project:
stage: building
dependencies:
- yarn
tags:
- ios
script:
- ln -s ~/app-envs/$IOS_SCHEME/.env .env
- ln -s ~/app-envs/$IOS_SCHEME/GoogleService-Info.plist ios/GoogleService-Info.plist
- yarn run pre-build
- cd ios
- agvtool new-version -all "$CI_COMMIT_TAG"
- mkdir -p build
- arch -x86_64 pod install
- xcodebuild -workspace $IOS_XCODE_WS -scheme $IOS_SCHEME -sdk iphoneos -configuration Release clean | xcpretty
- xcodebuild -workspace $IOS_XCODE_WS -scheme $IOS_SCHEME -sdk iphoneos -configuration Release -allowProvisioningUpdates archive -archivePath "./build/$IOS_SCHEME.xcarchive" | xcpretty | tail -n 200
- xcodebuild -exportArchive -archivePath "./build/$IOS_SCHEME.xcarchive" -exportOptionsPlist exportOptions.plist -exportPath "./build" -allowProvisioningUpdates | xcpretty | tail -n 200
artifacts:
paths:
- ios/build/$IOS_SCHEME.ipa
- ios/build/*/dSYMs/*.app.dSYM
expire_in: 1 day
only:
refs:
- tags
variables:
- $IOS_XCODE_WS != null
Android build step:
build_android_project:
stage: building
dependencies:
- yarn
tags:
- apps
script:
- ln -s ~/app-envs/$IOS_SCHEME/.env .env
- ln -s ~/app-envs/$IOS_SCHEME/google-services.json android/app/google-services.json
- yarn run pre-build
- cd android
- ./gradlew bundleProd -Pandroidci.signingProperties=$ANDROID_SIGNING_PROPERTIES -Pandroidci.versionCode=$CI_PIPELINE_ID
artifacts:
paths:
- android/app/build/generated/sourcemaps/react/prod/release/index.android.bundle.map
- android/app/build/outputs/bundle/prodRelease/app-prod-release.aab
expire_in: 1 day
only:
refs:
- tags
variables:
- $ANDROID_SIGNING_PROPERTIES != null
Shortcomings
There are a couple of things that break our workflow. Some of the issues we encounter(ed):
Xcode version issues
Apple has a track record of wanting developers to update fast and often. If you strictly keep to the Apple ecosystem this is generally not an issue. But especially when developing React Native apps, the project settings are generated by CocoaPods, and they need to play catch-up.
We end up running older versions of Xcode, at least one or two minor versions behind, sometimes even a full major version. An annoying downside of this is that iOS is generally automatically updated to the latest version (with good reason), but in order to debug apps on a physical device, you will also need the latest version of Xcode. Advice is to turn off automatic updates on your test devices, or, better yet, have multiple devices with different versions. At the time of writing, Xcode 13 beta just came out, Xcode 12.5 is the latest, and Xcode 12.3 is the version that we use.
Another downside of this is that our current CI only supports one version of Xcode (and CocoaPods) at a time, which means all your apps need to use about the same version of packages. Updating React might require a newer version of CocoaPods, so if you want to update one app, you need to update them all. Running multiple versions on a single system is possible, but an easier solution would be setting up multiple CI runners on separate devices.
CocoaPods version issues
At this moment there are two CocoaPods versions that we use: 1.9.8 and 1.10.0. The latest version builds just fine, but seems to create invalid binaries, giving the dreaded error: `error: archive at path '*.xcarchive' is malformed`. Building with 1.9.8 works just fine, except we need to remove all occurrences of `VALID_ARCHS` configuration options to prevent the following error: `Pods-appName-artifacts.sh: line 85: ARCHS: unbound variable`.
Apple Silicon architecture issues
In order to speed up our building process, we immediately ordered a new Mac Mini M1. Indeed, compile times were about 7 times faster compared to our older Mac Mini Intel i7. But with a new architecture come new problems. We ran into quite a few problems with dynamic library linking, especially when binary libraries are involved. We ran into several ‘unknown symbol’ errors.
Our current solution is to run most of the tools in x86_64 mode, especially when developing locally. This means running `arch -x86_64 pod install` and running Xcode in Rosetta mode:
Android issues
I will deny ever writing this, but it appears that Android came out as the most developer-friendly environment to work with. Once we wrote the Gradle plugin to work with our CI, Android builds have always been the most stable ones. There are some minor issues with getting the simulator up-and-running on Apple Silicon platforms, but there are no issues so far with purely building apps.
Conclusion
For our given situation, no turn-key solutions for CI/CD integration for apps were available. Investing time in setting up a custom system was definitely worth the effort. Changes to apps are now available for testing within minutes, which both improve our workflow and our quality. Implementation is fairly easy and saves our developers a lot of manual building and releasing, gives our testers so much more overview on what features can be tested, and detecting issues with changes is much easier having a new release version per pipeline.