caipivara
10/28/2014 - 11:57 PM
Android CI
Android CI
#Android and CI and Gradle (A How-To)
There are tech stacks in this world that make it dead simple to integrate a <abbr title="Continuous Integration">CI</abbr> build system. <br>
The Android platform is not one of them.
Although Gradle is getting better, it's still a bit non-deterministic, and some of the fixes you'll need will start to feel more like black magic than any sort of programming.
But fear not! It can be done!
Before we embark on our journey, you'll need a few things to run locally:
1. A (working) Gradle build
2. Automated tests (JUnit, Espresso, etc.)
If you don't have Gradle set up for your build system, it is highly recommend that you move your projects over. Android Studio has a built-in migration tool, and the [Android Dev Tools website](http://tools.android.com/tech-docs/new-build-system/intellij_to_gradle) has an excellent guide on how to migrate over to the Gradle build system, whether you're on Maven, Ant, or some unholy combination of all three.
A very general example of a `build.gradle` file follows:
```gradle
//build.gradle in /app
apply plugin: 'com.android.application'
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'com.android.tools.build:gradle:0.13.2'
}
}
//See note below
task wrapper(type: Wrapper) {
gradleVersion = '2.1'
}
android {
compileSdkVersion 19
buildToolsVersion "20.0.0"
defaultConfig {
applicationId "com.example.originate"
minSdkVersion 14
targetSdkVersion 19
versionCode 1
versionName "1.0"
testApplicationId "com.example.originate.tests"
testInstrumentationRunner "android.test.InstrumentationTestRunner"
}
buildTypes {
debug {
debuggable true
}
release {
debuggable false
}
}
}
dependencies {
compile project(':libProject')
compile com.android.support:support-v4:21.0.+
}
```
*(**NOTE**: the Gradle Wrapper task isn't strictly necessary, but a highly recommended way of ensuring you always know what version of Gradle you're using - both for futureproofing and for regressions)*
Check out the [Android Developers website](http://developer.android.com/sdk/installing/studio-build.html) for some good explanations and samples.
##Choose your weapon
Personally, I'm a fan of [CircleCI](circleci.com). They sport a clean, easy-to-use interface, support more languages than you could possibly care about. Plus, they are [free for open source Github projects!](http://blog.circleci.com/a-step-into-open-source/)<br>
(Other options include [TravisCI](https://travis-ci.org/), [Jenkins](http://jenkins-ci.org/), and [Bamboo](https://www.atlassian.com/software/bamboo))
In this guide, we'll be using CircleCI, but these instructions should translate readily to TravisCI.
##Configure all the things!
In order to use CircleCI to build/test your Android library, there's some configuration necessary.
Below are some snippets of some of the basis configurations you might want/do. About half of this comes from the [CircleCI docs](https://circleci.com/docs/android) and half of it comes from my blood, sweat, and tears.
At the end of this section, I'll include a complete `circle.yml` file.
###Machine
First, the code:
```yml
machine:
environment:
ANDROID_HOME: /home/ubuntu/android
java:
version: oraclejdk6
```
1. The setting of the `ANDROID_HOME` environment variable is necessary for the Android SDKs to function properly. It'll also be useful for booting up the emulator in later steps.
2. Although setting the JDK version isn't strictly necessary, it's nice to ensure that it doesn't change behind-the-scenes and possibly surprise-bork your build.
###Dependencies + Caching
```yml
dependencies:
cache_directories:
- ~/.android
- ~/android
override:
- (source scripts/environmentSetup.sh && getAndroidSDK)
```
1. By default, CircleCI will cache nothing. You might think this a non-issue right now, but you'll reconsider when each build takes 10+ minutes to inform you that you dropped a semicolon in your log statement.
<br>By caching `~/.android` and `~/android`, you can shave precious minutes off of your build time.
2. Android provides us with a nifty command-line utility called...`android` (inventive!). We can use this in little Bash script that we'll write in just a second. For now, just know that `scripts/environmentSetup.sh` can be whatever you want, as can the Bash function `getAndroidSDK`.
####Bash Scripts - a Jaunt into the CLI
Gradle is good at a lot of things, but it isn't yet a complete build system. Sometimes, you just need some good ol'fashioned bash scripting.
In this round, we'll download Android API 19 (Android 4.4 Jelly Bean) and create a hardware-accelerated Android AVD (Android Virtual Device - aka "emulator) image.
Note: If `android` commands confuse/scare you, check out the [`android` documentation](http://developer.android.com/tools/help/android.html).
```bash
#!/bin/bash
# Fix the CircleCI path
function getAndroidSDK(){
export PATH="$ANDROID_HOME/platform-tools:$ANDROID_HOME/tools:$PATH"
DEPS="$ANDROID_HOME/installed-dependencies"
if [ ! -e $DEPS ]; then
cp -r /usr/local/android-sdk-linux $ANDROID_HOME &&
echo y | android update sdk -u -a -t android-19 &&
echo y | android update sdk -u -a -t platform-tools &&
echo y | android update sdk -u -a -t build-tools-20.0.0 &&
echo y | android update sdk -u -a -t sys-img-x86-android-19 &&
#echo y | android update sdk -u -a -t addon-google_apis-google-19 &&
echo no | android create avd -n testAVD -f -t android-19 --abi default/x86 &&
touch $DEPS
fi
}
```
1. The `export PATH` line is to ensure we have access to all of the Android CLI tools we'll need later in the script.
2. The `DEPS=...` is used in the `if/then` block to determine if CircleCI has already provided us with cached dependencies. If so, there's no to download anything!
3. Note that we're explicitly requesting the x86 version of the Android 19 emulator image (`sys-img-x86-android-19`). The emulator is notoriously slow, and we should use the hardware-accelerated version if at all possible.
4. We create the AVD with the line `android create avd ...`, with a `target` of Android 19 and a name of `testAVD`.
5. If you don't need the Google APIs (e.g., Maps, Play Store, etc.), you don't need to download `addon-google_apis-google-19` - hence why it's commented out!
**CAVEAT** - Because of the way this caching works, if you ever change which version of Android you compile/run against, you need to click the "Rebuild & Clear Cache" button in CircleCI. If you don't, you'll never actually start compiling against the new SDK. You have been warned.
###You shall not pass! (until your tests have run)
This section will vary greatly depending on your testing setup, so, moreso than with the rest of this post, YMMV. <br>This section is assuming you're using a plain vanilla Android JUnit test suite.
```yml
test:
pre:
- $ANDROID_HOME/tools/emulator -avd testAVD -no-skin -no-audio -no-window:
background: true
- (./gradlew assembleDebug):
timeout: 1200
- (./gradlew assembleDebugTest):
timeout: 1200
- (source scripts/environmentSetup.sh && waitForAVD)
override:
- (./gradlew connectedAndroidTest)
```
1. The `$ANDROID_HOME/tools/emulator` starts a "headless" emulator - namely the one we just created. <br>
a. Running the emulator from the terminal is a blocking command. That's why we are setting the `background: true` attribute on the emulator command. Without this, we would have to wait anywhere between 2-7 minutes for the emulator to start and THEN build the APK, etc.
This way, we kick off the emulator and can get back to building.
2. The two subsequent `./gradlew` commands use the [Gradle wrapper](http://www.gradle.org/docs/current/userguide/gradle_wrapper.html) (**gradle** +<b>w</b>rapper) to build the code from your `/app` and `androidTest` directories, respectively.
3. See below for `environmentSetup.sh` Part II. Essentially, after building both the app and the test suite, we cannot continue without the emulator being ready. And so we wait.
4. Once the emulator is up and running, we run `gradlew connectedAndroidTest`, which, as its name suggests, run the tests on the connected Android device. If you're using Espresso or other test libraries, those commands would go here. <br>
4a. The CircleCI Android docs say that the "standard" way to run your tests is through ADB - ignore them. Gradle is the future and it elides all of those thorny problems that ADB tests have.
####Bash Round 2
As mentioned above, after Gradle has finished building your app and test suite, you'll kind of need the emulator to...y'know...run your tests.
This script relies on the currently-booting AVD's `init.svc.bootanim` property, which essentially tells us whether the boot animation has finished. Sometimes, it seems like it'll go on forever...<br>
<br><sup>*will the madness never stop?!*</sup>
This snippet can go in the same file as your previous bash script - in that case, you only need one `#!/bin/bash` - at the top of your file.
```bash
#!/bin/bash
function waitAVD {
(
local bootanim=""
export PATH=$(dirname $(dirname $(which android)))/platform-tools:$PATH
until [[ "$bootanim" =~ "stopped" ]]; do
sleep 5
bootanim=$(adb -e shell getprop init.svc.bootanim 2>&1)
echo "emulator status=$bootanim"
done
)
}
```
Note: This script was adapted from [this busy-wait script](http://blog.crowdint.com/2013/05/17/android-builds-on-travis-ci-with-maven.html).
###Results
By default, CircleCI will be fairly vague regarding your tests' successes and/or failures. You'll have to go hunting through the very ~~chatty~~ verbose Gradle loggings in order to determine exactly which tests failed. Fortunately, there's a better way - thanks to Gradle!
When you run `gradlew connectedAndroidTests`, Gradle, when finished, will create a folder called `/build/outputs/reports/**testFolderName**/connected` in whichever folder you have a `build.gradle` script in. <br>
So, for example, if your repo was in `~/username/awesomerepo`, with a local library in `awesome_repo/lib` and an app in `/awesome_repo/app`, the Gradle test artifacts should be in `/awesome_repo/app/build/outputs/reports/**testFolderName**/connected`.
In this directory, you'll find a little website that Gradle has generated, showing you which test packages and specific tests passed/failed.
If you like, you can tell CircleCI to grab this by placing the following at the top of your `circle.yml` file:
```yml
general:
artifacts:
-/home/ubuntu/**repo_name**/build/outputs/reports/**testFolderName**/connected
```
You can then peruse your overwhelming success under the **Artifacts** tab for your CircleCI build - just click on `index.html`.
It should pull up something like this:
##Security, Signing, and Keystores
The astute among you will notice that I haven't gone much into the process of signing an Android app. This is mainly for the reason that people trying to set up APK signing fall into 2 categories - Simple and Enterprise.
**Enterprise:** If you're programming Android for a company, you probably have some protocol regarding where your keystores/passwords can and cannot live - so a general guide such as this won't be much help for you.
**Simple:** If you're not Enterprise and you're not currently wearing a tinfoil hat, your security protocol is probably a little more lax.
In either case, [Google and StackOverflow are your friends](http://stackoverflow.com/questions/18328730/how-to-create-a-release-signed-apk-file-using-gradle#comment27871848_18329835).
My final word of advice is that CircleCI can encrypt things like keystore passphrases - stuff you might consider passing in plain-text in your buildscript files. Check out [CircleCI's Environment Variables doc](https://circleci.com/docs/environment-variables).
##Finally,
Go into your CircleCI settings, add a hook for your Github repo, and then do a `git push origin branchName`. If the Gradle Gods have smiled upon you, Circle should detect your config files and start building and testing!
Depending on your test suite, tests can take as little as a few minutes or as much as a half-hour to run. Try not to [slack off](http://xkcd.com/303/) in the meanwhile, but rejoice in having some solid continuous integration!
Stay tuned for a future blog post about using CircleCI to automagically deploy to MavenCentral!
##Flipping to the back of the book...
Below is the full `circle.yml` as well as `environmentSetup.sh` for your viewing/copying pleasure:
```yml
# Build configuration file for Circle CI
# needs to be named `circle.yml` and should be in the top level dir of the repo
general:
artifacts:
-/home/ubuntu/**repo_name**/build/outputs/reports/**testFolderName**/connected
machine:
environment:
ANDROID_HOME: /home/ubuntu/android
java:
version: oraclejdk6
dependencies:
cache_directories:
- ~/.android
- ~/android
override:
- (echo "Downloading Android SDK v19 now!")
- (source scripts/environmentSetup.sh && getAndroidSDK)
test:
pre:
- $ANDROID_HOME/tools/emulator -avd testAVD -no-skin -no-audio -no-window:
background: true
- (./gradlew assembleDebug):
timeout: 1200
- (./gradlew assembleDebugTest):
timeout: 1200
- (source scripts/environmentSetup.sh && waitForAVD)
override:
- (echo "Running JUnit tests!")
- (./gradlew connectedAndroidTest)
```
And the accompanying shell scripts:
```bash
#!/bin/bash
# Fix the CircleCI path
function getAndroidSDK(){
export PATH="$ANDROID_HOME/platform-tools:$ANDROID_HOME/tools:$PATH"
DEPS="$ANDROID_HOME/installed-dependencies"
if [ ! -e $DEPS ]; then
cp -r /usr/local/android-sdk-linux $ANDROID_HOME &&
echo y | android update sdk -u -a -t android-19 &&
echo y | android update sdk -u -a -t platform-tools &&
echo y | android update sdk -u -a -t build-tools-20.0.0 &&
echo y | android update sdk -u -a -t sys-img-x86-android-19 &&
#echo y | android update sdk -u -a -t addon-google_apis-google-18 &&
echo no | android create avd -n testAVD -f -t android-19 --abi default/x86 &&
touch $DEPS
fi
}
function waitForAVD {
(
local bootanim=""
export PATH=$(dirname $(dirname $(which android)))/platform-tools:$PATH
until [[ "$bootanim" =~ "stopped" ]]; do
sleep 5
bootanim=$(adb -e shell getprop init.svc.bootanim 2>&1)
echo "emulator status=$bootanim"
done
)
}
```
##References
- [Android Dev Tools - Migrating to Gradle](http://tools.android.com/tech-docs/new-build-system/intellij_to_gradle)
- [Android-Gradle Plugin User Guide](http://tools.android.com/tech-docs/new-build-system/user-guide)
- [Android Developers Gradle Project walkthrough](http://developer.android.com/sdk/installing/studio-build.html)
- [`android` CLI tool](http://developer.android.com/tools/help/android.html)
- [Gradle Wrapper docs](http://www.gradle.org/docs/current/userguide/gradle_wrapper.html)
- [CircleCI Android Docs](https://circleci.com/docs/android)
- [CircleCI background processes](https://circleci.com/docs/background-process)
- [`circle.yml` dissected](https://circleci.com/docs/configuration)
- [StackOverflow.com](http://stackoverflow.com/)