Fusing automated UI testing with scripts for effectively fuzzing Android apps

https://github.com/ecnusse/Kea2

Kea2 is an easy-to-use tool for fuzzing mobile apps. Its key novelty is able to fuse automated UI testing with scripts (usually written by human), thus empowering automated UI testing with human intelligence for effectively finding crashing bugs as well as non-crashing functional (logic) bugs.

Kea2 is currently built on top of Fastbot, an industrial-strength automated UI testing tool, and uiautomator2, an easy-to-use and stable Android automation library. Kea2 currently targets Android apps.

• Feature 1(查找稳定性问题): coming with the full capability of Fastbot for stress testing and finding stability problems (i.e., crashing bugs);

• Feature 2(自定义测试场景\事件序列\黑白名单\黑白控件¹): customizing testing scenarios when running Fastbot (e.g., testing specific app functionalities, executing specific event traces, entering specifc UI pages, reaching specific app states, blacklisting specific activities/UI widgets/UI regions) with the full capability and flexibility powered by python language and uiautomator2;

• Feature 3(支持断言机制²): supporting auto-assertions when running Fastbot, based on the idea of property-based testing inheritted from Kea, for finding logic bugs (i.e., non-crashing functional bugs).

  For Feature 2 and 3, Kea2 allows you to focus on what app functionalities to be tested. You do not need to worry about how to reach these app functionalities. Just let Fastbot help. As a result, your scripts are usually short, robust and easy to maintain, and the corresponding app functionalities are much more stress-tested!

The ability of the three features in Kea2

Kea2 currently works with:

• unittest as the testing framework to manage the scripts;
• uiautomator2 as the UI test driver;
• Fastbot as the backend automated UI testing tool.

In the future, Kea2 will be extended to support

• pytest, another popular python testing framework;
• Appium, Hypium (for HarmonyOS/Open Harmony);
• any other automated UI testing tools (not limited to Fastbot)

Running environment:

• support Windows, MacOS and Linux
• python 3.8+, Android 5.0+ (Android SDK installed)
• VPN closed (Features 2 and 3 required)

Install Kea2 by pip:

Find Kea2's options by running

Kea2 connects to and runs on Android devices. We recommend you to do a quick test to ensure that Kea2 is compatible with your devices.

1. Connect to a real Android device or an Android emulator (only one device is enough) and make sure you can see the connected device by running adb devices.

2. Run quicktest.py to test a sample app omninotes (released as omninotes.apk in Kea2's repository). The script quicktest.py will automatically install and test this sample app for a short time.

Initialize Kea2 under your preferred working directory:

This step is always needed if it is your first time to run Kea2.

Run the quick test:

If you can see the app omninotes is successfully running and tested, Kea2 works! Otherwise, please help file a bug report with the error message to us. Thank you!

Test your app with the full capability of Fastbot for stress testing and finding stability problems (i.e., crashing bugs);

To understand the meanings of the options, you can see our manual.

The usage is similar to the the original Fastbot's shell commands.

See more options by

When running any automated UI testing tools like Fastbot to test your apps, you may find that some specifc UI pages or functionalities are difficult to reach or cover. The reason is that Fastbot lacks knowledge of your apps. Fortunately, this is the strength of script testing. In Feature 2, Kea2 can support writing small scripts to guide Fastbot to explore wherever we want. You can also use such small scripts to block specific widgets during UI testing.

In Kea2, a script is composed of two elements:

• Precondition: When to execute the script.
• Interaction scenario: The interaction logic (specified in the script's test method) to reach where we want.

Assuming Privacy is a hard-to-reach UI page during automated UI testing. Kea2 can easily guide Fastbot to reach this page.

• By the decorator @precondition, we specify the precondition --- when we are at the Home page. In this case, the Home page is the entry page of the Privacy page and the Home page can be easily reached by Fastbot. Thus, the script will be activated when we are at Home page by checking whether a unique widget Home exists.
• In script's test method test_goToPrivacy, we specify the interaction logic (i.e., opening Drawer, clicking the option Setting and clicking Privacy) to guide Fastbot to reach the Privacy page.
• By the decorator @prob, we specify the probability (50% in this example) to do the guidance when we are at the Home page. As a result, Kea2 still allows Fastbot to explore other pages.

You can find the full example in script quicktest.py, and run this script with Fastbot by the command kea2 run:

Kea2 supports auto-assertions when running Fastbot for finding logic bugs (i.e., non-crashing bugs). To achieve this, you can add assertions in the scripts. When an assertion fails during automated UI testing, we find a likely functional bug.

In Feature 3, a script is composed of three elements:

• Precondition: When to execute the script.
• Interaction scenario: The interaction logic (specified in the script's test method).
• Assertion: The expected app behaviour.

In a social media app, message sending is a common feature. On the message sending page, the send button should always appears when the input box is not empty (i.e., has some message).

The expected behavior (the upper figure) and the buggy behavior (the lower figure).

For the preceding always-holding property, we can write the following script to validate the functional correctness: when there is an input_box widget on the message sending page, we can type any non-empty string text into the input box and assert send_button should always exists.

We use hypothesis to generate random texts.

You can run this example by using the similar command line in Feature 2.

You can find the user manual, which includes:

• Examples of using Kea2 on WeChat (in Chinese);
• How to define Kea2's scripts and use the decorators (e.g., @precondition、@prob、@max_tries);
• How to run Kea2 and Kea2's command line options
• How to find and understand Kea2's testing results
• How to whitelist or blacklist specific activities, UI widgets and UI regions during fuzzing

• Fastbot
• uiautomator2
• hypothesis

General and Practical Property-based Testing for Android Apps. ASE 2024. pdf

An Empirical Study of Functional Bugs in Android Apps. ISSTA 2023. pdf

Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning. ASE 2022. pdf

Guided, Stochastic Model-Based GUI Testing of Android Apps. ESEC/FSE 2017. pdf

Kea2 has been actively developed and maintained by the people in ecnusse:

• Xixian Liang (@XixianLiang)
• Bo Ma (@majuzi123)
• Chen Peng (@Drifterpc)
• Ting Su (@tingsu)

Zhendong Su, Yiheng Xiong, Xiangchen Shen, Mengqian Xu, Haiying Sun, Jingling Sun, Jue Wang have also been actively participated in this project and contributed a lot!

Kea2 has also received many valuable insights, advices, feedbacks and lessons shared by several industrial people from Bytedance (Zhao Zhang, Yuhui Su from the Fastbot team), OPay (Tiesong Liu), WeChat (Haochuan Lu, Yuetang Deng), Huawei, Xiaomi and etc. Kudos!

1. 不少UI自动化测试工具提供了“自定义事件序列”能力（如Fastbot 和AppCrawler），但在实际使用中存在不少问题，如自定义能力有限、使用不灵活等。此前不少Fastbot用户抱怨过其“自定义事件序列”在使用中的问题，如#209, #225, #286等。 ↩

2. 在UI自动化测试过程中支持自动断言是一个很重要的能力，但几乎没有测试工具提供这样的能力。我们注意到AppCrawler的开发者曾经希望提供一种断言机制，得到了用户的热切响应，不少用户从21年就开始催更，但始终未能实现。 ↩