Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Article

Android Build Times: The New Bottleneck

Android Build Best Practices

xkcd-code-is-compiling

© xkcd.com

As Android is becoming the platform of choice for everything from cell phones to home appliances to in-car management systems, the need to build and deploy faster has grown significantly.  Android device makers are under tremendous pressure to incorporate Google’s latest Android version – along with their own value add – in order to be competitive. One of the big challenges to improving the velocity of adopting new Android releases is the lengthening time required to build the Android operating system, which includes the Linux kernel.

As this table shows, the Android Open Source Project (AOSP) code size is increasing in both KLOC and the time it takes to build with gmake.

Android Release Initial Release Date Build Version Lines of Code Build Duration 1
Jelly Bean July 2012 4.3.1 29,725,532 35m35s
KitKat October 2013 4.4.4 46,328,885 61m58s
Lollipop November 2014 5.1.1 58,937,746 90m51s
Marshmallow October 2015 6.0.0 45,532,137 62m49s
Nougat August 2016 7.0.0 58,743,987 85m00s

 

1 Builds performed with gmake –j 8 on 32-core hyper-threaded PC with 256GB RAM and a striped RAID0 disk array

The problem is getting progressively worse with each feature-enhanced release, making Android builds the new bottleneck.  Dependencies in an Android build are very well maintained, so it is already possible to run the build with very high levels of parallelism – on the local machine.  However, 85 minutes is just too long for any developer to wait even if they are lucky enough to have a 32-core computer for the build.  Multiply that hour and a half by the size of the development team and a year’s worth of development time quickly gets sacrificed to the ping pong table.

So much for “market responsiveness.”

Google has done a tremendous job with maintaining the dependencies but not everybody has ready access to a 32- or 64-core machine on which to fully exploit that effort.  That much hardware is still a $25,000 or greater investment.  That’s not a viable alternative for most organizations.

So, if the goal is get that build time down to less than ten minutes, saving thousands of man-hours a year, then we need to look beyond simple dependency management and local parallelism to find more speed.

Where is the extra speed?

If parallelism is built in and buying more and more hardware is not a viable way to improve build times, where can time be squeezed out of the process?  Despite the impression that Google has already done everything that can be done, there are multiple ways Android builds can be accelerated.  The following table explains further:

What It Does Speeds Build Times…
Distributed Builds Exploits the parallelism native to AOSP by distributing the build process across all available resources – even to the cloud … due to more cores available than on the local machine
Resource and Schedule Optimization Ensures the best possible use of the CPU resources available in order to complete the build as quickly as possible … because it uses resources in the most efficient manner possible
Dependency Optimization Intelligently ignores overspecifieddependencies and therefore run more things in parallel than might otherwise be possible. … because unneeded dependencies are ignored

 

(see sidebar)

Intelligent Cache/reuse Caching facilities that enable the reuse of work from other builds in the current build, rather than redoing that work from scratch … because it avoids doing work that does not have to be done
Build Analytics and Visualization Sophisticated visualization of build parallelization and dependencies …because it enables focusing effort on actual bottlenecks rather than guessing where they are

None of these capabilities are native to Android, which means going outside to find a solution.

So what’s the big deal about dependencies?  Plenty.

The data clouds below represent the specified vs the actual dependencies in the Android Jellybean release.  If you dump all the makefile rules you will get about 100,000 files and targets and nearly 2,000,000 dependencies.  It turns out you can ignore more than 80% of the specified dependencies during a build.

You can’t manually eliminate all those superfluous dependencies but you could write a script in gmake to do that and hope it works.  ElectricAccelerator does it automatically and guarantees build accuracy.

specified-dependencies actual-dependencies
Specified
1,990,628 dependencies
Actual
288,804 dependencies

Buy or Build?

The below flow chart illustrates the decision tree for solving your acceleration issues with Android.  You can develop your own internal solution, you can turn to open source or you can look to commercially available solutions.  Each has its benefits and drawbacks.

build-vs-buy-ci-build-acceleration-android

Custom

While developing your own customized solution in-house from scratch may be an interesting and satisfying technical challenge the reality is this option doesn’t make much sense for most organizations.  The time, support burden, and costs of internal development typically mean this isn’t a viable option, especially for a non-core, non-revenue tool.

Open Source

“Let’s go with open source.  It’s free,” is a common thought.  Tools like ccache and distcc that help with caching and distributed builds are well-known, and are extensible – to a point.  Unfortunately, like most open-source projects, these solutions have only small, part-time development staffs to support you in case things go badly.  And they don’t offer anything in the way of adaptive schedule optimization, guaranteed build dependency accuracy, fault tolerance or support for JACK.  Developers are on their own to figure out how best to use them, where to get help, troubleshooting, etc.  That’s another way of saying free isn’t really free.

Commercial Products

In the commercial build acceleration world there are only two options of note: IncrediBuild XGE and ElectricAccelerator.  When it comes to Android, however, as of this writing, IncrediBuild is a non-starter because it only supports Windows development platforms.
That leaves ElectricAccelerator.  Let’s compare it to open source solutions ccache and distcc:

ElectricAccelerator Open Source

Distributed Builds

Sharing across multiple servers

Sharing to cloud

Not dynamically
Fault Tolerance

Resource and Schedule Optimization

Adaptive schedule optimization

Cluster Sharing

Centralized Management

Grid Integration

Policy-driven resource allocation

Dependency Optimization

In-flight dependency correction

Removing unneeded dependencies

Detecting missing dependencies

Intelligent Cache/Reuse

GCC

Yes, with ccache
JACK

Javac

Build Analytics

Visualize parallel builds

Graphically Identify bottlenecks

Troubleshooting builds

While open source solutions are popular and well-known, they come up short in reducing the time and accuracy of Android builds.  In addition to speed, there’s enterprise reliability, scalability and the efficiencies of component reuse that are also important aspects of speeding time to market.  That’s all built in to ElectricAccelerator.

The Results

As with any comparison, the proof is in the pudding.  We ran tests with two prominent cell phone manufacturers’ Android code with stunning results: ElectricAccelerator was three times faster running on 1/3 of the cores.  Builds that were taking ¾ of an hour on 64 cores with gmake were taking 15 minutes or less on only 24 cores.

acceleration-results

Benchmarks created using Android M, gmake using 64 cores and ccache, ElectricAccelerator using 24 cores

That’s over 300% faster with fewer than half the resources!

Open Source Isn’t Always the Best Solution

While open source is initially an attractive option for solving many technical issues, when it comes to accelerating Android builds, it just doesn’t compare to the benefits of ElectricAccelerator.

  • Safely parallelize and distribute tasks
  • Leverage peer-to-peer multi-core desktops or scalable cloud infrastructure
  • Schedule optimization, ensures the best possible use of the CPU resources
  • Dependency optimization, intelligently ignores overspecified dependencies
  • Intelligent caching reuses work from other builds
  • Faster builds with fewer resources

Yes, we’re blowing our own horn here but if you need to get more Android releases out daily or weekly, ElectricAccelerator should be in your tool kit.  It’s not just us.  Here’s what one of our larger customers had to say:

huaweiReducing build times was a key focus to enable faster time-to-market with new Android releases.  With the unique technology in ElectricAccelerator 9.0, we were able to dramatically improve Android build times: release builds now complete in one-third the time, and incremental builds finish in under one minute instead of over six minutes.

James Lin , Sr. Manager, Development Tools, Huawei Technologies

 

In future blogs, we will talk further about additional ways to shorten Android build times through Build Analytics and Visualization, and Jobcache.

@ThingsExpo Stories
SYS-CON Events announced today that Daiya Industry will exhibit at the Japanese Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ruby Development Inc. builds new services in short period of time and provides a continuous support of those services based on Ruby on Rails. For more information, please visit https://github.com/RubyDevInc.
As businesses evolve, they need technology that is simple to help them succeed today and flexible enough to help them build for tomorrow. Chrome is fit for the workplace of the future — providing a secure, consistent user experience across a range of devices that can be used anywhere. In her session at 21st Cloud Expo, Vidya Nagarajan, a Senior Product Manager at Google, will take a look at various options as to how ChromeOS can be leveraged to interact with people on the devices, and formats th...
SYS-CON Events announced today that Yuasa System will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Yuasa System is introducing a multi-purpose endurance testing system for flexible displays, OLED devices, flexible substrates, flat cables, and films in smartphones, wearables, automobiles, and healthcare.
Organizations do not need a Big Data strategy; they need a business strategy that incorporates Big Data. Most organizations lack a road map for using Big Data to optimize key business processes, deliver a differentiated customer experience, or uncover new business opportunities. They do not understand what’s possible with respect to integrating Big Data into the business model.
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
SYS-CON Events announced today that Taica will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Taica manufacturers Alpha-GEL brand silicone components and materials, which maintain outstanding performance over a wide temperature range -40C to +200C. For more information, visit http://www.taica.co.jp/english/.
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, will discuss how they b...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
Digital transformation is changing the face of business. The IDC predicts that enterprises will commit to a massive new scale of digital transformation, to stake out leadership positions in the "digital transformation economy." Accordingly, attendees at the upcoming Cloud Expo | @ThingsExpo at the Santa Clara Convention Center in Santa Clara, CA, Oct 31-Nov 2, will find fresh new content in a new track called Enterprise Cloud & Digital Transformation.