Drones and Artificial Intelligence

Starting Point

“Actionable Data” is next to a powerful and reliable drone probably the most important driver of the drone industry. Drones often generate large amounts of data – sometimes more than we can handle. Unmanned aerial vehicles only add value to the user if there are ways to process data quickly and without putting additional efforts into this process. The faster, the more accurate, and the easier the images can be evaluated, the better.

Combining drones and artificial intelligence seems to be the answer to the above-mentioned challenges. Nowadays, almost every company that deals with data processing, analytics or ‘autonomous’ flight control and claims the use of artificial intelligence, machine or deep learning. But what do these terms actually mean? How do they relate to each other and where do these “terms” apply? The following text aims to answer exactly these questions.

In general, AI describes the capability of machines that can perform sophisticated tasks which have characteristics of human intelligence and includes things like reasoning, problem-solving, planning, learning, and understanding and reading human languages as shown in following graphic below. Currently, AI in relation to Machine Learning, Deep Learning, and Motion Planning are the hottest topics and will be in the focus of this publication.

Terminology and Description

Machine Perception

Let us start with Machine Perception. Since many AI-related tasks for drones are dealing with image recognition, the unmanned aerial vehicle must be able to perceive and absorb the environment or objects in some way. This is usually done with sensors such as electro-optical, stereo-optical, and LiDAR. This process is referred to as Machine Perception.

Computer Vision (CV)

Once the drone has captured raw sensor data, it usually needs to be analyzed in some way to extract meaningful information for a certain purpose. This ability is called Computer Vision and is concerned with the automatic extraction, analysis, and understanding of useful information from one or more images.

Machine Learning (ML)

To optimize differentiable parameters, techniques of Machine Learning can be applied. Unlike software that has been programmed manually and performed tasks with specific instructions (like Computer Vision software), Machine Learning algorithms are designed in such a way that they can learn and improve over time when exposed to new data.

Deep Learning (DL)

Deep learning, on the other hand, is a specialized method of information processing and a subset of machine learning that uses neural networks and copious amounts of data for decision-making. The learning methods are based on the functioning of the human brain, which also consists of interconnected neurons. So-called Artificial Neural Networks consist of multiple layers of which each is connected to the next layer and is responsible for a certain task. This design makes it possible to combine and expand what has been learned with new content.

When using Deep Learning over Machine Learning or Computer Vision?

While many companies have moved from Computer Vision to conventional Machine Learning approaches, it appears that the first steps are being taken with deep learning algorithms in the drone industry. Recent developments in the tech industry, namely GPUs (Graphic Processing Units), have it made it possible to exploiting DL through its price-to-performance ratio as well required hardware infrastructure. Although much more computing power is available through GPUs, it still takes a reasonable amount of time to train DL algorithms, and mostly millions of images are needed to reliably perform a certain task with DL. Consequently, this means that if you have access to a big data set of images and sufficient processing power, DL methods might be the preferable choice since it usually outscores conventional ML and CV methods, especially in image recognition.

Motion Planning

Now let’s come to Motion Planning. It is a strong tool when it comes to situational awareness and, in a broader sense, Sense & Avoid technology and BVLOS flights. The prerequisite for motion planning is usually capturing the environment – that is, Machine Perception. To do so, the drone visualizes the environment, e.g. with SLAM (Simultaneous Localization and Mapping) technology. This gives the drone the capability to not necessarily identify what exactly is in the environment, but the distance to it. In the context of Motion Planning, Deep Learning is deployed to detect and recognize objects like humans, biker, or cars to and subsequently to create a corresponding flight route.

Is AI the answer to all questions?

Certainly not today. Good image processing solutions are in place and performing as good as ML and DL approaches for certain tasks. One must ask, what should be my result and what information/data is available? Always consider that any sort of ML approaches requires huge data sets and a lot of training. Hence, if you only have a limited number of pictures available image processing software might be the best solution. When huge datasets are available, and many different tasks should be processed, ML or DL approaches probably outperform image processing software solutions. In other words, the more tasks and the more complicated they are with image processing software, the more likely it is that ML/DL approaches are the best way to solve them, since your data size increases.

Most applications of ML and DL algorithms are currently found in the areas of inspection and maintenance. Companies such as Sky-Futures and Scopito are using various ML and/or DL approaches for different inspection tasks. For example, insulators of power lines are automatically detected on images and checked for anomalies, or algorithms are used to detect corrosion on metallic surfaces. According to Sky-Futures, their detection rate is at 80-90%. Another example is Ardenna, the American software company is currently upgrading from Computer Vision Software to an ML-based rail inspection of the railway company BNSF to detect over 30 damages at once. A lot of AI approaches are already used for data analysis in the energy, agriculture, real estate, construction, and forestry sector. The list of AI-based data analysis applications seems already to be endless and the just mentioned examples are only a small fraction of what is already available on the market.

Summary and Outlook

As already described above, the goal of drones and artificial intelligence is to make efficient use of large data sets (such as aerial images) as automated and seamless as possible. No one wants to look at 5000 plain white picture of a wind turbine and look for tiny cracks. Drones can only unlock their full potential when data acquisition and data analytics happen at a high (or someday full) degree of automation. Great potential to process this mass of data as automated as possible seems to be ML od DL approaches. Due to an immense and rapid increase in processing power, costs of storage and availability of digital data in the recent years, the utilization of complex AI algorithms have become feasible for drones, and first and solid solutions are already on the market. If AI develops as rapidly as we have seen in the recent years, we will soon find highly automated and comprehensive solutions that will further increase the added value of using drones. But companies must consider that drones and artificial intelligence only make sense if it saves the user money/time – in some cases, traditional Computer Vision (in combination with ML/DL) still might be an easier and better solution.