Marubeni to contribute through AI analysis and process visualization in efficiency of understaffed ground handling, improving the On-Time performance and enabling more slots
The advancement of digital transformation in the airport industry is highly demanded. Although the impact of the new CoronaVirus Disease (Covid19) has caused a sharp drop in passenger numbers and visitors to Japan, domestic travel demand has recovered and the government has turned to relaxing the border control measures.
The government has set a target of 40 million tourists by 2025 and 60 million tourists by 2030 and also the domestic aviation industry started moving towards reaching these targets.
Even before the pandemic, airport operations had been identified as one challenge. Being confronted with societal issues like dropping birth rate and aging population and thus shrinking population, many industries are facing the shortage of workforce. And also the airports, a welcoming gateway to many foreign tourists to Japan, are not an exception. With their shortage of staff they could well become the bottleneck in fulfilling the government set targets. By the way, in the post pandemic current run on flying in Europe and the US, the aftermath of the crisis driven reduction in staff is one of the major hindrances in recovery and cases of airlines being forced into suspension or cancellation of flights have been seen.
In this environment, major trading company Marubeni is aiming to sell the “ApronAI” system developed by the Zurich based IT venture Assaia into the Japanese market as their general representative.
The system uses Artificial Intelligence to analyse video imagery that is acquired from cameras at the apron. Through the remote transparency of the apron operations and the optimisation of the airport operations, the solution will help reduce the burden of ground staff and contribute to the labor shortage situation which in effect is linked directly to improving on-time performance. At overseas airports where this system has already been implemented, there are even cases where the expansion of arrival and departure slots during peak hours due to improved punctuality have been obtained as concrete results. The expansion of arrival and departure slots during peak hours is expected to have a high economic effect, such as directly leading to increased earnings for airlines and airports. Furthermore, for passengers, the effect of increasing convenience, such as increasing the number of flights, can be expected.
Also in Japan, especially international hub airports or congested domestic airports have the problem of expansion of arrival and departure slots during peak hours in common. Furthermore, they are facing issues such as labor shortages and operational efficiency. It can be expected that this new AI analysis technology will become a solution to these problems in the future.
Marubeni continues to take on the challenges of airport operations
Marubeni has been trying to solve various problems that airport operations have been facing and the “Marubeni Airlines First Division” is at the forefront of it. When speaking of Marubeni, it is clear that they have a wealth of experience and knowledge in ground handling services, for example the acquisition of the former Shin Meiwa Ground Handling with Swissport International and the establishment of Swissport Japan (Swissport International: 51%, Marubeni: 49%). Swissport Japan currently operates ground handling operations at six airports in Japan: Narita, Haneda, Chubu, Kansai, Fukuoka, and Okinawa.
Heading Marubeni's Aviation Section I, Koichi Tagei said looking back: "Under the Abe administration, the promotion of attracting tourists to Japan bore fruit, and the number of visitors to Japan expanded to about 32 million in 2019, before the coronavirus. Along with that, Swissport Japan also succeeded in increasing sales and the number of employees.”
“However, considering the characteristics of the industry where the turnover rate is high and recruiting is not easy as well as considering the increase in demand, it has been foreseen in 2019 already that securing staff with good language skills and maintaining training for ramp staff will be the hardships.” he said.
“With the government setting a target for the number of visitors to Japan, the question is, in order to achieve 60 million visitors in 2030, whether it is possible to achieve growth of just under 50% (in comparison to 2019) using conventional methods. Our conclusion is that the answer is 'NO'.". In order to prevent the shortage of ground handling personnel from becoming a bottleneck to increasing the number of visitors to Japan, also the government has started various demonstration experiments for the implementation of automation and labor-saving technologies at airports.
In these circumstances, Marubeni established AiRO, a joint venture with ZMP. Since 2018, they have been implementing proof of concepts for automated driving technology within the restricted areas of Narita Airport, Chubu Airport, and Haneda Airport, and are accelerating their research and development toward the government's goal of fully automated driving by 2025.
What kind of system is ApronAI and which functionality improves the operational efficiency?
Advancing similar efforts to the aforementioned initiatives is Assaia’s “ApronAI”. As mentioned earlier, Swiss based Assaia is headquartered in Zurich and holds a branch office in Texas. The 2018 founded venture company was at that time a 3 people organization, but by accumulating results they grew up to a 40 people business.
Assaia’s ApronAI utilises either the surveillance cameras that are set up in a VDGS (Visual Docking Guidance System), or they use the video footage of ordinary CCTV surveillance cameras. By analysing the captured images with the AI developed by Assaia, it is possible to determine the exact timestamp of when each handling event started, when it ended, and whether or not each task was being performed on time. The system visualises in real time what is going on at the apron and allows to get an understanding of the current situation and centrally manage the various situations on a shared platform.
One of the functions is automatically generating timestamps based on the comparison of the captured ground truth and a plan and evaluates whether the actual task is delayed or not. Conventionally, delays in ground handling operations are basically reported after the fact. Moreover, cases where the report is handwritten can be observed quite well. However, with the system that Assaia has developed based on their collected data, they take the current progress rate and predict the off-block time of the current turnaround.
According to Marubeni’s Yuhei Hashimoto, “Instead of receiving information about already happened delays, through analysing with Artificial Intelligence, it is possible to inform ground handlers even prior to the delay happening about the situation and allow them to allocate additional resources.
The second functionality is the improvement of safety. Normally, before a plane enters the gate, an FOD check is done. As a matter of fact in some cases these checks are not done and Assaia’s AI based image analytics can help to take precautions against FOD checks or even FOD being missed.
The third function of “Apron AI” is the automatic allocation function. If the turnaround handling being carried out at a certain spot is delayed and a following aircraft is assigned to that gate, there are cases where the following aircraft even tries to enter because the delay in the turnaround is not known.
By using Assaia’s "Apron AI", it is possible to predict how many minutes later the aircraft will be able to depart, thus depending on the degree of delay in the process, it is possible to allocate a different gate, resulting in reduction of wasted time.
Easy to blend with existing system at airports, subscription models as revenue stream
From the perspective of airport companies, they probably already manage operations through some kind of software or digital devices, such as the Amadeus operation data system.
“One of the major features of Assaia’s system is that it blends very well with these different systems. Just because you have implemented Assaia’s system doesn’t mean that you have to stop using your other systems. Since the necessary information can be extracted from ApronAI and added to the existing systems, major modifications to the existing system are not necessary.” says Hashimoto.
Furthermore, Assaia's "ApronAI" target customers are "Basically, airport companies that have video data will be contract partners, but not only airport companies but also airlines, ground handlers and catering companies. The tool is convenient in that it enables real-time information sharing with all different stakeholders." he comments. For this reason, referring to examples of implementations overseas, "Airport companies could provide data to airlines and handling companies in a paid form, for example, as a monthly subscription, or provide data as a free service.” is Hashimoto's point of view regarding the development.
Optimal allocation of valuable ground handling staff
Mr. Tagei said, "All stationed ground handlers are very valuable human resources as they all have special permits. On the other hand, the problem is exactly that we are tied by regulations and have no choice but to attach such valuable human resources.” and points out that the actual situation is that the efficient utilisation of these resources is an issue. The Civil Aviation Bureau has formulated a ground handling action plan in 2020, and is starting to solve the issues faced in the field by utilising technology.
With Assaia’s “ApronAI”, ‘Valuable human resources can be freed from tasks such as monitoring and can be engaged in more advanced and essential tasks.’ and thus the utilisation of such modern technology will optimise the allocation of resources and contribute to efficient airport operations and help alleviate the staff shortage.
On another note, in overseas airports, a person that is called Aircraft Dispatch Manager is allocated to each gate and monitors, among other duties, whether the aircraft can leave the gate on time. With the implementation of Assaia’s system, it was possible to have one dispatcher monitor 3 flights simultaneously from a remote dispatch control office.
Seattle using at all 120 gates, JFK and others evaluating, verifying in extreme cold, hot or foggy conditions
Assaia being a Swiss company having an outpost in the US, has so far been focussing on the American and European market with 3 airports in the US being implemented as of today (Seattle Tacoma, Cincinnati, John F. Kennedy).
Especially at the international airport of Seattle Tacoma, the implementation covers all of the 120 gates. Based on a successful proof of concept together with Alaska Airlines at Seattle Tacoma, the utilisation of Assaia’s system led to a 6 minute reduction in the overall turnaround time (TAT) for each gate for them. If the same outcome holds true for the entirety of 120 gates, the effect is far from being small.
JFK and Cincinnati are currently still in the verification phase with 3 to 4 gates each but with the evaluation of the effect one can anticipate the increase of gates with adoption.
On a further note, not only the consolidated management of multiple gates at one airport, but for example also when multiple airports are under a unified administration, it would even be possible to monitor all gates through means of a provisioned remote control center in real-time.
According to Hashimoto, pilot installations apart from the above mentioned in other parts of the world are under way: “For example, they are conducting demonstration experiments under various climates and weather conditions, such as airport environments in arid regions in the Middle East, Helsinki Airport in extremely cold weather, and demonstrations in bad weather such as rain and snow.”. These prove that the system can adapt to every environment.
In addition, since camera images are the base, “there are concerns that reliability may decline due to fog or rainfall, however, the data that is acquired as a base for AI analytics have actually improved reliability and generated good results through deep learning.”. Verifications are conducted that even in conditions like dense fog or rain, results with a high degree of reliability can be guaranteed. If the number of cameras that are setup at an airport is large, the reliability can be further enhanced.
