AI Captain – A new marine AI to navigate ships at sea autonomously

Marine research organization Promare, along with IBM, is all set to start the sea trials of ‘AI Captain,’ a new marine which enables the Mayflower Autonomous Ship (MAS) to sense, think and act autonomously at sea.

The trial, taking place on a manned research vessel off the coast of Plymouth in the UK, will evaluate how the AI Captain uses cameras, AI, and edge computing systems to safely navigate around ships, buoys, and other ocean hazards.

MAS will trace the route of the original 1620 Mayflower to commemorate the 400th anniversary of the famous voyage. Sailing from Plymouth, UK to Plymouth, Massachusetts, with no human captain or onboard crew, it will become one of the first full-sized, fully autonomous vessels to cross the Atlantic. The mission will further the development of autonomous commercial ships and help transform the future of marine research.

MAS will rely on IBM’s advanced AI and edge computing systems to sense, think, and make decisions at sea, even with no human intervention. With the three hulls of the trimaran, MAS currently reaching the final phase of construction in Gdansk, Poland, a prototype of the AI Captain will first take to the water on a manned vessel – the Plymouth Quest – a research ship owned and operated by the Plymouth Marine Laboratory in the UK. The March sea trials, to be conducted in Smart Sound Plymouth, will determine how the Mayflower’s AI Captain performs in real-world maritime scenarios, and provide valuable feedback to help refine the ship’s machine learning models.

Over the past two years, the Mayflower team has been training the ship’s AI models using over a million nautical images collected from cameras in the Plymouth Sound in the UK as well as open-source databases. To meet the processing demands of machine learning, the team used an IBM Power AC922 fuelled by IBM Power9 CPUs and NVIDIA V100 Tensor Core GPUs, the same technologies behind the world’s smartest AI supercomputers. Now, using IBM’s computer vision technology, the Mayflower’s AI Captain should be able to independently detect and classify ships, buoys, and other hazards such as land, breakwaters, and debris.

As the Mayflower will not have access to high-bandwidth connectivity throughout its transatlantic voyage, it will use a fully autonomous IBM edge computing system powered by several onboard NVIDIA Jetson AGX Xavier devices. While at sea, the Mayflower will process data locally on NVIDIA Jetson, increasing the speed of decision making and reducing the amount of data flow and storage on the ship.

How does the Mayflower sense, think, and act at sea?

Let’s assume that the Mayflower is in the open ocean, approaching Cape Cod, with no current satellite connectivity. In its path is a cargo ship that collided with a fishing vessel, spilling some of its load. In this hypothetical scenario, the Mayflower’s AI Captain will use the following technologies and processes to independently assess the situation, and decide what action to take:

  • Radar detects multiple hazards in MAS’s path, 2.5 nautical miles ahead.
  • Onboard cameras provide visual input to IBM computer vision system which identifies risks as a cargo ship, a fishing vessel and three partially submerged shipping containers floating in the water.
  • Automatic Identification System (AIS) provides specific information about the cargo ship’s class, weight, speed, cargo, etc.
  • GPS Navigation System – provides MAS’s current location, heading, speed and course.
  • MAS’s nautical chart server provides geospatial information about its chosen route.
  • Weather data provided by The Weather Company.
  • Attitude Sensors – assess local sea state (how MAS pitches and rolls due to waves).
  • Fathometer – provides water depth measurements.
  • Vehicle Management System – provides operational data such as MAS’s battery charge level, power consumption, communications, science payloads, etc.
  • IBM Operational Decision Manager (ODM) evaluates COLREGs concerning the other vessels in the vicinity and generates a risk map indicating an “unsafe” situation ahead
  • MAS’s AI Captain ingests the ODM recommendation, computer vision input, current, and forecasted weather and assesses several options to avoid hazard.
  • AI Captain determines the best action for MAS, in this hypothetical scenario, is to steer to starboard to avoid the unexpected navigation hazard.
  • MAS’s Safety Manager verifies the decision as safe.
  • AI Captain instructs MAS’s Vehicle Management system to change course and speed.

“While the autonomous shipping market is set to grow from $90BN today to over $130BN by 2030, many of today’s autonomous ships are really just automated – robots which do not dynamically adapt to new situations and rely heavily on operator override,” said Don Scott, CTO of the Mayflower Autonomous Ship.

“Using an integrated set of IBM’s AI, cloud, and edge technologies, we are aiming to give the Mayflower the ability to operate independently in some of the most challenging circumstances on the planet,” he added.

“Edge computing is critical to making an autonomous ship like the Mayflower possible. The Mayflower needs to sense its environment, make smart decisions and then act on the insights in the minimum amount of time, even in the presence of intermittent connectivity, while keeping data secure from cyber threats,” said Rob High, VP, and CTO for Edge Computing, IBM. “IBM’s edge computing solutions are designed to support mission-critical workloads like the Mayflower Autonomous Ship, extending the power of the cloud and the security and flexibility of Red Hat Enterprise Linux out to the edge of the network, even in the middle of the ocean.”

Besides following the overall mission objectives to reach Plymouth in the shortest time, the AI Captain will draw on IBM’s rule management system (Operational Decision Manager – ODM) to follow the International Regulations for Preventing Collisions at Sea (COLREGs) as well as recommendations from the International Convention for the Safety of Life at Sea (SOLAS). Used widely across the financial services industry, ODM is particularly suited to the Mayflower project as it provides an entirely transparent record of its decision-making process, avoiding ‘black box’ scenarios.

As the weather is one of the most significant factors impacting the success of the voyage, the AI Captain will use forecast data from The Weather Company to help make navigation decisions. A Safety Manager function (running on RHEL) will review all of the AI Captain’s decisions to ensure they are safe – for the Mayflower and other vessels in its vicinity.

The March sea trials will take place for approximately two months on the Plymouth Quest with the ship’s human captain and crew at the helm. In the first stage if testing, the Mayflower AI Captain’s inference engine will receive input from the Quest’s radar, AIS, GPS, and navigation system, as well as data about visibility. Cameras, computer vision, edge, and autonomy capabilities will be added in the next phase of testing from April.