­­­­­­­How DataArt Helped Inchcape Shipping Services to Revolutionize Document Processing on AWS

By Dmitry Maltsev, Solutions Architect at DataArt
By Ben Osborne, Sr. Product Manager at DataArt

Inchcape Shipping Services is a maritime services provider that can trace its roots back to 1847 when it began trading with the British East India Company as its first client.

Today, Inchcape is a global organization and leader in ships agency and maritime services, boasting more than 240 offices in 68 countries, covering around 2,500 ports.

The main thing that all interested parties need when a vessel is in port—and this has never changed—is information. Ship owners, charterers, shippers, and port agents all need to know what’s happening at the various stages of the port call to inform decisions, change plans, or save time and money.

Without complete transparency over the physical operation and financial process, it can be difficult to trust that process or identify and act on opportunities to optimize.

Inchcape has placed great emphasis on technology to drive transparency, quality service delivery, and insights in the maritime shipping industry. Partnering with DataArt, an AWS Advanced Consulting Partner, Inchcape reimagined their core operational platform, Optic, as a bespoke .NET Core-based microservices solution.

Optic builds trust through transparency of the vessel program, real-time updates, and standardized workflow and data across all port calls and locations. By capturing data from more than 70,000 port calls worldwide each year, Inchcape is able to benchmark operations, costs, and performance metrics to enable principals to optimize their fleet utilization and reduce port-related costs.

In this post, we will explore some of the core focuses and successes of the program, and move to look specifically at one area where all aspects of the port call from planning, execution, and financial reconciliation have benefitted—document management.

About Inchcape Optic

It was a ground-up undertaking to tool all aspects of the complex port call workflows, both operational and financial, and handle the user interactions and communication that accompany them in a modern, consumer-grade application.

Having launched successfully in 2019, Optic has been well received by Inchcape’s customers and internal teams alike, and is continuously evolving as a product in active and rapid development.

Originally developed as a cloud-agnostic application, Optic was soon migrated to Amazon Web Services (AWS) to take advantage of a broad range of cloud-native services. This resulted in reduced infrastructure costs and increased operational efficiencies.

The Challenge

One enduring aspect of the challenge which Inchcape—and the wider maritime shipping industry at large—has always faced is the effective management of huge volumes of documentation and paperwork which inescapably accompany their core business offering.

Every port call generates many hundreds of invoices, receipts, and operational documents or certifications. These all need to be made available to the many interested parties who have duties and responsibilities around them.

Traditional manual workflows have always proved to be a bottleneck on efficiency; they are slow, inconsistent, insecure, and error prone. Meeting this challenge was central to the design, implementation, and ultimately the success of the Optic application.

This post explores how Optic leverages a wide range of AWS capabilities to deliver a robust approach to document scanning, ingestion, storage, contextualization and digital annotation.

System Architecture

Being responsible for processing and storing thousands of business-critical documents, Optic needed a scalable and highly reliable architecture to effectively handle the high load of requests globally.

The DataArt team designed a distributed microservices solution leveraging a number of AWS native services, which fulfills business requirements and helps to minimize operational costs.

The application consists of a set of microservices related to specific business domains. Each service is deployed as a set of .NET Core-based Docker containers to Amazon Elastic Container Service (Amazon ECS) cluster spread across multiple Amazon Elastic Compute Cloud (Amazon EC2) instances.

Deployment employs multi-AWS Availability Zone setup to achieve high availability and fault-tolerance.

Figure 1 – High-level architecture of Inchcape Optic solution.

Inter-service communications, as well as inbound traffic for multiple client-facing applications and external integrations, get smartly routed by Elastic Load Balancers and Amazon Route 53 DNS service.

Optic uses Amazon RDS for PostgreSQL as the primary storage for application transaction data. Amazon RDS also empowers a bespoke data warehouse solution that provides rich reporting capabilities for end users. To minimize the time to first byte and content delivery with minimal latency, Optic aggressively caches highly accessed data with the help of Amazon ElastiCache for Redis.

A high volume of documents and files uploaded to the system on a daily basis are stored in Amazon Simple Storage Service (Amazon S3), effectively leveraging various storage classes to reduce the costs. Amazon S3 also serves as a hosting platform for static files for Optic frontend application, which gets delivered globally using the Amazon CloudFront content delivery network solution.

For a system that handles a massive amount of business-sensitive records, security is a top priority. The architecture of Optic implements strict policies for AWS WAF and applies a number of other security measures—like mandatory virus scan for all uploaded files—to protect application infrastructure and data from security threats.

Document Processing

Document uploads, in addition to the processing of email attachments and inline images in the application’s messaging module, are some of the most heavily used scenarios in Optic. These documents come from various external sources and, as such, the application must ensure a high level of security by running a proactive virus scan for submitted files.

Optic uses a third party on-premises anti-virus solution, which provides the ability to run document scans across multiple anti-virus engines to maximize the coverage for the most common security threats, as shown in the diagram below.

Figure 2 – Document anti-virus scan process.

Optic utilizes three dedicated Amazon S3 buckets to handle files scan:

• Every newly uploaded file goes into the Unsafe bucket, indicating it requires a virus check. Users will not be able to access the file until the check is completed.
• If anti-virus gives a green light for the file, it gets moved by the application to Safe bucket. From that point, the file is available for download and further processing.
• In rare instances, the file might fail the virus check and end up in a Quarantine bucket as potentially dangerous. Such files get completely isolated from the application with an immediate notification to the system support.

Virus scan is a time-consuming operation, and anti-virus needs rapid access to the file to minimize waiting time. The default solution for the anti-virus system utilized by Optic is to upload the file that needs a scan directly to the anti-virus instance as a stream of bytes.

This pattern was not ideal from the performance and system’s stability perspectives: additional space is required to store copy of the file, while the upload process is a long-running operation and requires implementing additional solutions to ensure fault-tolerance.

To address these challenges, Optic uses a different technique allowing it to mount S3 storage as a local file system. This way, the anti-virus instance can be simply pointed to the file location as if it was stored on its local drive, eliminating the need to copy anything.

The code below uses s3fs utility for mounting ‘optic-documents-unsafe’ S3 bucket to ‘/files/optic-documents-unsafe’ directory via FUSE:

/usr/bin/s3fs optic-documents-unsafe -o use_cache=/tmp -o passwd_file=/etc/passwd-s3fs-optic-documents -unsafe -o allow_other -o uid=996 -o gid=994 -o mp_umask=002 -o umask=004 -o multireq_max=5 /files/optic-documents-unsafe

The Optic solution is fast and helps ensure the scanned file is exactly the same as was uploaded by the user.

Data Retention

All documents uploaded to Optic hold business-sensitive data and, according to regulatory requirements, must be stored for an extended period of time.

The DataArt team implemented a sophisticated data retention policy that enables Optic to safely store all important documents and files for as long as 15 years.

Optic benefits from various Amazon S3 storage classes to reduce the storage costs for uploaded documents:

• When initially uploaded, the file gets assigned with S3 Standard storage class to ensure high performance for file retrievals.
• After 30 days, the number of file downloads usually goes down and it can be moved to S3 Intelligent-Tiering. This way, you don’t have to worry about assigning the right storage class as S3 will automatically select the most cost-effective one depending on the access patterns.
• Files older than 180 days get accessed by users very rarely and can be permanently assigned with S3 Standard-IA access storage class for further cost reduction.

In order to comply with regulatory requirements, Optic also implements long-term storage policy using Amazon S3 Glacier:

• At the end of each month, a full backup of operation documents is created and stored in Amazon S3 Glacier storage. The retention is configured to keep backup for one year after which it gets automatically deleted.
• Yearly backups aggregate data for the last 12 month and utilize S3 Glacier Deep Archive as a long-term storage with 15 years retention.

Disaster Recovery

Optic employs a fault tolerant setup using Amazon ECS multi-Availability Zone deployment. In addition, the DataArt team implemented a comprehensive disaster recovery strategy which was critical for the business.

A key part of the approach was to ensure the recovery region has all S3 documents already in place (alongside the database) in order to minimize the time needed to recover the system. This was achieved by enabling a cross-region replication for all necessary S3 buckets.

Customer Success

The Optic program has been a success in delivering both the efficiencies and customer experience benefits that Inchcape sought to unlock:

• Automation elements have improved internal productivity and reduced service failure.
• Existing customers have been been delighted
• New customers have been attracted.

“We are very happy with the outcomes that Optic has delivered for our business,” says Andy Ashwell, Group Information Officer at Inchcape Shipping Services. “By far, the biggest benefit is the strengthening of our partnerships with key customers.

“The high levels of engagement, feedback, and adoption that have been achieved from our global customers, combined with improved data quality and consistency, are enabling us to jointly optimize our own and our customers’ businesses,” Ashwell adds. “This is enabling us to continuously simplify and reduce our customers total cost of port operations.

“The most exciting part is that this is the just the start. DataArt, through Optic, has given us a platform we can scale and evolve for the continued benefit of our customers and Inchcape.”

Inchcape is now exploring the use of Amazon Textract as a natural extension of the existing solution. By leveraging the machine learning capabilities of this service, the desire is to automate the entire document contextualization process from start to finish.

This will enable the application to ingest, process, store, and re-distribute digital paperwork with next-to-no manual intervention, allowing Inchcape to focus on the service quality and business intelligence insights that the business and its customers desire.

Summary

Optic has played a central role in helping Inchcape realize its 360 vision, unlocking previously elusive data transparency and availability as a means of facilitating smarter decision making throughout the lifecycle of a port call.

Spanning the complex operational and financial aspects of the business —and tooling both the organization itself and its customers—the Optic platform enables Inchcape to build on these early successes and provide a foundation to service future needs.

.

.

DataArt – AWS Partner Spotlight

DataArt is an AWS Advanced Consulting Partner that specializes in cloud-native software engineering and application modernization for finance, retail, travel, media, healthcare and other sectors.

Contact DataArt | Partner Overview

*Already worked with DataArt? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.