DLT Labs™ & Walmart Canada Transform Freight Invoice Management with Hyperledger Fabric
Save money. Live better.
That’s not only Walmart’s iconic slogan; it’s the guiding principle for its multibillion-dollar operations to ensure the lowest everyday prices for its consumers.
Within Walmart Canada (Walmart), one aspect of fulfilling that promise involved reexamining its supply chain logistics, and specifically its freight transportation, which enables stocking more than 400 stores nationwide with nearly 120,000 different products. To do so, Walmart works with thousands of businesses to ensure that in aggregate, and with proper allocations to each store, it is sufficiently stocked to service more than 1.2 million customers every day.
“Walmart places enormous value on its partnership with third party transportation companies, referred to as ‘carriers’,” says Neeraj Srivastava, DLT Labs™ co-founder and chief technology officer. “The carriers are a vital link in Walmart’s supply chain backbone, and handle a vast array of transportation, so they need to ensure both efficiency and the best working relationships possible.”
At the heart of a constructive relationship is managing a massive and constant flow of information, while ensuring the carriers are paid on time. This may sound easy, but the industry is inherently complex.
Carriers work around the clock, and through all seasons, to deliver goods. It is both a capital-intensive business and they also face a high degree of volatility on each delivery (fuel prices, delays, unexpected events). The business model that has evolved shares the risk of price fluctuations and unexpected events between the carriers and the beneficial cargo owners (BCO’s) like Walmart. Each invoice has both fixed charges, and variable charges that are called ‘accessorial charges’. While in principle it forms a healthy partnership with aligned interests, in practice there is immense complexity based on the number of variables. It is not uncommon to have invoice disputes in the range of 70% or more over every load a carrier delivers. Despite various service providers trying to solve this puzzle using the full spectrum of existing technologies, typically by running a ‘match and compare’ analysis, disputes continue to plague the industry with high administrative costs and lengthy time delays, and invoice reconciliation often simply meant one side or the other capitulated.
Walmart Canada enlisted the help of DLT Labs™, a global leader in the development and deployment of an innovative enterprise-level platform using distributed ledger technology, to develop a solution that would solve this perennial problem once and for all.
The High Cost of Outdated Information Management Systems
What DLT Labs™ discovered in their market research was that invoicing problems are ubiquitous and enormous throughout the industry. “The retail industry is riddled with different, incompatible, and outmoded systems,” says Srivastava.
In the U.S. transportation industry alone, $140 billion is reputed to be tied up daily in invoice disputes. There are overpayments, underpayments, and non-payments. The only thing parties can agree on is that there are delayed payments, which in turn drives up working capital costs because of the need to borrow, often through accounts receivable financing, or ‘factoring’. But even factoring is difficult or impossible when the invoices are not yet approved. Assuming a 10% ‘cost of capital’ applied to these funds, that means $14 billion a year is captive and allocated to disputes; and this capital could clearly be put to more productive use.
While the amount of data continues to grow over time, the problems have largely increased as well. For example, IoT data can provide valuable information, but it is often not integrated with other information systems.
This is one of those situations where there were no winners, only losers. There is an assumption that non-payment allows a BCO to benefit by holding its capital longer (time value of money). But DLT’s analysis is this is a fallacy, and any small benefit is dramatically outweighed by the cost of administration, the burden on BCO-carrier relations and the fact that the overall costs are increased because carriers simply price-in the cost of any such delay into their bids.
At every turn the costs are enormous and wasteful. It is commonly accepted that administration costs represent 20% of transportation fees. And from another perspective, payment processing is reputed to be 5% the cost of processing the payment itself, and 95% earmarked for the supporting information (exactly who should be paid how much, when and where).
Now imagine this institutionalized yet inefficient cycle of transactions and multiply that by 70: as Walmart works with up to 70 different carriers in its supply chain, each of which has its own respective process for calculating shipping costs based on individual contracts that outline varying rates for fuel, line haul and other charges.
Along with 70 different contractual agreements and conditions, Walmart must juggle multiple systems to account for and track shipment data funneling in from each trailer, railcar or ship through different systems that are often incompatible. Each load delivered has extensive data such as load costs, carrier details, and tracking information that amounts up to approximately 220+ data points per load. With more than 500,000 loads moving across the country each year, the volume of information that needs to be managed and accounted for is staggering.
And the results were paradoxical: there was an endless amount of data, without the clarity and information needed to approve invoice amounts. In the end, Walmart and each carrier would have their own respective versions of the work accomplished and rates which, more than often than not, would not match. This would inevitably lead to rising tensions and frustrating delays for all parties as they struggled to find a resolution.
In Blockchain We Trust
The DLT Labs™ team was able to distill all the confusing and conflicting processes down to one problem: Trust.
“To make sure they can have the best relationship with their carriers, Walmart wanted carriers to trust them, and the carriers wanted Walmart to trust them as well,” says Srivastava. “In our view, there is no better technology than blockchain to improve trust among organizations.”
In partnership with DLT Labs™, Walmart was determined to eliminate the root problems that caused invoice disputes. Both parties agreed that leveraging the intrinsic benefits of distributed ledger technology (which includes an assurance of immutability and transparency while keeping data cryptographically secure), could provide the foundation of a solution.
After reviewing roughly 17 different technologies, and looking across the spectrum of available blockchains — the team built a stack that included Hyperledger Fabric — an enterprise-grade blockchain code base sponsored by the Linux Foundation.
“Hyperledger Fabric is the only platform in the market which is mature enough to run a production environment and meet the security standards of major enterprises,” says Srivastava. “And it was built with data security and privacy at its core.”
Rather than an open, permission-less system, Hyperledger Fabric offers a scalable, modular blockchain framework which can support private transactions and confidential contracts within a network of multiple vendors.
In short, Hyperledger Fabric allows Walmart to bring together all the carriers within its multi-partner freight operations under one architecture to automate and implement universal workflows across the network. At the same time, through Hyperledger Fabric’s unique “channels” feature, the solution allows independent and protected relationships for each organization directly between itself and Walmart, and the information is not accessible to other members.
Because Hyperledger Fabric is an open-source project, the DLT Labs™ team enhanced the code base as necessary, and to further stabilize and increase the throughput of the platform, “We leveraged our proprietary innovation called Application Specific Integrated Kernels , which helped us overcome network latency and integration challenges which arise due to the asynchronous nature of the blockchain. We had also developed a series of tools and frameworks to ensure that a comprehensive suite of all components necessary to run the network in production were readily available,” says Srivastava.
Another challenge was how to integrate the new platform with each company’s legacy systems. “We had approximately 70 carriers with 70 different systems, so we solved this problem by designing and developing a single Integration Hub and Supply Chain Gateway which inherently connects with the platform for smooth integration with any system,” says Srivastava. So, it is truly a seamless integration, and any organization can continue to follow their familiar processes without learning or investing in a new technology.
Introducing… DL Freight™
In early 2019, DLT Labs™ rolled out the world’s largest full production blockchain solution for any industrial application: DL Freight™. Remarkably, because the platform is fully configurable, DL Freight™ was configured for the demanding needs of Walmart in just 60 days. The new system tracks deliveries, verifies transactions, and automates invoices in real time. Reconciliation between Walmart and its fleet of carriers is no longer even necessary because they are all working off the same information and calculations by way of smart contracts.
Instead of reconciliation through a matching process, there is a unified, joint creation of each invoice. Walmart followed a tremendously careful and meticulous review through a live pilot that lasted into the fall and comprised an exhaustive array of security audits and analysis. At the end of the process is payment authorization, entirely through the solution, and Walmart relies on that authorization for payment with no further checks or balances.
“We started this journey with a simple challenge which was how to digitize and simplify the way we pay our carriers. There are lots of different ways to do that but, working with DLT, it was fast. We developed something from the ground up within nine months [duration of the pilot],” says John Bayliss, senior vice-president, logistics and supply chain at Walmart Canada. “It was efficient to maintain and so it has really streamlined the processing on our side and allowed us to consolidate the platform itself rather than dealing with a lot of intermediaries. Although we use this for payment processing, it is a platform we are using for other tools to get insight on how we run our business.”
DL Freight™ is a specific configuration of DLT Labs™’ configurable enterprise SaaS platform which consists of interoperable modules that address common business processes such as workflow, supply chain, credential information, privacy and consent, and value transfer. The module used in this case was DL Asset Track™.
Within DL Freight™, carriers are the peers, and the governance of the platform is controlled by the applicable contracts, as in any conventional business. The difference is that the freight, legal, and finance departments of Walmart as well as all the carriers have all agreed that the solution fairly and accurately represents and processes those agreements. As a result, it has removed the guesswork, and any real potential for dispute over the interpretation of agreements. All parties periodically review and renegotiate the terms / governance model every three years.
With Hyperledger Fabric as its foundation, all data is continuously distributed and accessible to Walmart and its full fleet of 70 carriers in real time. “The network is set up to make sure the system remains available 24/7,” says Srivastava.
Because of the massive logs of information collected from daily transactions, DLT Labs™ runs the platform on more than 600 virtual machines (VMs) to securely store and manage data points from thousands of transactions per day. But despite the heavy load, the system’s response time is easily on par with, if not superior to, comparable TMS offerings (Transportation Management Solutions).
“We have carefully chosen the datacenter locations which are dispersed across various geographical regions to make sure an outage at one location does not impact the system availability,” says Srivastava. “Each and every component of the system, whether it’s part of the blockchain or not, is distributed across these geographical locations.”
As a unique and powerful innovation, for security purposes, the entire network runs without direct access to the Internet. Instead, users have to go through a series of microservices and firewalls to connect. DLT Labs™ also adopted OIDC protocol for authentication and authorization for every member of DL Freight™. Finally, on the platform, all data at rest or in motion are encrypted and connections throughout the network are secured using bank grade encryption.
These measures ensured that DL Freight™ not only passed penetration testing by a third-party vendor, the platform was also approved by various Walmart security and compliance departments within its global network, each of which has its own set of extraordinarily high standards and checks.
Save Money. Pay Better.
While the economic return on investment (ROI) for Walmart is confidential, Walmart is an extremely financially disciplined organization. It would not have proceeded with this implementation without clear and very substantial financial benefits. In this case both the speed of payback (time to return of up-front capital invested), and annual benefits (return on operating cost/SaaS fees) exceeded a very high threshold.
Since DL Freight™ was introduced into its operations, Walmart has experienced a demonstrable transformation of freight management and carrier relations. What was previously an overwhelming number of invoice disputes in the range of 70% has now been virtually eliminated to only small number of discrepancies of approximately 1.5%. In the words of Francis Lalonde, Walmart VP Transportation, “Something unusual has to occur to have a dispute now. There are small items to resolve, but as we all have the same information, there is simply no need for disputes anymore.” The error threshold for each transaction went from $10 per invoice to $0 per invoice. And the timeline to agree on and approve carrier invoices that formerly varied from 6-8 weeks, but often extended over many months, went to less than one week. After approximately six months of national deployment, the platform has successfully processed over 200,000 invoices totaling payments in the hundreds of millions of dollars.
“The financial and operational incentives for adopting DL Freight™ are significant – they include cost savings and faster time to process payments,” says Srivastava. “And in our view, despite assertions that conventional systems and solutions could solve this puzzle, it would not have been possible without blockchain.”
Walmart has achieved a level of automation for invoice approvals that is unprecedented, with clear benefits to all parties including both Walmart and the carriers.
“Carriers are heavily motivated to get on-boarded so they can get paid much faster by using the DL Freight™ platform rather than older systems,” says Srivastava. “Without trust in the system, no carriers would have agreed to put their data onto the platform.”
“As a turnkey solution, DL Freight™ has already turned heads and provides a compelling demonstration of the power and central role blockchain can play in large scale industrial solutions,” says Srivastava. “The platform is simple and intuitive, and the adoption process is remarkably easy and flexible. DL Freight™ will run on any cloud infrastructure, and DLT Labs™ can host and manage all infrastructure needs. But to enter the mainstream, it was fundamental not to focus on the underlying technology and innovation, but instead simply concentrate on the substantial value proposition for enterprises”.
Eliminating Supply Chain Problems For Good
The roadmap for DL Freight™ includes the network effect of increased adoption. Onboarding is extremely rapid and, as mentioned, allows seamless integration with legacy systems. There are a host of ongoing major technical features and improvements anticipated, including even further improvement in transaction speed, integrating new in-house developed encryption algorithms to enhance the security, and improving smart algorithms for better prediction and analysis of data along with a multitude of other features to cater to various business needs. DL Freight™ has already established the first digital freight processing network, building on the underlying platform’s capabilities in providing and enabling true inter-company information exchange, the holy grail of information systems which has eluded the industry to date.
The vision for DLT Labs™ is to dramatically transform the way information is managed. “The core problem with information management derives from a simple but powerful historical reality. While organizations exist for the purpose of working with each other, information systems are designed and built to serve their owners,” says Srivastava. “At DLT Labs™, we provide a sustainable bridge between legacy systems and business partners that finally overcomes the ubiquitous problem of inter-organizational information exchange and sharing.
DLT Labs™ seeks to ensure that the power of distributed ledgers enables the secure, reliable inter-company information exchange on a global scale. It may sound dull outside the world of IT, or fanciful inside it, but the fact is this is now being done for the first time, at scale and in production with remarkable success. “Our vision is to allow organizations in all corners of the globe to collaborate and share information with each other without thinking twice about security or data breach,” says Srivastava. “Once organizations feel comfortable working with each other, they can tackle the other challenges related to supply chain management. We fervently believe the improvement in trust from this technology will be a major contribution to a better world.”
And blockchain will always have a major role in that. “The blockchain space has steadily grown since 2011 as blockchain has held enormous promise for enterprises,” says Srivastava. “We are on our way to becoming the standard platform for the development of applications and solutions in the enterprise blockchain space, and we believe the future of enterprise blockchain is in our collective hands.”
About DLT Labs™ Inc.
DLT Labs™ is a global leader in the development and delivery of enterprise blockchain technologies and solutions, as well as a pioneer in the creation and implementation of standards for application development. With a deep track record in innovation and one of the world’s largest pools of highly experienced blockchain experts, DLT Labs™ enables the transformation and innovation of complex multi-stakeholder processes. Follow along at DLTLabs.com,@dltlabs and linkedin.com/company/dlt-labs
About Walmart Canada
Walmart Canada operates a chain of more than 400 stores nationwide serving more than 1.5 million customers each day. Walmart Canada’s flagship online store, Walmart.ca is visited by more than 900,000 customers daily. With more than 90,000 associates, Walmart Canada is one of Canada’s largest employers and is ranked one of the country’s top 10 most influential brands. Walmart Canada’s extensive philanthropy program is focused on supporting Canadian families in need, and since 1994 Walmart Canada has raised and donated more than $400 million to Canadian charities. Additional information can be found at walmartcanada.ca, facebook.com/walmartcanada and at @walmartcanada
Hyperledger is an open source collaborative effort created to advance cross-industry blockchain technologies. It is a global collaboration including leaders in banking, finance, Internet of Things, manufacturing, supply chain, and technology. The Linux Foundation hosts Hyperledger under the foundation. To learn more, visit https://www.hyperledger.org/