Published: Invalid Date
5 min read
#React
At Fluent Commerce, we strive to provide our customers with the best solutions to stay ahead of their game. As a distributed order management platform, inventory management is crucial for all of our customers.
Failure to process inventory quickly and accurately can result in frequent occurrences of underselling or overselling, which in turn can lead to a cascade of negative consequences. These repercussions include cancelled orders, lost sales opportunities, and ultimately, dissatisfied customers.
We have introduced Global Inventory as a solution to meet our customers' diverse needs. With Global Inventory, customers have the flexibility to customize the logic and perform various actions, such as reserving, pre-selling, or updating inventory, based on their specific requirements.
Executing this logic can pose significant computational challenges, particularly when dealing with large datasets such as 20 million records. The processing speed may be compromised, and valuable resources could be wasted due to the repetitive nature of the data, where a small percentage (e.g., 5%) represents new information. This computational burden not only affects the efficiency of inventory processing for individual customers but can also have a negative impact on other customers operating within the same multi-tenant environment. When extensive resources are dedicated to processing redundant or unchanging data, it can disrupt the overall order and inventory management processes, impeding the smooth operation of other customers' businesses.
To enhance the speed of inventory processing while optimizing computing resources, We recently introduced a solution called Inventory Accelerator, specifically designed to address these challenges. This innovative feature employs data filtering techniques to examine incoming data before it undergoes further processing. By identifying and ignoring redundant or unchanged records, Inventory Accelerator significantly reduces the computational overhead associated with inventory processing. This also means that customers can process hundreds of millions of inventory updates efficiently, ensuring a seamless and expedited inventory processing workflow.
Before going into the details of the Inventory Accelerator, let us take a 30,000-foot view of the Fluent Commerce platform that is responsible for processing inventory batches.
The following diagram depicts the high-level execution flow of inventory batch processing with the Inventory Accelerator, followed by a short description of each step.
The Inventory Accelerator (aka Accelerator) is built on top of the AWS Glue service. AWS Glue is a fully managed extract, transform, and load (ETL) service provided by Amazon Web Services (AWS). It is designed to simplify and automate the process of preparing and loading data for analytics, machine learning, and other data-driven tasks.
The Accelerator consists of two main components that work together, each implemented using Apache Spark.
Here is how the components are tied together in the Accelerator. The flow starts from left to right.
The Accelerator execution is triggered either by the arrival of new inventory batch items or by a fixed scheduler, whichever happens first. Then, the data syncing module runs multiple parallel jobs to capture the latest snapshot of inventory data in the relational database. Once the three entity snapshots are loaded, the deduplication module starts filtering out the "changed inventory data" and produces events for all such batch items.
Before discussing the definition of a changed inventory data item, let us briefly introduce some terms used in the inventory management domain.
The inventory catalog serves as a structure that facilitates the grouping of inventory position data. Each inventory position calculates the "on-hand" inventory available based on its associated quantities and location. To represent specific actions that occurred during a particular period, various types of inventory quantities are utilized, including SALE, CORRECTION, LAST_ON_HAND, or DELTA. For example, the SALE inventory quantity type accurately records a sale event associated with the respective inventory position. In contrast, the CORRECTION type indicates a faulty item or any other issue that prevents the item from being sold. Lastly, the LAST_ON_HAND value represents the most recent on-hand value provided by the customer.
The Accelerator determines that the inventory batch item should be filtered if the batch item meets the two conditions below.
The current implementation is primarily designed to be compatible with the out-of-the-box Global Inventory workflows, offering limited customization options. It supports certain customizations, such as customizing transient and active statuses of inventory quantities.
To evaluate the improvements, a comprehensive set of performance tests was meticulously designed, implemented, and executed using 2000 batches, each containing a total of 1 million batch items. Let's now examine the insightful results obtained from these tests.
Scenario | E2E execution time w/o the accelerator | E2E execution time with the accelerator | improvement |
99% Redundant Inventory Data | 54 minutes | 34 minutes | 🔻37.00% |
92% Redundant Inventory Data | 58 minutes | 40 minutes | 🔻31.00% |
80% Redundant Inventory Data | 80 minutes | 57 minutes | 🔻28.75% |
50% Redundant Inventory Data | 65 minutes | 57 minutes | 🔻12.3% |
Based on the provided data, it is evident that an increased amount of redundant data leads to greater performance enhancements by the accelerator.
Steve Zhang
Senior Software Engineer II
Disclaimer: All information and resources found on community.fluentcommerce.com are based on the opinions of the author (unless otherwise noted). All information is intended to inspire and motivate well thought through technical decisions inline with Fluent Commerce recommended practices and principles.
Copyright © 2025 Fluent Commerce