NTUC Enterprise has switched cloud allegiances for its enterprise data stack, adopting a serverless architecture that needs less management overhead to support growing data consumption and analytics efforts.
Chief data officer Yiqun Hu told Google’s Data Cloud and Applied ML Summit that NTUC Enterprise - through its digital arm, NE Digital - first decided to "adopt more services from the Google Cloud Platform (GCP)" about two years ago.
“Previously, the data stack we used was quite heavy on AWS,” Hu said.
The co-operative - which runs social enterprises in grocery under the FairPrice brand, health, care and other household-oriented verticals - had been using Amazon Redshift as its data warehouse and data lake, but decided to migrate to the BigQuery cloud data warehouse on GCP instead.
It ran the data migration from Redshift to BigQuery with Google Cloud’s storage transfer service, and largely with its own internal staff.
“I think our migration we almost did [entirely] in-house with strong support from the GCP teams, so there was the opportunity for our staff to learn and acquire the new skills. They are quite good at the AWS cloud and some knowledge is transferable to the Google Cloud,” Hu said.
Big change
Hu said that BigQuery is “a big change” for NTUC Enterprise, being a “fully serverless service”.
“[But] it actually eases our pain about managing and sizing configuration of the data lake infrastructure.”
NTUC Enterprise then started to ingest “a lot more data into BigQuery”. It set up data pipelines that ran on either a batch processing or real-time basis, and these were orchestrated using Cloud Composer, a Google Cloud service built on the Apache Airflow open source project.
On top of that, NTUC Enterprise is visualising data in the lake using either an undisclosed “advanced visualisation tool” or alternatively Google Data Studio, which Hu described as “an easy tool for almost every staff [member] in our organisation.”
“We encourage them to try this out and to explore the data by themselves, and to build simple Google Data Studio dashboards,” he said.
The tool supported a broader “data democratisation” push by the co-operative to empower staff to analyse and seek insights from data holdings.
The co-operative is slowly making more use of the GCP stack for data, using Data Catalog for metadata management, and this year starting to explore Vertex AI, a Jupyter-based workbench for data scientists that NTUC Enterprise sees as potentially managing “end-to-end AI workflow”.
Hu said that NTUC Enterprise had some history of using data and analytics in its various social enterprises, and that AI and machine learning “exists in multiple forms in our organisations”.
Helping customers
He said that models were designed both to help end customers - such as those of its grocery businesses - as well as staff working in its operations.
“Take our grocery business as an example. We have a mobile app for the online channel, for the omni grocery business, but we use the customer data to understand the customer preference, customer behaviour, and to engage the customer in a more personalised way,” Hu said.
“The content in our mobile app adapts to different behaviour and different preferences of the customers.
“We built a recommendation engine to decide what categories and what SKUs [stock keeping units, otherwise known as products] we need to present to remind the customer to purchase.”
For staff in the grocery business or in fulfilment operations for online, NTUC Enterprise has focused on improving the operational efficiency of stock pickers.
A machine learning algorithm makes decisions about “optimally putting different orders together to assign to different pickers to make their work more productive,” Hu said.
“These are a few examples of the different applications using data analytics.”
Hu added that NTUC Enterprise’s data stack and architecture is still being enhanced.
“It’s a continuous innovation,” he said. “We are still trying to explore different kinds of possibilities.”
