With multi-cloud and hybrid-cloud gaining traction in the changing data and analytics landscape, organisations need to “figure out” how to get data spread across different locations and clouds and then consume it in a way that complies with all the regulations, according to HSBC's Chief Data Science Architect of Wholesale Banking, Thomas Qian.
Qian, speaking at a fireside chat organised by Databricks, said over the next two-three years, apart from data, there will be more emphasis on metadata dominance.
“The key to success for companies lies in how to stitch them together,” he said.
He added that over the past five years, the industry has witnessed a shift from traditional business intelligence (BI) applications to the use of artificial intelligence (AI) to accelerate business growth.
Also, employee roles are changing from being generalist data analysts, to more focused ones like data engineer, machine learning engineer, or data scientist, Qian added.
Talking about elements of architecture that companies need to change to adapt to the new big-data world, Qian said that each cloud provider has their own preferences, and it's hard to have a single design that will fit all the company's needs.
“Companies have their own direction of where to go, hence there is a need to find a balance between cloud-agnostic and cloud-native approach,” Qian said.
Better to fail early
While growing and scaling AI practice in an organisation, Qian said it was good to have an open mindset and understand capabilities. This can help quick prototyping that allows the opportunity to learn fast as well as fail fast.
“It's better to fail in the early stage instead of rolling into production,” he added.
Further, to “maximise investments” in AI and machine learning (ML), organisations should focus on user experience, rather than consider “exact numbers” (in terms of investment) as the only metrics,” Qian said.
Companies depend on the “architecture design” to find alignment in bringing data together, governing and controlling, and managing it.
However, when people talk about architecture, they always think in the long term, for five or ten years, he added.
“In this complex, changing world, the focus should be on the next 12 months and revisit the whole design every two to three years,” Qian said.
On the “data governance” front, in the old days, the focus was only on data access control and on data quality Qian said and added: “Today, organisations have to deal with technology and understand where and why people are using data”.
“This opens up a whole new world for data governance perspective on how companies are managing all those data, and technology,” he said.
