Dataiku has released survey findings that uncover a gap in businesses’ investments in Generative AI and the ability of senior IT professionals to operationalise investments at scale. Senior IT leaders will be bold with their Generative AI initiatives over the next 12 months, as nearly three quarters (73%) plan to spend more than $500,000 and around half (46%) will spend over $1 million. Findings also confirm IT stacks across organisations are not comparable to modern infrastructure standards to maximise effectiveness and manage runaway costs.
Dataiku surveyed 200 senior analytics and IT leaders at enterprise companies across the globe in April 2024 to explore enablement against execution of Generative AI. Most agreed the proliferation of disconnected tools, lack of tech optionality, and outdated processes will intensify current challenges, such as data quality, governance, and risk management:
- Nearly half of respondents (44%) indicated that their current data tools do not fit their analytics and AI needs, and 43% reported that their current data analytics stack does not meet modern infrastructure standards. Another 88% do not have specific tools or processes for managing LLMs.
- Over three quarters of IT leaders agree that to modernise their data stack it means adding AI capabilities, followed by tool consolidation (65%).
- A majority of IT leaders (60%) said they use more than five tools to perform each step in the analytics process, from data ingestion to MLOps and LLMOps. Another 71% want five or less tools to reduce the burden of scaling projects with cobbled-together systems.
- Lack of governance and usage control can lead to compounding operational risk with a worrying portion of respondents (74%) still relying on spreadsheets for quick analyses, even as 62% have faced serious issues due to spreadsheet errors.
- Data quality and usability remains the biggest data infrastructure challenge that IT leaders face (45%), even with the high number of tools in their data stack. Together with data access issues, which were cited by 27% of respondents, organisations still have not solved the data quality problem.
“The reality is that Generative AI will continue to shift and evolve, with different technologies and providers coming and going. How can IT leaders get in the game while also staying agile to what’s next?,” said Conor Jensen, Field CDO of Dataiku. “All eyes are on whether this challenge — in addition to spiraling costs and other risks — will eclipse the value production of Generative AI. Our survey reveals most data stacks are not built to meet these needs.”
Simplify Data Stack Complexities
The process to reshape the data and analytics stack within enterprises is a complex change that has compounded with the introduction of Generative AI. As a next step to extending its Unified AI Platform, Dataiku recently launched embedded, as-you-go data quality infrastructure to allow everyone within an organisation the ability to more effectively operationalise data quality across the analytics and AI lifecycle. Together with the expansion of its LLM Mesh, with a Generative AI cost monitoring solution and a way to build tailored conversational AI chatbots, Dataiku continues to evolve the platform to enable AI transformation across organisations responsibly, and at scale.
