‘AI for Data’ before ‘Data for AI’

Touted as the next big thing, if not already one, AI has pushed business leaders across industry segments to put AI on top of their strategic priorities. Over the past year, I tried to fathom the frenzy, reality and impact of AI, especially in the Banking and Financial Industry. While many organizations are keen to adopt AI for driving business growth, value extraction from data is still in its nascent stage, with significant business value yet to be unlocked. Despite ambitious plans to leverage AI for meeting business objectives faster, what is startling though is despite the hype, use cases being explored for AI adoption, at least in the Banking and Financial industry, are still gravitating around improving Operational Efficiency, Process Optimization, Customer Insights & Experience, Fraud Detection, Calls Diversion, Email Sorting etc. which somehow, undermine the potential of AI.

Another stark revelation has been the dismally low rate of proof of concepts going live. The most plausible explanation for the resistance for transition from Proof of Concepts to Full-Scale Production could be the hallucinations resulting from bad data. Organisations are struggling to make trusted and contextual data available for AI and analytical functions. Vision and reality seem to be at odds with each other.

While we are all convinced that Data is the bedrock of AI, the question that remains unanswered is how confident are we on the data that fuels the AI Algorithms, models and simulations? While data driven decisionmaking in data programs is the top goal for 70% of data and analytics professionals, 25% of users see inaccurate data as the biggest hurdle to Gen AI adoption.  Over 70% users also experienced difficulties in defining processes for data governance and developing the ability to quickly integrate data into AI models, having an insufficient amount of training data.

This makes one ponder if the exceptional strength of this technology not be used to overcome massive data dependencies, steep learning curves and remediate data issues rather than expecting traditional data management tools and techniques to solve these problems?

Organisations today encounter broadly two categories of data issues for full scale adoption of AI:

1. Perennial Data Issues that Demand Augmented Governance:

• Co-existence of Multi-Generational Data Stores
• Increase in Variety and Volume of Unstructured and Semi-Structured Data
• Multi-Fold Increase in Data Consumers
• Establishing Authoritative Data Stores on the back of Mergers and Acquisitions
• Stringent Data Privacy and Protection Laws Coming into Force

2. Data Issues Compounded by AI:

• Increased Instances of Identity Theft owing to Incorrect/Non-Data Classification
• Hallucinations and Wrong Interpretations and Analysis owing to Lack of Context
• Unethical Usage resulting from Uncontrolled Access and Data Processing
• Quality of Data from Semi and Unstructured Data leading to incorrect Hypothesis and Models

Traditional data governance and management approaches and technologies often fall short of managing non-standardized data and data in motion. Unless industries can leverage AI for better data governance and management, the true potential of AI for realising business value cannot be salvaged. There has been a renewed focus on augmented data governance, given the challenges faced by data professionals in integrating data into AI models with insufficient amount of training data. Given the increasingly sophisticated and complex data landscape, AI can augment data Management practices and techniques.

Broader role of AI in Data Management:

• Data Integration: Considering the variety and format of Data sources that businesses have to deal with today, its critical to consider AI to automate data integration from different sources, formats, and structures. ML models can help map and transform data, making it more consistent and analysable. 
• Data Quality: Data Quality scope issues can occur across the lifecycle and need continuous monitoring. Traditional data quality approaches and techniques need to be revisited. Data quality aspects need to get embedded into the data observability solutions and with the help of AI algorithms Data pipelines and integrations, rules and logic can be monitored continuously and intelligently. ML models need to be trained to find, predict and self-heal the data inconsistencies.
• Data Relevancy and Contextualisation: Data scientists often struggle with the contextual understanding of data as right context and relevance is essential to segregate noise from information. NLP and AI-powered search engines can help by providing context to data from business applications, Data models and functional specs. 
• Data Lineage, Traceability and Auditability: Regulatory focus around Data traceability and auditability is forcing organisations to explore innovative ways to build lineage. AI models can help scan through the Business term definitions, BRD, FSD, Technical design, Test cases, System metadata, Data Models and logs.
• Ensure Data Completeness and Accuracy: AI methods and techniques can help augment data sets by suggesting missing values and data. Models can also be used for auto creation of quality rules based on RCAs and history. Besides, AI can also help create synthetic data for model training, testing and development.
• Detect Trends and Remediation: With data explosion, it’s critical to predict volume trends and AI-driven analytics tools can do that with high degree of confidence. AI can also help understand correlations, and hidden patterns inside huge datasets to predict major chnages in the market. 
• Data Security: AI can help with classifictaion of data and manage sensitive data to comply with regulations like GDPR, HIPAA, CCPA etc besides aspects like AI ethics and responsive AI.  Machine learning models can also be levergaed to examine any abnormal network traffic and user activity to predict potential security breaches.

In a nutshell, many issues that organisations consider as impediment for broader and deeper adoption of AI can be addressed by AI techniques. Many Data Management technology companies have integrated AI as part of their product offering. With a robust and unified governance, AI can help organisations place game changing bets.