How to unlock the value of data with cloud modernization

A modern, cloud-based infrastructure helps companies harness surges of untapped data to find new revenue opportunities and efficiency gains.

The world’s path to digitization accelerated during the pandemic. As people spent more time shopping online and working from home, the amount of data they produced was staggering: 64.2 zettabytes, according to the International Data Corporation (IDC). Over the next five years, IDC predicts that the amount of data created will be over twice the amount of data created since the advent of digital storage.

Organizations face their own data deluge—from consumers who engage with their products and services online to employees who work across their digital platforms. But more than 44% of organizations do not have clear data-driven strategies, according to a 2021 IDG Data and Analytics Survey. 

Companies without clarity around their data were caught off guard by the sheer amount of it produced by the pandemic, says Rahul Pathak, vice president of analytics at Amazon Web Services (AWS), whose teams focus on helping businesses solve large-scale data problems. “As businesses have moved more of their activity online, we’ve seen an explosion in data volumes,” he says. “However, customers with legacy infrastructure have struggled to keep up with the increased volumes; their systems were either hard or expensive to scale.”

This creates a missed opportunity for these companies to harness a wealth of untapped data to find new revenue opportunities or efficiency gains, says Pathak. “Either they’ve had to discard large volumes of data or the new data has taken them longer to process than normal, delaying the quality and velocity of their responses.”

Data transformation starts with migration

In order to manage this surge in data, organizations are increasingly moving to the cloud. This allows them to manage the magnitude of data in a secure and actionable way, says Sandy Carter, vice president, Worldwide Public Sector Partners and Programs, at AWS. Her team focuses on accelerating data-led migrations and modernization in the cloud. “With the cloud, organizations can unify data that was previously siloed, making it easier to analyze and gain insights,” she says. 

The first step in data transformation, says Carter, is to migrate data to a data lake in the cloud: Data lakes are used to store large datasets in highly available, secure, and flexible environments. Handling structured and unstructured data at any scale, they enable analysis across formerly siloed data. Carter points to one example where an AWS partner utilized data lakes when helping to provide colleges and universities with fast, easy access to critical institutional data.

“AWS Partner NorthBay helped Evisions, which provides administration software to higher education, to create an AWS-based transformative platform that features a new data ingestion layer, a software-as-a-service data lake, and a data consumption and access application,” says Carter. “These solutions allow Evisions’ customers to expand data access more broadly across their campuses and automate repeatable tasks.”

Once data is stored in the cloud effectively, it’s time to analyze it

After the data is stored in a data lake, says Carter, an organizations’ next step is to set up an analytics engine. 

“Analytics tools help provide insight from data,” she says. “They search, explore, filter, aggregate, and visualize data in near real time for application monitoring, log analytics, and clickstream analytics.”

Carter cites Arkhotech, an AWS Advanced Partner, as an example of a company effectively harnessing analytics. When distributed and disjointed data from different government institutions slowed the Subsecretaria de Prevencion del Delito of Chile (SPD) from meeting their mission of crime prevention, Arkhotech worked with SPD to migrate nine sources of data to a data lake on AWS. Arkhotech then used data analytics to make data reporting and public data publishing simpler and supplied data visualization tools to SPD—and the time it took to make information available to employees decreased from two weeks to one day. 

“The analytics also enabled SPD to gain accurate and timely insights that could drive business decisions, such as resource allocation, citizen protection needs, and reveal operational inefficiencies,” she says.

Carter identifies the final step to data transformation as leveraging artificial intelligence technologies, including machine learning and deep learning. 

“The business landscape and the corresponding rules regarding data are changing so frequently that organizations need systems to be intelligent and agile enough to adapt to these changes at a quicker pace,” she says. “Therefore, machine learning has become a necessary component of advanced data analytics systems. Machine learning systems evaluate data, assess the quality, predict missing inputs, and provide recommendations.”

Managing huge amounts of data in real time

Recently, the technology companies OSRAM and Continental launched a joint venture to develop innovative automotive lighting systems to meet the needs of modern mobility concepts. Speed was imperative during the launch: There was only a six-month window between the joint venture’s announcement and the first day of operation. OSRAM Continental had to set up an IT infrastructure within this limited window.

AWS helped OSRAM Continental build entirely new and ready-to-run IT infrastructure based on VMware Cloud on AWS. Working through one vendor allowed OSRAM Continental to get everything it needed in a short period of time, they set up an entire data center in three to four hours. The setup allowed for a speedy implementation within the cloud data center, as well as cost savings, maximum flexibility, and centralized management. 

Going forward, trends such as projections in and outside of the vehicle will require huge volumes of data, and much of this data will be managed in real time. 

“Managing real-time and streaming data is critical,” says Pathak. “AWS provides a number of services, such as Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka, that make it easy and cost-effective for customers to work with massive volumes of real-time data for advanced analytics, machine learning, edge computing, Internet of things, and more.”

And while a modern, cloud-based data and analytics foundation is necessary to be truly “data-driven,” Pathak notes, companies also need to think about culture and skills.

“Leaders need to establish data as an organizational asset, ensure data is available in a secure and well-governed way to the folks in their organizations that need it, and set an expectation that people use data to back up their decisions and recommendations.”