qubole.com

A similar trend was observed among the prospective customers as well, some of the customers choose Qubole on top of their data lake to save cost.” Use of Qubole for machine learning for predicting and personalization of data science use cases will grow in 2021,” Purvang added. When a customer configures the Qubole Platform through AWS’s PrivateLink connectivity, the traffic between Qubole VPC and the customer’s VPC does not traverse through the public internet,” says Purvang. He highlighted the following capabilities that Qubole offers for data governance: • Granular and Efficient Updates and Deletes: ACID capabilities on data lake helps with the right to be forgotten and the right to be erased by making sure that data in the data lake is current and, if asked to be deleted, it is deleted.

As part of their mission in February 2020, Casey pointed out that AWS Data Exchange began working with their customers to source third-party data to help academics, researchers, and the healthcare community triage Covid-19 related issues. In one case, AWS’ customer Chan Zuckerberg BIOHUB were able to combine Covid-19 data from the AWS Data lake with their third-party data to help them in their predictive models to predict Covid-19 epidemiology. The AWS data lake team established a public data lake on Covid -19 in April 2020, leveraging data from AWS Data Exchange’s data providers. AWS Data exchange gives you single data ingestion and mechanism to feed data into your data lake or cloud data warehouse running on AWS.

However, use cases such as data exploration, Interactive Analytics, and Machine Learning require that the raw data is processed to create use-case driven trusted datasets. As a result, every data lake implementation must enable users to iterate between data engineering and use cases such as interactive analytics and Machine Learning. In a data lake, these pipelines are authored using standard interfaces and open source frameworks such as SQL, Python, Apache Spark, and Apache Hive. Also, a data lake supports programmatic access to data via standard programming languages such as R, Python, and Scala and standard libraries for numerical computation and machine learning such as TensorFlow, Apache Spark, MLib, MXNet, Tensorflow, Keras, and SciKit Learn.

The data management team, like other users within the organization, also needs functionality for usage monitoring to be able to analyze user-interaction and system performance data for uncovering past trends, report on this analysis, predict future outcomes and optimize system administration and management decisions taken by either humans or the machine. While data warehousing remains an important technology component of a comprehensive analytic data management platform, its capabilities must be extended with the latest cloud-native capabilities for supporting analysts and data scientists tasked with the cross-functional analysis of data from multiple internal and external sources, data arriving in batches and streams, and data residing in the cloud and on-premises. Modern analytic data management needs and requirements are driven by elevated awareness of the value of analytics in today’s uncertain market; growing volume, variety, and velocity of data; and availability of new data analysis and Artificial Intelligence and Machine Learning tools and techniques, many of which are based on open source technology. Additionally, it should support openness to use multiple storage formats, multiple data science and analytics tools and languages, and various AI frameworks, as well as the ability to develop multiple upstream and downstream data pipelines via connectors and application programming interfaces (APIs) and the ability to extend the solution by internal IT staff who are able to rely on industry-standard development tools and techniques.