AI-DRIVEN DATA OBSERVABILITY + FINOPS FOR AWS

Performance. Reliability. Cost-effectiveness.

Unravel’s automated, AI-powered data observability + FinOps platform for AWS and other modern data stacks provides 360° visibility to allocate costs with granular precision, accurately predict spend, run 50% more workloads at the same budget, launch new apps 3X faster, and reliably hit greater than 99% of SLAs.

Unravel Data Observability + FinOps for AWS you can:

• Launch new apps 3X faster: End-to-end observability of data-native applications and pipelines. Automatic improvement of performance, cost efficiency, and reliability.
• Run 50% more workloads for same budget: Break down spend and forecast accurately. Optimize apps and platforms by eliminating inefficiencies. Set guardrails and automate governance. Unravel’s AI helps you implement observability and FinOps to ensure you achieve efficiency goals.
• Reduce firefighting time by 99% using AI-enabled troubleshooting: Detect anomalies, drift, skew, missing and incomplete data end-to-end. Integrate with multiple data quality solutions. All in one place.
• Forecast budget with ⨦ 10% accuracy: Accurately anticipate cloud data spending to for more predictable ROI. Unravel helps you accurately forecast spending with granular cost allocation. Purpose-built AI, at job, user and workgroup levels, enables real-time visibility of ongoing usage.

To see Unravel Data for AWS in action contact: Data experts  | 650 741-3442

The post Data Observability for AWS Datasheet appeared first on Unravel.

Unravel Data Raises $50 Million Series D Funding to Accelerate the Next Generation of DataOps Observability

Third Point Ventures Leads Round to Enable Enterprises to
Unleash the Potential of the Modern Data Stack

Palo Alto, CA – September 27, 2022 – Unravel Data, the first DataOps observability platform built to meet the needs of modern data teams, today announced that it has closed a $50 million Series D round of funding to accelerate the next generation of DataOps observability. The round was led by Third Point Ventures, with participation from Bridge Bank and existing investors that include Menlo Ventures, Point72 Ventures, GGV Capital, and Harmony Capital, bringing the total amount of funding raised by Unravel Data to $107 million.

The investment comes as large enterprises face the challenge of operating an overwhelming number of data pipelines that are being used for data products, advanced modeling, and business-critical reporting, and at a time where the complexity of data systems is heightened by the shift to multi-cloud strategies and burdened by over-provisioned environments. As a result, data teams are struggling to deliver data outcomes in the time-efficient manner required and effectively manage the limitless rise in cloud compute and storage costs.

Unravel Data will use the investment to extend the Unravel Platform to help connect the dots from every system in the modern data stack within and across the most popular data ecosystems, including Databricks, Snowflake, Amazon EMR, BigQuery, and Dataproc. As the number of systems and data pipelines escalate, an entirely new way to manage and optimize the data pipelines that support the real-time analytics ambitions of the data-driven enterprise is needed.

“The DataOps observability market is poised to explode as enterprises invest in building data products that increase customers, revenue, and efficiencies,” said Curtis McKee, Partner at Third Point Ventures. “We’re excited to partner with Unravel Data, as the company has paved the way and established a proven track record of success helping some of the world’s most recognized brands simplify their data operations so they can bring new data-driven innovations to market.”

“Data engineers and data scientists currently spend more than half their day debugging and troubleshooting issues on the thousands of data pipelines in their environment,” said Kunal Agarwal, CEO of Unravel Data. “Just as the DevOps market united the practice of software development and operations a decade ago to transform the application lifecycle, data teams require the same kind of full-stack visibility, automation, and actionable intelligence that meet their needs around data pipeline performance, cost, and quality.”

Founded by Big Data pioneers Kunal Agarwal and Dr. Shivnath Babu, Unravel Data was born from the realization that the exponential growth of data combined with the broad adoption of the public cloud would require an entirely new way to manage and optimize the data pipelines that support the real-time analytics ambitions of the data-driven enterprise. Numerous Fortune 100 companies, including two of the top five global pharmaceutical companies and three of the top 10 financial institutions in the world, rely on Unravel Data to gain unprecedented visibility across their data stacks, proactively troubleshoot and optimize their data workloads, and define guardrails to govern costs and improve predictability. Customers who have deployed Unravel have been able to double productivity of data teams and ensure data applications run on time, while being able to scale cost efficiently on the cloud.

“At 84.51 (a Kroger company), we leverage cutting-edge data science and advanced analytics to create highly personalized experiences for millions of consumers. Our ability to proactively monitor the performance, cost, and quality of our data pipelines is foundational to our mission,” said Jeff Lambert, VP of Data Solutions at 84.51°, the wholly owned subsidiary of Kroger, the largest grocery chain in America. “Unravel enables us to see across our entire data environment, enabling our team to quickly understand which workloads are running well and which ones are having issues, and make recommendations on how we can fix the problems.”

As part of the new funding, Curtis McKee, Partner at Third Point Ventures, will be joining Unravel Data’s Board of Directors.

To learn more about how Unravel Data is helping data teams tackle some of today’s most complex big data challenges, visit: www.unraveldata.com. DataOps teams can preview how Unravel supports a wide variety of daily observability challenges by viewing the new library of quick demonstration videos on YouTube here.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading DataOps observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51 (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data Raises $50 Million Series D Funding appeared first on Unravel.

Current practice for Spark troubleshooting is messy. Part of this is due to Spark’s very popularity; it’s widely used on platforms as varied as open source Apache Spark, on all platforms; Cloudera’s Hadoop offerings (on-premises and in the cloud); Amazon EMR, Azure Synapse, and Google Dataproc; and Databricks, which runs on all three public clouds. (Which means you have to be able to address Spark’s interaction with all of these very different environments.)

Because Spark does so much, on so many platforms, “Spark troubleshooting” covers a wide range of problems – jobs that halt; pipelines that fail to deliver, so you have to find the issue; performance that’s too slow; or using too many resources, either in the data center (where your clusters can suck up all available resources) or in the cloud (where resources are always available, but your costs rise, or even skyrocket.)

Where Are the Issues – and the Solutions?

Problems in running Spark jobs occur at the job and pipeline levels, as well as at the cluster level, as described in Part 1 of this three-part series: the top ten problems you encounter in working with Spark. And there are several solutions that can help, as we described in Part 2: five types of solutions used for Spark troubleshooting. (You can also see our recent webinar, Troubleshooting Apache Spark, for an overview and demo.)

Table: What each level of tool shows you – and what’s missing

Existing tools provide incomplete, siloed information. We created Unravel Data as a go-to DataOps platform that includes much of the best of existing tools. In this blog post we’ll give examples of problems at the job, pipeline, and cluster levels, and show how to solve them with Unravel Data. We’ll also briefly describe how Unravel Data helps you prevent problems, providing AI-powered, proactive recommendations.

The Unravel Data platform gathers more information than existing tools by adding its own sensors to your stack, and by using all previously existing metrics, traces, logs, and available API calls. It gathers this robust information set together and correlates pipeline information, for example, across jobs.

The types of issues that Unravel covers are, broadly speaking: fixing bottlenecks; meeting and beating SLAs; cost optimization; fixing failures; and addressing slowdowns, helping you improve performance. Within each of these broad areas, Unravel has the ability to spot hundreds of different types of factors contributing to an issue. These contributing factors include data skew, bad joins, load imbalance, incorrectly sized containers, poor configuration settings, and poorly written code, as well as a variety of other issues.

Fixing Job-Level Problems with Unravel

Here’s an example of a Spark job or application run that’s monitored by Unravel.

In Unravel, you first see automatic recommendations, analysis, and insights for any given job. This allows users to quickly understand what the problem is, why it happened, and how to resolve it. In the example below, resolving the problem will take about a minute.

Let’s dive into the insights for an application run, as shown below.

You can see here that Unravel has spotted bottlenecks, and also room for improving the performance of this app. It has narrowed down what the particular problem is with this application and how to resolve it. In this case, it has recommended to double the number of executors and reduce the memory for each executor, which will improve performance by about 30%, meeting the SLA.

Additionally, Unravel has also spotted some bad joins which are slowing this application down, as shown below.

In addition to helping speed this application up, Unravel is also recommending resource settings which will lower the cost of running this application, as shown below – reductions of roughly 50% in executor memory and driver memory, cutting out half the total memory cost. Again, Unravel is delivering pinpoint recommendations. Users avoid a lengthy trial-and-error exercise; instead, they can solve the problem in about a minute.

Unravel can also help with jobs or applications that just didn’t work and failed. It uses a similar approach as above to help data engineers and operators get to the root cause of the problem and resolve it quickly.

In this example, the job or application failed because of an out of memory exception error. Unravel surfaces this problem instantly and pinpoints exactly where the problem is.

For further information, and to support investigation, Unravel provides distilled and easy-to-access logs and error messages, so users and data engineers have all the relevant information they need at hand.

And once data teams start using Unravel, they can do everything with more confidence. For instance, if they try to save money by keeping resource allocations low, but overdo that a little bit, they’ll get an out-of-memory error. Previously, it might have taken many hours to resolve the error, so the team might not risk tight allocations. But fixing the error only takes a couple of minutes with Unravel, so the data team can cut costs effectively.

Examples of logs that Unravel provides for easy access and error message screens follow.

Unravel strives to help users solve their problems with a click of a button. At the same time, Unravel provides a great deal of detail about each job and application, including displaying code execution, displaying DAGs, showing resource usage, tracking task execution, and more. This allows users to drill down to whatever depth needed to understand and solve the problem.

Task stage metrics in Unravel Data

As another example, this screen shows details for task stage information:

• Left-hand side: task metrics. This includes the job stage task metrics of Spark, much like what you would see from Spark UI. However, Unravel keeps history on this information; stores critical log information for easy access; presents multiple logs coherently; and ties problems to specific log locations.
• Right-hand side: holistic KPIs. Information such as job start and end time, run-time durations, I/O in KB – and whether each job succeeded or failed.

Data Pipeline Problems

The tools people use for troubleshooting Spark jobs tend to focus on one level of the stack or another – the networking level, the cluster level, or the job level, for instance. None of these approaches helps much with Spark pipelines. A pipeline is likely to have many stages, involving many separate Spark jobs.

Here’s an example. One Spark job can handle data ingest; a second job, transformation; a third job may send the data to Kafka; and a final job can be reading the data from Kafka and then putting it into a distributed store, like Amazon S3 or HDFS.

Airflow being used to create and organize a Spark pipeline.

The two most important orchestration tools are Oozie, which tends to be used with on-premises Hadoop, and Airflow, which is used more often in the cloud. They will help you create and manage a pipeline; and, when the pipeline breaks down, they’ll show you which job the problem occurred in.

But orchestration tools don’t help you drill down into that job; that’s up to you. You have to find the specific Spark run where the failure occurred. You have to use other tools, such as Spark UI or logs, and look at timestamps, using your detailed knowledge of each job to cross-correlate and, hopefully, find the issue. As you see, just finding the problem is messy, intense, time-consuming, expert work; fixing it is even more effort.

Oozie also gives you a big-picture view of pipelines.

Unravel, by contrast, provides pipeline-specific views that first connect all the components – Spark, and everything else in your modern data stack – and runs of the data pipeline together in one place. Unravel then allows you to drill down into the slow, failed, or inefficient job, identify the actual problem, and fix it quickly. And it gets even better; Unravel’s AI-powered recommendations will help you prevent a pipeline problem from even happening in the first place.

You didn’t have to look at Spark UI, plus dig through Spark logs, then check Oozie or Airflow. All the information is correlated into one view – a single pane of glass.

This view shows details for several jobs. In the graphic, each line has an instance run. The longest duration shown here is three minutes and 1 second. If the SLA is “under two minutes,” then the job failed to meet its SLA. (Because some jobs run scores or hundreds of times a day, missing an SLA by more than a minute – especially when that means a roughly 50% overshoot against the SLA – can become a very big deal.)

Unravel then provides history and correlated information, using all of this to deliver AI-powered recommendations. You can also set AutoActions against a wide variety of conditions and get cloud migration support.

Cluster Issues

Resources are allocated at the cluster level. The screenshot shows ResourceManager (RM), which tracks resources, schedules jobs such as Spark jobs, and so on. You can see the virtual machines assigned to your Spark jobs, what resources they’re using, and status – started or not started, completed or not completed.

Apache Hadoop ResourceManager

The first problem is that there’s no way to see what actual resources your job is consuming. Nor can you see whether those resources are being used efficiently or not. So you can be over-allocated, wasting resources – or running very close to your resources limit, with the job likely to crash in the future.

Nor can you compare past to present; ResourceManager does not have history in it. Now you can pull logs at this level – the YARN level – to look at what was happening, but that’s aggregated data, not the detail you’re looking for. You also can’t dig into potential conflicts with neighbors sharing resources in the cluster.

You can use site tools like Cloudwatch, Cloudera Manager or Ambari. They provide a useful holistic view, at the cluster level – total CPU consumption, disk I/O consumption, and network I/O consumption. But, as with some of the pipeline views we discussed above, you can’t take this down to the job level.

You may have a spike in cluster disk I/O. Was it your job that started that, or someone else’s? Again, you’re looking at Spark UI, you’re looking at Spark logs, hoping maybe to get a bit lucky and figure out what the problem is. Troubleshooting becomes a huge intellectual and practical challenge. And this is all taking away from time making your environment better or doing new projects that move the business forward.

It’s common for a job to be submitted, then held because the cluster’s resources are already tied up. The bigger the job, the more likely it will have to wait. But existing tools make it hard to see how busy the cluster is. So later, when the job that had to wait finishes late, no one knows why that happened.

A cluster-level view showing vCores, specific users, and a specific queue

By contrast, in this screenshot from Unravel, you see cluster-level details. This job was in the data security queue, and it was submitted on July 5th, around 7:30pm. These two rows show vCores – overall consumption on this Hadoop cluster’s memory. The orange line shows maximum usage, and the blue line shows what’s available.

At this point in time, usage (blue line) did not exceed available resources (orange line)

You can also get more granular and look at a specific user. You can go to the date and time that the job was launched and see what was running at that point in time. And voilà – there were actually enough resources available.

So it’s not a cluster-level problem; you need to examine the job itself. And Unravel, as we’ve described, gives you the tools to do that. You can see that we’ve eliminated a whole class of potential problems for this slowdown – not in hours or days, and with no trial-and-error experimentation needed. We just clicked around in Unravel for a few minutes.

Unravel Data: An Ounce of Prevention

For the issues above, such as slowdowns, failures, missed SLAs or just expensive runs, a developer would have to be looking at YARN logs, ResourceManager logs, and Spark logs, possibly spending hours figuring it all out. Within Unravel, though, they would not need to jump between all those screens; they would get all the information in one place. They can then use Unravel’s built-in intelligence to automatically root-cause the problem and resolve it.

Unravel Data solves the problem of Spark troubleshooting at all three levels – at the job, pipeline, and cluster levels. It handles the correlation problem – tying together cluster, pipeline, and job information – for you. Then it uses that information to give unique views at every level of your environment. Unravel makes AI-powered recommendations to help you head off problems; allows you to create AutoActions that execute on triggers you define; and makes troubleshooting much easier.

Unravel solves systemic problems with Spark. For instance, Spark tends to cause overallocation: assigning very large amounts of resources to every run of a Spark job, to try to avoid crashes on any run of that job over time. The biggest datasets or most congested conditions set the tone for all runs of the job or pipeline. But with Unravel, you can flexibly right-size the allocation of resources.

Unravel frees up your experts to do more productive work. And Unravel often enables newer and more junior-level people to be as effective as an expert would have been, using the ability to drill down, and the proactive insights and recommendations that Unravel provides.

Unravel even feeds back into software development. Once you find problems, you can work with the development team to implement new best practices, heading off problems before they appear. Unravel will then quickly tell you which new or revised jobs are making the grade.

The Unravel advantage – on-premises and all public clouds

Another hidden virtue of Unravel is: it serves as a single source of truth for different roles in the organization. If the developer, or an operations person, finds a problem, then they can use Unravel to highlight just what the issue is, and how to fix it. And not only how to fix it this time, for this job, but to reduce the incidence of that class of problem across the whole organization. The same goes for business intelligence (BI) tool users such as analysts, data scientists, everyone. Unravel gives you a kind of X-ray of problems, so you can cooperate in solving them.

With Unravel, you have the job history, the cluster history, and the interaction with the environment as a whole – whether it be on-premises, or using Databricks or native services on AWS, Azure, or Google Cloud Platform. In most cases you don’t have to try to remember, or discover, what tools you might have available in a given environment. You just click around in Unravel, largely the same way in any environment, and solve your problem.

Between the problems you avoid, and your new-found ability to quickly solve the problems that do arise, you can start meeting your SLAs in a resource-efficient manner. You can create your jobs, run them, and be a rockstar Spark developer or operations person within your organization.

Conclusion

In this blog post, we’ve given you a wide-ranging tour of how you can use Unravel Data to troubleshoot Spark jobs – on-premises and in the cloud, at the job, pipeline, and cluster levels, working across all levels, efficiently, from a single pane of glass.

In Troubleshooting Spark Applications, Part 1: Ten Challenges, we described the ten biggest challenges for troubleshooting Spark jobs across levels. And in Spark Troubleshooting, Part 2: Five Types of Solutions, we describe the major categories of tools, several of which we touched on here.

This blog post, Part 3, builds on the other two to show you how to address the problems we described, and more, with a single tool that does the best of what single-purpose tools do, and more – our DataOps platform, Unravel Data.

The post The Spark Troubleshooting Solution is The Unravel DataOps Platform appeared first on Unravel.

Mountain View, California — April 22, 2025 — Unravel Data, the first data actionability and FinOps platform built to address the cost, performance, and reliability of modern data platforms, today announced that it has been recognized on the CRN prestigious “Big Data 100” list for 2025. This annual list identifies the foremost technology vendors delivering innovative cloud-based big data analytics solutions that help enterprises harness the power of their data.

CRN^®, a brand of The Channel Company^®, has selected Unravel Data for its cutting-edge platform, which simplifies the management and optimization of complex data pipelines across multi-cloud and hybrid environments. The company’s inclusion on this year’s list underscores its commitment to helping organizations maximize performance, reduce costs, and increase the reliability of their modern data applications.

“We are honored to be recognized by CRN as one of the top cloud big data solution providers,” said Kunal Agarwal, CEO of Unravel Data. “This acknowledgment validates our mission to empower enterprises with the visibility and intelligence they need to succeed in today’s data-driven economy. As businesses continue to navigate the complexities of modern data stacks such as Databricks, Snowflake, and Google BigQuery, our platform ensures they can deliver on the promise of big data while controlling costs and maintaining operational excellence.”

Unravel Data’s platform stands out for its ability to:

• Provide end-to-end visibility across the entire data stack
• Automatically detect and remediate performance problems
• Optimize resource allocation and cloud spend
• Deliver AI-powered recommendations and automation for continuous improvement

“The companies on the CRN 2025 Cloud Big Data 100 list represent the cutting edge of innovation in cloud-based analytics and data management,” said Jennifer Follett, VP, US Content and Executive Editor, CRN. “These organizations are transforming how enterprises extract value from their data assets while addressing critical challenges around data governance, scalability, and cost optimization.”

Unravel Data continues to expand its platform capabilities and partner ecosystem to address the evolving needs of modern data teams. Recent enhancements include agentic automation to drive increased actionability, “two-click” insights, and real-time cost-performance evaluations down to job and user levels.

For more information about Unravel Data and its DataOps solutions, visit www.unraveldata.com.

About Unravel Data
Unravel Data is the leading provider of full-stack visibility and AI-powered insight and automation for modern data platforms. The Unravel FinOp and Data Actionability Platform leverages AI, machine learning, and advanced analytics to provide intelligent application performance management across the entire stack. Fortune 1000 companies rely on Unravel Data to drive self-service operations for their modern data applications. Founded in 2016, Unravel Data is headquartered in Mountain View, California, and backed by premier investors including Menlo Ventures, GGV Capital, and Harmony Partners.

Media Contact:
Keith Alsheimer
Chief Marketing Officer, Unravel Data
hello@unraveldata.com

About The Channel Company®
The Channel Company (TCC) is the global leader in channel growth for the world’s top technology brands. We accelerate success across strategic channels for tech vendors, solution providers, and end users with premier media brands, integrated marketing and event services, strategic consulting, and exclusive market and audience insights. TCC is a portfolio company of investment funds managed by EagleTree Capital, a New York City-based private equity firm. For more information, visit thechannelco.com.

Follow The Channel Company® on LinkedIn and X.

© 2025 The Channel Company, Inc. The Channel Company logo is a registered trademark of The Channel Company, Inc. All other trademarks and trade names are the properties of their respective owners. All rights reserved.

The Channel Company Contact:
Kristin DaSilva
The Channel Company
kdasilva@thechannelcompany.com

The post Unravel Data Honored With Recognition On The CRN Big Data 100 List For 2025 appeared first on Unravel.

Mountain View, California, January 21, 2025 — Unravel Data, the first Data Actionability and FinOps platform built to address the cost, performance, and reliability of modern data platforms, today announced that it has been recognized on the 2025 Cloud 100 list by CRN®, a brand of The Channel Company. This prestigious CRN list spotlights 100 leading channel-focused cloud companies across five key categories: cloud infrastructure, management, security, software, and storage.

CRN Cloud 100 companies demonstrate dedication to supporting channel partners and advancing innovation in cloud-based products and services. The list is the trusted resource for solution providers exploring cloud technology vendors that are well-positioned to help them build cloud portfolios that drive their success.

As cloud adoption accelerates, managing cloud costs, performance, and reliability has become increasingly crucial, with data management services emerging as the fastest-growing segment of cloud spending. Unravel Data is an advanced AI-driven data observability and FinOps solution that maximizes performance and cost efficiencies for cloud data analytics and AI workloads.

“Our platform can enable enterprises to more than double their workloads without increasing costs,” said Kunal Agarwal, CEO and co-founder of Unravel Data. “This achievement, coupled with our recognition as a CRN Cloud 100 Company for 2025, underscores our commitment to innovation and our ability to deliver substantial value to our customers and partners in the ever-evolving cloud landscape.”

Unravel’s cutting-edge AI and automation capabilities now offer predictive spend forecasting, proactive cost anomaly detection, and AI-generated optimization. These features provide real-time, user-level spend reporting and code-level optimization and implement automated spend controls that adapt to changing workload patterns. By leveraging advanced machine learning algorithms, Unravel empowers organizations to monitor data pipelines, detect code issues, and implement autonomous performance and infrastructure optimizations to increase ROI and ensure the reliability of SLAs.

At the core of the Unravel Data Actionability^TM Platform is the next-generation AI Insights Engine, designed to comprehend the intricacies of modern cloud data platforms, including Databricks, Snowflake, and BigQuery. Billions of data points are ingested in real-time, providing predictive analytics and autonomous optimization. This AI-driven approach allows Unravel to anticipate performance bottlenecks, preemptively optimize resource allocation, and automatically implement cost-saving measures, surpassing traditional manual management methods in today’s complex, hyper-scale data environments.

“As customer cloud needs accelerate, particularly in the face of expanding needs for digital transformation and AI-based solutions, cloud innovation has become more important than ever,” said Jennifer Follett, VP, U.S. Content, and Executive Editor, CRN, at The Channel Company. “Each company on this year’s Cloud 100 list are breaking new ground delivering products and services that empower solution providers to expand their cloud offerings and meet their customers’ requirements. We look forward to seeing how these companies continue to advance cloud computing in the coming year.”

CRN’s Cloud 100 list will be featured in the February 2025 issue of CRN magazine and online at www.crn.com/cloud100 beginning January 21.

About Unravel Data
Unravel Data radically transforms how businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Unravel’s market-leading data observability and FinOps platform, which has purpose-built AI for each data platform, provides actionable recommendations for cost and performance data and AI pipeline efficiencies. A winner of the Best Data Tool & Platform as part of the annual SIIA CODiE Awards, some of the world’s most recognized brands, like Maersk, Mastercard, and Equifax, rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Contact:
Keith Alsheimer
Unravel Data
hello@unraveldata.com

About The Channel Company
The Channel Company (TCC) is the global leader in channel growth for the world’s top technology brands. We accelerate success across strategic channels for tech vendors, solution providers, and end users with premier media brands, integrated marketing and event services, strategic consulting, and exclusive market and audience insights. TCC is a portfolio company of investment funds managed by EagleTree Capital, a New York City-based private equity firm. For more information, visit thechannelco.com.

Follow The Channel Company: LinkedIn, X, and Facebook.

© 2025 The Channel Company, Inc. CRN is a registered trademark of The Channel Company, Inc. All rights reserved.

The Channel Company Contact:
Kristin DaSilva
The Channel Company
kdasilva@thechannelcompany.com

The post Unravel Data Earns Spot on CRN’s Cloud 100 List for 2025 appeared first on Unravel.

UNRAVEL DATA SECURITY AND TRUST
ENABLE DATA ACTIONABILITY + FINOPS WITH CONFIDENCE

Privacy and security are top priorities for Unravel and our customers. At Unravel, we help organizations better understand and improve the performance, quality, and cost efficiency of their data and AI pipelines. As a data business, we appreciate the scope and implications of privacy and security threats.

This data sheet provides details to help information security (InfoSec) teams make informed decisions. Specifically, it includes:

• An overview of our approach to security and trust
• An architectural diagram with connectivity descriptions
• Details about Unravel compliance and certifications
• Common questions about Unravel privacy and security

For additional details, please reach out to our security experts.

The post Unravel Data Security and Trust appeared first on Unravel.

Unravel Data, the first AI-enabled data actionability and FinOps platform built to address the speed and scale of modern data platforms, today announced it has been included as a Sample Vendor in the Gartner® Hype Cycle for Container Technology, 2024 in the Augmented FinOps category.

Unravel’s Perspective

How Augmented FinOps Helps

Augmented FinOps empowers organizations by automating and enhancing financial operations through AI and automation. This innovative approach provides real-time insights into cloud spending, identifies cost-saving opportunities, and ensures budget adherence. By leveraging domain-specific knowledge and intelligent automation, Augmented FinOps reduces manual workload, improves financial accuracy, and optimizes resource allocation. Ultimately, it drives efficiency, enabling businesses to focus on strategic growth while ensuring financial health and governance.

Introducing Unravel’s New AI Agents

Unravel recently announced the three groundbreaking new AI agents: the Unravel FinOps AI Agent, the Unravel DataOps AI Agent, and the Unravel Data Engineering AI Agent. These AI agents are designed to transform how data teams manage and optimize their operations. The Unravel FinOps AI Agent helps automate financial governance, providing near real-time insights into cloud expenditures with showback and chargeback reports, identifying cost-saving opportunities and enabling action. The Unravel DataOps AI Agent streamlines data pipeline monitoring and anomaly detection, and troubleshooting, freeing up human experts for more strategic tasks. Meanwhile, the Unravel Data Engineering AI Agent enhances productivity by automating routine tasks, allowing data engineers to focus on high-value problem-solving. Together, these AI agents empower organizations to achieve greater efficiency, accuracy, and innovation in their data operations, driving transformative business outcomes.

Three Keys to Optimizing Containers for Your Modern Data Stack

In today’s fast-paced digital landscape, optimizing containers is crucial for the efficiency and scalability of your modern data stack. Here are three key strategies to ensure your containerized environments are performing at their best:

1. Implement FinOps for Containers:

FinOps principles like monitoring, optimization, automation, and cost allocation can be applied to container workloads to enhance their efficiency, scalability, and cost-effectiveness. Advanced observability tools, integrated with AI-driven insights, can proactively identify potential problems and suggest optimizations, keeping your data stack running efficiently.

2. Right-Size Your Containers:

Properly sizing your containers is essential to prevent resource wastage and ensure optimal performance. Over-provisioning can lead to unnecessary costs, while under-provisioning can cause performance bottlenecks. Utilize AI-powered tools and automation that provide real-time monitoring and analytics to understand your workloads’ demands and adjust resources accordingly. This dynamic approach helps maintain a balance between cost and performance, ensuring your applications run smoothly without incurring excessive expenses.

3. Automate Management and Scaling:

Automation is a game-changer in container management, allowing for seamless scaling and resource allocation based on real-time demands. Employ automation tools that can handle tasks such as load balancing, resource provisioning, and fault tolerance. Kubernetes, for example, offers powerful automation capabilities that can dynamically manage container orchestration, ensuring your data stack can scale efficiently as your workloads grow. By automating these processes, you reduce the risk of human error, increase operational efficiency, and ensure that your infrastructure can adapt to changing demands without manual intervention.

Optimizing containers using a FinOps approach enables teams to right-size, automate, and scale not only enhances performance and cost-efficiency but also ensures your modern data stack is resilient, scalable, and ready to meet the demands of today’s data-driven world.

Next Steps

Ready to optimize your data operations? Discover the transformative impact of Unravel’s new AI agents. Request a free health check to see how your organization can improve performance, efficiency, and cost management. Start your journey towards smarter, more actionable data insights today.

Gartner, Hype Cycle for Container Technology, By Dennis Smith, 20 June 2024
GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The post Unravel Data was Mentioned in the Gartner® Hype Cycle for Container Technology, 2024 appeared first on Unravel.

Unravel AI Agents Empower DataOps, FinOps and Data Engineering Teams to Move Beyond Observability to Achieve Data Actionability^TM

PALO ALTO, Calif.– June 12, 2024 – Unravel Data, the first AI-enabled data actionability^TM and FinOps platform built to address the speed and scale of modern data platforms, today announced the release of three groundbreaking new AI agents: the Unravel DataOps Agent, the Unravel FinOps Agent, and the Unravel Data Engineering Agent.

While generative AI capabilities such as Large Language Models allow users to ask general questions and receive broad answers, AI agents are designed to perform specific, actionable tasks within their domains. These AI agents leverage domain-specific knowledge graphs and advanced automation to tackle precise problems faced by data teams, significantly enhancing efficiency and accuracy. All three new AI agents are included as part of the latest version of Unravel Platform.

The introduction of AI agents has the potential to transform various data-centric disciplines where the complexity of managing and making decisions about data pipelines often requires significant human intervention. In DataOps, AI agents can automate routine tasks like data pipeline monitoring and anomaly detection, freeing up human experts for more strategic endeavors. Meanwhile, AI agents for FinOps teams can be deployed to continuously track and analyze cloud expenditures for data storage, processing, and analytics, identifying cost-saving opportunities and potential budget overruns.

“Over the past decade, Unravel has been at the forefront of data observability, continuously innovating to meet the evolving needs of our enterprise customers. With this launch, we are taking customers beyond observability, to actionability,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “The market is demanding solutions that don’t simply observe and tell you what’s happening, but make it actionable by telling you how to solve the problem. And better still, to fix it for them. This will allow resource crunched data teams to get more done through smart automation.”

AI agents hold immense potential for transforming data-related disciplines by automating the many time-consuming and tedious tasks that consume the time of under-resourced data teams. Some of the distinguishing capabilities of Unravel’s AI Agents include:

• Domain-Specific Design: Unravel’s AI agents leverage deep domain-specific knowledge graphs to address the specific challenges data teams face daily.
• Flexibility and Control: Users can choose their preferred level of AI automation, from highly supervised to fully automated, ensuring they are able to maintain the proper amount of oversight over critical data-driven processes.
• Enhanced Focus for Data Engineers: By automating mundane tasks, these AI agents enable data engineers to reduce boring and repetitive tasks, maximize productivity and focus on solving critical, high-value problems.
• Precision and Reliability: Built to provide accurate and reliable solutions, Unravel’s AI agents are finely tuned to handle precise data operations issues.
• Comprehensive Cost Management: For FinOps teams, the AI agents facilitate automated cost governance, uncover savings opportunities, and proactively manage budgets.

Maersk, one of the world’s largest logistics companies, recognized the benefits of Unravel’s AI agents to streamline their data processes, optimize their cloud spending, and enhance their operational efficiency. “Unravel enables developers to proactively manage complex data systems effortlessly,” said Peter Rees, Lead Architect for Enterprise Data at Maersk. “By providing clear, actionable information through a conversational interface, we look forward to using these AI agents to help us detect and troubleshoot issues faster, propose configuration changes, and allow developers to approve and apply those changes seamlessly.”

To learn more about how Unravel agents can deliver actionable insights into your data pipelines, visit: www.unraveldata.com

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Unravel’s market-leading data actionability^TM and FinOps platform with purpose-built AI for each data platform, provides prescriptive recommendations needed for cost and performance data and AI pipeline efficiencies. A recent winner of the Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards, some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

PR Contact:
Rob Nachbar
Kismet Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data Unveils Data Industry’s First Purpose-Built Autonomous AI Agents appeared first on Unravel.

Palo Alto, CA – Apr. 24, 2024 – Unravel Data is proud to announce A.P. Moller – Maersk, a global leader in container logistics, is participating at the upcoming Gartner Data & Analytics Summit. Scheduled from May 13-15, 2024, at ExCeL London, this summit is renowned for gathering visionaries and innovators in the realm of data and analytics.

Peter Rees, Lead Architect at Maersk, together with Mark Sear, Director of Insight Analytics, Data, and Integration at Maersk, will be leading a must-attend session titled “Unravel Data: How to stop burning money (or at least slow the burn).” This session is meticulously designed to address the escalating concern of unpredictable growth in data analytics costs which can significantly hinder the progression of data-driven innovation.

Session Overview:

Data analytics costs are spiraling, and businesses are searching for methodologies to efficiently manage and optimize these expenses without compromising on innovation or operational agility. Maersk, leveraging Unravel Data’s cutting-edge solutions, has pioneered a cost-optimization framework that not only streamlines development and data delivery processes but also aligns with the enterprise’s mission to deliver a more connected, agile, and sustainable future for global logistics.

During the session, attendees will gain exclusive insights into how Maersk has successfully harnessed the power of Unravel Data within its infrastructure to ensure the business remains at the forefront of cost efficiency while bolstering its data-driven decision-making capabilities.

Speaker Highlights:

• Peter Rees, as Maersk’s Lead Architect specializing in Enterprise Data & AI/ML, brings a wealth of knowledge in data mesh and event-driven architectures. His extensive track record in AI strategies and analytics, complemented by an innovative mindset, positions him as a cornerstone in the conversation on bridging data technology with business value.
• Mark Sear elevates the discourse with his profound expertise in digital transformation and business intelligence as Maersk’s Director of Insight Analytics, Data, and Integration. Mark’s academic and professional achievements underscore his commitment to leveraging data for actionable insights, thus fostering strategic business growth and operational efficiencies.

Event Details:

• What: Gartner Data & Analytics Summit
• When: May 13-15, 2024
• Where: ExCeL London, UK
• Session Title: Unravel Data: How to stop burning money (or at least slow the burn)

Unravel Data invites all attendees who are looking to navigate the challenges of data analytics cost growth to join this session. It promises to be an enlightening exploration of practical solutions, real-world applications, and visionary strategies for any organization aiming to optimize their data-driven initiatives and investments.

About A.P. Moller – Maersk:
A.P. Moller – Maersk is an integrated container logistics company working to connect and simplify its customers’ supply chains. As a global leader in shipping services, the company operates in 130 countries and employs over 80,000 people. For further information, visit www.maersk.com.

About Unravel Data:
Unravel’s automated, AI-powered data observability + FinOps provides 360° visibility to allocate costs with granular precision, accurately predict spend, run 50% more workloads at the same budget, launch new apps 3X faster, and reliably hit greater than 99% of SLAs. For further information, visit www.unraveldata.com.

Media Contact:
Keith Alsheimer
CMO, Unravel Data
hello@unraveldata.com

The post Maersk data leaders speaking at Gartner Data & Analytics Summit London appeared first on Unravel.

AI-DRIVEN DATA OBSERVABILITY + FINOPS FOR SNOWFLAKE DATA ENGINEERS

Snowflake data engineers are under enormous pressure to deliver results. This data sheet provides more context about the challenges data engineers face and how Unravel helps them address these challenges.

Specifically, it discusses:

• Key Snowflake data engineering roadblocks
• Unravel’s purpose-built AI for Snowflake
• Data engineering benefits

With Unravel, Snowflake data engineers can speed data pipeline development and analytics initiatives with granular and real-time cost visibility, predictive, predictive spend forecasting, and performance insights for their data cloud.

To see Unravel Data for Snowflake in action contact: Data experts

The post Data Observability + FinOps for Snowflake Engineers appeared first on Unravel.

PALO ALTO, CA — Jan. 22, 2024 – Unravel Data, the first AI-enabled data observability and FinOps platform built to address the speed and scale of modern data platforms, today announced that CRN^®, a brand of The Channel Company, has named the company to its annual Cloud 100 list. The list spotlights technology suppliers for their commitment to channel partners as well as their demonstrated innovation in cloud-based technology development and serves as a trusted resource for solution providers looking for technology vendors best positioned to support their cloud product and services needs. 

“As cloud costs continue to climb, managing cloud spend remains a top challenge with data management services leading as the fastest-growing category of cloud spending. With the Unravel Data Platform — the first data observability and FinOps platform that unlocks both the performance and cost efficiencies that data teams need to achieve speed, scale, and ROI for cloud data analytics and AI products — enterprises are able to achieve up to twice as many workloads for the same spend,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “We work closely with our channel partners to ensure that our joint customers are able to optimize not only the costs but the performance of their cloud data projects and so it’s extremely gratifying to be recognized by CRN.” 

Unravel leverages AI and automation to provide real-time, user-level spend reporting, code-level cost optimization tips, and automated spend controls designed to empower and unify data and FinOps teams. With Unravel, organizations can monitor data flows through their pipelines, and detect code, configuration, and infrastructure issues. By correlating and analyzing the full stack of telemetry metadata, Unravel provides easy-to-understand insights, actionable recommendations, and automation to optimize performance and cost before costs show up on an invoice.

At Unravel’s core is an AI-powered Insights Engine, which is purpose-built and trained to understand all the intricacies and complexities of specific modern cloud data platforms. Unravel provides platform-specific solutions for Databricks, Amazon EMR, Google Cloud BigQuery, and Snowflake. The Engine has been built to ingest and interpret the continuous millions of ongoing data streams to provide real-time insights into application and system performance, and recommendations to optimize costs, including right sizing instances and applying code recommendations for efficiencies. In addition, the Engine has been trained to understand optimal performance indicators and resulting financial implications. Without the power of the Insights Engine, the vast amount of data, system complexity, and data pipelines makes it impossible for people to manage data for performance and cost efficiency.

“As migration to the public cloud and cloud-based software accelerates, enterprises increasingly depend on innovative, secure cloud services to harness the cloud’s agility and scalability,” said Jennifer Follett, VP, US Content and Executive Editor, CRN, The Channel Company. “The companies selected for this year’s Cloud 100 list demonstrate a strong commitment to supporting cloud computing solution providers with leading-edge products and services. Congratulations to those on this year’s list! We look forward to seeing how they propel innovation and channel success in cloud computing throughout the year ahead.”

CRN’s Cloud 100 list honors the 100 leading cloud companies for 2024 across five key categories: infrastructure, monitoring and management, storage, software, and security. The list will be featured in the February 2024 issue of CRN magazine and online at www.crn.com/cloud100.

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Unravel’s market-leading data observability and FinOps platform with purpose-built AI for each data platform, provides actionable recommendations needed for cost and performance data and AI pipeline efficiencies. A recent winner of the Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards, some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

The post CRN Recognizes Unravel Data as a Cloud 100 Company for 2024 appeared first on Unravel.

Purpose-built AI provides real-time cost and performance insights and efficiency recommendations for Databricks users

Palo Alto, CA — December 5, 2023 — Unravel Data, the first AI-enabled data observability and FinOps platform built to address the speed and scale of modern data platforms, today announced that it has joined the Databricks Partner Program to deliver AI-powered data observability into Databricks for granular visibility, performance optimizations, and cost governance of data pipelines and applications. With this new partnership, Unravel and Databricks will collaborate on Go-To-Market (GTM) efforts to enable Databricks customers to leverage Unravel’s purpose-built AI for the Lakehouse for real-time, continuous insights and recommendations to speed time to value of data and AI products and ensure optimal ROI.   

With organizations increasingly under pressure to deliver data and AI innovation at lightning speed, data teams are on the front line of delivering production-ready data pipelines at an exponential rate while optimizing performance and efficiency to deliver faster time to value. Unravel’s purpose-built AI for Databricks integrates with Lakehouse Monitoring and Lakehouse Observability to deliver performance and efficiency needed to achieve speed and scale for data analytics and AI products. Unravel’s integration with Unity Catalog enables Databricks users to speed up lakehouse transformation by providing real-time, AI-powered cost insights, code-level optimizations, accurate spending predictions, and performance recommendations to accelerate data pipelines and applications for greater returns on cloud data platform investments. AutoActions and alerts help automate governance with proactive guardrails.

“Most organizations today are receiving unprecedented amounts of data from a staggering number of sources, and they’re struggling to manage it all, which can quickly lead to unpredictable cloud data spend. This combination of rapid lakehouse adoption and the hyperfocus companies have on leveraging AI/ML models for additional revenue and competitive advantage, brings the importance of data observability to the forefront,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “Lakehouse customers who use Unravel can now achieve the agility required for AI/ML innovation while having the predictability and cost governance guardrails needed to ensure a strong ROI.”

Unravel’s purpose-built AI for Databricks delivers insights based on Unravel’s deep observability at the job, user, and code level to supply AI-driven cost efficiency recommendations, including compute provisioning, query performance, autoscaling efficiencies, and more. 

Unravel for Databricks enables organizations to:

• Speed cloud transformation initiatives by having real-time cost visibility, predictive spend forecasting, and performance insights for their workloads 
• Enhance time to market of new AI initiatives by mitigating potential pipeline bottlenecks and associated costs before they occur
• Better manage and optimize the ROI of data projects with customized dashboards and alerts that offer insights on spend, performance, and unit economics

Unravel’s integration with popular DevOps tools like GitHub and Azure DevOps provides actionability in CI/CD workflows by enabling early issue detection during the code-merge phase and providing developers real-time insights into potential financial impacts of their code changes. This results in fewer production issues and improved cost efficiency.

Learn how Unravel and Databricks can help enterprises optimize their cloud data spend and increase ROI here.   

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Unravel’s market-leading data observability and FinOps platform with purpose-built AI for each data platform, provides actionable recommendations needed for cost and performance data and AI pipeline efficiencies. A recent winner of the Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards, some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

The post Unravel Data Partners with Databricks for Lakehouse Observability and FinOps  appeared first on Unravel.

Efficiency Recommendations for Infrastructure, Configuration, and Code

PALO ALTO, CA — November 14, 2023 – Unravel Data, the first AI-enabled data observability and FinOps platform built to address the speed and scale of modern data platforms, today announced the release of Unravel for Snowflake. By employing AI that is purpose-built for managing the Snowflake technology stack, cloud data cost management is put into the hands of Snowflake customers by providing them with granular insights into specific cost drivers, as well as AI-driven cost and performance recommendations for optimizing SQL queries and data applications. Unravel for Snowflake is the latest data observability and FinOps product from Unravel Data, adding to the portfolio of purpose-built AI solutions that include Databricks, EMR, Cloudera, and BigQuery. 

Today, companies are looking to AI to provide them with a competitive advantage, which is driving an exponential increase in data usage and workloads, use cases, pipelines, and generative AI/LLM models. In turn, companies are facing even greater problems with broken pipelines and inefficient data processing, slowing time-to-business value and adding to exploding cloud data bills. Unfortunately, most companies lack visibility into their data cloud spend or ways to optimize data pipelines/workloads to lower spend, speed innovation, and mitigate problems. 

Unravel’s purpose-built AI for Snowflake delivers insights based on Unravel’s deep observability at granular levels to deliver AI-driven cost optimization recommendations for warehouses and SQL that include: warehouse provisioning, run-time, auto-scaling efficiencies, and more. With Unravel, Snowflake users can see real-time cost usage by query, user, department, and warehouse, and set customized dashboards, alerts, and guardrails to enable accurate, granular cost allocation, trend visualization, and forecasting.  

“As companies double down on AI efforts, we can expect to see more wasted data cloud spend. Costs are incurred not only with infrastructure but with consumption, as most AI pipelines are created in ways that drive up unnecessary cloud data costs,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “Data engineering and architecture teams need an early warning system to alert them to out-of-control spending, an automated way to pinpoint the source of performance issues and cost overruns, and AI-driven recommendations to optimize code in ways that mitigate unnecessary costs, speed new development, and eliminate data pipeline problems.”

At the core of Unravel Data’s platform is its AI-powered Insights Engine, which has been trained to understand all the intricacies and complexities of modern data platforms and the supporting infrastructure. The Insights Engine has been built to ingest and interpret the continuous millions of ongoing data streams to provide real-time insights into application and system performance, and recommendations to optimize costs, including right-sizing instances and applying code recommendations for performance and financial efficiencies. When combined with Unravel’s automated guardrails and alerts, the Insights Engine enables organizations to achieve data cloud efficiency at scale.  

“Our latest research shows that the adopters of cloud data warehouses struggle with data pipeline complexity, lack of staff/expertise, and an inability to predict workloads,” says Kevin Petrie, VP of Research at The Eckerson Group. “FinOps platforms for cloud data analytics, such as Unravel, provide the granular visibility that stakeholders need to predict and monitor spending. This makes it easier for companies to optimize workloads, change user behavior, and get a handle on governing cloud costs.”

Unravel for Snowflake includes additional features such as:

• Visibility for cost allocation with chargeback/showback reports 
• Warehouse-level insights and recommendations relating to warehouse consolidation and underutilization efficiencies
• Compute + storage unit cost reporting with average cost per project, query, and user over time
• SQL-related insights and recommendations for optimizing queries by filters, joins, projection inefficiencies, anti-patterns, and more to improve query efficiency and increase capacity so that more users and requests can be served at the same spend
• Dashboard customization for at-a-glance summaries and drill-down insights for spend, performance, and unit costs
• Alert customization using OpenSearch-based alerts beyond Snowflake’s out-of-the-box alerts to enable early warnings of resource usage spikes before they hit the cloud bill

To learn more about how we are helping Snowflake customers optimize their data cloud costs and to request a complimentary “Health Check” – projected annual cost savings for your Snowflake warehouses using Unravel’s optimization insights and recommended actions to start saving–  visit Unravel for Snowflake.

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Unravel’s market-leading data observability and FinOps platform with purpose-built AI for each data platform, provides actionable recommendations needed for cost and performance data and AI pipeline efficiencies. A recent winner of the Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards, some of the world’s most recognized brands like Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com 

The post Unravel Data Launches Cloud Data Cost Optimization for Snowflake appeared first on Unravel.

New Functionality Delivers FinOps, AI-driven Cloud Cost Management and Performance Optimization for BigQuery Users

PALO ALTO, CA — August 10, 2023 – Unravel Data, the first AI-enabled data observability and FinOps platform built to address the speed and scale of modern data platforms, today announced the release of Unravel 4.8.1, enabling Google Cloud BigQuery customers to see and better manage their cloud data costs by understanding specific cost drivers, allocation insights, and performance and cost optimization of SQL queries. This launch comes on the heels of the recent BigQuery pricing model change that replaced flat-rate and flex slot pricing with three new pricing tiers, and will help BigQuery customers to implement FinOps in real time to select the right new pricing plan based on their usage, and maximize workloads for greater return on cloud data investments.

As today’s enterprises implement artificial intelligence (AI) and machine learning (ML) models to continually garner more business value from their data, they are experiencing exploding cloud data costs, with a lack of visibility into cost drivers and a lack of control for managing and optimizing their spend. As cloud costs continue to climb, managing cloud spend remains a top challenge for global business leaders. Data management services are the fastest-growing category of cloud service spending, representing 39% of the total cloud bill. Unravel 4.8.1 enables visibility into BigQuery compute and storage spend and provides cost optimization intelligence using its built-in AI to improve workload cost efficiency. 

Unravel’s AI-driven cloud cost optimization for BigQuery delivers insights based on Unravel’s deep observability of the job, user, and code level to supply AI-driven cost optimization recommendations for slots and SQL queries, including slot provisioning, query duration, autoscaling efficiencies, and more. With Unravel, BigQuery users can speed cloud transformation initiatives by having real-time cost visibility, predictive spend forecasting, and performance insights for their workloads. BigQuery customers can also use Unravel to customize dashboards and alerts with easy-to-use widgets that offer insights on spend, performance, and unit economics.

“As AI continues to drive exponential data usage, companies are facing more problems with broken pipelines and inefficient data processing which slows time to business value and adds to the exploding cloud data bills. Today, most organizations do not have the visibility into cloud data spend or ways to optimize data pipelines and workloads to lower spend and mitigate problems,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “With Unravel’s built-in AI, BigQuery users have data observability and FinOps in one solution to increase data pipeline reliability and cost efficiency so that businesses can bring even more workloads to the cloud for the same spend.” 

“Enterprises are increasingly concerned about lack of visibility into and control of their cloud-related costs, especially for cloud-based analytics projects,” says Kevin Petrie, VP of Research at The Eckerson Group. “By implementing FinOps programs, they can predict, measure, monitor, optimize and account for cloud-related costs related to data and analytics projects.”

At the core of Unravel Data’s platform is its AI-powered Insights Engine, purpose-built for data platforms, which understands all the intricacies and complexities of each modern data platform and the supporting infrastructure to optimize efficiency and performance. The Insights Engine ingests and interprets the continuous millions of ongoing metadata streams to provide real-time insights into application and system performance, and recommendations to optimize costs and performance for operational and financial efficiencies. 

Unravel 4.8.1 includes additional features, such as:

• Recommendations for baseline and max setting for reservations
• Scheduling insights for recurring jobs
• SQL insights and anti-patterns
• Recommendations for custom quotas for projects and users
• Top-K projects, users, and jobs
• Showback by compute and storage types, services, pricing plans, etc.
• Chargeback by projects and users
• Out-of-the-box and custom alerts and dashboards
• Project/Job views of insights and details
• Side-by-side job comparisons
• Data KPIs, metrics, and insights such as size and number of tables and partitions, access by jobs, hot/warm/cold tables

To learn more on how we are helping BigQuery customers optimize their data cost and management, or to partner with Unravel Data, please visit https://www.unraveldata.com/google-cloud-bigquery/.

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. A recent winner of the Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards, some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact

Blair Moreland

ZAG Communications for Unravel Data

unraveldata@zagcommunications.com 

The post Unravel Data Launches Cloud Data Cost Observability and Optimization for Google Cloud BigQuery appeared first on Unravel.

Unravel Data Recognized in Both Augmented FinOps and Data Observability Categories by Gartner

PALO ALTO, CA — July 20, 2023 – Unravel Data, the first data observability and FinOps platform built to address the speed and scale of modern data platforms, today announced it has been included as a Sample Vendor in the Gartner^® Hype Cycle for Emerging Technologies in Finance, 2023 for both Augmented FinOps and Data Observability. Aimed at helping Chief Financial Officers (CFOs) identify the top emerging technologies shaping the future of finance, this Hype Cycle offers a 10-year outlook on the most relevant technology and data trends, and provides recommendations for CFOs looking to increase flexibility and resiliency while also increasing productivity and profitability.

As more data workloads move to the cloud, IT and financial leaders must increasingly measure and optimize efficiency (both cost and performance). Both augmented financial operations (FinOps) and data observability are crucial elements to this process, helping organizations maximize the business value of their data. According to the report, “data observability is a technology that supports an organizations’ ability to understand the health of an organization’s data, data pipelines, data landscape, and data infrastructure by continuously monitoring, tracking, alerting and troubleshooting issues to reduce and prevent data errors or system downtime.”

The report further states that Augmented FinOps, “applies the traditional DevOps concepts of agility, continuous integration and deployment, and end-user feedback to financial governance, budgeting and cost optimization efforts. Augmented FinOps automates this process through the application of artificial intelligence (AI) and machine learning (ML) practices — predominantly in the cloud — to enable environments that automatically optimize cost based on defined business objectives expressed in natural language.”

“As enterprises continue to move to the cloud and integrate AI into their everyday business practices, there is more demand than ever for visibility into cloud data spend in order to maximize ROI and the impact of AI on their business,” said Kunal Agarwal, CEO and Co-founder of Unravel Data. “Unravel enables companies to visually see how their cloud data spend is trending for accurate forecasting and provides proactive alerts and guardrails to help govern that spend, as well as AI-driven automated suggestions for maximizing TCO of their cloud data platforms — enabling data-forward companies to lead their markets with their AI initiatives.”

The Unravel Data Platform provides organizations with data observability on data workload spending, and an early warning system for alerting on out-of-control spending, while offering an automated way to pinpoint the source of cost overruns. Unravel Data enables each of the three FinOps phases for modern data platforms, such as Databricks. These include:

• Inform: Unravel provides granular visibility required to manage allocation, consumption efficiency, and charge/showback at the job, user and workgroup levels
• Optimize: Unravel maximizes business impact of available budget and resources by optimizing data workloads to perform to required SLAs at most efficient cost
• Operate: Unravel delivers ongoing, continuous improvement through cost governance and self-service optimization to reliably predict costs and maximize value (ROI)

At the core of Unravel Data’s platform is its AI-powered Recommendation Engine, which understands all the intricacies and complexities of each modern data platform and the supporting infrastructure to optimize efficiency.

For more information on how Unravel Data is helping organizations around the world control cloud costs, please visit https://www.unraveldata.com/platform/finops-data-teams/.

Gartner, Hype Cycle for Emerging Technologies in Finance, 2023, By Mark D. McDonald, Published 11 July 2023

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally, and HYPE CYCLE is a registered trademark of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved.

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. 

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. A recent winner of the Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards, some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact

Blair Moreland

ZAG Communications for Unravel Data unraveldata@zagcommunications.com

The post Unravel Data Named a Sample Vendor in the Gartner^® Hype Cycle™ for Emerging Technologies in Finance, 2023 appeared first on Unravel.

Unravel Data Observability Platform earns prestigious industry recognition for Best Data Tool & Platform

Palo Alto, CA — June 21, 2023 —Unravel Data, the first data observability platform built to meet the needs of modern data teams, was named Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards. The prestigious CODiE Awards recognize the companies producing the most innovative Business Technology products across the country and around the world.

“We are deeply honored to win a CODiE Award for Best Data Tool & Platform. Today, as companies put data products and AI/ML innovation front and center of their growth and customer service strategies, the volume of derailed projects and the costs associated are ascending astronomically. Companies need a way to increase performance of their data pipelines and a way to manage costs for effective ROI,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “The Unravel Data Platform brings pipeline performance management and FinOps to the modern data stack. Our AI-driven Insights Engine provides recommendations that allow data teams to make smarter decisions that optimize pipeline performance along with the associated cloud data spend, making innovation more efficient for organizations.”

Cloud-first companies are seeing cloud data costs exceed 40% of their total cloud spend. These organizations lack the visibility into queries, code, configurations, and infrastructure required to manage data workloads effectively, which in turn, leads to over-provisioned capacity for data jobs, an inability to quickly detect pipeline failures and slowdowns, and wasted cloud data spend.

“The 2023 Business Technology CODiE Award Winners maintain the vital legacy of the CODiEs in spotlighting the best and most impactful apps, services and products serving the business tech market,” said SIIA President Chris Mohr. “We are so proud to recognize this year’s honorees – the best of the best! Congratulations to all of this year’s CODiE Award winners!”

The Software & Information Industry Association (SIIA), the principal trade association for the software and digital content industries, announced the full slate of CODiE winners during a virtual winner announcement. Awards were given for products and services deployed specifically for education and learning professionals, including the top honor of the Best Overall Business Technology Solution.

A SIIA CODiE Award win is a prestigious honor, following rigorous reviews by expert judges whose evaluations determined the finalists. SIIA members then vote on the finalist products, and the scores from both rounds are tabulated to select the winners.

Details about the winning products can be found at https://siia.net/codie/2023-codie-business-technology-winners/.

Learn more about Unravel’s award-winning data observability platform.

About the CODiE Awards

The SIIA CODiE Awards is the only peer-reviewed program to showcase business and education technology’s finest products and services. Since 1986, thousands of products, services and solutions have been recognized for achieving excellence. For more information, visit siia.net/CODiE.

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact

Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data Recognized by SIIA as Best Data Tool & Platform at 2023 CODiE Awards appeared first on Unravel.

Hosts Hyderabad’s first-ever conference on DataOps and FinOps, commits to serving the local tech community 

Hyderabad — June 16, 2023 — US-based Unravel Data, the first data observability platform built to meet the needs of modern data teams, today said it is planning to double its Hyderabad headcount in the next year as the company continues to expand and grow its local operations to better serve its customers. 

As a commitment to serving the local tech community in Hyderabad and nearby cities through continuous learning and development opportunities, Unravel today hosted the second Indian edition of its flagship DataOps Observability Conference in Hyderabad. This marked the first-ever conference on this subject ever hosted in the city. Over 200 senior data and technology professionals from data-leading companies attended the conference.

Speaking on the sidelines of the conference, Unravel co-founder and CTO Shivnath Babu said, “Hyderabad is a strategic location for Unravel as many of our customers around the world are relying on their data teams based in Hyderabad to accelerate their business-critical Big Data analytics projects. This is why we are not only committed to growing our team and operations in Hyderabad, but also to serve the local tech community through world-class learning and networking opportunities. Bringing our flagship DataOps Observability conference to Hyderabad within months of opening our office here underscores this commitment.”

Unravel has created the world’s first AI-enabled DataOps Observability Platform to meet the needs of modern data teams. The platform helps businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. The company counts several of the Fortune 500 companies as its clients, including Credit Suisse, Intel, Adobe, DBS Bank, Citi, Novartis, Maersk, and Mastercard, among others. 

Unravel’s VP of Engineering Operations Giriraj Bagdi said, “A majority of modern, data-driven businesses report that their data sources, workloads, and applications have grown exponentially in recent years. Many of them are consequently embracing DataOps Observability as a powerful means of streamlining and optimizing their data pipelines and applications. As pioneers of AI-powered DataOps observability, we are truly excited to share our latest insights and innovations with our tech community in Hyderabad, which is already at the front and center of implementing several of these innovations for businesses around the world.”  

Unravel has over 100 employees in India, out of which 25 are based in Hyderabad. The Second India DataOps Observability Conference has put Hyderabad firmly on the map as a key Asian destination for large global enterprises looking for skills and expertise as they race to prioritize DataOps as an established discipline across their organizations. The conference featured 14 high-profile speakers including Anuj Gupta, Chief Founder & CEO of Gradient Advisors, Snehith Allamraju, Senior Manager for Data and Analytics at Envista Corps, and others. Shivnath Babu delivered the keynote address. 

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data to Double Hyderabad Employee Count by Mid-2024 appeared first on Unravel.

Unravel Data Observability Platform recognized by peers as a market leader

Palo Alto, CA — May 11, 2023 — Unravel Data, the first data observability platform built to meet the needs of modern data teams, today announces that the Unravel Data Observability Platform was named a 2023 SIIA CODiE Award Finalist in the Best Data Tool category. CODiE finalists represent the best products, services, and people in the education and business technology industries.

The SIIA CODiE Awards, the long-running, premier awards program for the software and information industries, are produced by the Software & Information Industry Association (SIIA), the principal trade association for the software, education, media, and digital content industries. The Unravel Data Observability Platform was selected as a finalist across dozens of business technology and education technology categories, among hundreds of nominations.

“It is an understatement to say that we are thrilled to be recognized by SIIA CODiE as a finalist,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “The CODiE Awards are one of—if not the—most coveted awards in the industry, and it speaks to the power of the Unravel Platform. Cloud costs, if they aren’t already, are becoming a company’s single biggest and fastest growing IT budget expense. This moves data observability from being a nice-to-have to a must-have. Unravel offers the only platform that provides data teams with AI-powered answers and precise recommendations, making it easier for teams to work together, make smarter decisions, and ultimately optimize cloud data spend.”

Unravel enables data teams to optimize the cost of their data operations intelligently, run faster data pipelines, and troubleshoot mission-critical applications. Instead of just showing an overwhelming set of KPIs that often require domain expertise to discern, Unravel correlates and analyzes a full stack of telemetry metadata to provide easy-to-understand insights, actionable recommendations, and automation to optimize both performance and cost.

Designed for modern data stacks and supporting all major cloud and on-premises platforms, Unravel enables full-stack visibility, AI insights, and automation for modern data applications. It is uniquely suited to remove the blindspots in enterprise data pipelines and provide a single-pane-of-glass view to manage single, multi-, or hybrid-cloud environments. And as more enterprises tackle the challenge of ensuring reliable data results while taming unpredictable costs, the Unravel Platform does the heavy lifting.

“The 2023 CODiE Award finalists highlight the products and people who are out in front, leading their industries forward,” said SIIA President Chris Mohr. “These honorees continue the proud tradition of CODiE Award finalists of recognizing the most impactful products, services, and leaders of their time, setting a foundation for the next generation of innovators. Congratulations to all who received this well-earned acknowledgment.”

The SIIA CODiE Awards are the industry’s only peer-recognized awards program. Finalists are determined by industry experts. CODiE Award winners will be announced during the Virtual Celebrations June 21-22, 2023 at 1pm EST.

Details about each finalist are listed at https://siia.net/codie/celebrate-finalists/.

About the SIIA CODiE Awards
The SIIA CODiE Awards is the only peer-reviewed program to celebrate the vision, talent, and advances in building quality products in the Tech Industry. Since 1986, thousands of products, services and solutions have been recognized for Leading Innovation and Achieving Excellence. For more information, visit https://siia.net/codie/.

About Software and Information Industry Association (SIIA)
SIIA is the only professional organization connecting more than 700 data, financial information, education technology, specialized content and publishing, and connects learners and educators. Our diverse members manage the global financial markets, develop software that solves today’s challenges through technology, provide critical information that helps inform global businesses large and small, and innovate for better communication across the information ecosystem.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

SIIA Communications
codieawards@siia.net

The post Unravel Data Named 2023 SIIA CODiE Award Finalist in Best Data Tool appeared first on Unravel.

Here at Unravel, we are all about data observability. We live it. We breathe it. And increasingly, others are recognizing the vital role it plays in not only controlling costs, but helping companies realize and optimize their investments in cloud data projects. It’s become such a critical component of the data space that CRN created a new Data Observability category in this year’s BigData 100 list. We are excited, therefore, to announce that not only have we been named to the list for the fifth consecutive year, but we were among the inaugural group of Coolest Data Observability Companies of 2023.

CRN’s BigData 100 is an annual list that recognizes the technology vendors that go above and beyond by delivering innovation-driven products and services that solution providers can use to help organizations of all sizes gain the competitive advantages of becoming data-driven companies. The list identifies IT vendors that have consistently made technical innovation a top priority through their portfolios of big data management and integration tools; systems and platforms; and data operations and observability offerings.

We provide the market with the only platform that delivers AI-powered answers and precise recommendations for data teams. It’s no wonder that increasing numbers of data-driven enterprises rely on Unravel to help them bring their cloud costs under control. It’s safe to say that data observability has become a must-have for companies to succeed in today’s data-driven and cost-conscious environment.

Not only does data observability improve the speed and quality of data delivery, but it makes it easier for IT and business teams to work together and make smarter data-driven decisions, faster. By giving teams a unified view of data performance, cost, and quality across the entire organization, it breaks down silos and empowers individuals to make their own decisions on how to best reduce and/or optimize their cloud spend while still meeting SLAs.

Unravel provides data observability for the modern data stack. Learn more about how Unravel can help modern enterprises understand, troubleshoot, and optimize their data workloads.

The post Data Observability Is Big(Data 100) News appeared first on Unravel.

As usual, the latest release of Unravel (4.7.8.0) introduces new features and a slew of improvements and enhancements. As always, these new capabilities reflect the emerging needs of data team practitioners and their executive teams. Three of the new features are summarized below. Complete Release Notes for Unravel v4.7.8.0 can be found here.

The top three things any data-forward enterprise cares about are Performance, Cost, and Quality. That reliable results get delivered on time, every time, in the most cost-effective manner. Unravel’s strength has always been in performance observability, and we’ve taken the lead in cost governance for data workloads (DataFinOps). Now we bring in the third pillar: data quality.

Unravel integrates with Data Quality solutions

Unravel now integrates and correlates external data quality checks within its AI-driven insights. Unravel has taken the path of integrating data quality checks that run outside of Unravel—all the homegrown checks, assertions built in any number of excellent solution vendor tools—rather than building yet another data quality tool inside Unravel. 

The first data quality integration is with the open source leader, Great Expectations.

Now data teams have insights and details about performance, cost, and quality in a single pane of glass. No more jumping from tool to tool. And as different personas care about these different dimensions, everybody is working from the same single source of truth. With one click, drill down into quality check results, timelines, lineage, partitions, schema, usage, size, users, and more—right alongside AL recommendations and insights for performance and cost.

AutoActions for Amazon EMR help control cloud costs

AutoActions can now monitor EMR apps and clusters, and you can set the AutoAction policy to generate alerts when EMR exceeds a set dollar threshold, a running time threshold, or when it’s been idle for a long time.

Unravel gets EMR cost data while clusters are running, so you get a notification in real time that the cluster is exceeding a threshold that impacts costs—not after the fact.

You can get a list of all threshold violations across the various clusters, their cost, with drill-down details into each cluster.

Check out this video with a sample use case.

Get proactive budget-tracking alerts and notifications

Unravel can now proactively send notifications—via email or Slack—to specified individuals or groups whenever certain conditions are triggered. Perhaps most usefully, this feature gives both individual users and their managers a heads-up when projected costs are estimated to be at risk of exceeding budget (or have already done so). A common, reusable notification module can be used for multiple purposes, and a single notification may be used by multiple budgets.

Costs may be expressed in DBUs (for Databricks) or dollars (for Amazon EMR). To avoid alert storms and notification fatigue, only one alert is generated per month for budgets at risk, and one per month for exceeded budgets. (But if budgets are changed, alerts can be triggered again for the same month.)

Explore this feature further in this video.

The post What’s New In Unravel 4.7.8.0 appeared first on Unravel.

Third Annual Survey from Data Teams Summit Event Reveals Key Priorities and Strategies for Enterprise Data Teams in 2023

Palo Alto, CA – April 4, 2023 – Unravel Data, the first data observability platform built to meet the needs of modern data teams, today released key findings from a survey administered to more than 350 data professionals who attended the 2023 Data Teams Summit event in order to gauge the priorities, challenges, and benchmark the progress of enterprise data teams. This year’s survey asked data team stakeholders – including a cross-section of data scientists, data engineers, data analysts and the various business stakeholders who rely upon real-time data insights to inform their decisions – to assess how they are meeting their big data analytics objectives and define the practices they are adopting to conquer the complexities of the modern data stack. To view and download an infographic of the key findings from this survey, click here.

“For the third year in a row we’ve had the opportunity to take the pulse of enterprise data teams to better understand the daily challenges they face as they accelerate their ambitious big data analytics programs,” said Kunal Agarwal, co-founder and CEO of Unravel Data. “In just the course of a year we’ve seen a significant shift in how these growing, cross-functional teams are prioritizing DataOps as an established discipline across their organizations in a similar way that DevOps became an entrenched practice among software teams a decade ago. But despite this progress, this year’s survey also demonstrates that issues like FinOps, cloud utilization, and data security continue to present unique challenges to data teams.”

Some of the key findings collected from the most recent survey include:

• Cloud spending is now a critical KPI for the majority of data teams: More than two-thirds of data teams surveyed said that cloud spending has become a KPI of high strategic importance. When responses were broken down by role, almost 80% of business stakeholders said cloud spending was a critical KPI while just over half (55%) of data practitioners indicated the same.
• Cloud resources are being underutilized: In addition to cloud spending being elevated as a top KPI, almost half (44%) of all respondents in this year’s survey also reported that they believe that they are leaving money on the table when it comes to their public cloud utilization. Alarmingly, almost a quarter of respondents (23%) said they were unable to even estimate what percentage of their cloud resources went unused.
• FinOps interest is high yet adoption lags: Despite the fact that data teams have reported a lack of visibility into cloud spending, the adoption of mature FinOps practice was not viewed as an immediate priority among respondents with just over 20% reporting that their data teams have an established FinOps practice while a third of data teams reported that they are still in the early planning phase of implementing FinOps.
• DataOps as a practice is maturing: This year, more than 44% of respondents reported they are actively employing DataOps methodologies, compared to just less than a quarter (21%) of respondents in 2022, representing a 110% increase from the year prior. Further demonstrating the maturing DataOps practice, only 20% of respondents in this year’s survey said they were at the beginning stage compared to 41% last year.
• Data reliability emerges as the top challenge: This year when participants were asked what they viewed as the top challenge with operating their data stack, 41% respondents cited the lack of data quality as their most significant obstacle while 35% noted that the lack of visibility across their environments was the second biggest obstacle to managing their data stack.

Now in its third year, the Data Teams Summit is a one-day virtual conference that aims to build a vibrant peer-based community for DataOps professionals to share best practices. Data professionals can productively collaborate with one another and deliver on the promise of what it means to be a data-led company. This past year’s event attracted more than 2,300 data professionals and featured presentations from data leaders at some of the most data-forward enterprise organizations in the world. Recordings of all 24 sessions are available to watch for free and on-demand on the Data Teams Summit site at: www.datateamssummit.com.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Press Contact:
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post New Survey Finds Two Thirds of Data Teams Report Cloud Spending as a Critical KPI appeared first on Unravel.

As DataOps moves along the maturity curve, many organizations are deciphering how to best balance the success of running critical jobs with optimized time and cost governance.

Watch the fireside chat from Data Teams Summit where Mark Sear, Head of Data Platform Optimization for Maersk, shares how his team is driving towards enabling strong engineering practices, design tenets, and culture at one of the largest shipping and logistics companies in the world. Transcript below.

Transcript

Kunal Agarwal:

Very excited to have a fireside chat here with Mark Sear. Mark, you’re the director of data integration, AI, machine learning, and analytics at Maersk. And Maersk is one of the largest shipping line and logistics companies in the world. Based out of Copenhagen, but with subsidiaries and offices across 130 countries with about 83,000 employees worldwide. We know that we always think about logistics and shipping as something just working harmoniously, transparently in the background, but in the recent past, given all of the supply chain pressures that have happened with the pandemic and beyond, and even that ship getting stuck in the Suez Canal, I think a lot more people are paying attention to this industry as well. So I was super excited to have you here, Mark, to hear more about yourself, you as the leader of data teams, and about what Maersk is doing with data analytics. Thank you so much for joining us.

Mark Sear:

It’s an absolute pleasure. You’ve just illustrated the perils of Wikipedia. Maersk is not just one of the largest shipping companies in the world, but we’re also actually one of the largest logistics companies in the world. We have our own airline. We’ve got hundreds of warehouses globally. We’re expanding massively, so we are there and of course we are a leader in decarbonation. We’ve got a pledge to be carbon-neutral way before just about anybody else. So it’s a fantastic company to work at. Often I say to my kids, we don’t just deliver stuff, we are doing something to help the planet. It’s a bigger mission than just delivering things, so it’s a pleasure to be here.

Kunal Agarwal:

That’s great. Mark, before we get into Maersk, we’d love to learn about you. So you have an amazing background and accumulation of all of these different experiences. Would you help the audience to understand some of your interests and how you got to be in the role that you currently are at? And what does your role comprise inside of Maersk?

Mark Sear:

Wow. It’s a long story. I’m an old guy, so I’m just couple of years over 60 now, which you could say you don’t look it, but don’t worry about it.

Kunal Agarwal:

You don’t look it at all, only 40.

Mark Sear:

I’m a generation that didn’t, not many of us went to university, so let me start there. So I left school at 18, did a bit of time in the basic military before going to what you would call, I suppose fundamentally, a crypto analyst school. They would detect how smart you were, whether you had a particular thing for patents, and they sent me there. Did that, and then since then I’ve worked in banking, in trading in particular. I ran a big trading group for a major bank, which was great fun, so we were using data all the time to look for both, not just arbitrage, but other things. Fundamentally, my life has been about data.

Kunal Agarwal:

Right.

Mark Sear:

Even as a kid, my dad had a very small business and because he didn’t know anything about computers, I would do the computing for him and work out the miles per gallon that his trucks were getting and what the trade-in was.

Kunal Agarwal:

Sure.

Mark Sear:

And things like that. So data’s been part of my life and I love everything about data and what it can do for people, companies, everything. Yeah, that’s it. Data.

Kunal Agarwal:

That’s great, Mark. Obviously this is a conference spot, a data team, so it’s great to hear from the data guy who’s been doing it for a really long time. So, Mark, to begin, Maersk, as you said, is one of the largest shipping and logistics companies in the world. How has data transformed your company?

Mark Sear:

One thing, this is a great question. How has it transformed and how will it transform?

Kunal Agarwal:

Yes.

Mark Sear:

I think that for the first time in the last couple of years, and I’ve been very lucky, I’ve only been with the company three years, but shortly after I joined, we had a new tech leader, a gentleman called Navneet Kapoor. The guy is a visionary. If you imagine shipping was seen for many years, there’s a bit of a backwater really. You move containers from one country to another country on ships, that was it. Navneet has changed the game for us all and made people realize that data is pervasive in logistics. It’s literally everywhere. If you think about our biggest ship, ship class, for example, it’s called an E-Class. That can take over 18,000 shipping containers on one journey from China to Europe, 18,000.

Kunal Agarwal:

Oh wow.

Mark Sear:

Think about that. So that’s absolutely huge. Now, to put that into context, in one journey, one of those ships will move more goods than was moved in the entire 19th century between continents, one journey. And we got six of them and they’re going backwards and forwards all the time. So the data has risen exponentially and what you can do with it, we are now just starting to get to grips with it, that’s what so exciting. Consider, we have companies that do want to know how much carbon is being produced as part of their products. We have things like that. We just have an incredibly diverse set of products.

To give you an example, I worked on a project about 18 months ago where we worked out, working in tandem with a couple of nature organizations, that if a ship hits a whale at 12 knots and above, that whale will largely die. If you hit it below 12 knots, it will live. It’s a bit like hitting an adult at 30 miles an hour versus 20. The company puts some money in so we could use the data for where the whales were to slow the ships down. So this is an example of where this company doesn’t just think about what can we do to make money. This is a company that thinks about how can we use data to better the company, make us more profitable, and at the same time, put back into the planet that gave us the ability to have this business.

Kunal Agarwal:

Let’s not forget that we’re human, most importantly.

Mark Sear:

Yeah, it’s super exciting, right? You can make all the money in the world. If you trash the planet, there’s not a lot left to enjoy as part of it. And I love that about this company.

Kunal Agarwal:

Absolutely. And I’m guessing with the pandemic and post-pandemic, and all of the other data sets that you guys are gathering anyways from sensors or from the shipping lines or from all the efficiencies, with all the proliferation of all this data inside your organization, what challenges has your team faced or does the Maersk data team face?

Mark Sear:

Well, my team is in the enterprise architecture team. We therefore deal with all the other teams that are dealing with data, and I think we got the same challenges as everybody. We’ve got the data quality right? Do we know where that data comes from? Are we processing it efficiently? Do we have the right ideas to work on the right insights to get value out of that data? I think they’re common industry things, and as with everything, it’s a learning process. So one man’s high-quality data is another woman’s low-quality data.

And depending on who you are and what you want to do with that data, people have to understand how that quality affects other people downstream. And of course, because you’re quite right, we did have a pandemic, and in the pandemic shipping rates went a little bit nuts and they’re normalizing now. But, of course, if you think about introducing predictive algorithms where the price is going vertically and the algorithm may not know that there’s a pandemic on, it just sees price. So I think what we find is challenging, same as everybody else, is how do you put that human edge around data? Very challenging. How do you build really high-performing teams? How do you get teams to truly work together and develop that esprit de corps? So there are a lot of human problems that go alongside the data problems.

Kunal Agarwal:

Yeah. Mark, give us a sense of your size. In terms of teams, applications, whatever would help us understand what you guys were, where you guys are, and where you guys headed.

Mark Sear:

Three years ago when I joined there were 1,900 people in tech; we’ve now got nearly 6,000. We had a huge amount of outsourcing; now we’re insourcing, we’re moving to an open source first event-based company. We’ve been very inquisitive. We’ve bought some logistics companies, so we’ve gone on the end-to-end journey now with the logistics integrator of choice globally. We’ve got our own airline. So you have to think about a lot of things that play together.

My team is a relatively tiny team. We’ve got about 12, but we liaise with, for example, our global data and analytics team that has got 600 people in it. We then organized into platforms, which are vertically problem solving, but fully horizontally integrated passing events between them. And each one of those has their own data team in it as well. So overall, I would guess we’ve got 3,000 people working directly with data in IT and then of course many thousands more.

Kunal Agarwal:

Wow.

Mark Sear:

Out in the organization. So it’s big organizations. Super exciting. Should say, now I’m going to get a quick commercial in. If you are watching this and you are a top data talent, please do hit me up with your resume.

Kunal Agarwal:

There’s a couple of thousand people watching this live, so you’ll definitely.

Mark Sear:

Hey, there you go, man. So listen, as long as they’re quality, I don’t care.

Kunal Agarwal:

From Mark, he’s a great boss as well. So when you think about the maturity curve of data operations, where do you think Maersk is at and what stands in your way to be fully matured?

Mark Sear:

Okay, so let’s analyze that. I think the biggest problem in any maturity curve is not defining the curve. It’s not producing a pyramid to say we are here and a dial to say, well, you rank as a one, you want to rank as a five.

Kunal Agarwal:

Sure.

Mark Sear:

The biggest problem to me is the people that actually formulate that curve. Now everyone’s got staff turnover and everyone or the majority of people know that they’re part of a team. But the question is how do you get that team to work with other teams and how do you disseminate that knowledge and get that group think of what is best practice for DataOps? What is best practice for dealing with these problems?

Kunal Agarwal:

It’s almost a spectrum on the talent side, isn’t it?

Mark Sear:

It’s a spectrum on the talent side, there’s a high turnover because certainly in the last 12 to 18 months, salaries have been going crazy, so you’ve had crazy turnover rates in some areas, not so much in other areas. So the human side of this is one part of the problem, and it’s not just the human side to how do you keep them engaged, it’s how do you share that knowledge and how do you get that exponential learning organization going?

And perhaps when we get into how we’ve arrived at tools like Unravel, I’ll explain to you what my theory is on that, but it’s almost a swarm learning that you need here, an ants style learning of how to solve problems. And that’s the hardest thing, is getting everybody in that boat swimming in the same direction before you can apply best practices because everybody says this is best practice. Sure, but if it was as simple as looking at a Gartner or whoever thing and saying, “Oh, there are the five lines we need to do,” everybody would do it. There’d be no need for anybody to innovate because we could do it; human beings aren’t very good at following rules, right?

Kunal Agarwal:

Yeah. So what kind of shifts and changes did you have to make in your big data operations and tools that you had to put into place for getting that maturity to where you expected it to be?

Mark Sear:

I think the first thing we’ve got to do, we’ve got to get people thinking slightly shorter timeframe. So everybody talks about Agile, Agile, Agile.

Kunal Agarwal:

Right.

Mark Sear:

Agile means different things to different people. We had some people who thought that Agile was, “Well, you’re going to get a fresh data set at the end of the day, so what the heck are you complaining about? When I started 15 years ago, you got it weekly.” That’s not agile. Equally, you’ve got people who say, I need real-time data. Well, do you really need real-time data if you’re actually dealing with an expense account? You probably don’t.

Kunal Agarwal:

Right.

Mark Sear:

Okay, so the first thing we’ve got to do is level set expectations of our users and then we’ve got to dovetail what we can deliver into those. You’ve got to be business focused, you’ve got to bring value. And that’s a journey. It’s a journey for the business users.

Kunal Agarwal:

Sure.

Mark Sear:

It’s a journey for our users. It’s about learning. So that’s what we’re doing. It’s taking time. Yeah, it’s taking time, but it’s like a snowball. It is rolling and it is getting bigger and it’s getting better, getting faster.

Kunal Agarwal:

And then when you think about the tools, Mark, are there any that you have to put into place to accelerate this?

Mark Sear:

I mean, we’ve probably got one of everything to start and now we’re shrinking. If I take . . . am I allowed to talk about Unravel?

Kunal Agarwal:

Sure.

Mark Sear:

So I’ll talk about–

Kunal Agarwal:

As much as you would.

Mark Sear:

–Unravel for a few seconds. So if you think about what we’ve got, let’s say we’ve got 3,000 people, primarily relatively young, inexperienced people churning out Spark code, let’s say Spark Databricks code, and they all sit writing it. And of course if you are in a normal environment, you can ask the person next to you, how would you do this? You ask the person over there, how would you do this? We’ve had 3,000 engineers working from home for two years, even now, they don’t want to come into the office per se, because it’s inconvenient, number one, because you might be journeying in an hour in and an hour home, and also it’s not actually, truly is productive. So the question is how do you harvest that group knowledge and how do people learn?

So for us, we put Unravel in to look at and analyze every single line of code we write and come up with those micro suggestions and indeed macro suggestions that you would miss. And believe me, we’ve been through everything like code walkthroughs, code dives, all those things. They’re all standard practice. If you’ve got 2,000 people and they write, let’s say, 10 lines of code a day each, 20,000 lines of code, you are never going to walk through all of that code. You are never going to be able to level set expectations. And this is key to me, be able to go back to an individual data engineer and say, “Hey, dude, listen, about these couple of lines of code. Did you realize if you did it like this, you could be 10 times as efficient?” And it’s about giving that feedback in a way that allows them to learn themselves.

And that’s what I love about Unravel: you can get the feedback, but it’s not like when you get that feedback, nobody says, “Come into my office, let’s have a chat about these lines of code.” You go into your private workspace, it gives you the suggestions, you deal with the suggestions, you learn, you move on, you don’t make the mistakes again. And they may not even be mistakes. They might just be things you didn’t know about.

Kunal Agarwal:

Right.

Mark Sear:

And so because Unravel takes data from lots of other organizations as well, as I see it, we’re in effect, harvesting the benefits of hundreds of thousands of coders globally, of data engineers globally. And we are gaining the insights that we couldn’t possibly gain by being even the best self-analysis on the planet, you couldn’t do it without that. And that to me is the advantage of it. It’s like that swarm mentality. If you’ve ever, anybody watching this, had a look at swarm AI, which is to predict, you can use it to predict events. It’s like if you take a soccer game, and I’ve worked in gambling, if you take a soccer match and you take a hundred people, I’ll call it soccer, even though the real name for is football, you Americans.

Kunal Agarwal:

It’s football, I agree too.

Mark Sear:

It’s football, so we’re going to call it football, association football to give you it’s full name. If you ask a hundred football fans to predict a score, you’ll get a curve, and you’ll generally, from that predictor, get a good result. Way more accurate than asking 10 so-called experts, such as with code. And that’s what we’re finding with Unravel is that sometimes it’s the little nuances that just pop up that are giving us more benefits.

Kunal Agarwal:

Right.

Mark Sear:

So it’s pivotal to how we are going to get benefits out over the longer term of what we’re doing.

Kunal Agarwal:

That’s great. And we always see a spectrum of skills inside an organization. So our mission is trying to level the playing field so anybody, even a business user, can log in without knowing the internals of all of these complex data technologies. So it’s great to hear the way Maersk is actually using it. We spoke a little bit about making these changes. We’d love to double click on some of these hurdles, right? Because you said it was a journey to get to people to this mature or fast-moving data operations, if you may, or more agile data operations if you may. If we can double click for a second, what has been the biggest hurdle? Is it the mindset? Is it managing the feedback loop? Is it changing the practices? Is it getting new types of people on board? What has been the biggest hurdle?

Mark Sear:

Tick all of the above.

Kunal Agarwal:

Okay.

Mark Sear:

But I think–

Kunal Agarwal:

Pick for option E.

Mark Sear:

Yeah, so let me give you an example. There are several I’ve had with people that have said to me, “I’ve been doing this 25 years. There’s nothing, I’ve been doing it 25 years.” That presupposes that 25 years of knowledge and experience is better than 10 minutes with a tool that’s got 100,000 years of learning.

Kunal Agarwal:

Right.

Mark Sear:

Over a 12-month period. So that I classify that as the ego problem. Sometimes people need their ego brushing, sometimes they need their ego crushing. It’s about looking the person in the eye, working out what’s the best strategy of dealing with them and saying to them, “Look, get on board.” This isn’t about saying you are garbage or anything else. This is about saying to you, learn and keep mentoring other people as you learn.

Kunal Agarwal:

Yeah.

Mark Sear:

I remember another person said to me, “Oh my god, I’ve seen what this tool can do. Is AI going to take my job?” And I said to them, no, AI isn’t going to take your job, but if you’re not careful, somebody, a human being that is using AI will take it, and that doesn’t apply to me. That applies just in general to the world. You cannot be a Luddite, you cannot fight progress. And as we’ve seen with Chat GPT and things like that recently, the power of the mass of having hundreds and thousands and millions of nodes analyzing stuff is precisely what will bring that. For example, my son who’s 23, smart kid, well, so he tells me. Smart kid, good uni, good university, blah blah blah. He said to me, “Oh Tesla, they make amazing cars.” And I said to him, Tesla isn’t even a car company. Tesla is a data company that happens to build a fairly average electric car.

Kunal Agarwal:

Absolutely.

Mark Sear:

That’s it. It’s all about data. And I keep saying to my data engineers, to be the best version of you at work and even outside work, keep picking up data about everything, about your life, about your girlfriend, the way she feels. About your boyfriend, the way he feels. About your wife, your mother. Everything is data. And that’s the mindset. And the biggest thing for me, the biggest issue has been getting everybody to think and recognize how vital data is in their life, and to be open to change. And we all know throughout go through this cycle of humanity, a lack of openness to change is what’s held humanity back. I seek to break that as well.

Kunal Agarwal:

I love that Mark. Switching gears, we spoke a little bit about developer productivity. We spoke about agility and data operations. Maersk obviously runs, like you were explaining, a lot of their data operations on the cloud. And as we see a lot of organizations when they start to get bigger and bigger and bigger in use cases on the cloud, cost becomes a front and center or a first-class citizen conversation to have. Shed some light on that for us. What is that maturity inside of Maersk, or how do you think about managing costs and budgets and forecast on the cloud, and what’s the consequence of not doing that correctly?

Mark Sear:

Well, there are some things that I can’t discuss because they’re obviously internal, but I think, let’s say I speak to a lot of people in a lot of companies, and there seem to be some themes that run everywhere, which is there’s a rush towards certain technologies, and people, they test it out on something tiny and say, “Hey, isn’t this amazing? Look how productive I am.” Then they get into production and somebody else says, “That’s really amazing. You were very productive. But have you seen what comes out the other end? It’s a cost, a bazillion dollars an hour to run it.” Then you’ve got this, I think they called it the Steve Jobs reality distortion field, where both sets of people go into this weird thing of, “Well, I’m producing value because I’m generating code and isn’t it amazing?” And the other side is saying, “Yeah, but physically the company’s going to spend all its money on the cloud. We won’t be able to do any other business.”

Kunal Agarwal:

Yeah.

Mark Sear:

So we are now getting to a stage where we have some really nice cost control mechanisms coming in. For me, it’s all in the audit. And crucial to this is do it upfront. Do it in your dev environment. Don’t go into production, get a giant bill and then say, how do I cut that bill? Which is again, where we’ve put Unravel now, right in the front of our development environment. So nothing even goes into production unless we know it’s going to work at the right cost price. Because otherwise, you’ve just invented the world’s best mechanism for closing the stable door after the cost horse has bolted, right?

Kunal Agarwal:

Right.

Mark Sear:

And then that’s always a pain because post-giant bill examinations are really paying, it’s a bit like medicine. I don’t know if you know, but in China, you only pay a doctor when you are well. As soon as you are sick, you stop paying bills and they have to take care of you. So that to me is how we need to look at cost.

Kunal Agarwal:

I love that. Love that analogy.

Mark Sear:

Do it upfront. Keep people well, don’t ever end up with a cost problem. So that’s again, part of the mindset. Get your data early, deal with it quickly. And that’s the level of maturity we are getting to now. It’s taking time to get there. We’re not the only people, I know it’s everywhere. But I would say to anybody, I was going to say lucky enough to be watching this, but that’s a little bit cocky, isn’t it? Anybody watching this? Whatever you do, get in there early, get your best practice in as early as possible. Go live with fully costed jobs. Don’t go live, work out what the job cost is and then go, how the hell do I cut it?

Kunal Agarwal:

Yeah.

Mark Sear:

Go live with fully costed jobs and work out well, if it costs this much in dev test, what’s it going to cost in prod? Then check it as soon as it goes live and say, yeah, okay, the delta’s right, game on. That’s it.

Kunal Agarwal:

So measure twice, cut once, and then you’re almost shifting left. So you’re leaving it for the data engineers to go and figure this out. So there’s a practice that’s emerging called FinOps, which is really a lot of these different groups of teams getting together to exactly solve this problem of understand what the cost is, optimize what the cost is, and then govern what the cost is as well. So who within your team does what I’m sure the audience would love to hear that a little bit.

Mark Sear:

Pretty much everybody will do everything, every individual data engineer, man, woman, child, whatever will be, but we’re not using child labor incidentally, that was.

Kunal Agarwal:

Yeah, let’s clarify that one for the audience.

Mark Sear:

That’s a joke. Edit that out. Every person will take it on themselves to do that because ultimately, I have a wider belief that every human being wants to do the right thing, given everything else being equal, they want to do the right thing. So I will say to the people that I speak to as data engineers, as new data engineers, I will say to them, we will empower you to create the best systems in the world. Only you can empower yourself to make them the most efficient systems in the world.

Kunal Agarwal:

Interesting.

Mark Sear:

And by giving it to them and saying, “This is a matter of personal pride, guys,” at the end of the day, am I going to look at every line of your code and say, “You wouldn’t have got away with that in my day.” Of course not. When I started in it, this is how depressingly sad it is. We had 16K of main memory on the main computer for a bank in an IBM mainframe, and you had to write out a form if you wanted 1K of disk. So I was in a similar program in those days. Now I’ve got a phone with God knows how much RAM on it.

Kunal Agarwal:

Right, and anybody can spin up a cloud environment.

Mark Sear:

Absolutely. I can push a button, spin up whatever I want.

Kunal Agarwal:

Right.

Mark Sear:

But I think the way to deal with this problem is to, again, push it left. Don’t have somebody charging in from finance waiving a giant bill saying, “Guys, you are costing a fortune.” Say to people, let’s just keep that finance dude or lady out of the picture. Take it on yourself, yourself. Show a bit of pride, develop this esprit de corps, and let’s do it together.

Kunal Agarwal:

Love it. Mark, last question. This is a fun one and I know you’re definitely going to have some fun answer over here. So what are your predictions for this data industry for this year and beyond? What are we going to see?

Mark Sear:

Wow, what do I think? Basically–

Kunal Agarwal:

Since you’ve got such a pulse on the overall industry and market.

Mark Sear:

So to me, the data industry, obviously it’ll continue to grow. I don’t believe that technology in many levels, I’ll give you over a couple of years, technology in many levels, we’re actually a fashion industry. If the fashion is to outsource, everybody outsource. So the fashion is to in-source, everybody does. Women’s skirts go up, fashion changes, they come down. Guys wear flared trousers, guy wears wear narrow trousers and nobody wants to be out of fashion. What I think’s going to happen is data is going to continue to scale, quantum computing will take off within a few years. What’s going to happen is your CEO is going to say, “Why have I got my data in the cloud and in really expensive data centers when someone has just said that I can put the whole of our organization on this and keep it in the top drawer of my desk?”

And you will have petabyte, zettabyte scale in something that can fit in a shoebox. And at that point it’ll change everything. I will probably either be dead, or at least hopefully retired and doing something by then. But I think it is for those people that are new to this industry, this is an industry that’s going to go forever. I personally hope I get to have an implant in my head at some point from Elon. I will be going for, I’m only going to go for version two. I’m not going for version one and hopefully–

Kunal Agarwal:

Yeah, you never want to go for V1.

Mark Sear:

Exactly, absolutely right. But, guys, ladies, everybody watching this, you are in the most exciting part, not just of technology, of humanity itself. I really believe that, of humanity itself, you can make a difference that very few people on the planet get to make.

Kunal Agarwal:

And on that note, I think the big theme that we have going on this series, we strongly feel that data teams are running the world and will continue to run the world. Mark, thank you so much for sharing this, exciting insights, and it’s always fun having you. Thanks you for making the time.

Mark Sear:

Complete pleasure.

The post Enabling Strong Engineering Practices at Maersk appeared first on Unravel.

Palo Alto, CA – Jan. 31, 2023 – Unravel Data, the first data observability platform built to meet the needs of modern data teams, announced today that it was recognized as a Representative Vendor in the DataOps Market in the 2022 Gartner Market Guide for DataOps Tools.

“In recent years, organizations have witnessed significant increases in the number of data sources and data consumers as they deploy more applications and use cases. DataOps has risen as a set of best practices and technologies to streamline and efficiently handle this onslaught of new upcoming workloads,” said Sanjeev Mohan, principal with SanjMo. “Just as DevOps helped streamline web application development and make software teams more productive, DataOps aims to do the same thing for data applications.”

According to a Gartner strategic planning assumption from this Market Guide, “by 2025, a data engineering team guided by DataOps practices and tools will be 10 times more productive than teams that do not use DataOps.” In this Market Guide for DataOps Tools, Gartner examines the DataOps market and explains the various capabilities of DataOps tools, paints a picture of the DataOps tool landscape, and offers recommendations.

“Data teams are struggling to keep up with the increased volume, velocity, variety, and complexity of their data applications and data pipelines. These teams are facing many of the same generic challenges that software teams did 10-plus years ago,” said Kunal Agarwal, CEO of Unravel Data. “We’re proud to be recognized by Gartner in the DataOps Market. Unravel Data provides unprecedented visibility across users’ data stacks, proactively troubleshooting and optimizing data workloads, and defining guardrails to govern costs and improve predictability.”

Today’s modern data stack is a complex collection of different systems, platforms, technologies, and environments. Enterprises need a DataOps solution that works with every type of workload and addresses the performance, cost, and quality issues facing data teams today. Founded by data pioneers Kunal Agarwal and Dr. Shivnath Babu, Unravel Data was created on the notion that the exponential growth of data combined with the broad adoption of the public cloud requires an entirely new way to manage and optimize the data pipelines that support the real-time analytics needs of data-driven enterprises.

Customers who have deployed the Unravel platform have been able to double the productivity of data teams and ensure that data applications run on time, while scaling costs efficiently in the cloud. To learn more about how Unravel is leading the DataOps observability space, visit the new library of demonstration videos.

*Gartner, “Market Guide for DataOps Tools”, December 5, 2022.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Gartner^® Market Guide for Dataops Tools… Unravel Data Highlighted appeared first on Unravel.

Palo Alto, CA – Jan. 12, 2023 – Unravel Data, the first DataOps observability platform built to meet the needs of modern data teams, today announced that Zia Mansoor has joined the company’s Board of Directors. Mansoor currently serves as Microsoft’s Vice President of Data & AI, and has extensive expertise in go-to-market strategy and providing product and solution leadership across operational database, analytics, AI/Machine Learning, data governance, and business intelligence solutions.

“We are excited to welcome Zia to our Board. He’s had tremendous success enabling customers to build efficient and scalable solutions while instilling a strong data culture within organizations. His appointment reflects our strong relationship with M12, Microsoft’s venture fund, a key investor in Unravel,” said Kunal Agarwal, CEO of Unravel Data. “We look forward to working together as Unravel continues to help organizations efficiently deliver data outcomes and effectively control skyrocketing cloud compute and storage costs.”

Mansoor has held various positions at Microsoft for the last 15 years. Prior to his current role leading Worldwide Data and AI commercial business, he led Microsoft Canada’s largest team of technical professionals as Vice President of Customer Success. Mansoor has helped numerous organizations accelerate their journey to the cloud and realize potential across all Microsoft commercial solution areas, including Modern Workplace, Azure (App Development and Infrastructure, Data and AI), Business Applications, and Security.

“I am excited to work with Unravel Data, which has proven itself to be a fast-growing data observability company. Alongside my fellow advisors, I look forward to helping drive innovation in its data operations solutions. I have long admired the team, their work, and how they are leveraging AI to meet the growing opportunities for the modern data stack,” said Mansoor.

Founded by data pioneers Kunal Agarwal and Dr. Shivnath Babu, Unravel Data understands that the exponential growth of data combined with the broad adoption of the public cloud requires an entirely new way to manage and optimize the data pipelines that support the real-time analytics needs of today’s data-driven enterprises. Numerous Fortune 100 companies, including two of the top five global pharmaceutical companies and three of the top 10 financial institutions in the world, rely on Unravel Data to gain unprecedented visibility across their data stacks, proactively troubleshoot and optimize their data workloads, and define guardrails to govern costs and improve predictability. Customers who have deployed Unravel have been able to double productivity of data teams and ensure data applications run on time, while scaling costs efficiently in the cloud.

To learn more about how Unravel is leading the DataOps observability space, visit the
new library of demonstration videos.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading DataOps observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Microsoft Exec Zia Mansoor Joins Unravel Data Board of Directors appeared first on Unravel.

Observability designed for data teams

The Unravel platform harnesses full-stack visibility, contextual awareness, AI-powered
intelligence, and automation to go “beyond observability”— to not only show you what’s
going on, but why and exactly how to make things better and then keep them that way
proactively. Unravel is designed for every member of your data team.

Key Use Cases

• Advanced cost governance
• AI-enabled optimization
• Automated troubleshooting
• Flexible data quality
• Cloud migration

Read this datasheet to learn more.
Get the help you need to simplify your modern dataops.

The post Data Observability for Azure Datasheet appeared first on Unravel.

The Data Teams Summit 2023 is just around the corner!

This year, on January 25, 2023, we’re taking the peer-to-peer empowerment of data teams one step further, transforming DatOps Unleashed into Data Teams Summit to better reflect our focus on the people—data teams—who unlock the value of data.

Data Teams Summit is an annual, full-day virtual conference, led by data rockstars at future-forward organizations about how they’re establishing predictability, increasing reliability, and creating economic efficiencies with their data pipelines.

Join us for sessions on:

• DataOps best practices
• Data team productivity and self-service
• DataOps observability
• FinOps for data teams
• Data quality and governance
• Data modernizations and infrastructure

The peer-built agenda is packed, with over 20 panel discussions and breakout sessions. Here’s a sneak peek at some of the most highly anticipated presentations:

Keynote Panel: Winning strategies to unleash your data team

Great data outcomes depend on successful data teams. Every single day, data teams deal with hundreds of different problems arising from the volume, velocity, variety—and complexity—of the modern data stack.

Learn best practices and winning strategies about what works (and what doesn’t) to help data teams tackle the top day-to-day challenges and unleash innovation.

Breakout Session: Maximize business results with FinOps

As organizations run more data applications and pipelines in the cloud, they look for ways to avoid the hidden costs of cloud adoption and migration. Teams seek to maximize business results through cost visibility, forecast accuracy, and financial predictability.

In this session, learn why observability matters and how a FinOps approach empowers DataOps and business teams to collaboratively achieve shared business goals. This approach uses the FinOps Framework, taking advantage of the cloud’s variable cost model, and distributing ownership and decision-making through shared visibility to get the biggest return on their modern data stack investments.

See how organizations apply agile and lean principles using the FinOps framework to boost efficiency, productivity, and innovation.

Breakout Session: Going from DevOps to DataOps

DevOps has had a massive impact on the web services world. Learn how to leverage those lessons and take them further to improve the quality and speed of delivery for analytics solutions.

Ali’s talk will serve as a blueprint for the fundamentals of implementing DataOps, laying out some principles to follow from the DevOps world and, importantly, adding subject areas required to get to DataOps—which participants can take back and apply to their teams.

Breakout Session: Becoming a data engineering team leader

As you progress up the career ladder for data engineering, responsibilities shift as you start to become more hands-off and look at the overall picture rather than a project in particular.

How do you ensure your team’s success? It starts with focusing on the team members themselves.

In this talk, Matt Weingarten, a lead Data Engineer at Disney Streaming, will walk through some of his suggestions and best practices for how to be a leader in the data engineering world.

Attendance is free! Sign up here for a free ticket

The post Sneak Peek into Data Teams Summit 2023 Agenda appeared first on Unravel.

Palo Alto, CA – January 5, 2023 – Unravel Data, the first DataOps observability platform built to meet the needs of modern data teams, today announced that CEO and co-founder Kunal Agarwal will deliver the keynote in the upcoming virtual Data Teams Summit 2023, which will take place on Jan. 25, 2023.

Great data outcomes depend on successful data teams. Every day, data teams deal with hundreds of different problems arising from the volume, velocity, variety—and complexity—of the modern data stack. In his keynote, Winning Strategies to Unleash Your Data Team, Agarwal discusses best practices and winning strategies about what works (and what doesn’t) to help data teams tackle the top day-to-day challenges and unleash innovation. Agarwal will be joined by Sanjeev Mohan, principal, SanjMo and Benjamin Rogojan, owner & data consultant, Seattle Data Guy at the keynote.

Clinton Ford, director of product marketing for Unravel, is also presenting at the event and will speak to how a FinOps approach empowers DataOps and business teams to collaboratively achieve shared business goals. In the session Maximize Business Results with FinOps, Ford, along with Thiago Gil, FinOps Foundation ambassador, will address how organizations can apply the FinOps framework to get the biggest return on their modern data stack investments.

WHAT: Data Teams Summit (formerly DataOps Unleashed)
WHEN: Wednesday, Jan. 25, 2023
ATTENDEE REGISTRATION: Data Teams Summit Registration

To learn more about how Unravel is leading the DataOps observability space, visit the new library of demonstration videos. To register for the Data Teams Summit, visit https://datateamssummit.com.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading DataOps observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post CEO Kunal Agarwal to Deliver Keynote at Upcoming Data Teams Summit appeared first on Unravel.

What are the (rapidly) evolving trends in the data space? And how is Unravel responding—especially for FinOps? What’s on the product roadmap for 2023? How can you get the most out of your Unravel deployment today?

Unravel leadership presented a virtual year-end update to talk about emerging industry developments, give a sneak peek into what’s in store for 2023, and share our customers’ best practices on how they’re using Unravel to maximize business value.

Here are some of the highlights, with video clips.

The rapid rise of FinOps

CEO Kunal Agarwal discusses his vision for Unravel and how cost governance and cost optimization has become a top challenge for data teams running workloads in the cloud. Some 60% of organizations encounter cloud cost overruns. Some of that is due to increased workloads, but another big reason is that companies just have difficulty understanding how to run more efficiently. But that kind of “actionable intelligence” is the foundation of FinOps—engineering, finance, and business teams making collaborative data-driven decisions about spending.

It’s the same kind of insights—at both a granular and holistic level—you need to tackle performance issues and/or to migrate to the cloud reliably and efficiently. Kunal shows how customers are using Unravel to realize tangible benefits to optimize performance and cost, and recaps our recent $50 million Series D funding

Unravel roadmap for 2023

Vice President of Product Eric Chu gives a glimpse into what’s ahead for the Unravel platform. Over the next year, you’ll see Unravel expand its capabilities to include integration with data quality solutions, expand its coverage on the modern data stack, and introduce additional FinOps observability, intelligence, and automation features. Specifically:

• Unravel will support Snowflake
• Databricks SQL and Unity Catalog support
• Amazon EMR Serverless, as well as Glue and Kubernetes
• Integration with data quality solutions, starting with Great Expectations
• Integration with ServiceNow, PagerDuty, Jira, and more
• Top-down FinOps dashboards, as well as enhanced AI insights

We are collaborating with customers now to get design input and user feedback, enabling early access to preview features.

Unlocking value with Unravel

There are probably features in Unravel you already have access to but may not be leveraging to their fullest extent. Chris Santiago, Unravel VP of Solutions Engineering, digs into some hidden treasures and best practices that other customers use on a daily basis.

Chris walks through a FinOps-focused use case demo to show how Unravel capabilities helps at each stage of the cost-optimization lifecycle process: reporting (visualize, track, allocate), self-service insights (reduce, improve, empower), and guardrails & automation (control, prevent).

His demo takes us from a bird’s-eye view overview of Databricks costs, then walks through the various Unravel reports you can access to drill down into get precise insights—cost breakdown, user and usage, node downsizing recommendations, event analysis—into where the spend is going at a granular level, where there’s waste, where optimization can have the greatest impact, etc. Then Chris touches on how Unravel’s  AI recommendations fit in, as well as run through how to set up automated guardrails and real-time budget-tracking alerts to proactively control cloud costs. 

The demo is specific to Databricks, but there are dozens of reports available for any environment Unravel supports. Some of them are in beta, so reach out to your technical account manager (TAM) for access to or training on these reports. In the meantime, explore these key Unravel features.

New professional services offering

Unravel’s Global Head of Customer Success, Matt Sangar, introduces our new professional services to help you leverage Unravel’s full capabilities. Whether it’s improving application performance, lowering cloud costs, reducing MTTR and support ticket volume, or improving data team productivity, Unravel professional services can help you get the most out of Unravel quickly. He has added two new roles to your Customer Success team, an Unravel engineer and a project manager, that will work with you as an extension of your team.

Expanded training options

Unravel is also introducing new on-site training enablement for in-person learning, at basic, advanced, and custom level, for Admin and Ops teams, AppDev and Support teams, and FinOps—budget management training for Finance counterparts. 

Sometimes you might need extra help, or there are other teams who want/need to learn about Unravel. We now offer half- or full-day Unravel Data Days for on-site, in-person (or Zoom) workshops. We are also introducing office hours, when users can sit down with Unravel engineers to troubleshoot apps in real time.

For more information about our new training programs, professional services, access to beta reports, or anything else, please contact your TAM and we’ll take care of you!

The post 5 highlights from the Unravel Roadmap 2023 preview appeared first on Unravel.

Keith Roseland-Barnes Joins Unravel Data as Chief Revenue Officer

Palo Alto, CA – December 1, 2022 – Unravel Data, the first DataOps observability platform built to meet the needs of modern data teams, today announced that Keith Roseland-Barnes has joined the company as its new Chief Revenue Officer. In this new role, he will lead the go-to-market strategy for sales, solution engineering, and the channel. The addition of Roseland-Barnes follows the company’s recent announcements of Series D funding and Ravi Vedantam joining as Vice President, Global Partnerships.

“Data teams across organizations in every industry are looking to evolve from legacy data technologies and get more out of the data they have. They need to optimize cost and performance to efficiently deliver data outcomes and effectively manage the limitless rise in cloud compute and storage costs,” says Kunal Agarwal, CEO of Unravel Data. “Keith has spent most of his 20+ year career in technology sales leadership and has gone from traditional data to AI/ML and now to cloud and DataOps. He’s the perfect person to help lead our team as we continue to grow and enable companies to drive more value from internal and external data.”

Roseland-Barnes brings more than 20 years of technology sales, operations, and leadership experience, including roles at DataRobot, Cloudera, Hortonworks, and Oracle. He has a strong track record of helping customers overcome complex data challenges, establishing and delivering on their objectives, and building strong relationships. He has consistently exceeded revenue targets through negotiating high-value contracts and structuring multimillion-dollar deals across new and existing customers all around the world.

“After spending more than a decade in the traditional data, big data, and data science arenas, it felt like the right time for me to shift to DataOps observability, as enterprises continue to be overwhelmed by the number of data pipelines in their environments,” said Roseland-Barnes. “Founders Kunal Agarwal and Dr. Shivnath Babu had the foresight to realize that the exponential growth of data combined with the broad adoption of the public cloud would require an entirely new way of managing and optimizing data pipelines. I’m excited to be part of this growing team and the DataOps observability space.”

Numerous Fortune 100 companies, including two of the top five global pharmaceutical companies and three of the top 10 global financial institutions, trust Unravel Data for unprecedented visibility across their data stacks, proactively troubleshoot and optimize their data workloads, and define guardrails to govern costs and improve predictability. Customers who have deployed the Unravel platform have been able to double their data teams’ productivity and ensure data applications run on time while efficiently scaling cost on the cloud. Unravel delivers these benefits to its customers by collaborating closely with its partners, helping each customer drive cloud adoption and cloud migration with a plan to identify efficiencies and provide ongoing governance.

To learn more about how Unravel supports a wide variety of daily observability challenges, visit the new library of quick demonstration videos.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading DataOps observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Keith Roseland-Barnes Joins Unravel Data as CRO appeared first on Unravel.

Unravel Data Accelerates DataOps Time to Value with Launch of its 2022 Fall Release

Latest Edition of the Unravel Platform Includes BigQuery Support and Cost 360 for Amazon EMR to Help Customers Optimize the Cost, Efficiency, and Performance of their Data Applications

Palo Alto, CA – Nov 9, 2022 – Unravel Data, the first DataOps observability platform built to meet the needs of modern data teams, today announced the general availability of its 2022 Fall Release of the Unravel Platform. With this new release, users of the Unravel Platform are now able to leverage several new capabilities including support for Google Cloud BigQuery and Cost 360 for Amazon EMR. These new capabilities are designed to help users boost the efficiency of their public cloud spend, simplify troubleshooting across their big data ecosystem, and improve the overall performance of their business-critical data applications.

“Data teams have a clear mandate to ensure that the data pipelines that support their data analytics programs are fully optimized, running efficiently and staying within budget. However, given the complexity of their data ecosystem, getting answers about the health and performance of their data pipelines is harder than ever,” said Kunal Agarwal, founder and CEO of Unravel Data. “Whether they’re migrating more workloads to platforms like BigQuery or Amazon EMR or already running it as part of their data ecosystem, enterprise data teams are struggling to control costs and accurately forecast their resource requirements that are ultimately impacting their ability to execute on their strategic data analytics initiatives. With this latest edition, Unravel customers will be better able to gain the full-stack observability they need to optimize performance and manage their costs according to budget.”

Some of the new capabilities in the Fall Edition of the Unravel Platform include:

• Full Cost Governance for Amazon EMR: With Unravel’s new ‘Cost 360 for Amazon EMR’ capability, customers can enjoy full visibility and gain critical insights into their spending on Amazon EMR. A new cost page has been added that enables customers to observe their EMR cost and usage trends and identify anomalies, slice and dice chargeback views with a variety of filters, and actively monitor costs against user-defined budgets that can automatically trigger warnings if costs are projected to exceed predefined thresholds.
• Advanced EMR Cluster Management: To streamline the management of EMR clusters, a new Clusters page is available inside the Unravel UI for Amazon EMR, allowing users to actively monitor all their EMR clusters and cost data from a single location. The Clusters page also helps users quickly visualize, debug, and troubleshoot issues at both the cluster and application levels.
• Unified View into BigQuery Deployments: Unravel enables customers to now view all information about any number of Google Cloud Platform projects and all BigQuery queries in these respective projects from a single interface. Key performance indicators and metadata are collected for every query. The new edition now supports advanced relevance-based ranked search, faceted search based on metadata and performance indicators, time-based search, as well as drill downs into the individual query level.
• Enhancements for Databricks: When scheduling a job to extract metadata from Databricks, Unravel users can now specify multiple database names and can extract up to 25,000 Delta tables in a single job. Other Databricks enhancements include custom pricing for workload and tier combinations, APIs to detect anomalies and metrics, as well as the ability to encrypt passwords and tokens to harden application security.

More than 20 enterprise companies in the Fortune 100, including two of the top five global pharmaceutical companies and three of the top 10 financial institutions in the world, today rely on Unravel Data to gain unprecedented visibility across their data stacks, proactively troubleshoot and optimize their data workloads, and define guardrails to govern costs and improve predictability.

Data teams are invited to sign up for a free trial of the Unravel Platform at: https://www.unraveldata.com/create-account/

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading DataOps observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51 (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data Accelerates DataOps Time to Value appeared first on Unravel.

Unravel Data to Host India’s First DataOps Observability Conference in Bangalore

The day-long conference on Friday, October 14, will decode DataOps observability and showcase the latest trends and developments in observability

October 12, 2022, Bangalore: Unravel Data, the first DataOps observability platform built to meet the needs of modern data teams, today announced that it is hosting India’s first DataOps Observability Conference on Friday, October 14, 2022 in Bangalore. The day-long conference is hosted in partnership with JetBrains and The Fifth Elephant. Hundreds of data engineers, data leaders, architects, and operations engineers from leading technology companies will be attending the conference.

DataOps and observability are two of the biggest trends in modern data management. This first-of-its-kind conference dedicated to these trends will explain why DataOps observability has become a mission-critical requirement for managing modern data stacks.

The conference will feature leading data professionals from companies like Uber, Dell Technologies, Infosys and Intuit as speakers. The agenda for the event includes panel discussions and talks on managing the modern data stack and applying DataOps observability to all data and analytics use cases.

Dr. Shivnath Babu – co-founder and CTO of Unravel Data will deliver the keynote address at the event. Industry experts who will be speaking on DataOps are Anuj Gupta – chief AI officer at Vahan Inc, Atri Sharma – Senior Staff Engineer at Uber, Sona Samad, Staff Engineer at Intuit and Pallavi Angraje – Software Engineer at Intuit. A Panel discussion on “DataOps Observability: Old Wine in a New Bottle or the Next Big Thing?” will be hosted by Dr. Shivnath Babu. The panelists include Govinda Sambamurthy – Head Of Engineering, Enterprise Data Portability at Atlassian, Akhila Prabhakaran – Consultant Data Scientist at Dell and Giriraj Bagdi – VP Engineering Operations, Data/AI Platform at Unravel.

“We are excited to host the first ever DataOps Observability Conference for data leaders and professionals in India and demonstrate the different aspects of DataOps observability, including application and pipeline performance, operational observability into how the entire platform or system is running end-to-end, and business observability aspects such as ROI and—most significantly—FinOps insights to govern and control escalating cloud costs,” said Shivnath.

Unravel Data estimates that data engineers and operations teams spend around 70-75% of their time tracking down and resolving problems through manual detective work and time-consuming trial and error. Further, as more workloads migrate to the cloud, team leaders are finding that costs are getting out of control—often leading to migration initiatives stalling out completely because of budget overruns.

“Data is eating the world, but complexity is eating data teams who are struggling with cloud costs, pipeline performance, and data quality issues. True DataOps observability can empower these teams to easily tackle even the most complex modern data stack challenges, so they can spend more time on innovation and less time troubleshooting issues,” Shivnath added.

Unravel’s DataOps Observability Platform extracts, correlates, and analyzes information from every system in the modern data stack, within and across some of the most popular data ecosystems. The platform is designed to show data application and pipeline performance, cloud costs, and quality; and proactively suggests precise, prescriptive fixes to quickly and efficiently solve problems. The AI-powered solution helps enterprises realize greater return on their investment in the modern data stack by delivering faster troubleshooting, better performance to meet service level agreements, self-service features that allow applications to get out of development and into production faster and more reliably, and reduced cloud costs.

Data teams can preview how Unravel supports a wide variety of daily observability challenges by viewing the new library of quick demonstration videos on YouTube here.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading DataOps Observability Platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51 (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Bhawana Singh
+917204277001
bhawanas@evoc.in

The post India’s First DataOps Observability Conference in Bangalore appeared first on Unravel.

The post DataOps Observability Designed for Data Teams appeared first on Unravel.

Join us at an event near you or attend virtually to discover our DataOps observability platform, discuss your challenges with one of our DataOps experts, go under the hood to check out platform capabilities, and see what your peers have been able to accomplish with Unravel.

September 21-22: Big Data LDN (London) 

Big Data LDN is the UK’s leading free-to-attend data & analytics conference and exhibition, hosting leading data and analytics experts, ready to arm you with the tools to deliver your most effective data-driven strategy. Stop by the Unravel booth (stand #724) to see how Unravel is observability designed specifically for the unique needs of today’s data teams. 

Register for Big Data LDN here

And be sure to stop by the Unravel booth at 5PM on Day 1 for the Data Drinks Happy Hour for drinks and snacks (while supplies last!)

October 5-6: AI & Big Data Expo – North America (Santa Clara) 

The world’s leading AI & Big Data event returns to Santa Clara as a hybrid in-person and virtual event, with more than 5,000 attendees expected to join from across the globe. The expo will showcase the most cutting-edge technologies from 250+ speakers sharing their unparalleled industry knowledge and real-life experiences, in the forms of solo presentations, expert panel discussions and in-depth fireside chats.

Register for AI & Big Data Expo here

And don’t miss Unravel Co-Founder and CEO Kunal Agarwal’s feature presentation on the different challenges facing different AI & Big Data team members and how multidimensional observability (performance, cost, quality) designed specifically for the modern data stack can help.

October 10-12: Chief Data & Analytics Officers (CDAO) Fall (Boston)

The premier in-person gathering for data & analytics leaders in North America, CDAO Fall offers focus tracks on data infrastructure, data governance, data protection & privacy; analytics, insights, and business intelligence; and data science, artificial intelligence, and machine learning. Exclusive industry summit days for data and analytics professionals in Financial Services, Insurance, Healthcare, and Retail/CPG.

Register for CDAO Fall here

October 14: DataOps Observability Conf India 2022 (Bengaluru)

India’s first DataOps observability conference, this event brings together data professionals to collaborate and discuss best practices and trends in the modern data stack, analytics, AI, and observability.

Join leading DataOps observability experts to:

• Understand what DataOps is and why it’s important
• Learn why DataOps observability has become a mission-critical need in the modern data stack
• Discover how AI is transforming DataOps and observability

Register for DataOps Observability Conf India 2022 here

November 1-3: ODSC West (San Francisco)

The Open Data Science Conference (ODSC) is essential for anyone who wants to connect to the data science community and contribute to the open source applications it uses every day. A hybrid in-person/virtual event, ODSC West features 250 speakers, with 300 hours of content, including keynote presentations, breakout talk sessions, hands-on tutorials and workshops, partner demos, and more. 

Register for ODSC West here

Sneak peek into what you’ll see from Unravel 

Want a taste of what we’ll be showing? Check out our 2-minute guided-tour interactive demos of our unique capabilities. Explore features like:

• Full-stack data pipeline observability
• Automated root cause analysis, at both the job and pipeline level
• Job-level AI optimization recommendations
• Automated cloud cluster optimizations
• Pinpoint-precise chargeback reports
• Automated budget tracking
• Proactive cost governance alerts
• Cloud migration workload fit reports
• Automated cluster discovery

Explore all our interactive guided tours here.

The post Unravel Goes on the Road at These Upcoming Events appeared first on Unravel.

Security is top of mind for every enterprise these days. There are so many threats they can hardly be counted, but one commonality exists: data is always the target. Unravel’s mission is to help organizations better understand and improve the performance of their data. We’re a data business, so we appreciate the scope and implications of these threats. Unravel Data is dedicated to making sure that both our current and potential customers are aware of the advanced features and security of Unravel Data’s AI-enabled DataOps Observability for modern data stacks. We are committed to having our company’s policies and procedures examined and verified by impartial third parties as part of that commitment. That’s why we’re excited to announce that the Unravel Data platform has earned a Service Organization Control (SOC) 2, Type II certification for 2022.

What is SOC 2?

The SOC 2 certification is the most rigorous and respected security certification any software platform can earn. The certification process is governed by the American Institute of CPAs (AICPA). According to the organization, the SOC 2 certification is “intended to meet the needs of a broad range of users that need detailed information and assurance about the controls at a service organization relevant to security, availability, and processing integrity of the systems the service organization uses to process users’ data and the confidentiality and privacy of the information processed by these systems.” The SOC 2, Type II certification focuses on “management’s description of a service organization’s system and the suitability of the design and operating effectiveness of controls.” This basically means, does a company have the right policies and procedures in place to keep its users safe from attacks and vulnerabilities and maintain their data privacy?

Why is this important?

High-leverage data teams choose Unravel as a data observability platform that helps them save time and preserve trust. Customers that use our platform can trust that their metadata is secure.

For Unravel to achieve this certification, independent auditor A-LIGN Assurance performed a thorough test of the product and examined the company’s operations to ensure Unravel has adequate safeguards to prevent or mitigate any security threats. The process was meticulous and comprehensive, taking several months to complete. Essentially, the SOC 2, Type II certification means that Unravel customers can rest assured knowing that our product is safe and that we take significant proactive measures to protect user data.

If you would like to view the full Unravel full SOC 2 report under NDA, please contact us here.

The post Unravel Data Is SOC 2 Type II Compliant appeared first on Unravel.

Benchmarked Survey from 2022 DataOps Unleashed Event Highlights the Key Priorities and Pressing Challenges Facing the Burgeoning DataOps Market

Palo Alto, CA – April 14, 2022 – Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today released key findings from a survey administered to several hundred attendees at the most recent DataOps Unleashed event in order to gauge the priorities, challenges, and progress of leading data teams as they seek to modernize their big data management and analytics capabilities so they can fully realize their real-time data analytics ambitions. To view and download an infographic of the key findings from this survey, click here.

“Data is the lifeblood of the modern enterprise and those organizations who have dedicated the resources and budget to modernizing their data stacks are the ones who will be best positioned to drive innovations in the coming decade,” said Kunal Agarwal, co-founder and CEO of Unravel Data. “The results of this latest survey show just how complex the modern data stack has become and illustrates the many unanticipated challenges that come with efficiently managing and optimizing data pipelines across multiple public cloud providers and platforms. It also serves to validate that there is an obvious demand for a purpose-built solution that can help these teams gain the critical visibility they need to drive the most value from their data operations.”

Some of the key findings collected from the most recent survey, which benchmarks responses from the year prior, include:

• DataOps as a practice is hitting an inflection point: There was an almost 80% increase from the year prior of respondents who said they are in the active stage of adopting a formal DataOps approach to manage and optimize their data pipelines. This year, more than 41% of attendees reported they are actively employing DataOps methodologies, compared to just less than a quarter (24%) in 2021.
• Visibility into data pipelines remains the top challenge: For the second year in a row, when participants were asked what they viewed as the top challenge with operating their data stack, respondents cited the lack of visibility across their environment as their most significant obstacle. Whereas in the previous year respondents reported that “controlling runaway costs” was the second biggest, this year the “lack of proactive alerts” was noted as the second most challenging aspect.
• Complexity of cloud migrations is more time consuming than previously thought: Sixty percent of respondents from this year’s event estimated that their cloud migration project would take between 12-24 months, representing a 150% increase over the prior year’s projection. The challenge of forecasting the duration of these cloud migration initiatives reveals the vast amount of complexity and uncertainty that data teams face when attempting to map out these critical projects.
• Automation continues to be a key driver: When asked about the role of automation in managing their DataOps pipelines, three in four DataOps professionals in both years reported that the ability to “automatically test and verify before moving jobs/pipelines to production” was the most important automation priority when compared to other aspects such as automating troubleshooting of platform or pipeline issues.
• Data teams spend more time building than deploying/managing pipelines: For both years, data professionals reported that they spent the majority of their day building their data pipelines (39% in 2021 and 43% in 2022). In 2022, respondents reported spending slightly less time maintaining or troubleshooting their pipelines (30%) than the year prior (34%) while the time spent deploying data pipelines remained the same at 27% for both years.

Now in its second year, DataOps Unleashed is a one-day virtual conference underwritten by Unravel Data, that seeks to build a vibrant peer-based community for DataOps professionals to share best practices. Data professionals can productively collaborate with one another and deliver on the promise of what it means to be a data-led company. This past year’s event attracted more than 2,300 data professionals and featured presentations from data leaders at some of the most data-forward enterprise organizations in the world, including AWS, DBS, Google, Johnson & Johnson, and many others. Recordings of all 20 sessions are available to watch for free and on-demand on the DataOps Unleashed site at: www.dataopsunleashed.com.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post New Survey Reveals Poor Visibility and Lack of Prescriptive Insights as Top Challenges for DataOps in 2022 appeared first on Unravel.

Today, we’re excited to announce the Unravel Winter Release [4730]!

This winter release introduces major enhancements and improvements across the platform, including comprehensive cost management for Databricks, support for Delta Lake on Databricks, data observability for Google BigQuery, interactive pre-check before installation and upgrade. Also, major enhancements include deep dive into Spark applications and YARN resource manager, a view of all Databricks applications from a single tab, workload fit and cluster discovery report for cloud data migration, multi-cluster support for dashboard app on the Unravel app store and more.

These innovations will enable businesses to manage and simplify the complexities of modern data operations, optimize their costs, and derive more value from their data.

Here are the innovations we introduced in eight key areas of Unravel:

New features and improvements:
1. Cost 360 for Modern Data Stack: Understand and optimize your cloud data infrastructure costs to get the most out of your data cloud investments. Data operations cost control is a major concern, especially for cloud users. Most of the time the costs get unpredictable and unexpectedly overshoot the budgets.

What is Cost 360 for Modern Data Stack?
Cost 360 for Modern Data Stack refers to a comprehensive, end-to-end, single view of the costs associated with an enterprise’s data operations along with granular details about the user, team, data workload, usage type, data job, data application, compute, and resources consumed to execute the data application. Also, it ensures that every data operations team member in the organization has access to the same version of the truth about the cost of data operations today and projected future costs.

Unravel provides DataOps and data engineering teams with full-stack visibility into their complex data pipelines and the ability to see the costs incurred by different users and data applications. With this release, organizations now have a better view of the cost incurred across data operations in Databricks and can optimize their data workload on Databricks.

Better governance over Databricks cost: Unravel gives Databricks users meaningful insights into Databricks costs.

Trends: Get a better understanding of your existing cost trends and forecasts.

• View your cost trends for DBUs as well as the overall number of clusters used. This allows you to uncover interesting time periods with anomalies such as
  unexpected spikes. you can further investigate the chargeback function or optimize the resources.
• View the trends of DBUs cost and number of clusters for a specific user, workspace, or team. This will assist in identifying interesting time periods with anomalies for that entity and swiftly optimize its cost.

Chargeback: With Unravel you can leverage the chargeback feature for Databricks to allocate costs to internal users to improve accountability, cut waste, and empower data teams to spend efficiently.

Budgets and Forecasts: Unravel allows you to create customized budgets that notifies you if you exceed (or if you are expected to exceed) the limits that you have set.

• Set customized budgets that fit your goals to keep your Databricks spending on track. You can define a target budget for individual users, groups, or workloads.
• Set role-based permission to make the budget view visible only to those users with the right permissions, so that not every user can see it. It’s only exposed to admin and read-only admin.

Optimization: Unravel provides you actionable insights to optimize Databricks workloads for efficiency and cost optimization.

• Insights and recommendations to optimize compute [clusters] and jobs. Visibility into estimated cost improvements to prioritize workload optimization
• Insights and recommendations to improve compute [clusters] and jobs performance. Visibility into estimated cost savings to prioritize workload optimization.

Cost 360 degree for Databricks video

2. Databricks Delta Lake: Support for Delta Lake. Now with Unravel, you can configure the application to fetch the metadata of the Databricks Delta tables and monitor them from the Data tab of the Unravel UI. With this, you can view Delta tables related data insights on the tables page, like., Delta table format, size of tables, number of partitions created over time, table metadata etc.

3. Unravel Role-Based Access Control (RBAC): Unravel’s role-based access control allows admins to manage who has access to various views and features of Unravel and what areas they have access to. This empowers users and admins to view more and do more.

Support for custom roles and permissions:

• Define custom roles beyond the default — admin, read-only, and user.
• Define views that a role can see.
• Define data filters to apply. You can choose from user tags, app tags, app data fields, and even write an elasticsearch query filter to meet your requirements.
• Generate user tags using user tagging script.

4. Google Cloud [Dataproc and BigQuery]: Gain complete visibility into your Google Cloud Dataproc and Google Cloud BigQuery projects [private preview].

Google Cloud Dataproc Observability:

• View and understand cluster resources, jobs, and stages across Google Dataproc instances and workspaces.
• Visibility into data usage, including tables accessed by user or by application, and identification of hot/cold/warm tables and data.
• Pinpoint accuracy for runtime tunable parameters gets maximum performance from Google Cloud Dataproc.
• Unravel provides meta-intelligence about cloud server instances to right-size infrastructure for the best performance of Spark and Hadoop apps.
• Faster diagnosis and resolution of application failures, slowdowns in Spark and Hadoop applications and data pipelines
• Reduced mean time to identification and recovery from failures (MTTI/MTTR)

Google Cloud BigQuery Observability [private preview]: Unravel for Google BigQuery is available in private preview, with public preview coming shortly. Now Unravel helps you to gain insights and improve performance of your BigQuery applications, users, and infrastructure.

A private preview of support for Google BigQuery includes:

• Observability: Observe the performance of Google BigQuery and troubleshoot the issues before they impact the application underneath and the users.
• View the jobs running on the cluster.
• View details of errors encountered by the jobs.
• Governance: Google BigQuery can quickly get expensive if you are managing massive amounts of data every day and do not use cost optimization techniques.
• Gain the chargeback view of resources used in BigQuery.
• Associate jobs with business priority tags.
• Optimization: Monitor and detect slow Google BigQuery queries, and optimize inefficient queries and operations.
• Analysis of the job execution based on the resource usage and time spent.
• Visibility into data/tables usage (hot, warm, cold).

5. Cloud Data Migration: Cloud data migration can be difficult. If an enterprise does not have complete insight into its data, data users, data applications, and data dependencies, the process can become time-consuming and costly. With a data-driven assessment plan in place, an organization can minimize the time and cost of cloud data migration.

Unravel enables you to get full visibility and understanding of an organization’s data environment, highlighting the best data applications to migrate, removing migration bottlenecks, maximizing cost savings, and simplifying the process.

Cloud data migration assessment report

• Assess (basic assessment) — Make a well-informed go/no go decision to the cloud.
• Plan (deep assessment) — When an organization has decided to migrate to the cloud, deep assessment will help them to plan for a smoother and faster migration.

6. UI Enhancements: Unravel now has improved its UI to boost usability.

• Budget tab: View a list of different aspects of budgets and quickly compare budgets between different periods.
• Alert tab: Set custom alert notification to email or Slack.

7. Unravel’s Billing Services: The Billing tab in Unravel shows the charges of Unravel for its support for AWS EMR.
The following pricing plans are supported:

• Pay-as-you-go: As per this plan, Unravel tracks the number of instance hours that you have incurred and shows the charges based on the usage.
• Pay-in-advance: As per this plan, you can pay in advance for a specific number of instance hours. Credits will be taken based on the usage, with the remaining credits shown on the Billing tab daily. You can monitor when they run out of credits and plan accordingly.

8. AppStore Page: The App store feature is added from where you can manage all the Unravel applications. From the app store, you can install an application, run the administrative tasks for managing your apps, navigate to different apps, open and run the apps. Now you can get different views and insights on the telemetry data for your cluster that Unravel has collected.

What’s next for you?

Read the detailed release notes here.
Create a free account to get started.

The post Announcing the Unravel Winter Release appeared first on Unravel.

Unravel Data CEO to Keynote Second “DataOps Unleashed” Virtual Event on February 2, 2022

Peer-to-Peer Event Dedicated to Helping Data Professionals Untangle the Complexity of Modern Data Clouds and Simplify their Data Operations

WHAT: Unravel Data announced that its CEO and Co-founder, Kunal Agarwal will deliver the keynote address at the “DataOps Unleashed” event, a free, full-day virtual summit will showcase some of the most prominent voices from across the burgeoning DataOps community that will be taking place on February 2nd, 2022.

The most successful enterprise organizations of tomorrow will be the ones who can effectively harness data from a broad array of sources and operating environments and rapidly transform it into actionable intelligence that supports data-driven decision making. DataOps Unleashed is a growing, cross-industry community where data professionals can productively collaborate with one another, share industry best practices, and deliver on the promise of what it means to be a data-led company.

The event will feature compelling presentations, conversation, and peer-sharing between technical practitioners, data scientists, and executive strategists from some of the most data-forward enterprise organizations in the world, including Slack, Zillow, Cisco, IBM, and many other recognized global brands. Over the course of the day, attendees will learn how a modernized approach to DataOps can transform their operations, improve data predictability, increase reliability, and create economic efficiencies with their data pipelines. Leading vendors from the DataOps market including Census, Metaplane, Airbyte, and Manta will be joining Unravel Data in supporting this community event.

Speakers at the DataOps Unleashed event to include:

• Kunal Agarwal, Co-founder & CEO & Shivnath Babu, Co-founder & CTO, Unravel Data
• Ryan Kinney, Senior Data Engineer, Slack
• Andrei Lopatenko, VP of Engineering, Zillow
• Disha Ahuja, Senior Manager, Cisco
• Torsten Steinbach, Lead Architect, IBM

More information about the event can be found at: https://dataopsunleashed.com/

WHEN: Wednesday, February 2nd beginning at 9AM PST

COST: Free

WHERE: Register at: https://dataopsunleashed.com/

WHO: Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Adobe and Deutsche Bank. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

The post DataOps Unleashed 2022 Keynote appeared first on Unravel.

Thank you for your interest in the 451 Research Report, Data Science and analytics in the cloud set to grow three times faster than on-premises.

You can download it here.

451 Research: Data Science and analytics in the cloud set to grow three times faster than on-premises
Published: September, 28 2021

Introduction
Data science and analytics, as well as the data abstraction and acceleration offerings that underpin them, represented a $29bn market in 2020, according to 451 Research’s Data, AI & Analytics Market Monitor: Data Science & Analytics. Moreover, this market is growing thanks to the critical role of data, AI and analytics in speeding up enterprise data-driven decision-making for faster time to insight. Cloud services, in particular, are exhibiting strong growth – a trend that has been underway for some time and has been accelerated by the COVID-19 pandemic.

Get the 451 Take. Download Report.

The post Data Science and Analytics in the Cloud appeared first on Unravel.

In a press release delivered today, Unravel Data announced its appearance on CRN’s Big Data 100 list for 2021. Unravel’s entry appears in the Data Management and Integration category. Also featured in this category are other rising stars such as Confluent, Fivetran, Immuta, and Okera, all of whom spoke at new industry conference DataOps Unleashed, held in March.

The list recognizes vendors for their “innovation, insight, and industry expertise,” according to Blaine Raddon, CEO of the Channel Company, owners of CRN. Unravel Data CEO Kunal Agarwal cited the company’s mission, “to empower organizations to unleash innovation” by modernizing their approach to DataOps.

Other categories in the list are Business Analytics, Data Science and Machine Learning, Database Systems, and Systems and Platforms. Taken together, the Big Data 100 list provides a snapshot of rapid change taking place in big data, streaming data, and modern data solutions.

The post Unravel Data Featured in CRN’s 2021 Big Data 100 List appeared first on Unravel.

Company Named as One of the “Coolest Data Management Vendors in 2021”

Palo Alto, CA – May 6, 2021 — Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, announced today that CRN, a brand of The Channel Company, has recognized Unravel Data on its 2021 Big Data 100 list. This annual list recognizes the technology vendors that deliver outstanding, innovation-driven products and services for solution providers, who in turn help enterprise organizations better manage and utilize the massive amounts of data that large businesses generate.

A team of CRN editors compiled this year’s Big Data 100 list by identifying IT vendors that have consistently made technical innovation a top priority through their offering of products and services. Unravel Data has been named as one of the “Coolest Data Management and Integration Software Companies” for 2021. Other categories recognized by CRN include business analytics, systems and platforms, database systems, and data science and machine learning. Over the years, the Big Data 100 list has become an invaluable resource for solution providers that trust CRN to help them find vendors that provide crucial solutions for use in data intelligence, insights, and analytics.

Unravel Data removes the blind spots in enterprise data pipelines, providing full-stack visibility and AI-powered recommendations that drive more reliable performance for modern data applications, in the cloud, across clouds, and on-premises. More than 20 Fortune 100 brands rely on Unravel Data to modernize their DataOps systems, proactively monitor the performance of their big data applications, and unleash the power of their data applications, so they can act intelligently and decisively on insights derived from their data.

“IT vendors featured on CRN’s 2021 Big Data 100 list have demonstrated a proven ability to bring much-needed innovation, insight, and industry expertise to the solution providers and customers that need it most,” said Blaine Raddon, CEO of The Channel Company. “I am honored to recognize these companies for their unceasing commitment toward elevating and improving the ways businesses gain value from their data.”

“It’s truly an honor for Unravel Data to be included by CRN on their 2021 Big Data 100 list, alongside iconic technology vendors in one of the fast-growing segments of the enterprise software market,” said Kunal Agarwal, founder and CEO of Unravel Data. “We started Unravel Data with an ambitious mission: to empower organizations to unleash innovation by modernizing how they manage and optimize their data operations. This mission continues to serve as our guiding principle, and we remain committed to transforming the DataOps category.”

The 2021 Big Data 100 list is available online at www.CRN.com/BigData100.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

About The Channel Company®
The Channel Company enables breakthrough IT channel performance with our dominant media, engaging events, expert consulting and education, and innovative marketing services and platforms. As the channel catalyst, we connect and empower technology suppliers, solution providers, and end users. Backed by more than 30 years of unequaled channel experience, we draw from our deep knowledge to envision innovative new solutions for ever-evolving challenges in the technology marketplace. www.thechannelco.com

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post CRN Recognizes Unravel Data on Its 2021 Big Data 100 List appeared first on Unravel.

Mastercard is one of the world’s top payment processing platforms, with more than 700 million cards in use worldwide. In the US, nearly 40% of American adults hold a Mastercard-branded card. And the company is going from strength to strength; despite a dip in valuation of more than a third when the pandemic hit, the company has doubled in value three times in the last nine years, recently reaching a market capitalization of more than $350B dollars. 

The importance of cards has soared as a result of the pandemic. Cash use has declined sharply as less purchasing is done in person, and even in-person shopping has shifted toward cards, for reasons of hygiene. For Mastercard, keeping their back-end machinery running well has been vital, both to maintain business results and for consumer and payment network confidence. Mastercard is a strong user of artificial intelligence, in particular for fraud reduction, and all sorts of reporting, business intelligence support, and advanced analytics are necessary to meet business needs and regulatory requirements. 

At DataOps Unleashed, Mastercard’s Chinmay Sagade, Principal Engineer, and Srinivasa Gajula, Big Data Engineer, described a specific use of Unravel Data to increase platform resiliency. Mastercard uses Unravel to reject potentially harmful workloads on Hadoop, which improves job quality over time and keeps the platform available for all users. 

Ad Hoc Query Loads vs. Hadoop, Impala, Spark, and Hive

“They were able to see the platform resiliency and availability improved.” – Chinmay Sagade

Mastercard relies primarily on Hadoop for core big data needs, having first adopted the platform ten years ago. Their largest cluster has hundreds of nodes, and they have petabytes of data. Thousands of users access the platform, with much usage being ad hoc. Impala and Spark are widely used, with some Hive usage in the mix. 

There are several problems that are common in big data setups like the one in use at Mastercard – exacerbated, in this case, by the sheer scale at which Mastercard operates:

• These big data technologies are not easy for casual users to query correctly
• Poorly structured queries cause big system impacts
• Disks fill up, network pipes clog, and daemons are disabled, with unpredictable results
• Impacts include application failures, system slowdowns, system crashes, and resource bottlenecks

Before the pandemic, Mastercard’s big data stack was already at capacity. Rogue jobs were not only failing, but also affecting other jobs. One Hive query ran for 24 hours, pushing out other users. The need was strong to improve operational effectiveness, reduce resource utilization, and make room for growth, without additional infrastructure cost.

As Chinmay Sagade describes it, there is a “butterfly effect” – that is, “Big data means that even a small problem can cause a large impact.” He describes the situation as “a recipe for disaster,” as productivity plummets and SLAs are not met. He even cites receding hairlines as an occupational hazard for stressed Hadoop administrators. 

Unravel Data (and Smart Operators) to the Rescue

“We saw an immediate positive impact on the platform.” – Chinmay Sagade

At Mastercard, the problems with query processing became so serious that user and management trust in the platform was in decline. A new solution to these problems was sorely needed. 

Initial use of Unravel Data proved fruitful. For instance, Unravel Data identified that more than a third of data tables stored across technologies were unused. Removing these tables freed up resources. Repeating this scan now takes minutes, with actionable results, where it previously took days, and produced unreliable results.

Unravel Data is now used for several layers of defense against rogue queries:

• User and application owner self-tuning of their own query workloads
• Automated monitoring to alert on “toxic” workloads in progress
• Further monitoring to prevent the most hazardous workloads from running at all

Unravel helps improve resource usage, pre-empt many previous problems, and reduce mean time to remediation (MTTR) for the problems that remain.

Users who want to avoid problems can use Unravel Data to tune their own workloads. Their jobs then run faster, with far less chance of disruption, and they avoid automated alerts or even workload shutdowns. 

But users can be in a hurry. They may not know how to check their own workloads, or they may make mistakes despite the availability of checking. The Mastercard data team needed another layer of defense. 

Using Runtime Data to Manage Outliers

“We will ask them to take the necessary actions, like tuning the quarry and resubmitting again.” – Srinivasa Gajula

Mastercard took additional steps to monitor and act on toxic workloads. They created a Python-based framework which collects application data at runtime. Anomaly detection algorithms scan relevant metrics and flag toxic workloads. All of this connects to Unravel. 

“We use the Impala and Yarn APIs to collect metrics, along with HDFS metadata,” says Srinivasa Gajula. They produce summary reports to note the number and percentage of workloads that fail with out-of-memory errors, syntax errors, and other causes. They detect excessive numbers of small files and calculate both mean time between failures (MTBF) and mean time to repair (MTTR). This information is shared with users and application owners, helping them to make proper use of the platform. 

They also detect different types of joins and identify, as the join proceeds, whether it’s likely to make excessive demands on the system. When a user provides compute stats to Impala, for instance, then Impala can identify whether specific tables should be broadcasted, or shuffled, and how to filter data for optimal performance. And users can provide hints in the query to, for example, broadcast a small table, or shuffle a larger one. 

But many users run their queries without providing this helpful information. Impala may then broadcast a large table, for example, causing a performance slowdown or even a crash. 

So Mastercard now identifies these issues as they begin to occur. They build a tree from operator dependencies and predict whether a large table, for instance, is likely to be broadcast. If so, the user is asked to tune the query, and submit it again. 

They can even identify whether a particular query is CPU-bound or I/O bound. Where a cross join, for instance, is causing the number of rows produced to grow exponentially, in a way that is likely to cause performance issues, or even stability problems for the platform. They can alert the user or, in extreme cases, kill the query. 

Unravel is now part of the software development life cycle (SDLC) process at Mastercard. Application quality increases up front, and the ability to fix remaining problems in production is greatly improved as well.

Business Impacts of Pre-Checking and Pre-Emption

“Now administrators can spend their time in value-added activities.” – Chinmay Sagade

Mastercard has racked up many benefits by empowering users to check their own queries and pre-empting the remainder that are not “fixed” and are still problematic:

• Less time spent troubleshooting
• Greater reliability
• Resources not over-allocated, so resources are freed up
• Infrastructure costs reduced through appropriate use or resources

Not all of this has been easy. Users needed plenty of notice and detailed documentation. And they can only be expected to learn so much about how to right-size their own queries. But users have actually supported restrictions on unbalanced jobs, as they see the benefits of better query performance and a more reliable platform for everyone.

This blog post is a good starting point, but it’s worth taking the time to watch the Mastercard presentation yourself. And you can view all the videos from DataOps Unleashed here. You can also download The Unravel Guide to DataOps, which was made available for the first time during the conference.

The post Mastercard Reduces MTTR and Improves Query Processing with Unravel Data appeared first on Unravel.

The DataOps Unleashed conference, founded this year by Unravel Data, was full of interesting presentations and discussions. We described the initial keynote and some highlights from the first half of the day in our DataOps Unleashed Part 1 blog post. Here, in Part 2, we bring you highlights through the end of the day: more about what DataOps is, and case studies as to how DataOps is easing and speeding data-related workflows in big, well-known companies.

You can freely access videos from DataOps Unleashed – most just 30 minutes in length, with a lot of information packed into hot takes on indispensable technologies and tools, key issues, and best practices. We highlight some of the best-received talks here, but we also urge you to check out any and all sessions that are relevant to you and your work.  

Mastercard Pre-Empts Harmful Workloads

See the Mastercard session now, on-demand. 

Chinmay Sagade and Srinivasa Gajula of Mastercard are responsible for helping the payments giant respond effectively to the flood of transactions that has come their way due to the pandemic, with cashless, touchless, and online transactions all skyrocketing. And much of the change is likely to be permanent, as approaches to doing business that were new and emerging before 2020 become mainstream. 

Hadoop is a major player at Mastercard. Their largest cluster is petabytes in size, and they use Impala for SQL workloads, as well as Spark and Hive. But they have in the past been plagued by services being unavailable, applications failing, and bottlenecks caused by suboptimalsub-optimal use of workloads. 

Mastercard has used Unravel to help application owners self-tune their workloads and to create an automated monitoring system to detect toxic workloads and automatically intervene to prevent serious problems. For instance, they proactively detect large broadcast joins in Impala, which tend to consume tremendous resources. They also detect cross-joins in queries. 

Their work has delivered tremendous business value:

• Vastly improve reliability
• Better configurations free up resources
• Reduced problems free up time for troubleshooting recurring issues
• Better capacity usage and forecasting saves infrastructure costs

To the team’s surprise, users were hugely supportive of restrictions, because they could see the positive impact on performance and reliability. And the entire estate now works much better, freeing resources for new initiatives. 

Gartner CDO DataOps Panel Shows How to Maximize Opportunities

See the Gartner CDO Panel now, on-demand.

As the VP in charge of big data and advanced analytics for leading industry analysts, Gartner, Sanjeev Mohan has seen it all. So he had some incisive questions for his panel of four CDOs. A few highlights follow. 

Paloma Gonzalez Martinez is CDO at AlphaCredit, one of the fastest-growing financial technology companies in Latin America. She was asked: How has data architecture evolved? And, if you had a chance to do your big data efforts over again, how would you do things differently? 

Paloma shared that her company actually is revisiting their whole approach. The data architecture was originally designed around data; AlphaCredit is now re-architecting around customer and business needs. 

David Lloyd is CDO at Ceridian, a human resources (HR) services provider in the American Midwest. David weighed in on the following: What are the hardest roles to fill on your data team? And, how are these roles changing?

David said that one of the guiding principles he uses in hiring is to see how a candidate’s eyes light up around data. What opportunities do they see? How do they want to help take advantage of them? 

Kumar Menon is SVP of Data Fabric and Decision Science Technology at Equifax, a leading credit bureau. With new candidates, Kumar looks for the combination of the intersection of engineering and insights. How does one build platforms that identify crucial features, then share them quickly and effectively? When does a model need to be optimized, and when does it need to be rebuilt?

Sarah Gadd is Head of Semantic Technology, Analytics and Machine Intelligence at Credit Suisse. (Credit Suisse recently named Unravel Data a winner of their 2021 Disruptive Technologies award.) Technical problems disrupted Sarah from participating live, but she contributed answers to the questions that were discussed. 

Sarah looks for storytellers to help organize data and analytics understandably, and is always on the lookout for technical professionals who deeply understand the role of the semantic layer in data models. And in relation to data architecture, the team faces a degree of technical debt, so innovation is incremental rather than sweeping at this point. 

84.51°/Kroger Solves Contention and the Small Files Problem with DataOps Techniques

See the 84.51/Kroger session now, on-demand. 

Jeff Lambert and Suresh Devarakonda are DataOps leaders at the 84.51° analytics business of retailing giant Kroger. Their entire organization is all about deriving value from data and delivering that value to Kroger, their customers, partners, and suppliers. They use Yarn and Impala as key tools in their data stack. 

They had a significant problem with jobs that created hundreds of small files, which consumed system resources way out of proportion to the file sizes. They have built executive dashboards that have helped stop finger-pointing, and begin solving problems based on shared, trusted information. 

Unravel Data has been a key tool in helping 84.51° to adopt a DataOps approach and get all of this done. They are expanding their cloud presence on Azure, using Databricks and Snowflake. Unravel gives them visibility, management capability, and automatically generated actions and recommendations, making their data pipelines work much better. 84.51 has just completed a proof of concept (PoC) for Unravel on Azure and Databricks, and are heavily using recently introduced Spark 3.0 support. 

Resource contention was caused by a rogue app that spiked memory usage. Using Unravel, 84.51° quickly found the offending app, killed it, and worked with the owner to prevent the problem in the future. 84.51 now proactively scans for small files and concerning issues using Unravel, heading off problems in advance. Unravel also helps move problems up to a higher level of abstraction, so operations work doesn’t require that operators be expert in all of the technologies they’re responsible for managing. 

At 84.51°, Unravel has helped the team improve not only their own work, but what they deliver to the company:

• Solving the small files problem improves performance and reliability
• Spotting and resolving issues early prevents disruption and missed SLAs
• Improved resource usage and availability saves money and increases trust 
• More production from less investment allows innovation to replace disruption 

Cutting Cloud Costs with Amazon EMR and Unravel Data

See the AWS session now, on-demand. 

As our own Sandeep Uttamchandani says, “Once you get into the cloud, ‘pay as you go’ takes on a whole new meaning: as you go, you pay.” But AWS Solutions Architect Angelo Carvalho is here to help. AWS understands that their business will grow healthier if customers are wringing the most value out of their cloud costs, and Angelo uses this talk to help people do so. 

Angelo describes the range of AWS services around big data, including EMR support for Spark, Hie, Presto, HBase, Flink, and more. He emphasized EMR Managed Scaling, which makes scaling automatic, and takes advantage of cloud features to save money compared to on-premises, where you need to have enough servers all the time to support peak workloads that only occur some of the time. (And where you can easily be overwhelmed by unexpected spikes.)  

Angelo was followed by Kunal Agarwal of Unravel Data, who showed how Unravel optimizes EMR. Unravel creates an interconnected model of the data in EMR and applies predictive analytics and machine learning to it. Unravel automatically optimizes some areas, offers alerts for others, and provides dashboards and reports to help you manage both price and performance. 

Kunal then shows how this actually works in Unravel, demonstrating a few key features, such as automatically generated, proactive recommendations for right-sizing resource use and managing jobs. The lessons from this session apply well beyond EMR, and even beyond the cloud, to anyone who needs to run their jobs with the highest performance and the most efficient use of available resources. 

Microsoft Describes How to Modernize your Data Estate

See the Microsoft session now, on-demand.

According to Priya Vijayarajendran, VP for Data and AI at Microsoft, a modern, cloud-based strategy underpins success in digital transformation. Microsoft is enabling this shift for customers and undertaking a similar journey themselves. 

Priya describes data as a strategic asset. Managing data is not a liability or a problem, but a major opportunity. She shows how even a simplified data estate is very complex, requiring constant attention. 

Priya tackled the “what is DataOps” challenge, using DevOps techniques, agile, and statistics, processes, and control methodologies to intelligently manage data as a strategic asset. She displayed a reference architecture for continuous integration and continuous delivery on the Azure platform. 

Priya ended by offering to interact with the community around developing ever-better answers to the challenges and opportunities that data provides, whether on Microsoft platforms or more broadly. Microsoft is offering multi-cloud products that work on AWS and Google Cloud Platform as well as Azure. She said that governance should not be restrictive, but instead should enable people to do more. 

A Superset of Advanced Topics for Data Engineers

See the Apache Superset session now, on-demand.

Max Beauchemin is the creator of Airflow and currently CEO at Preset, the company that is bringing Apache Superset to the market. Superset is the leading open-source analytics platform and is widely used at major companies. Max is the original creator of Apache Airflow, mentioned in our previous Unleashed blog post, as well as Superset. Preset makes Superset available as a managed service in the cloud. 

Max discussed high-end, high-impact topics around Superset. He gave a demo, then demonstrated SQL Lab, a SQL development environment built in React. He then showed how to build a visualization plugin; creating alerts, reports, charts and dashboards; and using the Superset Rest API. 

Templating is a key feature in SQL Lab, allowing users to build a framework that they can easily adapt to a wide variety of SQL queries. Built on Python, Jinja allows you to use macros in your SQL code. Jinja integrates with Superset, Airflow, and other open source technologies. A parameter can be specified as type JSON, so values can be filled in at runtime. 

With this talk, Max gave the audience the information they need to plan, and begin to implement, ambitious Superset projects that work across a range of technologies. 

Soda Delivers a Technologist’s Call to Arms for DataOps

See the Soda DataOps session now, on-demand.

What does DataOps really mean to practitioners? Vijay Karan, Head of Data Engineering at Soda, shows how DataOps applies at each stage of moving data across the stack, from ingest to analytics. 

Soda is a data monitoring platform that supports data integrity, so Vijay is in a good position to understand the importance of DataOps. He discusses core principles of DataOps and how to apply those principles in your own projects. 

Vijay begins with the best brief description of DataOps, from a practitioner’s point of view, that we’ve heard yet:

What is DataOps?

A process framework that helps data teams deliver high quality, reliable data insights with high velocity.

At just sixteen words, this is admirably concise. In fact, to boil it down to just seven words, “A process framework that helps data teams” is not a bad description.

Vijay goes on to share half a dozen core DataOps principles, and then delivers a deep dive on each of them. 

Here at Unravel, part of what we deliver is in his fourth principle:

Improve Observability

Monitor quality and performance metrics across data flows

Just in this one area, if everyone did what Vijay suggests around this – defining metrics, visualizing them, configuring meaningful alerts – the world would be a calmer and more productive place. 

Conclusion

This wraps up our overview description of DataOps Unleashed. If you haven’t already done so, please check out Part 1, highlighting the keynote and talks discussing Adobe, Cox Automotive, Airflow, Great Expectations, DataOps, and moving Snowflake to space. 

However, while this blog post gives you some idea as to what happened, nothing can take the place of “attending” sessions yourself, by viewing the recordings. You can view the videos from DataOps Unleashed here. You can also download The Unravel Guide to DataOps, which was made available for the first time during the conference. 

Finding Out More

Read our blog post Why DataOps Is Critical for Your Business.

The post DataOps Has Unleashed, Part 2 appeared first on Unravel.

DataOps Unleashed launched as a huge success, with scores of speakers, thousands of registrants, and way too many talks for anyone to take in all at once. Luckily, as a virtual event, all sessions are available for instant viewing, and attendees can keep coming back for more. (You can click here to see some of the videos, or visit Part 2 of this blog post.)

Kunal Agarwal, CEO of Unravel Data, kicked off the event with a rousing keynote, describing DataOps in depth. Kunal introduced the DataOps Infinity Loop, with ten integrated and overlapping stages. He showed how teams work together, across and around the loop, to solve the problems caused as both data flow and business challenges escalate. 

Kunal introduced three primary challenges that DataOps addresses, and that everyone assembled needs to solve, in order to make progress:

• Complexity. A typical modern stack and pipeline has about a dozen components, and the data estate as a whole has many more. All this capability brings power – and complexity. 
• Crew. Small data teams – the crew – face staggering workloads. Finding qualified, experienced people, and empowering them, may be the biggest challenge. 
• Coordination. The secret to team productivity is coordination. DataOps, and the DataOps lifecycle, are powerful coordination frameworks. 

These challenges resonated across the day’s presentations. Adobe, Kroger, Cox Automotive, Mastercard, Astronomer, and Superconductive described the Unravel Data platform as an important ally in tackling complexity. And Vijay Kiran, head of data engineering at Soda, emphasized the role of collaboration in making teams effective. The lack of available talent to expand one’s Crew – and the importance of empowering one’s team members – came up again and again. 

There were many highlights per presentation. A few that stand out from the morning sessions are Adobe, moving their entire business to the cloud; Airflow, a leader in orchestration; Cox Automotive, running a global business with a seven-person data team; and Great Expectations, which majors in data reliability. 

How Adobe is Moving a $200B Business to the Cloud

Adobe was up next, with an in-depth cloud migration case study, covering the company’s initial steps toward cloud migration. Adobe is one of the original godfathers of today’s digital world, powering so much of the creativity seen in media, entertainment, on websites, and everywhere one looks. The company was founded in the early 1980s, and gained their original fame with Postscript and the laser printer. But the value of the business did not really take off until the past ten years, which is when the company moved from boxed software to a subscription business, using a SaaS model. 

Now, Adobe is moving their entire business to the cloud. They describe four lessons they’ve learned in the move:

• Ignorance is not bliss. In planning their move to the cloud, Adobe realized that they had a large blind spot about what work was running on-premises. This may seem surprising until you check and realize that your company may have this problem too. 
• Comparison-shop now. You need to compare your on-premises cost and performance per workload to what you can expect in the cloud. The only way to do this is to use a tool that profiles your on-premises workloads and maps each to the appropriate infrastructure and costs on each potential cloud platform. 
• Optimize first. Moving an unoptimized workload to the cloud is asking for trouble – and big bills. It’s critically important to optimize workloads on-premises to reduce the hassle of cloud migration and the expense of running the workload in the cloud. 
• Manage effectively. On-premises workloads may run poorly without too much hassle, but running workloads poorly in the cloud adds immediately, and unendingly, to costs. Effective management tools are needed to help keep performance high and costs under budget. 

Kevin Davis, Application Engineering Manager, was very clear that Adobe has only gained the clarity they need through the use of Unravel Data for cloud migration, and for performance management, both on-premises and in the cloud. Unravel allows Adobe to profile their on-premises workloads; map each workload to appropriate services in the cloud; compare cost and performance on-premises to what they could expect in the cloud; optimize workloads on-premises before the move; and carefully manage workload cost and performance, after migration, in the cloud. Unravel’s automatic insights increase the productivity of their DataOps crew. 

Cox Automotive Scales Worldwide with Databricks

Cox Automotive responded with a stack optimization case study. Cox Automotive is a global business, with wholesale and retail offerings supporting the lifecycle of motor vehicles. Their data services team, a mighty team of only seven people, supports the UK and Europe, offering reporting, analytics, and data sciences services to the businesses. 

The data services team is moving from Hadoop clusters, deployed manually, to a full platform as a service (PaaS setup) using Databricks on Microsoft Azure. As they execute this transition, they are automating everything possible. Databricks allows them to spin up Spark environments quickly, and Unravel helps them automate pipeline health management. 

Data comes from a range of sources – mostly as files, with streaming expected soon. Cox finds Unravel particularly useful for optimization: choosing virtual machine types in Databricks, watching memory usage by jobs, making jobs run quicker, optimizing cost. These are all things that the team has trouble finding through other tools, and can’t readily build by themselves. They have an ongoing need for the visibility and observability that Unravel gives them. Unravel helps with optimization and is “really strong for observability in our platform.” 

Great Expectations for Data Reliability

Great Expectations shared best practices on data reliability. Great Expectations is the leading open-source package for data reliability, which is crucial to DataOps success. An expectation is an assertion about data; when data falls outside these expectations, an exception is raised, making it easier to manage outliers and errors. Great Expectations integrates with a range of DataOps tools, making its developers DataOps insiders.SuperConductive provides support for Great Expectations and is a leading contributor to the open source project. 

The stages where Great Expectations works map directly to the DataOps infinity loop. Static data must be prepared, cleaned, stored, and used for model development and testing. Then, when the model is deployed, live data must be cleansed, in real time, and run into and through the model. Results go out to users and apps, and quality concerns feed back to operations and development. 

Airflow Enhances Observability

Astronomer conducted a master class on observability. Astronomer was founded to help organizations adopt Apache Airflow. Airflow is open source software for programmatically creating, scheduling, and monitoring complex workflows, including core DataOps tasks such as data pipelines used to feed machine learning models. To construct workflows, users create task flows called directed acyclic graphs (DAGs) in Python. 

The figure shows a neat integration between Airflow and Unravel. Specifically, how Unravel can provide end-to-end observability and automatic optimization for Airflow pipelines. In this example it’s a simple DAG containing a few Hive and Spark stages. Data is passed from Airflow into Unravel via REST APIs, which helps create an easy to understand interface and then allows Unravel to generate automated insights for these pipelines. 

DataOps Powers the New Data Stack

Unravel Data co-founder Shivnath Babu described and demystified the new data stack that is the focus of today’s DataOps practice. This stack easily supports new technologies such as advanced analytics and machine learning. However, this stack, while powerful, is complex, and operational challenges can derail its success. 

Shivnath showed an example of the new data stack, with Databricks providing Spark support, Azure Data Lake for storage, Airflow for orchestration, dbt for data transformation, and Great Expectations for data quality and validation. Slack provides communications and displays alerts, and Unravel Data provides end-to-end observability and automated recommendations for configuration management, troubleshooting, and optimization. 

In Shivnath’s demo, he showed pipelines in danger of missing performance SLAs, overrunning on costs, or hanging due to data quality problems. Unravel’s Pipeline Observer allows close monitoring, and alerts feed into Slack. 

The goal, in Shivnath’s talk and across all of DataOps, is for companies to move up the data pipeline maturity scale – from problems detected after the fact, and fixed weeks later, to problems detected proactively, RCA’d (the root cause analyzed and found) automatically, and healing themselves. 

OneWeb Takes the Infinity Loop to the Stars

To finish the first half of the day, OneWeb showed how to take Snowflake beyond the clouds – the ones that you can see in the sky over your head. OneWeb is a global satellite network provider that takes Internet connectivity to space, reaching anywhere on the globe. They are going to near-Earth orbit with Snowflake, using a boost from DataOps.live. 

OneWeb connects to their satellites with antenna arrays that require lots of room, isolation – and near-perfect connectivity. Since customer connectivity can’t drop, reliability is crucial across their estate, and a DataOps-powered approach is a necessity for keeping OneWeb online. 

One More Thing…

All of this is just part of what happened – in the first half of the day! We’ll provide a further update soon, including – wait for it – the state of DataOps, how to create a data-driven culture, customer case studies from 84.51°/Kroger and Mastercard, and much, much more. You can view the videos from DataOps Unleashed here. You can also download The Unravel Guide to DataOps, which was made available for the first time during the conference. 

Finding Out More

Read our blog post Why DataOps Is Critical for Your Business.

The post DataOps Has Unleashed, Part 1 appeared first on Unravel.

Thank you for your interest in the Unravel Data2021 Infographic.

You can download it here.

The post Unravel Data2021 Infographic appeared first on Unravel.

The post How DBS Bank Leverages Unravel Data appeared first on Unravel.

Palo Alto, CA — January 12, 2021 — Credit Suisse today named five winners of its 2021 Disruptive Technology Recognition (DTR) Program, an annual program that highlights some of the best disruptors of traditional enterprise information technology. The program gives a chance for participants to collaborate to promote innovation at the bank and partner firms.

This is the third year of the program, which allows Credit Suisse to exchange ideas and philosophies with companies leading technology changes that will disrupt the existing framework and shape the future for businesses across the spectrum.

“We saw last year’s winners have a significant impact across industry verticals, and we expect this year’s nominees to follow in these big footsteps. We have high expectations for the new award nominees as we take the DTR program into 2021 and are excited to see how they will raise the bar for the future of enterprise technology,” said Laura Barrowman, Group Information Officer for Credit Suisse.

Credit Suisse partners with technology companies in all stages of business. The bank has found the DTR Program helps foster a closer partnership with companies to actively drive change through advances in enterprise technology.

This year’s DTR Program partners are:

AttackIQ equips cybersecurity teams with a Security Optimization Platform for automating security control validation, improving security program effectiveness, and using insights to make better decisions.

Immuta provides an Automated Data Governance platform that powers compliant BI, analytics and data science for data-driven organizations by automating data access control and privacy protections.

MURAL is a digital workspace for visual collaboration, helping enterprise teams imagine together from anywhere to unlock new ideas, solve hard problems, and innovate faster using its inclusive, simple-to-use platform.

Unravel Data radically simplifies DataOps by providing unified observability and AI-enabled operations across the modern data stack, allowing data teams to run their data pipelines reliably and cost-effectively in all cloud environments.

WalkMe is a code-free software that enables organizations to measure, drive, and act to ultimately maximize the impact of their digital transformation and accelerate the return on their software investment.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Credit Suisse AG Names Unravel Data A Disruptive Tech Winner appeared first on Unravel.

Also see our blog post with stats from our Untold conference polls, “More than 60% of our Pipelines have SLAs…”

Unravel Data recently held our first-ever customer conference, Untold. Untold was a four-hour virtual event exclusively for and by Unravel customers, with five talks and an audience of 200 enthusiastic customers. The event featured powerful presentations, lively Q&A sessions, and interactive discussions on a special Slack channel. 

Unravel customers can view the recorded presentations and communicate with the speakers, as well as their peers who attended the event. If you’re among the hundreds of existing Unravel customers, including three of the top five North American banks, reach out to your customer success representative for access. If you are not yet a customer, you can create a free account or contact us. 

The talks included: 

• Using Unravel for DataOps and SDLC enhancement
• Keeping complex data pipelines healthy
• Solving the small files problem (with great decreases in resource usage and large increases in throughput)
• Managing thousands of nodes in scores of ever-busier modern data  clusters

One speaker even described a successful cloud migration and the key pain points that they had to overcome.

In a “triple threat” presentation, with three speakers addressing different use cases for Unravel in the same organization, the middle speaker captured much of the flavor of the day in a single comment: “Especially with cloud costs, I think Unravel could be a game changer for every one of us.” 

Another speaker summed up “the Unravel effect” in their organization: “We can invest those dollars that we’re saving elsewhere, which makes everybody in our organization super happy.”

Brief descriptions of each talk, with highlights from each of our speakers, follow. Look for case studies on these uses of Unravel in the weeks ahead. 

Using Unravel to Improve the Software Development Life Cycle and DataOps

A major bank uses DataOps and Unravel to support the software development life cycle (SDLC) – to create, validate, and promote apps to production. And Unravel helps operators to orchestrate the apps in production. Unravel significantly shortens development time and improves software quality and performance. 

Highlights from the presentation include:

• • “We have Unravel integrated with our entire software development life cycle. Developers get all the insights into how well their code will perform before they go to production.”
  • “The Unravel toolset is something like a starting point to log into our system, and do all the work, from starting building software to deployment level to production.”
  • “When less experienced users look at Spark code, they don’t understand what they’re seeing. When they look at Unravel recommendations, they understand quickly, and then they go fix it.”
  • “As we move to the cloud, Unravel’s APIs will be useful. Today you run your code and spend $10. Tomorrow, you’ll check your code in Unravel, you’ll implement all these recommendations, you’ll spend $5 instead of $10.”

Managing Mission-Critical Pipelines at Scale

A leading software publisher manages thousands of complex, mission-critical pipelines, with tight SLAs – and costs that can range up into the millions of dollars associated with any failures. Unravel gives this organization fine-grained visibility into their pipelines in production, allowing issues to be detected and fixed before they become problems. 

Key points from the talk:

• “At my company, this is a simple pipeline. (Speaker shows slide with Rube Goldberg pipeline image – Ed.) One of our complicated pipelines has a 50-page user manual. We partnered with the Unravel team to stitch these pipelines together.” 
• “Unravel is helping us to actually predict the problems in advance and early in the process, ensuring we can bring these pipelines’ health back to normal, and making sure our SLAs are met.”
• “Last year, to be frank, we were around 80% SLA accomplishments. But, through the two quarters, we are 98% and above.”

Easing Operations and Cutting Costs 

A fast-growing marketing services company uses Unravel to manage operations, reduce costs, and support leadership use cases, all while orchestrating and managing cloud migration – and saves hugely on resources by solving the small files problem. 

This talk actually involved three separate perspectives. A few highlights follow:

• “On-prem Hadoop, that’s where the Unraveled story started for us.”
• “We had around 1,800 cores allocated. After the config changes recommended by Unravel, the cores allocated went down to 120.”
• “Unravel helped us build an executive dashboard… the response time from Unravel on it was great. The whole entire experience was wonderful for me.”
• “Proactive alerting – AutoActions – catches runaway queries, rogue jobs, and resource contention issues. It makes our Hadoop clusters more efficient, and our users good citizens.” 
• “We were over-allocating for jobs in Spark, and I’m not an expert Spark user. Unravel recommendations and AutoActions helped solve the problem of over-allocation, without my having to learn more about Spark.”
• “Having the data from Unravel makes our conversations with our tenants much more productive, and has helped to build trust among our teams.” 
• “We can invest those dollars that we’re saving elsewhere, which makes everybody in our organization super happy.”

A Successful Move to Cloud 

A leading provider of consumer financial software moved to the cloud successfully –  but they had to use inadequate tools, including tracking the move in Excel spreadsheets and saving data in CSV files. They were able to complete the move in less than a year, “leaving no app behind.” 

As the speaker described, it was a huge job: 

• “Think of our move from on-premises to the cloud as changing the engines of the plane while the plane is flying.”
• “We actually started the exercise with more than 20,000 tables, but careful analysis showed that half of them were unused.”
• “We have many, many critical pipelines, and there were millions of dollars at stake for these things not to be correct, complete, or within SLA.” 
• “The cloud is not forgiving when it comes to cost. Linear increases in usage lead to linear increases in costs.” 
• “The cloud offers pay as you go, which sounds great. But when you go, you pay.” 

The move made clear the need for Unravel Data as a supportive platform for assessing the on-premises data estate, matching on-premises software tools to their closest equivalents on each of the major public cloud platforms, and tracking the success of the move. 

Unravel Right-Sizes Resource Consumption – as the Pandemic Sends Traffic Soaring

One of the world’s largest financial services companies sees usage passing critical thresholds – and then the pandemic sends their data processing needs through the roof. They used Unravel to steadily reduce memory and CPU usage, even as data processing volumes grew rapidly. 

• “Before Unravel, our memory and CPU allocations were way greater than our actual usage. And we spent many hours troubleshooting problems because we couldn’t see what was going on inside our apps.” 
• “With Unravel, we saw that a lot of vCores were unused. And we were able to drop almost 40,000 tables… that helped us a lot.”
• “Before Unravel, we were uncomfortably past the 80% line in capacity, and memory was always pegged. With Unravel, we were able to cut usage roughly in half for the same throughput.”
• “Before Unravel, we couldn’t give users – including the CEO – a real good reason on why they weren’t getting what they wanted.”
• “Comprehensive job visibility, such as the configuration page in Unravel, has improved resolution times.”
• “(Unravel) provides us a reasonable rate of growth in our use of resources compared to workloads processed – a rate which I can sell to my management team.”
• “I’m sleeping a lot better than I was a year ago. My Fitbit sleep score is now 88. It’s been a good journey so far.” (A Fitbit sleep score of 88 is well above most Fitbit users – good, bordering on excellent – Ed.)

Finding Out More

Unravel customers can view the talks, communicate with industry peers who gave and attended the talks, and more. (There may still be some swag available!) If you’re interested, you can create a free account or contact us. 

The post “My Fitbit Score is Now 88” – Sharing Success Stories at the First Unravel Customer Conference appeared first on Unravel.

Unravel provides full-stack coverage and a unified, end-to-end view of everything going on in your environment, plus recommendations from our rules-based model and our AI engine. Unravel works on-premises, in the cloud, and for cloud migration. 

Unravel provides direct support for platforms such as Cloudera Hadoop (CDH), HortonWorks Data Platform (HDP), Cloudera Data Platform (CDP), and a wide range of cloud solutions, including AWS infrastructure as a service (IaaS), Amazon EMR, Microsoft Azure IaaS, Azure HDInsight, and DataBricks on both cloud platforms, as well as GCP IaaS, Dataproc, and BigQuery. We have grown to supporting scores of well-known customers and engaging in productive partnerships with both AWS and Microsoft Azure. 

We have an ambitious engineering agenda and a relatively large team, with more than half the company in the engineering org. We want our engineering process to be as forward-looking as the product we deliver. 

We constantly strive to develop adaptive and end-to-end testing strategies. For testing, Unravel had started with a modest customer deployment. We now support scores of large customer deployments with 2000 nodes and 18 clusters. We had to conquer the giant challenges posed by this massive increase in scale. 

Since testing is an integral part of every release cycle, we give top priority to developing a systematic, automated, scalable, and yet customizable approach for driving the entire release cycle. As a new startup comes up, the obvious and quickest approach one is tempted to follow is the traditional testing model, and to manually test and certify a module/product. 

Well, this structure sometimes works satisfactorily when the features in the product are few. However,  a growing customer base, increasing features, and the need for supporting multiple platforms give rise to proportionally more and more testing. At this stage, testing becomes a time-consuming and cumbersome process. So if you and your organization are struggling with the traditional, manual testing approach for modern data stack pipelines, and looking for a better solution, then read on. 

In this blog, we will walk you through our journey about:

• How we evolved our robust testing strategies and methodologies.
• The measures we took and the best practices that we applied to make our test infrastructure the best fit for our increasing scale and growing customer base.

Evolution of Unravel’s Test Model

Like any other startup, Unravel had a test infrastructure that followed the traditional testing approach of manual testing, as depicted in the following image:           

Initially, with just a few customers, Unravel mainly focused on release certification through manual testing. Different platforms and configurations were manually tested, which took roughly ~4-6 weeks of release cycle. With increasing scale, this cycle became endless, which made the release train longer and unpredictable. 

This type of testing model has quite a few stumbling blocks and does not work well with scaling data sizes and product features. Common problems with the traditional approach include:

•  Late discovery of defects, leading to:         
  • • Last-minute code changes and bug fixes    
    • Frantic communication and hurried testing  
    • Paving the way for newer regressions
•  Deteriorating testing quality due to:
  • • Manual end-to-end testing of the modern data stack pipeline, which becomes error-prone and tends to miss out on corner cases, concurrency issues, etc.
    • Difficulty in capturing the lag issues in modern data stack pipelines
• Longer and unpredictable release trains that leads to:
  • • Stretched deadlines, since testing time increases proportionally with the number of builds across multiple platforms.
    • Increased cost due to high resource requirements such as more man-hours, heavily equipped test environments, etc.

Spotting the defects at a later stage becomes a risky affair, since the cost of fixing defects increases exponentially across the software development life cycle (SDLC). 

While the traditional testing model has its cons, it also has some pros. A couple of key advantages are that:

• Manual testing can reproduce customer-specific scenarios 
• It can also catch some good bugs where you least expect them to be

So we resisted the temptation to move fully to what most organizations now implement, a completely mechanized approach. To cope with the challenges faced in the traditional model, we introduced a new test model, a hybrid approach that has, for our purposes, the best of both worlds. 

This model is inspired by the following strategy which is adaptive, to scale with a robust testing framework.

Our Strategy

Unravel’s hybrid test strategy is the foundation for our new test model.

New Testing Model 

Our current test model is depicted in the following image:

This approach mainly focuses on end-to-end automation testing, which provides the following benefits:

• Runs automated daily regression suite on every new release build, with end-to-end tests for all the components in the Unravel stack
• Provides a holistic view of the regression results using a rich reporting dashboard 
• The Automation framework works for all kinds of releases (point release, GA release), making it flexible, robust, and scalable. 

A reporting dashboard and an automated regression summary email are key differentiators of the new test model. 

The new test model provides a lot of key advantages. However, there are some disadvantages too.

KPI Comparison – Traditional Vs New Model

The following bar chart is derived on the KPI values for deployment and execution time, which is captured for both the traditional as well as the new model.

The following graph showcases the comparison of deployment, execution, and resource time savings:

Release Certification Layout

The new testing model comes with a new Release Certification layout, as shown in the image below. The process involved in the certification of a release cycle is summarized in the Release Cycle Summary table. 

Release Cycle Summary

Conclusion

Today, Unravel has a rich set of unit tests; more than 1000 tests are run for every commit, along with the CI/CD pipeline in place. This includes functional sanity test cases (1500+) and can cover our end-to-end data pipelines as well as the integration test cases. Such a testing strategy can significantly reduce the impact on integrated functionality by proactively highlighting issues in pre-checks. 

Cutting a long story short, It is indeed a difficult and tricky task to build a flexible, robust, and scalable test infrastructure that caters to varying scales, especially for a cutting-edge product like Unravel, and with a team that strives for the highest quality in every build. 

In this post, we have highlighted commonly faced hurdles in testing modern data stack pipelines. We have also showcased the reliable testing strategies we have developed to efficiently test and certify modern data stack ecosystems. Armed with these test approaches, just like us, you can also effectively tame the scaling giant!

Reference Links (clip art images)

Unravel’s Hybrid Test Strategy:

• End to End testing: https://www.lambdatest.com/blog/all-you-need-to-know-about-end-to-end-testing/
• Integrated testing:  https://professionalqa.com/types-of-integration-testing
• Load testing:  https://www.kiwiqa.com/getting-to-know-the-fundamentals-of-performance-testing-a-guide-for-amateurs-and-professionals/ 
• Weekly E2E pipeline: https://www.pngkit.com/view/u2r5a9y3q8y3r5y3_smart-end-to-end-process-icon/ 
• Parallelism:  https://pages.tacc.utexas.edu/~eijkhout/istc/html/parallel.html    (2.6.1.1 The Fork-Join Mechanism)

Exponential Cost of Fixing Defects:

• Developer (Male computer user vector): image:https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS1E42d7aLu5TRDcxzmpFD2N5b3YGo4E3C2dt1fcGONICav8jTteF-mpOM&s
• Developer (Female computer user vector image): https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTX-Q6L_n9Bd5zdQ4J3haaO9hkMJKX3Cpg7RY9Ml8bq56YOmXqo2i5rrA&s
• Build: https://icon-library.com/icon/automation-icon-2.html

Unravel’s Test Model:

• QA Testing – Manual (Computer Users): https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSqI4svILCrc77R2-K0q5jys4PCXDiJFnEVASBiF5A_FNdnSBbdA_kD9Q&s

The post Unravel’s Hybrid Test Strategy to Conquer the Scaling Data Giant appeared first on Unravel.

THE NEW UNRAVEL 4.6.2 RELEASE INCREASES THE POWER OF UNRAVEL WHILE MAKING THAT POWER SIGNIFICANTLY EASIER TO ACCESS

The Unravel 4.6.2.0 release, now generally available, builds on our previous 4.6 release with a new UI/UX, multi-cluster support, monitoring for ELK (Elasticsearch, Logstash, and Kibana), and a new installer that makes Unravel available in minutes.

Note: The new UI/UX for Unravel and multi-cluster support have been among the most requested features from Unravel’s delighted customers.

NEW UI/UX

Unravel 4.6.2 adds a new graphical user interface as the visible front end of a new user experience. The previous interface was respected for its ability to bring information from disparate systems to a single pane of glass, for its ability to deliver root cause analysis data crisply, and for the way in which it delivered monitoring information directly from sensors alongside AI-driven insights.

The new UI is faster, cleaner, and combines information that was formerly on several pages into a unified, single-page view. Now, you can also find the appropriate, persona-based experience inside Unravel for data operations people, data analysts, data scientists, pipeline owners, and chief data officers (CDOs). In addition to the UI, Unravel continues to support the Unravel RESTful API for programmatic access to Unravel functionality.

The new UI, showing the new main dashboard and multi-cluster support (arrow).

MULTI-CLUSTER SUPPORT

With Unravel 4.6.2, you can now use a single Unravel instance to monitor multiple independent on-premise clusters for Cloudera Distributed Hadoop (CDH) and Hortonworks Data Platform (HDP). Multi-cluster support helps to create a “single source of truth” for all connected CDH and HDP clusters.

ELK SUPPORT

Unravel 4.6.2 supports metrics and graphical presentation of KPIs for the Elasticsearch/Logstash/Kibana (ELK) stack. This extends Unravel’s platform support, which continues to include Kafka, Spark, Pig, Cascading, Hadoop, Impala, HBASE, and relational databases supporting SQL.

Cluster details for Elasticsearch (also see the Unravel documentation).

MUCH MORE

I’ve highlighted just some of the key features in Unravel 4.5. Please see the release documentation for 4.6.2.0 for further information. Create a free account to experience the Unravel product.

The post Unravel Data Release 4.6.2 Features New UI and Multi-Cluster Support appeared first on Unravel.

We’ve been busy speaking to our customers and thought leaders in the industry and have rounded up the key takeaways from our latest CDO sessions. Here are some of the top takeaways and advice gained from these sessions with big data leaders, Kumar Menon from Equifax, Anheuser-Busch’s Harinder Singh, Sandeep Uttamchandani from Unravel, and DBS Bank’s Matteo Pelati:

1. DataOps is the end to end management and support of a data platform, data pipelines, data science models, and essentially the full lifecycle of consumption and production of information.

2.  You need to incorporate a multitude of different factors, such as compliance, cost, root cause analysis, tuning, and so on, early on so that DataOps is seamless and you can avoid surprises.

3. It is important to strike the right balance between governance and time to market. When you have to move fast, governance always slows down. And governance doesn’t just refer to the required regulation or compliance. It’s also just good data hygiene, maintaining the catalog, and maintaining the glossaries.

4. Building your company’s platform and services as a product can be extremely beneficial and pay you back after some time. You need time for the investment to return, but once you get to that stage, you’ll get your ROI.

Since for many companies, the pandemic has accelerated the movement to the Cloud, these big data leaders gave plenty of insights on Cloud migration:

1. Moving to the Cloud is a double edged sword. While it’s convenient when it’s time to market fast, you also have to be very careful about security in terms of how you manage, configure, and enforce it.

2. When you think about moving to the Cloud, it’s a highly non trivial process. On one side, you have your data and thousands of data sets, then on the other side, you have these active pipelines that run daily, dashboards, and ML models feeding into the product. You have to figure out the best sequence to move these.

3. When moving to the Cloud, you have to have a different philosophy when you’re building Cloud native applications versus when you’re building on-prem. You must improve the skill sets of your people to think more broadly.

4. A big challenge when moving to the cloud is accessing data. However, you can use encryption and tokenization of data at a large scale and expand the use throughout the entire data platform.

They also provided businesses with thoughtful, yet practical, advice on what they should be doing in order to not only stay afloat, but grow, during this COVID-19 pandemic:

1. Try to understand your internal business partners and customers’ needs. Everybody is in a unique situation right now, not just your company, so focus on your internal customers and what they need from you in terms of data analytics.

2. Consider changing the delivery model of your product or service and meet the customer where they are instead of expecting customers to come to you.

3. Make sure that you connect a lot more with your customers and your coworkers to keep the momentum going. This ecosystem, however, is not just your customers, but potentially your customers’ customers as well.

4. Focus on data literacy and explainable insights within your organization. Not everyone understands data the way you do, but data professionals have a unique opportunity here to really educate and build that literacy within their enterprise for better decision making.

5. Keep an eye out for how fast regulations are changing. It’s very likely that new data residency requirements, regulations, and privacy laws will emerge as a result of the pandemic, so make sure that the architecture you build today is adaptable and flexible in order to withstand the challenge of the time.

Matteo Pelati spoke on how DBS Bank has leveraged Unravel:

1. DBS has leveraged Unravel to analyze jobs, analyze previous runs of a job and block the promotion of a job if it doesn’t satisfy certain criteria.

2. Unravel has become really useful to understand the impact of users’ queries on the system and to let users understand the impacts of the operation that they’re orchestrating.

The above takeaways just scratch the surface of the insights that these CDO’s have to offer. As skilled and experienced big data leaders, they contribute valuable knowledge to the big data community. To hear more from them, you can watch the webinars or read the transcripts from our Getting Real with Data Analytics and Transforming DataOps in Banking sessions.

The post Top Takeaways From CDO Sessions: Customers and Thought Leaders appeared first on Unravel.

This is my first post since landing at Unravel and I couldn’t be more energized about what’s to come. Our industry is being re-architected in new and exciting ways. Many industry observers would say it’s about time! The landscape is littered with unfulfilled promises and unsolved complexity. However, one fact remains, without data and more critically, what that data can tell us, we are flying blind amidst a business climate that has never moved faster and has never faced more uncertainty.

Many data ‘swamps’ have frankly failed to deliver, not because the data was not of value, but rather it has been too complex to wrangle and actionable insights have been too hard to extract. This has now changed. Our collective industry ability to apply more discipline and rigor to Data Operations and the maturity of Machine Learning and AI developments fuel our unwavering belief in the promise of Data.

To get the full benefits of modern data applications, you need reliable, optimal performance from those applications. And that’s no easy task when everything is running on what has been a massive, ungainly stack – different technologies, stretching across on-premises and cloud environments. You have not had the visibility or the resources to monitor each element, understand how they’re all working together, find and resolve issues, and optimize for the greatest efficiency and effectiveness.

It’s time to stop being reactive and start getting proactive. Data is only useful if you can depend on it. I joined Unravel not only because of their strong belief system around the promise of data but also in their ability to help you build understanding, remediate issues, uncover opportunities, and ensure your applications remain effective and reliable.

Ultimately, Unravel is here to help you get ahead of the game, avoid nasty surprises, and be ready for whatever new needs and technologies you’ll have in the future. In addition to providing recommendations for today’s needs, we also help you plan for what’s next – modeling how you’re going to grow, for instance, and what you need to do to keep up with that growth. I couldn’t be more excited about our industry at this time and of course the phenomenal team I have the privilege to be joining.

The full press release can be viewed below.
————————————————————————————————————————————–

Unravel Hires Enterprise Sales Leader as New Vice President of Worldwide Sales.

The new VP Worldwide Sales will draw on experience working with global enterprises to help Unravel customers accelerate time to value for their business critical modern data cloud workloads

PALO ALTO, CALIFORNIA – June 23, 2020 – Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced that it has hired Roman Orosco as its new Vice President of Worldwide Sales. Orosco will scale out and lead the company’s sales and revenue operations teams as Enterprises across all industries leverage and increasingly depend on data to provide competitive advantage, improve operating efficiencies and re-architect their business to deal with macro-environment changes such as Covid-19.

Orosco brings over 20 years of demand generation, channel, sales and operational leadership to his new role at Unravel. Orosco most recently served as the Vice President of Americas at BlueCat Networks, where he built a high-performing sales organization that fueled the company’s growth from $20M to $100M. Prior to BlueCat, he led revenue teams and delivered company changing revenue for over 15 years at SAP, NICE, Documentum, i2 Technologies, and other software companies.

“We’re excited to have an accomplished sales leader like Roman join the Unravel team. Roman has a great combination of domain expertise and functional experience but most importantly he always puts the customer at the forefront of his decision making. At a time like we are in right now, it has never been more important for us to have high empathy with our customer’s business needs and focus on driving near term impact,” said Kunal Agarwal, CEO, Unravel Data. “Roman is the right person, at the right time to lead our sales organization through a period of growth and customer transition to modern data clouds.”

“My North star has always been for my teams and I to deliver tangible business value for our clients. Enterprises continue to scale their Investments in data and adopting best practice disciplines in Data Operations to ensure their data driven applications are high performing, reliable and operating cost effectively,” said Roman Orosco, VP Worldwide Sales, Unravel Data. “I look forward to working closely with our internal teams and partners to remove operational friction and where possible, automate key Data Operations workflows such as Performance tuning, Troubleshooting and Cost optimization for cloud and on-premise workloads.”

About Unravel Data
Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement
The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

PR Contact
Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post The Promise of Data and Why I Joined Unravel appeared first on Unravel.

New Offering Accelerates Unravel’s Mission to Support Big Data Workloads Wherever they Reside, Including on-Premises, Public Cloud, Multiple Cloud and Hybrid Environments

PALO ALTO, CALIFORNIA – May 27, 2020 – Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced Unravel for AWS Databricks, a solution to deliver comprehensive monitoring, troubleshooting, and application performance management for AWS Databricks environments. Unravel for AWS Databricks leverages Unravel’s AI-powered data operations platform to accelerate performance of Spark on AWS while providing unprecedented visibility into runtime behavior, resource usage, and cloud costs.

“As business needs evolve, data workloads are moving to a growing variety of settings, stretching across on-prem environments, public clouds, multiple clouds and a hybrid mix of all of these. It’s important that organizations can get the same performance, reliability and value out of their data applications no matter where they are,” said Kunal Agarwal, CEO, Unravel Data. “Unravel for AWS is our latest effort to expand the platform to accommodate Big Data wherever it exists. With this addition, Unravel now supports Databricks in both AWS and Azure, and the Unravel platform is broadly available in every major public cloud as well as on-premises and in hybrid settings. We were always committed to being infrastructure-agnostic and this is another milestone in that mission.”

The announcement is the latest development in a long relationship between Unravel and AWS. Unravel already supports Amazon EMR, as well as Cloudera/Hortonworks on IaaS for AWS. This release provides further support for customers deploying modern data apps on AWS. In addition, Unravel is an existing member of the AWS Partner Network and member of AWS global startup program.

AWS Databricks is a unified data analytics platform for accelerating innovation across data science, data engineering, and business analytics, integrated with AWS infrastructure. Unravel for AWS Databricks helps operationalize Spark apps on the platform: AWS Databricks customers will shorten the cycle of getting Spark applications into production by relying on the visibility, operational intelligence, and data driven insights and recommendations that only Unravel can provide. Users will enjoy greater productivity by eliminating the time spent on tedious, low value tasks such as log data collection, root cause analysis and application tuning.

Key features of Unravel for AWS Databricks include:

• Application Performance Management for AWS Databricks – Unravel delivers visibility and understanding of Spark applications, clusters, workflows, and the underlying software stack
• Automated root cause analysis of Spark apps – Unravel can automatically identify, diagnose, and remediate Spark jobs and the full Spark stack, achieving simpler and faster resolution of issues for Spark applications on AWS Databricks clusters
• Comprehensive reporting, alerting, and dashboards – AWS Databricks users can now enjoy detailed insights, plain-language recommendations, and a host of new dashboards, alerts, and reporting on chargeback accounting, cluster resource usage, Spark runtime behavior and much more

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data Expands Support for Modern Data Workloads in the Cloud with Introduction of Unravel for AWS Databricks appeared first on Unravel.

Retailers are using big data to report on consumer demand, inventory availability, and supply chain performance in real time. Big data provides a convenient, easy way for retail organizations to quickly ingest petabytes of data and apply machine learning techniques for efficiently moving consumer goods. A top supermarket retailer has recently used Unravel to monitor its vast trove of customer data to stock the right product for the right customer, at the right time.

The supermarket retailer needed to bring point-of-sale, online sales, demographic and global economic data together in real-time and give the data team a single tool to analyze and take action on the data. The organization needed all the systems in their data pipelines to be monitored and managed end-to-end to ensure proper system and application performance and reliability. Existing methods were largely manual, error prone and lacked actionable insights.

Unravel Extensible Data Operations Platform

After failing to find alternative solutions to cluster performance management, the customer chose Unravel to help remove risks in their cloud journey. During implementation, Unravel worked closely with the ITOps team to support the customer, providing support and iterating in collaboration based on the insights and recommendations provided by Unravel. This enabled both companies to triage issues, and troubleshoot issues faster.

Bringing All The Data Into A Single Interface

The customer utilized a number of modern open source data projects in its data engineering workflow – Spark, MapReduce, HBase, YARN and Kafka. These components were needed to ingest and properly process millions of transactions a day. Hive query performance was a particular concern, as numerous downstream business intelligence reports depended on timely completion of these queries. Previously, the devops team spent several days to a week troubleshooting job failure issues, often blaming the operations team of improperly cluster configuration settings. The operations teams would in turn ask the devops team to re-check SQL query syntax for cartesian joins and other inefficient code. Unravel was able to shed light to these types of issues, providing usage based reporting which helped both teams pinpoint inefficiencies quickly.

Unravel was able to leverage its AI and automated recommendations engine to clean up hundreds of Hive tables, greatly enhancing performance. A feature that the company found particularly useful is the ability to generate custom failure reports using Unravel’s flexible API. In addition to custom reports, Unravel is able to deliver timely notifications through e-mail, serviceNOW, and PagerDuty.

Happy with the level of control Unravel was able to provide for Hive, the customer deployed Unravel for all other components – Spark, MapReduce, HBase, YARN and Kafka and made it a standard tool for DataOps management across the organization. Upon deploying Unravel, the team was presented with an end-to-end dashboard of insights and recommendations across the entire stack (Kafka, Spark, Hive/Tez, HBase) from a single interface, which allowed them to correlate thousands of pages of logs automatically. Previously, the team performed this analysis manually, with unmanaged spreadsheet tracking tools.

In addition to performance management, the organization was looking for an elegant means to isolate users who were consistently wasteful with the compute resources on the Hadoop clusters. Such reporting is difficult to put together, and requires cluster telemetry to not only be collected across multiple components, but also correlated to a specific job and user. Using Unravel’s chargeback feature, the customer was able to report not only the worst offenders who were over-utilizing resources, but the specific cost ramifications of inefficient Hive and Spark jobs. It’s a feature that enabled the company to recoup any procurement costs in a matter of months.

Examples of cluster utilization showing in the Unravel UI

Scalable modern data applications on the cloud are critical to the success of retail organizations. Using Unravel’s AI-driven DataOps platform, a top retail organization was able to confidently optimize its supply chain. By providing full visibility of their applications and operations, Unravel helped the retail organization to ensure their modern data apps are effectively architected and operational. This enabled the customer to minimize excess inventory and deliver high demand goods on time (such as water, bread, milk, eggs) and maintain long term growth.

FINDING OUT MORE

Download the Unravel Guide to DataOps. Contact us or create a free account.

The post Supermarkets Optimizing Supply Chains with Unravel DataOps appeared first on Unravel.

Data is the new oil and a critical differentiator in generating retrospective, interactive, and predictive ML insights. There has been an exponential growth in the amount of data in the form of structured, semi-structured, and unstructured data collected within the enterprise. Harnessing this data today is difficult — typically data in the lakes is not consistent, interpretable, accurate, timely, standardized, or sufficient. Scully et. al. from Google highlight that for implementing ML in production, less than 5% of the effort is spent on the actual ML algorithms. The remaining 95% of the effort is spent on data engineering related to discovering, collecting, preparing data, as well as building and deploying the models in production.

As a result of the complexity, enterprises today are data rich, but insights poor. Gartner predicts that 80% of analytics insights will not deliver business outcomes through 2022. Another study, highlights that 87% of data projects never make it production deployment.

Over the last two years, I have been leading an awesome team in the journey to democratize data at Intuit Quickbooks. The focus has been to radically improve the time it takes to complete the journey map from raw data into insights (defined as time to insight). Our approach has been to systematically break the jou and automate corresponding data engineering patterns, making it self-service for citizen data users. We modernized the data fabric to leverage the cloud, and developed several tools and frameworks as a part of the overall self-serve data platform.

The team has been sharing the automation frameworks both as talks in key conferences and 3 open-source projects. Checkout the list of talks and open-source projects at the end of the blog. It makes me really product on how the team has truly changed the trajectory of the data platform. A huge shoutout and thank you to the team — all of you rock!

In the journey to democratize data platforms, I recently moved to Unravel Data. Today, there is no “one-size-fits-all” requiring enterprises to adopt polyglot datastores and query engines both on-premise as well as the cloud. Configuring and optimizing queries to run seamlessly for performance, SLAs, cost, and root-cause diagnosis is highly non-trivial requiring deep understanding. Data users such as data analysts, scientists and data citizens essentially need a turn-key solution to analyze and automatically configure their jobs and applications.

I am very excited to be joining the Unravel Data driving the technology of AI-powered data operations platform for performance management, resource and cost optimization, and cloud operations and migration. The mission to democratize data platforms continues …

The full press release can be viewed below.

————————————————————————————————————————————–

Unravel Hires Data Industry Leader with over 40 Patents as New Chief Data Officer and VP of Engineering

The new CDO will draw on experience from IBM, VMware and Intuit QuickBooks to help Unravel customers accelerate their modern data workloads

PALO ALTO, CALIFORNIA – April 1, 2020 – Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced that it has hired Sandeep Uttamchandani as its new Chief Data Officer and VP of Engineering. Uttamchandani will help boost Unravel’s capabilities for optimizing data apps and end-to-end data pipelines, with special focus on driving innovations for cloud and machine learning workloads. He will also lead and expand the company’s world-class data and engineering team.

Uttamchandani brings over 20 years of critical industry experience in building enterprise software and running petabyte-scale data platforms for analytics and artificial intelligence. He most recently served as Chief Data Architect and Global Head of Data and AI at Intuit QuickBooks, where he led the transformation of the data platform used to power transactional databases, data analytics and ML products. Before that, he held engineering leadership roles for over 16 years at IBM and VMWare. Uttamchandani has spent his career delivering innovations that provide tangible business value for customers.

“We’re thrilled to have someone with Sandeep’s track record on board the Unravel team. Sandeep has led critical big data, AI and ML efforts at some of the world’s biggest and most successful tech companies. He’s thrived everywhere he’s gone,” said Kunal Agarwal, CEO, Unravel Data. “Sandeep will make an immediate impact and help advance Unravel’s mission to radically simplify the way businesses understand and optimize the performance of their modern data applications and data pipelines, whether they’re on-premises, in the cloud or in a hybrid setting. He’s the perfect fit to lead Unravel’s data and engineering team in 2020 and beyond.”

In addition to his achievements at Intuit QuickBooks, IBM and VMWare, Uttamchandani has also led outside the office. He has received 42 total patents involving systems management, virtualization platforms, and data and storage systems, and has written 25 conference publications, including an upcoming O’Reilly Media book on self-service data strategies. Uttamchandani earned a Ph.D. in computer science from the University of Illinois at Urbana-Champaign, one of the top computer science programs in the world. He currently serves as co-Chair of Gartner’s CDO Executive Summit.

“My career has always been focused on developing customer-centric solutions that foster a data-driven culture, and this experience has made me uniquely prepared for this new role at Unravel. I’m excited to help organizations boost their businesses by getting the most out of their modern data workloads,” said Sandeep Uttamchandani, CDO and VP of Engineering, Unravel Data. “In addition to driving product innovations and leading the data and engineering team, I look forward to collaborating directly with customer CDOs to assist them in bypassing any roadblocks they face in democratizing data platforms within the enterprise.”

About Unravel Data
Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement
The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

PR Contact
Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post The journey to democratize data continues appeared first on Unravel.

Last year, Cloudera released the Cloudera Data Platform, an integrated data platform that can be deployed in any environment, including multiple public clouds, bare metal, private cloud, and hybrid cloud. Customers are increasingly demanding maximum flexibility to adhere to multi-cloud, hybrid data management demands. Unravel has from the beginning has made it a core strategy to support the full modern data stack, on any cloud, hybrid as well as on-premises.

Today we are pleased to announce that Unravel is now certified on Cloudera Data Platform (both CDP public cloud as well as CDP Data Center). This marks an important milestone in our continued partnership with Cloudera, bolstered by a growing demand among Cloudera users for our AI-driven performance optimization solution for modern data clouds.

The certification ensures that Unravel is integrated seamlessly with Cloudera Data Platform, providing customers with an intelligent solution to improve the reliability and performance of their modern data stack applications and operations, and optimize costs through data driven insights. We look forward to continually supporting Cloudera customers, on CDP as well as CDH and HDP.

The full press release can be viewed below.

————————————————————————————————————————————————

Unravel Data Earns Certification for Cloudera Data Platform

Unravel Supports Cloudera Data Platform in the public cloud, on-premises and in hybrid environments, continuing Unravel’s mission to simplify and optimize modern data applications wherever they exist

PALO ALTO, CALIFORNIA – March 25, 2020– Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced that it has been certified on the Cloudera Data Platform. Cloudera Data Platform manages data in any environment, including multiple public clouds, bare metal deployments, private clouds and hybrid clouds. The certification further advances Unravel’s mission to simplify and optimize modern data apps wherever they exist, with this move particularly bolstering Unravel’s support for hybrid cloud and multi-cloud environments.

“Data apps, especially AI and ML apps, are increasingly being spread across a mix of on-premise and public cloud environments. These highly distributed hybrid cloud deployments provide unique advantages and greater flexibility compared to all-in approaches that put everything either in the cloud or in a private datacenter,” said Kunal Agarwal, CEO, Unravel Data. “Organizations are also deploying more data apps in multiple clouds, which allows them to leverage the specific strengths of each cloud and provides unique app functionality. However, the growing distribution of data apps in hybrid and multi-cloud settings introduces operational complexity and naturally makes it harder to optimize and monitor these apps. Unravel ensures that enterprises have both a clear line of sight into these apps and automated recommendations to troubleshoot and maximize their performance, no matter where the apps are located.”

In order to earn this certification, Unravel maintained their silver partnership status through the Cloudera Connect partner program, built new integrations for Cloudera Data Platform (for both the public cloud and on-premise version), then documented and tested those integrations. Cloudera worked closely with Unravel during the entire process.

The certification is the latest milestone in a long relationship between Unravel and Cloudera. Unravel was previously certified on Cloudera Enterprise and the two share many joint customers. This integration will ensure legacy CDH and HDP customers who migrate to Cloudera CDP will continue to enjoy Unravel’s solutions to simplify data operations on AWS, Azure and GCP in addition to on-premises.

About Unravel Data
Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe and Deutsche Bank. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement
The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

PR Contact
Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post Unravel Data Now Certified on Cloudera Data Platform appeared first on Unravel.

PALO ALTO, Calif. – February 25, 2020– Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, introduced new migration, cost analytics and architectural mapping capabilities for Unravel for Azure Databricks, which is now generally available from Unravel and in the Azure Marketplace. The move further solidifies Unravel’s mission to support modern data workloads wherever they exist, whether on-premises, in the public cloud or a hybrid setting.

“With more and more big data deployments moving to the public cloud, Unravel has spent the last several years helping to simplify the process of cloud migration as well as improving the management and optimization of modern data workloads once in the cloud. We have recently introduced platforms for all major public cloud platforms,” said Bala Venkatrao, Chief Product Officer, Unravel Data. “This release, highlighted by the industry’s only slice and dice migration capabilities, makes it easier than ever to move data workloads to Azure Databricks, while minimizing costs and increasing performance. The platform also allows enterprises to unify their data pipelines end-to-end, such as Azure Databricks and Azure HDInsight.”

Unravel for Azure Databricks delivers comprehensive monitoring, troubleshooting, and application performance management for Azure Databricks environments. The new additions to the platform include:

• Slice and dice migration support – Unravel now includes robust migration intelligence to help customers assess their migration planning to Azure Databricks in version 4.5.5.0. Slice and dice migration support provides impact analysis by applications and workloads. It also features recommended cloud cluster topology and cost estimates by service-level agreement (SLA), as well as auto-scaling impact trend analysis as a result of cloud migration.
• Cost analytics – Unravel will soon add new cost management capabilities to help optimize Azure Databricks workloads as they scale. These new features include cost assurance, cost planning and cost forecasting tools. Together, these tools provide granular detail of individual jobs in Azure Databricks, providing visibility at the workspace, job, and job-run level to track costs or DBUs over time.
• Detailed architectural recommendations: Unravel for Azure Databricks will soon include right-sizing, a feature that recommends virtual machine or workload types that will achieve the same performance on cheaper clusters.

Unravel for Azure Databricks helps operationalize Spark apps on the platform: Azure Databricks customers can shorten the cycle of getting Spark applications into production by relying on the visibility, operational intelligence, and data driven insights and recommendations that only Unravel can provide. Users enjoy greater productivity by eliminating the time spent on tedious, low value tasks such as log data collection, root cause analysis and application tuning.

In addition to being generally available directly from Unravel, Unravel for Azure Databricks is also available on the Azure Marketplace, where users can try a free trial of the platform and get $2000 in Azure credits. Cloud marketplaces are quickly becoming the preferred way for organizations to procure, deploy and manage enterprise software. Unravel for Azure Databricks on Azure Marketplace offers one-click deployment of Databricks performance monitoring and management in Azure.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Introduces Workload Migration and Cost Analytics Solution for Azure Databricks appeared first on Unravel.

Thank you for your interest in Unravel Understanding Cloud Data Services.

You can download it here.

The post Unravel Understanding Cloud Data Services appeared first on Unravel.

At Unravel, we have been working with AWS for some time to help our joint customers simplify data migrations to the cloud. Announced at AWS re:Invent, we are proud to be partnering with AWS on the newly created AWS Partner Network Global Startup Program. Unravel provides detailed insights and recommendations to help enterprises of all sizes slice and dice application workloads to ensure an effective cloud migration strategy.

Once data workloads are running on AWS, Unravel provides a solution for rapid troubleshooting and performance optimization that enables you to get more cluster for your money and enables ops and apps teams to meet their SLAs targets with confidence.

In addition, Unravel provides full visibility of your Amazon EMR resource consumption and accounting for costs and chargeback reporting for all your application stakeholders.

For a more detailed view on how we do what we do under the covers take a look here.

If you are ready to try out Unravel and see for yourself, we are available on the AWS Marketplace and offer a $2000 incentive to try us out!

The full Press Release can be viewed below:

————————————————————————————————————————————————

Unravel Joins  AWS Partner Network Global Startup Program

PALO ALTO, Calif. – December 18, 2019– Unravel Data, a data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, announced its membership in the Amazon Web Services (AWS) Partner Network (APN) Global Startup Program. The APN Global Startup Program is a unique “white glove” support and go-to-market program for selected startup APN Partners, allowing members to build on their AWS expertise, better serve shared customers, and accelerate their growth. To be selected for the APN Global Startup Program, Unravel had to meet predefined criteria, including a clear, demonstrated product market fit for an innovative enterprise tech product, be backed and recommended by a top-tier venture capital firm, and demonstrate a strategic commitment to building their AWS and cloud expertise.

The APN Global Startup Program enables qualifying startups to gain product design wins, visibility, exposure, leads, and commercial opportunities made possible with exclusive APN resources and dedicated Startup Partner Development Managers (PDM) with deep AWS knowledge and startup business experience, that guide startups in their growth journey with APN. By becoming an APN Global Startup Partner, Unravel will receive benefits ranging from a tailor-made plan for mapping the startup needs and opportunities to a selection of AWS services and APN programs, promotion support to drive visibility and awareness around the startup offering, to resources for helping startups sell and deploy innovative solutions on behalf of AWS shared customers.

“Unravel is proud to be part of the APN and the newly launched APN Global Startup Program,” said Kunal Agarwal, CEO of Unravel Data. “Our team is dedicated to helping companies simplify their cloud data operations by leveraging the agility, breadth of services, and pace of innovation that AWS provides access to.”

The Unravel platform is designed to accelerate the adoption of big data workloads in AWS. By supporting Amazon EMR, Unravel allows users to connect to a new or existing Amazon EMR cluster with just one click. Unravel for Amazon EMR can improve the productivity of big data teams with a simple, intelligent, self-service performance management capability. Unravel for Amazon EMR is engineered to:

• Automatically fix slow, inefficient and failing Spark, Hive, MapReduce, HBase, and Kafka applications
• Right size AWS cloud expenses by automatically adjusting resource consumption by users and applications
• Get a detailed view of consumption to understand cluster resource usage by user, department, or project and enable chargeback accounting
• Using AI, machine learning, and other advanced analytics, Unravel can assure service level agreements (SLAs) and optimizes compute, I/O, and storage costs. Furthermore, Unravel can reduce operational overhead through advanced automation and predictive maintenance, enabled by unified observability and AIOps capabilities.

Joining the APN Global Startup Program will allow more customers to discover the benefits Unravel has to offer for modern data stack environments to monitor, manage and improve data pipelines build on AWS.

AWS is enabling scalable, flexible, and cost-effective solutions from startups to global enterprises. The APN is a global program helping partners build a successful AWS-based business by aiding organizations to build, market, and sell their offerings. The APN provides valuable business, technical, and marketing support, empowering startups to achieve exponential growth.

_

About Unravel Data – Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement

The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

Contacts

Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post Unravel Joins AWS Partner Network Global Startup Program appeared first on Unravel.

Unsurprisingly, Modern data apps have become crucial across all areas of the financial industry, with apps for fraud detection, claims processing and compliance amongst others playing a business-critical role. Unravel has been deployed by several of the world’s largest financial services organizations to ensure these critical apps perform reliably at all times. One recent example is one of America’s ten largest banks, a corporation that encompasses over 3,000 retail branches, 5000 ATMs and over 70,000 employees. This is what happened.

This bank has been using big data for a variety of purposes, but its two most important apps are fraud detection and compliance. They deployed Informatica broadly in order to run ETL jobs. This was a massive focus for the bank’s DataOps team, which had many workflows running multiple Hive queries. They also made heavy use of Spark and Kafka Streaming in order to process tons of real-time streaming data for their fraud detection app.

Unravel Kafka Dashboard (main)

The bank suffered constant headaches before they deployed Unravel. First, their data apps tended to be slow and failed frequently. In order to figure out why, they had to dig through an avalanche of raw data logs, a process that could take weeks. Once they had identified the problem, they would have to do a long trial-and-error process to determine how to fix it. This again could take weeks, if they were fortunate enough to even find a fix for the issue.

There was another monitoring issue at the cluster usage level. They knew they weren’t optimally consuming their compute resources but had no visibility into how to improve utilization. The team only fully became aware of how critical their compute utilization problem was when it caused a critical data app to fail.

After deploying Unravel, the bank was able to quickly alleviate these problems. To begin, the platform’s reporting capabilities changed things dramatically. The team was able to monitor and understand its many different modern data stack technologies (Hive, Spark, Workflows, Kafka) from a single interface rather than relying on siloed views that didn’t enable correlation or many useful insights. The bank’s Kafka Streaming deployment had been particularly hard to monitor due to the massive input of streaming data. In addition, they previously had no way to track if Informatica and Hive queries for ETL jobs were hitting SLAs. Unravel changed all of that, delivering detailed insights that told the team how every workload was performing.

The insights were just as valuable at the cluster usage level, with Unravel providing cluster optimization opportunities to further boost performance and reduce wasteful resource consumption. This was the first time the bank really felt they understood what was happening in each and every cluster.

On top of the monitoring and visibility capabilities, Unravel yielded a significant boost in app performance. This is where the platform’s AI and automated recommendations were huge boon for the customer. After first automatically diagnosing several root cause issues, Unravel delivered cleanup recommendations for almost half million Hive tables, resulting in tremendous performance improvements. The platform also enabled the team to set notifications for specific failures and gave them the option to run automated fixes in these circumstances.

Examples of “Stalled” and “Lagging” Consumer Groups (name = “demo”) showing in the Unravel UI

While the bank isn’t currently deploying data apps in the cloud, they do have plans to migrate soon. One of the hardest parts of any cloud migration is the planning phase. Unravel’s cloud assessment capabilities give the bank detailed insights to streamline this painstaking preliminary phase: The assessment mapped out the bank’s on-premises big data pipelines and then told them which apps are best fit for the cloud and how those apps should be configured using specific instance type recommendations and forecasting costs and consumption. This move saved the customer from having to hire an expensive consulting firm to evaluate and advise their move to the cloud, accelerated their decision timeline and critically provided data driven insights instead of relying on guesswork.

Modern data apps are the backbone of any major financial institution. Unravel’s AI-driven DataOps platform allowed this bank to leverage these critical data apps to their full potential for the first time. Unravel has been so transformative that the customer has been able to open their data lake to broader business users, democratizing data apps so they provide value to the team outside of the developer and IT operations staff. In the bank’s own words, Unravel is helping drive a cultural shift by ensuring big data delivers on its true potential and is future proofing architectural decisions as hybrid cloud deployments are evaluated.

The post Top 10 Bank Leverages Unravel’s AIOps To Tame Fraud Detection and Compliance Performance appeared first on Unravel.

As technology evolves at a rapid pace, the healthcare industry is transforming quickly along with it. Tech breakthroughs like IoT, advanced imaging, genomics mapping, artificial intelligence and machine learning are some of the key items re-shaping the space. The result is better patient care and health outcomes. To facilitate this shift to the next generation of healthcare services – and to deliver on the promise of improved patient care – organizations are adopting modern data technologies to support new use cases.

We are a large company operating healthcare facilities across the US and employing over 20,000 people. Despite our size, we understand that we must be nimble and adapt fast to keep delivering cutting edge healthcare services. We only began leveraging big data about three years ago, we’ve grown fast and built out a significant modern data stack, including Kafka, Impala and Spark Streaming deployments, among others. Our focus here was always on the applications, developer needs and, ultimately, business value.

We’ve built a number of innovative data apps on top of our growing data pipelines, providing great new services and insights for our customers and employees alike. During this process, though, we realized that it’s extremely difficult to manually troubleshoot and manage such data apps. We have a very developer-focused culture –the programmers are building the very apps that are ushering in the next generation of healthcare, and we put them front and center. We were concerned when we noticed these developers were sinking huge chunks of their time fixing and diagnosing failing apps, taking their focus away from creating new apps to drive core business value.

Impala and Spark Streaming are two modern data stack technologies that our developers increasingly employed to support next generation use cases. These two technologies are commonly used to build apps that leverage streaming data, which is prevalent in our industry. Unfortunately, both Impala and Spark Streaming are difficult to manage. Apps built with these two were experiencing frequent slowdowns and intermittent crashes. Spark Streaming, in particular, was very hard to even monitor.

Our key data apps were not performing the way we expected and our programmers were wasting tons of time trying to troubleshoot them. When we deployed Unravel Data, it changed things swiftly, providing new insights into aspects of our data apps we previously had no visibility of and drastically improving app performance.

Impala – improving performance by 12-18x !

Impala is a distributed analytic SQL engine running on the Hadoop platform. Unravel provided critical metrics that helped us to better understand how Impala was being used, including:

• Impala memory consumption
• Impala queries
• Detail for queries using drill-down functionality
• Recommendations on how to make queries run faster, use data across nodes more efficiently, and more

Unravel analyzed the query pattern (insert, select, data, data locality across the Hadoop cluster) against Impala and offered a few key insights. For one, Unravel saw most of the time was spent scanning Hadoop’s file system across nodes and combining the results. After computing stats on the underlying table – a simple operation – we were able to dramatically improve performance by 12-18x.

Unravel provides detailed insights for Impala

Spark Streaming – Reducing memory requirements by 80 percent!

Spark Streaming is a lightning quick processing and analytics engine that’s perfect for handling enormous quantities of streaming data. As with Impala, Unravel provided insights and recommendations that alleviated the headaches we were having with the technology. The platform told us we didn’t have the memory for many Spark Streaming jobs, which was ultimately causing the all the slowdowns and crashes. Unravel then provided specific recommendations on how to re-configure Spark Streaming, a process that’s typically complicated and replete with costly missteps. In addition, Unravel found that we could save significant CPU resources by sending parallel tasks to cores.

Overall, two critical Spark Streaming jobs saw memory reductions of 74 percent and 80 percent. Unravel’s parallelism recommendations saved us 8.63 hours of CPU per day.

Spark Streaming performance recommendation

The Bigger Picture

Unravel is straight forward to implement and immediately delivers results. The platform’s recommendations are all configuration changes and don’t require any changes to coding. We were stunned that we could improve app performance so considerably without making a single tweak to the code, yielding an immediate boost to critical business apps. Unravel’s full-stack platform delivers insights and recommendations for our entire modern data stack deployment, eliminating the need to manage any data pipeline with a siloed tool.

Modern data apps are fueling healthcare’s technical transformation. By improving data app performance, We have been able to continue delivering a pioneering healthcare experience, achieving better patient outcomes, new services and greater business value. Without a platform like Unravel, our developers and IT team would be bogged down troubleshooting these apps rather than creating new ones and revolutionizing our business. Unravel has helped create a deep cultural shift to do more with our data and evolve to a DataOps mindset.

The post Doing more with Data and evolving to DataOps appeared first on Unravel.

We are very excited to announce that Unravel is now available for Google Cloud Dataproc. If you are already on GCP Dataproc or on a journey to migrate on-premises data workloads to GCP Dataproc then Unravel is now available immediately to accelerate and de-risk your Dataproc migration and ensure performance SLAs and cost targets are achieved.

We introduce our support for Dataproc as Google Cloud enters what is widely referred to as ‘Act 2’ under the watch of new CEO Thomas Kurian. We can expect an acceleration in new product announcements, engagement initiatives with the partner ecosystem and a restructured enterprise focused go-to-market model. We got a feel for this earlier this year at the San Francisco Google Next event and we can expect a lot more coming out of the highly anticipated Google Cloud Next ’19 event  in London coming in November.

As cloud services adoption continues to accelerate, the complexity will only continue to present Enterprise buyers with a bewildering array of choice. As outlined in our Understanding Cloud Data Services blog, wherever you are on your cloud adoption, Platform evaluation, or workload migration journey, now is the time to start to accelerate your strategic thinking and execution planning for Cloud based data services.

We are already helping customers run their big data workloads on GCP IaaS and with this new addition, we now support cloud native big data workloads running on Dataproc. In addition, Unravel plans to cover other important analytics platforms (including Google BigQuery) as part of our roadmap. This ensures Unravel provides an end-to-end, ‘single pane of glass’ for enterprises to manage their data pipelines on GCP.

Learn more about Unravel for Google Dataproc here and as always please provide feedback so we can continue to deliver for your platform investments.

Press Release quoted below.

————————————————————————————————————————————————

Unravel Data Launches Performance Management and Cloud Migration Assessment Solution for Google Cloud Dataproc

PALO ALTO, Calif. – October 22, 2019 —Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today introduced a performance management solution for the Google Cloud Dataproc platform making data workloads running on the top of the platform simpler to use and cheaper to run.

Unravel for Cloud Dataproc, which is available immediately, can improve the productivity of data teams with a simple and intelligent self-service performance management capability, helping dataops teams:

• Optimize data pipeline performance and ensure application SLAs are adhered to
• Monitor and automatically fix slow, inefficient and failing Spark, Hive, HBase and Kafka workloads
• Maximize cost savings by containing resource-hogging users or applications
• Get a detailed chargeback view to understand which users or departments are utilizing the system resources

For enterprises powered by modern data applications that rely on distributed data systems, the Unravel platform accelerates new cloud workload adoption by operationalizing a reliable data infrastructure, and it ensures enforceable SLAs and lower compute and I/O costs, while drastically lowering storage costs. Furthermore, it reduces operational overhead through rapid mean time to identification (MTTI) and mean time to resolution (MTTR), enabled by unified observability and AIOps capabilities.

“Unravel simplifies the management of data apps wherever they reside – on-premises, in a public cloud, or in a hybrid mix of the two. Extending our platform to Google Cloud Dataproc marks another milestone on our roadmap to radically simplify data operations and accelerate cloud adoption,” said Kunal Agarwal, CEO of Unravel Data. “As enterprises plan and execute their migrations to the cloud, Unravel enables operations and app development teams to improve the performance and reduce the risks commonly associated with these migrations.”

In addition to DataOps optimization, Unravel provides a cloud migration assessment offering to help organizations move data workloads to Google Cloud faster and with lower cost. Unravel has built a goal-driven and adaptive solution that uniquely provides comprehensive details of the source environment and applications running on it, identifies workloads suitable for the cloud and determines the optimal cloud topology based on business strategy, and then computes the anticipated hourly costs. The assessment also provides actionable recommendations to improve application performance and enables cloud capacity planning and chargeback reporting, as well as other critical insights.

“We’re seeing an increased adoption of GCP services for cloud-native workloads as well as on-premises workloads that are targets for cloud migration. Unravel’s full-stack DataOps platform can simplify and speed up the migration of data-centric workloads to GCP giving customers peace of mind by minimizing downtime and lowering risk.” said Mike Leone, Senior Analyst, Enterprise Strategy Group. “Unravel adds operational and business value by delivering actionable recommendations for Dataproc customers. Additionally, the platform can troubleshoot and mitigate migration and operational issues to boost savings and performance for Cloud Dataproc workloads.”

Unravel for Google Cloud Dataproc is available now.

Create a free account. Sign up and get instant access to the Unravel environment. Take a guided tour of Unravel’s full-stack monitoring, AI-driven recommendations, automated tuning and remediation capabilities.

Share this: https://ctt.ac/3Q3qc

About Unravel Data
Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement
The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

Contacts
Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post Unravel Launches Performance Management and Cloud Migration Assessment Solution for Google Cloud Dataproc appeared first on Unravel.

Monitor, Troubleshoot and Optimize Apache Spark Applications Using Microsoft Azure Databricks 

We are super excited to announce our support for Azure Databricks! We continue to build out the capabilities of the Unravel Data Operations platform and specifically support for the Microsoft Azure data and AI ecosystem teams.  The business and technical imperative to strategically and tactically architect the journey to cloud for your organization has never been stronger. Businesses are increasingly dependent on data for decision making and by extension the services and platforms such as Azure HDI and Azure Databricks that underpin these modern data applications.

The large scale industry adoption of Spark, and Cloud services from Azure and other platforms. represent the heart of the modern data operations program for the next decade. The combination of Microsoft and Databricks and resulting Azure Databricks offering is a natural response to deliver a deployment platform for AI, machine learning, and streaming data applications.

Spark has largely eclipsed Hadoop/MapReduce as the development paradigm of choice to develop a new generation of data applications that provide new insights and user experiences. Databricks has added a rich development and operations environment for running Apache Spark applications in the cloud, while Microsoft Azure has rapidly evolved into an enterprise favorite for migrating and running these new data applications in the cloud. 

It is against this backdrop that Unravel announces support for the Azure Databricks platform to provide our AI-powered data operations solution for Spark applications and data pipelines running on Azure Databricks. While Azure Databricks provides a state of the art platform for developing and running Spark apps and data pipelines, Unravel provides the relentless monitoring, interrogating, modeling, learning, and guided tuning and troubleshooting to create the optimal conditions for Spark to perform and operate at its peak potential.

Unravel is able to ask and answer questions about Azure Databricks that are essential to provide the levels of intelligence that are required to:

• Provide a unified view across all of your Azure Databricks instances and workspaces
• Understand Spark runtime behavior and how it interacts with Azure infrastructure, and adjacent technologies like Apache Kafka
• Detect and avoid costly human error in configuration, tuning, and root cause analysis 
• Accurately report cluster usage patterns and be able to adjust resource usage on the fly with Unravel insights
• Set and guarantee enterprise service levels, based on correlated operational metadata

The Unravel Platform is constantly learning and our training models adapting. The intelligence you glean from Unravel today continues to extend and adapt over time as application and user behaviors themselves change and adapt to new business demands. These in-built capabilities of the Unravel platform and our extensible APIs enable us to move fast to support customer demands to support an expanding range of Data and AI services such as Azure Databricks.  More importantly though it provides the insights, recommendations and automation to assure your journey to cloud is accelerated and your ongoing Cloud operations is fully optimized for cost and performance.

Read on to learn more about today’s news from Unravel.

Unravel Data Introduces AI-powered Data Operations Solution to Monitor, Troubleshoot and Optimize Apache Spark Applications Using Microsoft Azure Databricks

New Offering Enables Azure Databricks Customers to Quickly Operationalize Spark Data Engineering Workloads with Unprecedented Visibility and Radically Simpler Remediation of Failures and Slowdowns

PALO ALTO, Calif. – Sep. 4, 2019 —Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced Unravel for Azure Databricks, a  solution to deliver comprehensive monitoring, troubleshooting, and application performance management for Azure Databricks environments. The new offering leverages AI to enable Azure Databricks customers to significantly improve performance of Spark jobs while providing unprecedented visibility into runtime behavior, resource usage, and cloud costs.

“Spark, Azure, and Azure Databricks have become foundational technologies in the modern data stack landscape, with more and more Fortune 1000 organizations using them to build their modern data pipelines,” said Kunal Agarwal, CEO, Unravel Data. “Unravel is uniquely positioned to empower Azure Databricks customers to maximize the performance, reliability and return on investment of their Spark workloads.”

Unravel for Azure Databricks helps operationalize Spark apps on the platform: Azure Databricks customers will shorten the cycle of getting Spark applications into production by relying on the visibility, operational intelligence, and data driven insights and recommendations that only Unravel can provide. Users will enjoy greater productivity by eliminating the time spent on tedious, low value tasks such as log data collection, root cause analysis and application tuning.

“Unravel’s full-stack DataOps platform has already helped Azure customers get the most out of their cloud-based big data deployments. We’re excited to extend that relationship to Azure Databricks,” said Yatharth Gupta, principal group manager, Azure Data at Microsoft. “Unravel adds tremendous value by delivering an AI-powered solution for Azure Databricks customers that are looking to troubleshoot challenging operational issues and optimize cost and performance of their Azure Databricks workloads.”

Key features of Unravel for Azure Databricks include:

• Application Performance Management for Azure Databricks – Unravel delivers visibility and understanding of Spark applications, clusters, workflows, and the underlying software stack
• Automated root cause analysis of Spark apps – Unravel can automatically identify, diagnose, and remediate Spark jobs and the full Spark stack, achieving simpler and faster resolution of issues for Spark applications on Azure Databricks clusters
• Comprehensive reporting, alerting, and dashboards – Azure Databricks users can now enjoy detailed insights, plain-language recommendations, and a host of new dashboards, alerts, and reporting on chargeback accounting, cluster resource usage,  Spark runtime behavior and much more.

Azure Databricks is a Spark-based analytics platform optimized for Microsoft Azure. Azure Databricks provides one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

An early access release of Unravel for Azure Databricks available now.

About Unravel Data

Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement

The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

Contacts

Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post Unravel and Azure Databricks: Monitor, Troubleshoot and Optimize in the Cloud appeared first on Unravel.

Today, Unravel announced a new cloud migration assessment offer to accelerate the migration of data workloads to Microsoft Azure, Amazon AWS, or Google Cloud Platform. Our latest offer fills a significant gap in the cloud journey, equips enterprises with the tools to deliver on their cloud strategy, and provides the best possible transition with insights and guidance before, during, and after migration. Full details on the assessment and business value are detailed below in our announcement below.

So, why now?

The rapid increase in data volume and variety has driven organizations to rethink enterprise infrastructures and focus on longer-term data growth, flexibility, and cost savings. Current, on-prem solutions are too complicated, inflexible, and are not delivering on expected value. Data is not living up to its promise.

As an alternative, organizations are looking to cloud services like Azure, AWS, and Google Cloud to provide the flexibility to accommodate modern capacity requirements and elasticity. Unfortunately, organizations are often challenged by unexpected costs and a lack of data and insights to ensure a successful migration process. If left unaddressed, organizations will struggle with the complexity of these projects that don’t fulfill their expectations and frequently result in significant cost overruns.

The cloud migration assessment offer provides details of the source environment and applications running on it, identifies workloads suitable for the cloud, and computes the anticipated hourly costs. It offers granular metrics, as well as broader insights, that eliminate transition complexity and deliver migration success.

Customers can be confident that they’re migrating the right data apps, configuring them properly in the cloud, meeting performance service level agreements, and minimizing costs. Unravel can provide an alternative to what is frequently a manual effort fraught with guesswork and errors.

The two approaches can be characterized per the diagram below

Still unsure how the migration assessment will provide value to your business? Drop us a line to learn more about the offer – or download a sample cloud migration assessment report here.

————-

Read on to learn more about today’s news from Unravel.

Unravel Introduces Cloud Migration Assessment Offer to Reduce Costs and Accelerate the Transition of Data Workloads to Azure, AWS or Google Cloud

New Offer Builds a Granular Dependency Map of On-Premises Data Workloads and Provides Detailed Insights and Recommendations for the Best Transition to Cloud

PALO ALTO, Calif. – July, 31, 2019 —Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced a new cloud migration assessment offer to help organizations move data workloads to Azure, AWS or Google Cloud faster and with lower cost. Unravel has built a goal-driven and adaptive solution that uniquely provides comprehensive details of the source environment and applications running on it, identifies workloads suitable for the cloud and determines the optimal cloud topology based on business strategy , and computes the anticipated hourly costs. The offer also provides actionable recommendations to improve application performance and enables cloud capacity planning and chargeback reporting, as well as other critical insights.

“Managing the modern data stack on-premises is complex and requires expert technical talent to troubleshoot most problems. That’s why more enterprises are moving their data workloads to the cloud, but the migration process isn’t easy , as there’s little visibility into costs and configurations,” said Kunal Agarwal, CEO, Unravel Data. “Unravel’s new cloud migration assessment offer delivers actionable insights and visibility so organizations no longer have to fly blind. No matter where an organization is in its cloud adoption and migration journey, now is the time to accelerate strategic thinking and execution, and this offering ensures the fastest, most cost effective and valuable transition for the full journey-to-cloud lifecycle.”

“Companies have major expectations when they embark on a journey to the cloud. Unfortunately, organizations that migrate manually often don’t fulfill these expectations as the process of transitioning to the cloud becomes more difficult and takes longer than anticipated. And then once there, costs rise higher than forecasted and apps are difficult to optimize,” said Enterprise Strategy Group senior analyst Mike Leone. “This all results from the lack of insight into their existing data apps on-premises and how they should map those apps to the cloud. Unravel’s new offer fills a major gap in the cloud journey, equipping enterprises with the tools to deliver on their cloud goals.”

The journey to cloud is technically complex and aligning business outcomes with a wide array of cloud offerings can be challenging. Unravel’s cloud migration assessment offer takes the guesswork and error-prone manual processes out of the equation to deliver a variety of critical insights. The assessment enables organizations to:

• Discover current clusters and detailed usage to make an effective and informed move to the cloud
• Identify and prioritize specific application workloads that will benefit most from cloud-native capabilities, such as elastic scaling and decoupled storage
• Define the optimal cloud topology that matches specific goals and business strategy, minimizing risks or costs. Users get specific instance types recommendations on the amount of storage needed with the option to choose between local attached and object storage
• Obtain the hourly costs expected to incur when moving to the cloud, allowing users to compare and contrast the costs for different cloud providers and services and for different goals
• Compare costs for different cloud options (across IaaS and Managed Hadoop/Spark PaaS services). Includes the ability to override default on-demand prices to incorporate volume discounts users may have received
• Optimize cloud storage tiering choices for hot, warm, and cold data

The Unravel cloud assessment service encompasses four phases. The first phase is a discovery meeting in which the project is scoped, stakeholders identified and KPIs defined. Then during technical discovery, Unravel works with customers to define use cases, install the product and begin gathering workload data. Following, is the initial readout, as enterprises receive a summary of their infrastructure and workloads along with fresh insights and recommendations for cloud migration. Then comes the completed assessment, including final insights, recommendations and next steps.

Unravel is building a rapidly expanding ecosystem of partners to provide a portfolio of data operations and migration services utilizing the Unravel Data Operations Platform and cloud migration assessment offer.

Enterprises can find a sample cloud migration assessment report here.

About Unravel Data
Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern data stack leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement
The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

PR Contact
Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post Accelerate and Reduce Costs of Migrating Data Workloads to the Cloud appeared first on Unravel.

Modern data apps are increasingly going to the cloud due to their elastic compute demands, skills shortages and the complexity of managing big data on premises. And while more and more organizations are taking their data apps to the cloud to leverage its flexibility, they’re also finding that it is very challenging to assess application needs and how to migrate and optimize their data to ensure performance and cost targets are not compromised. This complexity underscores how critical a platform like Unravel is for businesses in any industry to get the most out of their cloud-based data apps.

The Unravel Partner Program we introduced today helps make the journey and experience in the cloud even easier. The program enables customers of world-leading technology innovators like Microsoft Azure, AWS, Informatica, Attunix, Dell, Clairvoyant and RCG Global Services, to solve their data operations challenges and accelerate their cloud adoption cycles.

I have the pleasure of leading this new program and working with the Unravel community. It’s thrilling to be at the starting line with our partners and customers to help them future-proof their data workloads in the cloud. Full details on platform, solution partnerships and program benefits are below in our company announcement and here, and I encourage you to drop us a line to learn more about the Unravel Partner Program.

————-

Read on to learn more about today’s news from Unravel.

Unravel Data Announces Partner Program and New Executive Hire to Meet Demand for Cloud Migration and Data Operations Initiatives

Unravel Partner Ecosystem Encompasses an Expansive Network of Innovators to Help Businesses Optimize Their Data Systems and Future-Proof Cloud Migration Strategies

PALO ALTO, Calif.,—June 25, 2019—Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced the Unravel Partner Program, which brings together world-leading technology innovators that support Unravel’s mission to radically simplify the way businesses optimize the performance of their modern data applications and the complex pipelines that power those applications. The new program debuts with the appointment of Mark Wolfram as vice president of business development and partnerships, who will direct the program’s strategy and expansion.

Unravel’s Partner Program ecosystem includes a robust list of brands including Informatica, Attunix, Dell and RCG Global Services to help their customers solve data operations challenges and optimize their cloud adoption plans :

• • Technology and Platform Partners:
    Unravel’s partnerships with leading cloud platforms, data management, data operations and analytics partners guarantee the performance, reliability and cost of data systems. Customers benefit from operationalized data application workloads and the ability to address performance and reliability challenges while in production. Opportunities for optimization are pinpointed to ensure that service level agreements are met and resources are optimized. And as data workloads move to the cloud and hybrid deployment models, Unravel can accelerate decision making while eliminating risk.
  • Solution Partners:
    Unravel’s partnerships with integrators and solution providers deliver validated data operations solutions to their customers to guarantee the performance, reliability and cost of their data systems either on premises or in the cloud. Customers benefit from access to highly differentiated Unravel technology to solve a variety of data operations and cloud migration challenges. Unravel’s training and support accelerates time to value, as does its direct and early access to product, engineering and go-to market teams for deal acceleration and collaborative support in solution design.

“We share with Unravel the common goal of building a culture of innovation and helping customers benefit from all of their data, not just the data for which they have ready-made solutions or tools,” stated Rick Skriletz, Global Managing Principal at RCG Global Services. “Although it’s not a universal truth, we typically find the more data that can be brought to bear on a problem, the better the results. Our complementary solution sets – and tight integration between them – are geared to bring automation to big data workflows, which is the one need we most often hear from customers.”

“It is a daunting task to migrate and optimize your data to the cloud as part of a digital transformation journey and yet we see a growing demand from our customers to do so from energy to financial services, manufacturing, government and retail,” said Matt O’Donnell, President of Cloud Services at Redapt Attunix. “The impetus of demand and complexity has set us on the path to partner with Unravel to offer our customers the sophisticated tools, support, vision and platform expertise they need to migrate with confidence, and from there monitor, manage and improve their data pipelines to achieve more reliable performance in the applications that power their business. Unravel’s compatibility and collaboration with leading platforms is a significant advantage to our customers who seek this synergy to deliver value to their customers.”

Unravel’s ecosystem partners benefit from enablement, go-to-market and post-sales support. Exclusive technical and best-practices and sales and product training are made available to the partners. Partners can take advantage of opportunities that can help build their business through joint marketing activities, co-marketing collateral and registered partner lead and deal registration. Through the program, partners have access to tools and APIs for customer account management, workflow integration, global dedicated technical support, and recognition as an official partner in the Unravel community.

Mark Wolfram, an accomplished technology executive with more than 20 years of experience in sales, business development, and management, leads the Unravel Partner Program. Wolfram brings a strong background in cloud and data analytics honed at Microsoft and Azuqua (acquired by Okta).

“Unravel Data is a unique company that has given more than just lip service about how to address the immense challenges inherent when enterprises look to move data workloads to the cloud. It is actually taking the significant steps necessary to truly partner with businesses during this complex migration process and to ensure continuous availability of modern data applications,” said Mark Wolfram, vice president of business development and partnerships at Unravel Data. “Unravel has developed the industry’s only AI-powered platform for planning, migrating and managing modern data apps in the cloud. I passionately support Unravel’s approach and look forward to sharing my expertise as together we grow the Unravel Partner Program to serve more enterprises around the world.”

To learn more about Unravel’s Partner Program, please visit: https://www.unraveldata.com/company/partners/.

About Unravel Data
Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyrights
The name Unravel Data is a trademark of Unravel Data. Informatica and Informatica World are trademarks or registered trademarks of Informatica in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners.

###

PR Contact
Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post Making Data Work With the Unravel Partner Program appeared first on Unravel.

A household name in global media analytics – let’s call them MTI – is using Unravel to support their data operations (DataOps) on Amazon EMR to establish and protect their internal service level agreements (SLAs) and get the most out of their Spark applications and pipelines. MTI runs 10’s of thousands of jobs per week, about 70% of which are Spark, with the remaining 30% of workloads running on Hadoop, or more specifically Hive/MapReduce.

Among the most common complaints and concerns about optimizing big data clusters and applications is the amount of time it takes to root-cause issues like application failures or slowdowns or to figure out what needs to be done to improve performance. Without context, performance and utilization metrics from the underlying data platform and the Spark processing engine can laborious to collect and correlate, and difficult to interpret.

Unravel employs a frictionless method of collecting relevant data about the full data stack, running applications, cluster resources, datasets, users, business units and projects. Unravel then aggregates and correlates this data into the Unravel data model and then applies a variety of analytical techniques to put that data into a useful context.

Unravel architecture for Amazon AWS/EMR

MTI has prioritized their goals for big data based on two main dimensions that are reflected in the Unravel product architecture: Operations and Applications.

Optimizing data operations

For MTI’s cluster level SLAs and operational goals for their big data program, they identified the following requirements:

• Reduce time needed for troubleshooting and resolving issues.
• Improve cluster efficiency and performance.
• Improve visibility into cluster workloads.
• Provide usage analysis

Reducing time to identify and resolve issues

One of the most basic requirements for creating meaningful SLAs is to set goals for identifying problems or failures – known as Mean Time to Identification (MTTI) – and the resolution of those problems – known as Mean Time to Resolve (MTTR). MTI executives set a goal of 40% reduction in MTTR.

One of the most basic ways that Unravel helps reduce MTTI/MTTR is through the elimination of the time-consuming steps of data collection and correlation. Unravel collects granular cluster and application-specific runtime information, as well as metrics on infrastructure, resources using native Hadoop APIs and via lightweight sensors that only send data while an application is executing. This alone can save data teams hours – if not days – of data collection by, capturing application and system log data, configuration parameters, and other relevant data.

Once that data is collected, the manual process of evaluating and interpreting that data has just begun. You may spend hours charting log data from your Spark application only to find that some small human error, a missed configuration parameter, and incorrectly sized container, or a rogue stage of your Spark application is bringing your cluster to its knees.

Unravel top level operations dashboard

Improving visibility into cluster operations

In order for MTI to establish and maintain their SLAs, they needed to troubleshoot cluster-level issues as well as issues at the application and user levels. For example, MTI wanted to monitor and analyze the top applications by duration, resources usage, I/O, etc. Unravel provides a solution to all of these requirements.

Cluster level reporting

Cluster level reporting and drill down to individual nodes, jobs, queues, and more is a basic feature of Unravel.Unravel cluster infrastructure dashboard

Application and workflow tagging

Unravel provides rich functionality for monitoring applications and users in the cluster. Unravel provides cluster and application reporting by user, queue, application type and custom tags like Project, Department etc.. These tags are preconfigured so that MTI can instantly filter their view by these tags. The ability to add custom tags is unique to Unravel and enables customers to tag various applications based on custom rules specific to their business requirements (e.g. Project, business unit, etc.).

Unravel application tagging by department

Usage analysis and capacity planning

MTI wants to be able to maintain service levels over the long term, and thus require reporting on cluster resource usage, and data on future capacity requirements for their program. Unravel provides this type of intelligence through the Chargeback/showback reporting.

Unravel chargeback reporting

You can generate ChargeBack reports in Unravel for multi-tenant cluster usage costs associated by the Group By options: application type, user, queue, and tags. The window is divided into three (3) sections,

• Donut graphs showing the top results for the Group by selection.
• Chargeback report showing costs, sorted by the Group By choice(s).
• List of Yarn applications running.

Unravel chargeback reporting

Improving cluster efficiency and performance

MTI wanted to be able to predict and anticipate application slowdowns and failures before they occur. by using Unravel’s proactive alerting and auto-actions so that they could, for example, find runaway queries and rogue jobs, detect resource contention, and then take action.

Unravel Auto-actions and alerting

Unravel Auto-actions are one of the big points of differentiation over the various monitoring options available to data teams such as Cloudera Manager, Splunk, Ambari, and Dynatrace. Unravel users can determine what action to take depending on policy-based controls that they have defined.

Unravel Auto-actions set up

The simplicity of the Auto-actions screen belies the deep automation and functionality of autonomous remediation of application slowdowns and failures. At the highest level, Unravel Auto-actions can be quickly set up to alert your team via email, PagerDuty, Slack or text message. Offending jobs can also be killed or moved to a different queue. Unravel can also create an HTTP post that gives users a lot of powerful options.

Unravel also provide a number of powerful pre-built Auto-action templates that can give users a big head start on crafting the precise automation they wish for their environment.

Preconfigured Unravel auto-action templates

Applications

Turning to MTI’s application-level requirements, the company was looking at improving overall visibility into their data application runtime performance, and to encourage a self-service approach to tuning and optimizing their Spark applications.

Increased visibility into application runtime and trends

MTI data teams, like many, are looking for that elusive “single pane of glass” for troubleshooting slow and failing Spark jobs and applications. They were looking to:

• Visualize app performance trends, viewing metrics such as applications start time, duration, state, I/O, memory usage, etc.
• View application component (pipeline stages) breakdown and their associated performance metrics
• Understand execution of Map Reduce jobs, Spark applications and the degree of parallelism and resource usage as well as obtain insights and recommendations for optimal performance and efficiency

Because typical data pipelines are built on a collection of distributed processing engines (Spark, Hadoop, et al.), getting visibility into the complete data pipeline is a challenge. Each individual processing engine may have monitoring capabilities, but there is a need to have a unified view to monitor and manage all the components together.

Unravel monitoring, tuning and troubleshooting

Intuitive drill-down from Spark application list to an individual data pipeline stage

Unravel was designed with an end-to-end perspective on data pipelines. The basic navigation moves from the top level list of applications to drill down to jobs, and further drill down to individual stages of a Spark, Hive, MapReduce or Impala applications.

Unravel Gantt chart view of a Hive query

Unravel provides a number of intuitive navigational and reporting elements in the user interface including a Gantt chart of application components to understand the execution and parallelism of your applications.

Unravel self-service optimization of Spark applications

MTI has placed an emphasis on creating a self-service approach to monitoring, tuning, and management of their data application portfolio. They are for development teams to reduce their dependency on IT and at the same time to improve collaboration with their peers. Their targets in this area include:

• Reducing troubleshooting and resolution time by providing self-service tuning
• Improving application efficiency and performance with minimal IT intervention
• Provide Spark developers performance issues and relate directly to the lines of code associated with a given step.

MTI has chosen Unravel as a foundational element of their self-service application and workflow improvements, especially taking advantage of application recommendations and insights for Spark developers.

Unravel self-service capabilities

Unravel provides plain language insights as well as specific, actionable recommendations to improve performance and efficiency. In addition to these recommendations and insights, users can take action via the auto-tune function, which is available to run from the events panel.

Unravel provides intelligent recommendations and insights as well as auto-tuning.

Optimizing Application Resource Efficiency

In large scale data operations, the resource efficiency of the entire cluster is directly linked to the efficient use of cluster resources at the application level. As data teams can routinely run hundreds or thousands of job per day, an overall increase in resource efficiency across all workloads improves the performance, scalability and cost of operation of the cluster.

Unravel provides a rich catalog of insights and recommendations around resource consumption at the application level. To eliminate resource wastage Unravel can help you run your data applications more efficiently by providing you AI driven insights and recommendations to do show:

Unravel Insight: Under-utilization of container resources, CPU or memory

Unravel Insight: Too few partitions with respect to available parallelism

Unravel Insight: Mapper/Reducers requesting too much memory

Unravel Insight: Too many map tasks and/or too many reduce tasks

Solution Highlights

Work on all of these operational goals is ongoing with MTI and Unravel, but to date, they have made significant progress on both operational and application goals. After running for over a month on their production computation cluster, MTI were able to capture metrics for all MapReduce and Spark jobs that were executed.

MTI also got great insights on the number and causes of inefficiently running applications. Unravel detected a significant number of inefficient applications. Unravel detected 38,190 events after analyzing 30,378 MapReduce jobs that they executed. They were also able to detect 44,176 events for 21,799 Spark jobs that they executed. They were also able to detect resource contention which causing Spark jobs to get stuck in “Accepted” state, rather than running to completion.

During a deep dive on their applications, MTI found multiple inefficient jobs where Unravel provided recommendations for repartitioning the data. They were also able to Identify many jobs which waste CPU and memory resources.

The post Case Study: Meeting SLAs for Data Pipelines on Amazon EMR appeared first on Unravel.

Off the back of a breakout year where we grew revenue 500%, today we announced the latest milestone in the Unravel journey – we closed a $35M Series C funding round.

Along with our data ecosystem partners, we are seeing unprecedented demand for solutions to complex, business-critical challenges in dealing with data.

Consider this. Data Engineers walk into work every day knowing they’re fighting an uphill battle. The root of the problem – or at least one problem – is that modern data systems are becoming impossibly complex. The burgeoning amount of data being processed in organizations today is staggering, where annual data growth is often measured in high double-digit percentages. Just a year ago, Forbes reported that 90% of the world’s data was created in the previous two years.

And with that data growth has come rapid growth in the number of applications for ingesting, correlating and analyzing that data. Each component of a data pipeline is by nature a specialist, and it takes lots of specialists to make data deliver results – and, more importantly, insights. This is a problem that touches virtually every corner of the world of business. And the pressure to perform and make data “work” is unrelenting.

In our own research from November 2018, Unravel found that three-quarters of businesses expect their modern data stack to drive profitable business applications by the end of 2019 – but only 12% were seeing this value at the time.

The media is rife with stories prophesying magnificent discoveries to be made when data converges with artificial intelligence-driven models. Some of these discoveries have been made, but many more are still to come. Too often, these discoveries are over the horizon or well beyond the horizon, as data practitioners struggle with data systems that create more hurdles than they knock down.

Old Technologies Cannot Solve New Problems

Model-driven insights from data is what every business aspires to. The need for reliable and scalable application performance spawned the development of Application Performance Management (APM) and log management tools, two pioneering technologies in the race to make sense of new, multi-tier web architectures. The problem is that those technologies fell short because they were not designed and built for modern data systems. From the standpoint of the Data Engineer, the metrics and graphs those technologies deliver fall flat, when the team needs actual recommendations and answers to the issues faced multiple times every day.

“It’s clear that enterprises continue to struggle with dealing with the enormous amount of data that fuels their businesses. Legacy approaches have failed, and they need to modernize their systems or risk being made irrelevant,” said Venky Ganesan, managing director, Menlo Ventures.

Dealing with Overwhelming Complexity

Although it might be trite, it’s worth mentioning that every business is becoming a data business. That’s why most businesses consider data management systems such as Spark, Kafka, hadoop and NoSQL as their critical systems of record.

Data pipelines are so complex that they are outgrowing our ability to manage them. That’s because these systems have so many interdependencies that solutions lie beyond human intuition or deduction. And that’s why Unravel talks a lot about the importance of full-stack visibility for optimizing the performance of data-driven applications. We obsess over the need to explore, correlate, and analyze everything in your modern data stack environment, search for dependencies and issues, understand how data and resources are being used, and discover how to troubleshoot and remediate issues.

And we believe in the promise of AI. That’s why Unravel integrated a powerful AI engine to deliver recommendations that drive more reliable performance in modern data applications.

Cloud Complicates Everything

As businesses migrate their data-focused applications and their data to the cloud, they face the fact that many cloud platforms provide only minimal siloed tools for managing these workloads.

In response, Unravel, unveiled its newest version of the Unravel platform, which focuses squarely on the unique requirements of hosting data-focused applications in the cloud. That release took the AI, machine learning, and predictive analytics that are the hallmarks of the platform and enabled users to assess which apps are the best candidates to move to the cloud – based on the customer’s own defined criteria.

The release also gave users the tools to validate the success of their cloud migration and predict capacity based on their specific application workloads. At the time, I noted that many unknowns around cost, visibility and migration had prevented this transition to the cloud from occurring more quickly. But that is no more.

Continuous Improvement

Continuous improvement: although the term is dated, the concept is still as timely as ever. And it’s a mantra of many businesses today that are never content, even with their highest achievements.

Continuous improvement is just the latest growth driver in modern data systems as well, and it’s being built on models. In turn, these models are built on closed-loop data. “When built right, these models create a reinforcing cycle: Their products get better, allowing them (businesses) to collect more data, which allows them to build better models, making their products better, and onward,” said Steven Cohen and Matthew Granade of Point72 Ventures, an investor in Unravel Data.

If anything is keeping CIOs from meeting their OKRs, #1 on that list is likely data system complexity. Well, complexity is here to stay! In our data-driven world, gains come when we deal with the inevitable complexities and move beyond them. At Unravel, we think big data can do better, and we’re here to help it along. By radically simplifying the way you do data operations, how your models perform and ensuring big data lives up to your expectations – both today and tomorrow.

————-

Read on to learn more about today’s news from Unravel.

Unravel Data Grows Revenue 500% Year-Over-Year, Secures $35M in Series C Funding

Point72 Ventures leads funding round to address performance and complexity challenges of modern data applications and cloud migration initiatives

PALO ALTO, Calif.,—May 14, 2019—Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced it has raised $35 million in an oversubscribed Series C funding round. Point72 Ventures, founded by renowned hedge fund investor Steve Cohen, led the round with participation from Harmony Partners, and existing Unravel investors Menlo Ventures, GGV Capital and M12 (Microsoft Ventures).

“At first glance, application performance management (APM) may seem like a problem that has been addressed by the APM and log management vendors such as AppDynamics, New Relic, Splunk, and DataDog. The reality is that these solutions were not initially built for modern data systems. By being natively built with modern IT in mind, Unravel can cost-effectively deliver the data application awareness and AI-powered recommendations, resolutions, and answers that organizations demand,” said Mike Leone, senior analyst at ESG. “Additionally, as workloads migrate to cloud platforms, the complexity of multiple systems, locality of services and technologies, different operating and pricing models, and constantly changing dependencies throughout the data pipeline add higher levels of risk to migrations, performance optimization challenges, and cost concerns.”

“Most industry-leading companies are now software businesses, and a majority of those businesses are running on top of mission-critical big data applications,” said David Dubick, Partner, Point72 Ventures. “These big data tailwinds have created a demand for tools to monitor, optimize and secure these systems, and Unravel is uniquely positioned to address this need in the marketplace.”

“CIOs in our network told us story after story of traditional application monitoring tools failing in a big data context because those tools were designed for the world of the past. And we didn’t just hear this problem from third parties, we were seeing it at Point72 as well,” said Matthew Granade, Chief Market Intelligence Officer at Point72 and Managing Partner of Point72 Ventures. “This new architecture requires a different product, one built from the ground up to focus on the unique challenges posed by big data applications. Unravel is poised to capture this emerging big-data APM market.”

Company Momentum Highlights

Today’s funding news follows a year of significant momentum for Unravel as evidenced by a series of milestones:

• Annual Recurring Revenue (ARR) growth of 500%
• Cloud Services Product and Partner Momentum
  • Microsoft Azure Cloud Partner Ecosystem — Unravel introduced support for Azure services and uses operational data from Azure HDInsight, Spark, Kafka, Hadoop, Hive, and HBase to automatically troubleshoot on-going issues that reduce confidence and performance on customers’ clusters. Unravel also correlates this full-stack data to help in migration to Azure. Unravel is available on the Microsoft Azure Marketplace and is co-sell ready.
  • AWS Cloud Partner Ecosystem — Unravel introduced its platform across the Amazon ecosystem supporting Amazon AWS, Amazon EMR, Cloudera EDH for AWS, Hortonworks Data Cloud on AWS, and MapR CDP on AWS, providing critical operational intelligence. Unravel is an AWS Advanced Partner and is available in the AWS marketplace
• Industry Accolades – Gartner named Unravel Data to its list of “Cool Vendors” for 2018 in Performance Analysis; Analytics and Containers. CRN awarded Unravel as a ‘Top 100 Coolest Cloud Computing Company,’ and Unravel made CNBC’s Upstart 100 list.

“Every business is becoming a data business, and companies are relying on their data applications such as machine learning, IoT, and customer analytics, for better business outcomes using technologies such as Spark, Kafka, and NoSQL,” said Kunal Agarwal, CEO, Unravel Data. “We are making sure that these technologies are easy to operate and are high performing so that businesses can depend on them. We partner with our customers through their data journey and help them successfully run data apps on various systems whether on-premises or in the cloud.”

Customer Validation

See https://www.unraveldata.com/customers/

Unravel Reviews on Gartner Peer Insights:
“Key Software Product for Today’s Modern Data Applications And Systems” March, 2019
https://www.gartner.com/reviews/review/view/785382#

Market Validation

“Enterprises are turning to technologies like Spark, Kafka, MPP, and NoSQL to embrace a data-centric approach to their business. The challenge is that there are massive skills shortages associated with architecting, managing, and optimizing all these integrated tools supported by numerous vendors across a data pipeline. In fact, on average, organizations work with 37 different vendors across their data pipeline today. Many of the technologies they rely on have their own monitoring and management tools, and this exacerbates the problem, creating operational silos and ultimately preventing end-to-end insight,” said Mike Leone, senior analyst at ESG. “How can organizations effectively utilize a wide range of applications like customer analytics, fraud prevention, and predictive maintenance that rely heavily on next-generation technology like AI and machine learning? By turning to a comprehensive data operations platform. Unravel allows customers to manage and optimize all their data pipelines from one location. By using AI-driven recommendations and automation, a high percentage of manual troubleshooting can simply be eliminated, enabling data operations teams to be proactive in preventing future issues.”

“As enterprises of every size choose the Azure Cloud platform to build and deliver their modern data, Unravel has proven an important tool to help enterprises operationalize this data and drive tangible value to the business,” said Rashmi Gopinath, partner, M12 Ventures. “Azure and Unravel have worked closely on product development and go-to market execution and are well positioned to meet this market demand.”

Investor Quotes

“There’s a tremendous need to enable organizations to maximize the value of their data infrastructure investments,” said Mark Lotke, founder and managing partner, Harmony Partners. “Unravel fills that gap perfectly as the only company that is truly using machine learning and an AI-driven platform to optimize and operationalize data-driven applications and the data systems they depend on at scale. The Unravel team demonstrated incredible growth in 2018 and is poised for an even bigger year in 2019 as demand for data operations solutions accelerates.”

“It’s clear that enterprises continue to struggle with dealing with the enormous amount of data that fuels their businesses. Legacy approaches have failed and they need to modernize their systems or risk being made irrelevant. Unravel is leading the pack in providing technology innovations that provide this competitive edge and fuel the next generation of cloud and hybrid cloud data services,” commented Venky Ganesan, managing director, Menlo Ventures.

“Since we led the company’s B round, we have been blown away by the market momentum that Unravel has achieved in a short space of time. There is clearly an unmet need in large enterprises for solving the complexity and operational challenges they face as they transition to being data driven and cloud first,” said Glenn Solomon, Managing Partner, GGV Capital. “Current approaches are failing these data ops teams and Unravel has come to market with technology innovation and go-to-market execution that is solving real world problems, today.”

About Unravel Data
Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement
The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

PR Contact
Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post Software Ate the World and Now the Models are Running It appeared first on Unravel.

It’s indisputable, new data-driven applications are moving to, or starting life running in the cloud. The increasing automation and resilience of current cloud infrastructure is an ideal environment for running modern data pipelines. For many companies and institutions, their cloud first strategy is becoming a cloud only strategy.

Native online business’ such as Netflix as well as mainstream Enterprises such as Capital One have multi $Billion valuations and almost no physical data centers. Public cloud providers will account for over 60% of all capital expenditures on cloud infrastructure – disks, CPUs, network switches and the like. Given this momentum, there is increased pressure on IT teams to prove that they are getting the most of their cloud and big data investments.

Against this backdrop Unravel Introduces the Industry’s First AI-Powered Cloud Platform Operations and Workload Migration Solution for Data Applications, delivering new AI-powered and Unified performance optimization for planning, migrating, and managing modern data applications on the AWS, Azure and Google Cloud Platforms.

Some of the new capabilities that IT teams will gain from this latest release include:

Unified management of the full modern data stack on all deployment platforms – Unravel Cloud Migration covers AWS, Azure and Google clouds, as well as on-premises, hybrid environments and multi-cloud settings. Customers get AI-powered troubleshooting, auto-tuning and automated remediation of failures and slowdowns with the same user interface.

Full stack  visibility – Unravel uses automation to provide detailed reports and metrics on app usage, performance, cost and chargebacks in the cloud.

Recommendations for the best apps to migrate – Unravel baselines on-premises performance of the full modern data stack and uses AI to identify the best app candidates for migration to cloud. Organizations can avoid migrating apps that aren’t ideal for the cloud and having to repatriate them later.

Mapping on-premises infrastructure to cloud server instances – Unravel helps customers choose cloud instance types for their migration based on three strategies:

• Lift and shift – A one-to-one mapping from physical to virtual servers ensures that a  cloud deployment will have the same (or more) resources available. This minimizes any risks associated with migrating to the cloud.
• Cost reduction – Provides the most cost-effective instance recommendations based on detailed dependency understanding for minimizing wasted capacity and over provisioning.
• Workload fit – Takes into account data collected over time from the on-premises environment, making recommendations for instance types based on the actual workload of applications running in a  data center. These recommendations will be based on the VCore, memory, and storage requirements of a customer’s typical runtime environment.

Cloud capacity planning and chargeback reporting – Unravel can predict cloud storage requirements up to six months out and can provide a detailed accounting of resource consumption and chargeback by user, department or other criteria.

Migration validation  – Unravel can provide a before and after assessment of cloud applications by comparing on-premises performance and resource consumption to the same metrics in the cloud, thereby validating the relative success of the migration.

All indications point to a massive shift in data deployments to the cloud, but there are too many unknowns around cost, visibility and migration that have prevented this transition to the cloud from occurring more quickly.

We are incredibly proud of this latest release and the value we believe it can deliver as organizations either begin their cloud journey for their modern data applications or look to optimize performance and cost efficiencies for those data workloads already operating in the cloud.

The post Unravel Introduces the First AI Powered DataOps Solution for the Cloud appeared first on Unravel.

Unravel lucked out with the quality and strategic Impact of our advisory board. Collectively, they hold a phenomenal track record of entrepreneurship, leadership, and product innovation, and I am pleased to introduce them to the Unravel community. Looking into the year ahead, we asked two of our advisors for their perspective on what 2019 holds as we dive into the next few quarters.. Our first guest, Herb Cunitz, was featured in Part 1 of our Prediction series (read it here) and discussed breakout modern data stack technologies, the role of artificial intelligence (AI) and automation in the modern data stack, and the increasing modern data stack skills gap. Now, in Part 2, Tasso Argyros, founder, and CEO of ActionIQ will outline his take on the upcoming trends for 2019.

Tasso is a widely recognized and respected innovator, with awards and accolades from the World Economic Forum, BusinessWeek and Forbes, and has a storied career with more than a decade’s experience working with data-focused organizations and advising companies on how to accelerate growth and become market leaders. He is a CEO and Founder at ActionIQ, a company giving Marketers direct access to their customer data, and previously founded Aster Data, a pioneer in the modern data stack, which was ultimately acquired by Teradata. He was also a Founder of early-stage Big Data seed fund, Data Elite, that helped incubate Unravel Data.

In 2018, big data matured and hit a point of inflection, where an increasing number of Fortune 1000 enterprises deployed critical modern data applications that depend on the modern data stack, into production. What effect will this have on the product innovation pipeline and adoption for 2019 and beyond? This is an excerpt of my recent conversation with Tasso:

Unravel: Looking back in the ‘rear-view mirror’ to the past year, what were the most exciting developments and innovations in the modern data stack?

TA: While some say innovation has slowed down in big data, I’m seeing the opposite and believe it has accelerated. When we started Aster Data in 2005, many thought that database innovation was dead. Between Oracle, IBM, and some specialty players like Teradata, investors and pundits believed that all problems had been solved and that there was nothing else to do. Instead, it was the perfect time to start a database company as the seeds of the Big Data revolution were about to be planted.

Since then, the underlying infrastructure have experienced massive, continual changes in about 3 to 4-year intervals. For example, in the mid-2000s, the primary industry trend was moving from expensive proprietary hardware to more cost-effective commodity hardware, and in the early-2010s, the industry spotlighted open source data software. Now, for the past few years, the industry has been focused on the introduction of and transition to cloud solutions, the increasing volume of streaming data and debut of Internet-of-Things (IoT) technologies.

As we focus on finding better ways to manage data, introduce new technologies and databases, and explore the ecosystem that lays on top of the big data layer, these will be the underlying trends that will continue to drive innovation in 2019 and beyond. Whereas initially, big data was more about collection, aggregation and experimentation, in 2018, it became clear that big data is a crucial, mission-critical aspect to the next generation of applications – and there is much more to learn.

Unravel: What breakout technology will move to the forefront in 2019?

TA: There has been a definite increase in the number and variety of data-centric applications (versus data infrastructure) that are being created and in-use today. As a result, there is a rising interest in the industry in learning how to manage data for specific systems and in different environments, including on-premises, hybrid, and across multiple clouds. In 2019, the industry will start empowering these organization with tools that help non-experts become self-sufficient at managing their data operations processes across their end-to-end irrespective of where code is executing.

Unravel: Which industries or use cases have delivered the most value and have seen the most significant adoption of a mature modern data ecosystem?

TA: Digital-native organizations were the first companies to jump in at-scale – which is not a surprise as they have historically advanced more rapidly and been ahead of those who have some form of legacy to consider. Although heavily regulated, financial services institutions saw the value of an effective modern data strategy early-on as well as those industries that struggled with the cost and complexity of traditional data warehousing approaches when the 2008-2009 recession hit. In fact, few realize that the big recession was one of the key catalysts that accelerated the adoption of new modern data stack technologies.

Big data started with a heavy analytics focus– and then, as it matured, turned operational. Now, it’s coming to the point where streaming data is driving innovation, and many different industries and verticals are set to benefit from this next step. For example, one compelling modern data use case is delivering improved customer experiences through real-time customer data gathering, inference and personalization.

Moreover, the convergence of data science and big data has accelerated adoption as it activates the use of big data for critical business decision-making through optimized machine learning. By offering the ability to filter and prepare data, extract insights from large data sets, and capture complex patterns and develop models, big data becomes a critical value driver for modern data application areas like fraud and risk detection, or industries telecom and Healthcare.

Unravel: Is 2019 the year where ‘Big data’ gives way to just ‘Data’, as the lines and technologies between the 2 become increasingly hard to separate. A data-pipeline is a data pipeline after all.

TA: In the early days, there was confusion between big data and data warehousing. Data warehousing was the buzzword during the two decades prior, whereas big data became the hot trend more recently. A data warehouse is a central repository of integrated data – it is rigid and organized. A technology category, such as big data, is instead a means to store and manage large amounts of data – from many different sources, at a lower cost– to make better decisions and more accurate predictions. In short, modern data stack technologies have been more efficient at processing today’s high-volume, highly variable data pipelines that continue to grow at ever increasing rates.

With that in mind however, nothing stands still for very long, especially with technology innovation. The delineation between categories, as with any maturing market continues to evolve and high degrees of fragmentation, often lead by Open Source committers is often juxtaposed with the evolution of existing adopted technologies. SQL is a good example of this where the traditional the landscape of SQL, NoSQL, NewSQL and serverless solutions like AWS Athena start to blur the lines between what is ‘big’ and what is just ‘data’. One thing is for sure, we have come a long way in a short space of time and ‘Big Data’ is much more that on-premises Hadoop.

Unravel: What role will AI and automation, and capabilities like AIOps, play in the modern data stack in the coming year?

TA: Technologies like Spark, Hive and Kafka, are very complex under the hood, and when an application fails, it requires a specialist with a unique skill set to comb through massive amounts of data and determine the cause and solution. Data Operations frameworks need to mature to permit separation of roles rather than relying on a single Data engineer to solve all of the problems. Self-service for the applications owners will relieve part of this bottleneck but fully operationalizing a growing number of production data pipelines will require a different approach that relies heavily on Machine Learning and Artificial Intelligence.

In 2019, as the industry continues to strive for higher efficiency, automation will rise as a solution to the modern data stack skills problem. For example, AI for Operations (AIOps), which combines big data, artificial intelligence, and machine learning functionality, can augment and replace many IT operations processes to e.g. Accelerate the time it takes to identify performance issues, proactively tune resources to reduce cost or Automate configuration changes to prevent an app failures proactivly.

Unravel: What major vendor shake-ups do you predict in 2019?

TA: The industry now understands that there is more to a big data ecosystem than just Hadoop. Hadoop, for many years, was the leading open source framework, but Spark and Kafka’s increasing rise in popularity has proven that the stack will continue to rapidly evolve in ways we have not yet thought of. Complexity will be with us for a very long time and along with that some incredible new innovative companies, a new emerging incumbent (Cloudera/Hortonworks) and the Cloud giants will jockey for customer mindshare.

The post Modern Data Stack Predictions With Unravel Data Advisory Board Member, Tasso Argyros appeared first on Unravel.

The post Unravel Named to CRN’s List of 100 Coolest Cloud Computing Companies appeared first on Unravel.

The post Introducing Unravel 4.5 appeared first on Unravel.

The post Learn how Unravel Complements Apache Ambari for Hortonworks appeared first on Unravel.

The post How Unravel Partners with Hortonworks on HDP appeared first on Unravel.

In essence, Unravel Data makes processing Big Data easier. The program was designed to resolve the complicated and disconcerting problems that emerge when processing Big Data. These applications can become confusing and difficult to operate. Never-before-seen challenges arise with chronic regularity, leaving research teams constantly struggling with issues such as allocating resources, scheduling, and debugging. These redundant issues act to slow down the actual processing, and can make it difficult to use Big Data Applications effectively, or to even profit from it. Unravel Data streamlines the process by detecting, and correcting, concealed defects that block advertising and analytics tracking, and online conversions.

The post Big Data Applications: Managing Complexity with Success appeared first on Unravel.

Unravel Data, which provides an application performance management (APM) platform designed to simplify DataOps, has unveiled a new set of automated actions for improving modern data stack operations and performance.

Based on its work with more than 100 enterprise customers and prospects, the 4.0 release helps make DataOps more proactive and productive by automating problem discovery, root-cause analysis, and resolution across the entire modern data stack, while improving ROI and time to value of big data investments.

The post Unravel Data Advances Application Performance Management for Big Data appeared first on Unravel.

Unravel Data Systems Inc. is updating its application performance management system for modern data stack environments with improved detection of runaway applications, better service level agreement management and enhanced ability to diagnose and recommend fixes for slowdowns and other performance problems.

The post Unravel Data upgrades performance monitor for modern data stacks applications appeared first on Unravel.

The post Unravel Named to CNBC’s 2018 Upstart 100 List appeared first on Unravel.

We are pleased to announce that Unravel is now MapR certified. This marks an important milestone in our continued partnership with MapR, underpinned by a growing demand among MapR users for our modern data stack Application Performance Management (APM) intelligence product.

The certification ensures that Unravel is integrated seamlessly with MapR Converged Data Platform, providing customers with an intelligent product to improve the reliability and performance of their modern data stack applications and optimize overall cluster resource usage.

Deploying Unravel on a MapR cluster is a simple two-step process:

• Step 1: Deploy Unravel Server on a client node machine which connects to relevant services like YARN, Hive, Oozie, Spark etc., and enables access to the Unravel Web UI.
• Step 2: Deploy Unravel Sensors (for Hive and Spark). Please note that the sensors are simply jar files (that provide additional metadata on Hive and Spark applications) and do not run as root on the deployed machines.

Unravel works smoothly on MapR environments with MapR Tickets/Certificates, encryption and other security configurations enabled. In our experience, getting Unravel up and running on MapR environments takes less than an hour, even for large scale clusters greater than 100+ nodes.

Learn more about why APM for the modern data stack is mission critical for big data in production.

The post Unravel Now Certified on MapR Converged Data Platform appeared first on Unravel.

Big Data is no longer a side project. Hadoop, Spark, Kafka, and NoSQL systems are slowly becoming part of the core IT fabric in most organizations. From product usage reports to recommendation engines, organizations are now running various types of Big Data applications in production to provide their customers greater value than ever before.

But Big Data applications are not easy to run, and ongoing operations management presents organizations with never-before-seen challenges. Application developers often complain about applications missing their delivery commitments (SLAs) or failing outright. Meanwhile, Big Data operations teams struggle with everyday tasks such as debugging, scheduling, and allocating resources. These ongoing management and performance challenges make it difficult for organizations to rely on their Big Data investment – let alone profit from it.

Shivnath Babu and I saw that managing the chaos and complexity of Big Data systems was taking up the majority of the time spent by Big Data professionals — versus working to deliver results to the business from this Big Data stack. We also saw that these problems weren’t unique to one organization, but were common in companies employing Big Data technology. Complicating matters, the Big Data ecosystem is expanding at such a rapid pace that practitioners are unable to keep up. Lack of expertise is often cited as one of the primary reasons for Big Data projects failing or slowing down.

We thought there had to be a better way to cope with this complexity so that enterprises could focus their attention on delivering value quickly using their Big Data stack, and set out on a mission to radically simplify Big Data operations.

Companies have engineers responsible for Big Data operations. Their job is to keep clusters healthy and Big Data applications performing reliably and efficiently. Today, they use a fragmented set of tools, command-line interfaces, and home-developed scripts to keep an eye on their Big Data stack. However, these fragmented tools fail to show the complete picture, making it very hard to root-cause and solve problems. For example, to troubleshoot the slowdown in the performance of a critical Big Data application, engineers have to look at the entire stack since the root cause could be in the application code, configuration settings, data layout, or resource allocation.

Therefore, we had to create a management solution that wasn’t looking only at one part of the stack, but one that was holistic. We also had to do more than simply provide charts and dashboards, which most users cannot make sense of; we had to provide ‘performance intelligence’ which would simplify the process of ongoing management and make data teams more productive.

Creating software like Unravel requires a mix of both industry experience as well as deep scientific knowledge. Therefore, we have assembled a team of innovators who previously worked at companies such as Cloudera, Oracle, IBM, Netflix, and scientists from Duke University, IIT, MIT, and Stanford. Together the Unravel team brings the needed experience in distributed computing and enterprise software that is crucial to solving this major problem that our industry faces today.

I am excited to announce that Unravel is already being used in production by several leading web and Fortune 100 companies today. We couldn’t be happier to see that Unravel is helping organizations rely on Big Data by ensuring that applications are fast and error-free and that the underlying cluster is being utilized to its full potential.

See how Unravel is mission critical to run big data in production here!

The post Why We Created Unravel Data for Big Data APM appeared first on Unravel.