As organizations invest ever more heavily in modernizing their data stacks, data teams—the people who actually deliver the value of data to the business—are finding it increasingly difficult to manage the performance, cost, and quality of these complex systems.

Data teams today find themselves in much the same boat as software teams were 10+ years ago. Software teams have dug themselves out the hole with DevOps best practices and tools—chief among them full-stack observability.

But observability for data applications is a completely different animal from observability for web applications. Effective DataOps demands observability that’s designed specifically for the different challenges facing different data team members throughout the DataOps lifecycle.

That’s data observability for DataOps.

In this white paper, you’ll learn:

• What is data observability for DataOps?
• What data teams can learn from DevOps
• Why observability designed for web apps is insufficient for data apps
• What data teams need from data observability for DataOps
• The 5 capabilities data observability for DataOps

Download the white paper here.

The post Data Observability: The Missing Link for Data Teams appeared first on Unravel.

Spark has become one of the most important tools for processing data – especially non-relational data – and deriving value from it. And Spark serves as a platform for the creation and delivery of analytics, AI, and machine learning applications, among others. But troubleshooting Spark applications is hard – and we’re here to help.

In this blog post, we’ll describe ten challenges that arise frequently in troubleshooting Spark applications. We’ll start with issues at the job level, encountered by most people on the data team – operations people/administrators, data engineers, and data scientists, as well as analysts. Then, we’ll look at problems that apply across a cluster. These problems are usually handled by operations people/administrators and data engineers.

For more on Spark and its use, please see this piece in Infoworld. And for more depth about the problems that arise in creating and running Spark jobs, at both the job level and the cluster level, please see the links below.

Five Reasons Why Troubleshooting Spark Applications is Hard

Some of the things that make Spark great also make it hard to troubleshoot. Here are some key Spark features, and some of the issues that arise in relation to them:

Memory-resident

Spark gets much of its speed and power by using memory, rather than disk, for interim storage of source data and results. However, this can cost a lot of resources and money, which is especially visible in the cloud. It can also make it easy for jobs to crash due to lack of sufficient available memory. And it makes problems hard to diagnose – only traces written to disk survive after crashes.

Parallel processing

Spark takes your job and applies it, in parallel, to all the data partitions assigned to your job. (You specify the data partitions, another tough and important decision.) But when a processing workstream runs into trouble, it can be hard to find and understand the problem among the multiple workstreams running at once.

Variants

Spark is open source, so it can be tweaked and revised in innumerable ways. There are major differences between the Spark 1 series, Spark 2.x, and the newer Spark 3. And Spark works somewhat differently across platforms – on-premises; on cloud-specific platforms such as AWS EMR, Azure HDInsight, and Google Dataproc; and on Databricks, which is available across the major public clouds. Each variant offers some of its own challenges and a somewhat different set of tools for solving them.

Configuration options

Spark has hundreds of configuration options. And Spark interacts with the hardware and software environment it’s running in, each component of which has its own configuration options. Getting one or two critical settings right is hard; when several related settings have to be correct, guesswork becomes the norm, and over-allocation of resources, especially memory and CPUs (see below) becomes the safe strategy.

Trial and error approach

With so many configuration options, how to optimize? Well, if a job currently takes six hours, you can change one, or a few, options, and run it again. That takes six hours, plus or minus. Repeat this three or four times, and it’s the end of the week. You may have improved the configuration, but you probably won’t have exhausted the possibilities as to what the best settings are.

Sparkitecture diagram – the Spark application is the Driver Process, and the job is split up across executors. (Source: Apache Spark for the Impatient on DZone.)

Three Issues with Spark Jobs, On-Premises and in the Cloud

Spark jobs can require troubleshooting against three main kinds of issues:

1. Failure. Spark jobs can simply fail. Sometimes a job will fail on one try, then work again after a restart. Just finding out that the job failed can be hard; finding out why can be harder. (Since the job is memory-resident, failure makes the evidence disappear.)
2. Poor performance. A Spark job can run slower than you would like it to; slower than an external service level agreement (SLA); or slower than it would do if it were optimized. It’s very hard to know how long a job “should” take, or where to start in optimizing a job or a cluster.
3. Excessive cost or resource use. The resource use or, especially in the cloud, the hard dollar cost of a job may raise concerns. As with performance, it’s hard to know how much the resource use and cost “should” be, until you put work into optimizing and see where you’ve gotten to.

All of the issues and challenges described here apply to Spark across all platforms, whether it’s running on-premises, in Amazon EMR, or on Databricks (across AWS, Azure, or GCP). However, there are a few subtle differences:

• Move to cloud. There is a big movement of big data workloads from on-premises (largely running Spark on Hadoop) to the cloud (largely running Spark on Amazon EMR or Databricks). Moving to cloud provides greater flexibility and faster time to market, as well as access to built-in services found on each platform.
• Move to on-premises. There is a small movement of workloads from the cloud back to on-premises environments. When a cloud workload “settles down,” such that flexibility is less important, then it may become significantly cheaper to run it on-premises instead.
• On-premises concerns. Resources (and costs) on-premises tend to be relatively fixed; there can be a leadtime of months to years to significantly expand on-premises resources. So the main concern on-premises is maximizing the existing estate: making more jobs run in existing resources, and getting jobs complete reliably and on-time, to maximize the pay-off from the existing estate.
• Cloud concerns. Resources in the cloud are flexible and “pay as you go” – but as you go, you pay. So the main concern in the cloud is managing costs. (As AWS puts it, “When running big data pipelines on the cloud, operational cost optimization is the name of the game.”) This concern increases because reliability concerns in the cloud can often be addressed by “throwing hardware at the problem” – increasing reliability but at a greater cost.
• On-premises Spark vs Amazon EMR. When moving to Amazon EMR, it’s easy to do a “lift and shift” from on-premises Spark to EMR. This saves time and money on the cloud migration effort, but any inefficiencies in the on-premises environment are reproduced in the cloud, increasing costs. It’s also fully possible to refactor before moving to EMR, just as with Databricks.
• On-premises Spark vs Databricks. When moving to Databricks, most companies take advantage of Databricks’ capabilities, such as ease of starting/shutting down clusters, and do at least some refactoring as part of the cloud migration effort. This costs time and money in the cloud migration effort, but results in lower costs and, potentially, greater reliability for the refactored job in the cloud.

All of these concerns are accompanied by a distinct lack of needed information. Companies often make crucial decisions – on-premises vs. cloud, EMR vs. Databricks, “lift and shift” vs. refactoring – with only guesses available as to what different options will cost in time, resources, and money.

The Biggest Spark Troubleshooting Challenges in 2024

Many Spark challenges relate to configuration, including the number of executors to assign, memory usage (at the driver level, and per executor), and what kind of hardware/machine instances to use. You make configuration choices per job, and also for the overall cluster in which jobs run, and these are interdependent – so things get complicated, fast.

Some challenges occur at the job level; these challenges are shared right across the data team. They include:

1. How many executors should each job use?
2. How much memory should I allocate for each job?
3. How do I find and eliminate data skew?
4. How do I make my pipelines work better?
5. How do I know if a specific job is optimized?

Other challenges come up at the cluster level, or even at the stack level, as you decide what jobs to run on what clusters. These problems tend to be the remit of operations people and data engineers. They include:

1. How do I size my nodes, and match them to the right servers/instance types?
2. How do I see what’s going on across the Spark stack and apps?
3. Is my data partitioned correctly for my SQL queries?
4. When do I take advantage of auto-scaling?
5. How do I get insights into jobs that have problems?

Section 1: Five Job-Level Challenges

These challenges occur at the level of individual jobs. Fixing them can be the responsibility of the developer or data scientist who created the job, or of operations people or data engineers who work on both individual jobs and at the cluster level.

However, job-level challenges, taken together, have massive implications for clusters, and for the entire data estate. One of our Unravel Data customers has undertaken a right-sizing program for resource-intensive jobs that has clawed back nearly half the space in their clusters, even though data processing volume and jobs in production have been increasing.

For these challenges, we’ll assume that the cluster your job is running in is relatively well-designed (see next section); that other jobs in the cluster are not resource hogs that will knock your job out of the running; and that you have the tools you need to troubleshoot individual jobs.

1. How many executors and cores should a job use?

One of the key advantages of Spark is parallelization – you run your job’s code against different data partitions in parallel workstreams, as in the diagram below. The number of workstreams that run at once is the number of executors, times the number of cores per executor. So how many executors should your job use, and how many cores per executor – that is, how many workstreams do you want running at once?

A Spark job uses three cores to parallelize output. Up to three tasks run simultaneously, and seven tasks are completed in a fixed period of time. (Source: Lisa Hua, Spark Overview, Slideshare.)

You want high usage of cores, high usage of memory per core, and data partitioning appropriate to the job. (Usually, partitioning on the field or fields you’re querying on.) This beginner’s guide for Hadoop suggests two-three cores per executor, but not more than five; this expert’s guide to Spark tuning on AWS suggests that you use three executors per node, with five cores per executor, as your starting point for all jobs. (!)

You are likely to have your own sensible starting point for your on-premises or cloud platform, the servers or instances available, and the experience your team has had with similar workloads. Once your job runs successfully a few times, you can either leave it alone or optimize it. We recommend that you optimize it, because optimization:

• Helps you save resources and money (not over-allocating)
• Helps prevent crashes, because you right-size the resources (not under-allocating)
• Helps you fix crashes fast, because allocations are roughly correct, and because you understand the job better

2. How much memory should I allocate for each job?

Memory allocation is per executor, and the most you can allocate is the total available in the node. If you’re in the cloud, this is governed by your instance type; on-premises, by your physical server or virtual machine. Some memory is needed for your cluster manager and system resources (16GB may be a typical amount), and the rest is available for jobs.

If you have three executors in a 128GB cluster, and 16GB is taken up by the cluster, that leaves 37GB per executor. However, a few GB will be required for executor overhead; the remainder is your per-executor memory. You will want to partition your data so it can be processed efficiently in the available memory.

This is just a starting point, however. You may need to be using a different instance type, or a different number of executors, to make the most efficient use of your node’s resources against the job you’re running. As with the number of executors (see the previous section), optimizing your job will help you know whether you are over- or under-allocating memory, reduce the likelihood of crashes, and get you ready for troubleshooting when the need arises.

For more on memory management, see this widely read article, Spark Memory Management, by our own Rishitesh Mishra.

3. How do I handle data skew and small files?

Data skew and small files are complementary problems. Data skew tends to describe large files – where one key-value, or a few, have a large share of the total data associated with them. This can force Spark, as it’s processing the data, to move data around in the cluster, which can slow down your task, cause low utilization of CPU capacity, and cause out-of-memory errors which abort your job. Several techniques for handling very large files which appear as a result of data skew are given in the popular article, Data Skew and Garbage Collection, by Rishitesh Mishra of Unravel.

Small files are partly the other end of data skew – a share of partitions will tend to be small. And Spark, since it is a parallel processing system, may generate many small files from parallel processes. Also, some processes you use, such as file compression, may cause a large number of small files to appear, causing inefficiencies. You may need to reduce parallelism (undercutting one of the advantages of Spark), repartition (an expensive operation you should minimize), or start adjusting your parameters, your data, or both (see details here).

Both data skew and small files incur a meta-problem that’s common across Spark – when a job slows down or crashes, how do you know what the problem was? We will mention this again, but it can be particularly difficult to know this for data-related problems, as an otherwise well-constructed job can have seemingly random slowdowns or halts, caused by hard-to-predict and hard-to-detect inconsistencies across different data sets.

4. How do I optimize at the pipeline level?

Spark pipelines are made up of dataframes, connected by transformers (which calculate new data from existing data), and Estimators. Pipelines are widely used for all sorts of processing, including extract, transform, and load (ETL) jobs and machine learning. Spark makes it easy to combine jobs into pipelines, but it does not make it easy to monitor and manage jobs at the pipeline level. So it’s easy for monitoring, managing, and optimizing pipelines to appear as an exponentially more difficult version of optimizing individual Spark jobs.

Existing Transformers create new Dataframes, with an Estimator producing the final model. (Source: Spark Pipelines: Elegant Yet Powerful, InsightDataScience.)

Many pipeline components are “tried and trusted” individually, and are thereby less likely to cause problems than new components you create yourself. However, interactions between pipeline steps can cause novel problems.

Just as job issues roll up to the cluster level, they also roll up to the pipeline level. Pipelines are increasingly the unit of work for DataOps, but it takes truly deep knowledge of your jobs and your cluster(s) for you to work effectively at the pipeline level. This article, which tackles the issues involved in some depth, describes pipeline debugging as an “art.”

5. How do I know if a specific job is optimized?

Neither Spark nor, for that matter, SQL is designed for ease of optimization. Spark comes with a monitoring and management interface, Spark UI, which can help. But Spark UI can be challenging to use, especially for the types of comparisons – over time, across jobs, and across a large, busy cluster – that you need to really optimize a job. And there is no “SQL UI” that specifically tells you how to optimize your SQL queries.

There are some general rules. For instance, a “bad” – inefficient – join can take hours. But it’s very hard to find where your app is spending its time, let alone whether a specific SQL command is taking a long time, and whether it can indeed be optimized.

Spark’s Catalyst optimizer, described here, does its best to optimize your queries for you. But when data sizes grow large enough, and processing gets complex enough, you have to help it along if you want your resource usage, costs, and runtimes to stay on the acceptable side.

Section 2: Cluster-Level Challenges

Cluster-level challenges are those that arise for a cluster that runs many (perhaps hundreds or thousands) of jobs, in cluster design (how to get the most out of a specific cluster), cluster distribution (how to create a set of clusters that best meets your needs), and allocation across on-premises resources and one or more public, private, or hybrid cloud resources.

The first step toward meeting cluster-level challenges is to meet job-level challenges effectively, as described above. A cluster that’s running unoptimized, poorly understood, slowdown-prone, and crash-prone jobs are impossible to optimize. But if your jobs are right-sized, cluster-level challenges become much easier to meet. (Note that Unravel Data, as mentioned in the previous section, helps you find your resource-heavy Spark jobs, and optimize those first. It also does much of the work of troubleshooting and optimization for you.)

Meeting cluster-level challenges for Spark may be a topic better suited for a graduate-level computer science seminar than for a blog post, but here are some of the issues that come up, and a few comments on each:

6. Are Nodes Matched Up to Servers or Cloud Instances?

A Spark node – a physical server or a cloud instance – will have an allocation of CPUs and physical memory. (The whole point of Spark is to run things in actual memory, so this is crucial.) You have to fit your executors and memory allocations into nodes that are carefully matched to existing resources, on-premises, or in the cloud. (You can allocate more or fewer Spark cores than there are available CPUs, but matching them makes things more predictable, uses resources better, and may make troubleshooting easier.)

On-premises, poor matching between nodes, physical servers, executors, and memory results in inefficiencies, but these may not be very visible; as long as the total physical resource is sufficient for the jobs running, there’s no obvious problem. However, issues like this can cause data centers to be very poorly utilized, meaning there’s big overspending going on – it’s just not noticed. (Ironically, the impending prospect of cloud migration may cause an organization to freeze on-prem spending, shining a spotlight on costs and efficiency.)

In the cloud, “pay as you go” pricing shines a different type of spotlight on efficient use of resources – inefficiency shows up in each month’s bill. You need to match nodes, cloud instances, and job CPU and memory allocations very closely indeed, or incur what might amount to massive overspending. This article gives you some guidelines for running Apache Spark cost-effectively on AWS EC2 instances and is worth a read even if you’re running on-premises, or on a different cloud provider.

You still have big problems here. In the cloud, with costs both visible and variable, cost allocation is a big issue. It’s hard to know who’s spending what, let alone what the business results that go with each unit of spending are. But tuning workloads against server resources and/or instances is the first step in gaining control of your spending, across all your data estates.

7. How Do I See What’s Going on in My Cluster?

“Spark is notoriously difficult to tune and maintain,” according to an article in The New Stack. Clusters need to be “expertly managed” to perform well, or all the good characteristics of Spark can come crashing down in a heap of frustration and high costs. (In people’s time and in business losses, as well as direct, hard dollar costs.)

Key Spark advantages include accessibility to a wide range of users and the ability to run in memory. But the most popular tool for Spark monitoring and management, Spark UI, doesn’t really help much at the cluster level. You can’t, for instance, easily tell which jobs consume the most resources over time. So it’s hard to know where to focus your optimization efforts. And Spark UI doesn’t support more advanced functionality – such as comparing the current job run to previous runs, issuing warnings, or making recommendations, for example.

Logs on cloud clusters are lost when a cluster is terminated, so problems that occur in short-running clusters can be that much harder to debug. More generally, managing log files is itself a big data management and data accessibility issue, making debugging and governance harder. This occurs in both on-premises and cloud environments. And, when workloads are moved to the cloud, you no longer have a fixed-cost data estate, nor the “tribal knowledge” accrued from years of running a gradually changing set of workloads on-premises. Instead, you have new technologies and pay-as-you-go billing. So cluster-level management, hard as it is, becomes critical.

8. Is my data partitioned correctly for my SQL queries? (and other inefficiencies)

Operators can get quite upset, and rightly so, over “bad” or “rogue” queries that can cost way more, in resources or cost, than they need to. One colleague describes a team he worked on that went through more than $100,000 of cloud costs in a weekend of crash-testing a new application – a discovery made after the fact. (But before the job was put into production, where it would have really run up some bills.)

SQL is not designed to tell you how much a query is likely to cost, and more elegant-looking SQL queries (ie, fewer statements) may well be more expensive. The same is true of all kinds of code you have running.

So you have to do some or all of three things:

• Learn something about SQL, and about coding languages you use, especially how they work at runtime
• Understand how to optimize your code and partition your data for good price/performance
• Experiment with your app to understand where the resource use/cost “hot spots” are, and reduce them where possible

All this fits in the “optimize” recommendations from 1. and 2. above. We’ll talk more about how to carry out optimization in Part 2 of this blog post series.

9. When do I take advantage of auto-scaling?

The ability to auto-scale – to assign resources to a job just while it’s running, or to increase resources smoothly to meet processing peaks – is one of the most enticing features of the cloud. It’s also one of the most dangerous; there is no practical limit to how much you can spend. You need some form of guardrails, and some form of alerting, to remove the risk of truly gigantic bills.

The need for auto-scaling might, for instance, determine whether you move a given workload to the cloud, or leave it running, unchanged, in your on-premises data center. But to help an application benefit from auto-scaling, you have to profile it, then cause resources to be allocated and de-allocated to match the peaks and valleys. And you have some calculations to make because cloud providers charge you more for spot resources – those you grab and let go of, as needed – than for persistent resources that you keep running for a long time. Spot resources may cost two or three times as much as dedicated ones.

The first step, as you might have guessed, is to optimize your application, as in the previous sections. Auto-scaling is a price/performance optimization, and a potentially resource-intensive one. You should do other optimizations first.

Then profile your optimized application. You need to calculate ongoing and peak memory and processor usage, figure out how long you need each, and the resource needs and cost for each state. And then decide whether it’s worth auto-scaling the job, whenever it runs, and how to do that. You may also need to find quiet times on a cluster to run some jobs, so the job’s peaks don’t overwhelm the cluster’s resources.

To help, Databricks has two types of clusters, and the second type works well with auto-scaling. Most jobs start out in an interactive cluster, which is like an on-premises cluster; multiple people use a set of shared resources. It is, by definition, very difficult to avoid seriously underusing the capacity of an interactive cluster.

So you are meant to move each of your repeated, resource-intensive, and well-understood jobs off to its own, dedicated, job-specific cluster. A job-specific cluster spins up, runs its job, and spins down. This is a form of auto-scaling already, and you can also scale the cluster’s resources to match job peaks, if appropriate. But note that you want your application profiled and optimized before moving it to a job-specific cluster.

10. How Do I Find and Fix Problems?

Just as it’s hard to fix an individual Spark job, there’s no easy way to know where to look for problems across a Spark cluster. And once you do find a problem, there’s very little guidance on how to fix it. Is the problem with the job itself, or the environment it’s running in? For instance, over-allocating memory or CPUs for some Spark jobs can starve others. In the cloud, the noisy neighbors problem can slow down a Spark job run to the extent that it causes business problems on one outing – but leaves the same job to finish in good time on the next run.

The better you handle the other challenges listed in this blog post, the fewer problems you’ll have, but it’s still very hard to know how to most productively spend Spark operations time. For instance, a slow Spark job on one run may be worth fixing in its own right and may be warning you of crashes on future runs. But it’s very hard just to see what the trend is for a Spark job in performance, let alone to get some idea of what the job is accomplishing vs. its resource use and average time to complete. So Spark troubleshooting ends up being reactive, with all too many furry, blind little heads popping up for operators to play Whack-a-Mole with.

Impacts of these Challenges

If you meet the above challenges effectively, you’ll use your resources efficiently and cost-effectively. However, our observation here at Unravel Data is that most Spark clusters are not run efficiently.

What we tend to see most are the following problems – at a job level, within a cluster, or across all clusters:

• Under-allocation. It can be tricky to allocate your resources efficiently on your cluster, partition your datasets effectively, and determine the right level of resources for each job. If you under-allocate (either for a job’s driver or the executors), a job is likely to run too slowly or crash. As a result, many developers and operators resort to…
• Over-allocation. If you assign too many resources to your job, you’re wasting resources (on-premises) or money (cloud). We hear about jobs that need, for example, 2GB of memory but are allocated much more – in one case, 85GB.

Applications can run slowly, because they’re under-allocated – or because some apps are over-allocated, causing others to run slowly. Data teams then spend much of their time fire-fighting issues that may come and go, depending on the particular combination of jobs running that day. With every level of resource in shortage, new, business-critical apps are held up, so the cash needed to invest against these problems doesn’t show up. IT becomes an organizational headache, rather than a source of business capability.

Conclusion

To jump ahead to the end of this series a bit, our customers here at Unravel are easily able to spot and fix over-allocation and inefficiencies. They can then monitor their jobs in production, finding and fixing issues as they arise. Developers even get on board, checking their jobs before moving them to production, then teaming up with Operations to keep them tuned and humming.

One Unravel customer, Mastercard, has been able to reduce usage of their clusters by roughly half, even as data sizes and application density has moved steadily upward during the global pandemic. And everyone gets along better, and has more fun at work, while achieving these previously unimagined results.

So, whether you choose to use Unravel or not, develop a culture of right-sizing and efficiency in your work with Spark. It will seem to be a hassle at first, but your team will become much stronger, and you’ll enjoy your work life more, as a result.

You need a sort of X-ray of your Spark jobs, better cluster-level monitoring, environment information, and to correlate all of these sources into recommendations. In Troubleshooting Spark Applications, Part 2: Solutions, we will describe the most widely used tools for Spark troubleshooting – including the Spark Web UI and our own offering, Unravel Data – and how to assemble and correlate the information you need.

The post The Biggest Spark Troubleshooting Challenges in 2024 appeared first on Unravel.

Read The Unravel Guide to DataOps

The Unravel Guide to DataOps gives you the information and tools you need to sharpen your DataOps practices and increase data application performance, reduce costs, and remove operational headaches. The ten-step DataOps lifecycle shows you everything you need to do to create data products that run and run.

The Guide introduces Unravel Data software as a powerful platform, helping you to create a robust and effective DataOps culture within your organization. Three customer use cases show you the power of DataOps, and Unravel, in the hands of your data teams. As one Unravel customer says in the Guide, “I’m sleeping a lot easier than I did a year ago.”

Download the Guide to help you as you master the current state, and future possibilities, of DataOps adoption in organizations worldwide.

The post Download The Unravel Guide to DataOps appeared first on Unravel.

Spark applications are easy to write and easy to understand when everything goes according to plan. However, it becomes very difficult when Spark applications start to slow down or fail. (See our blog Spark Troubleshooting, Part 1 – Ten Biggest Challenges.) Sometimes a well-tuned application might fail due to a data change, or a data layout change. Sometimes an application that was running well so far, starts behaving badly due to resource starvation. The list goes on and on.

It’s not only important to understand a Spark application, but also its underlying runtime components like disk usage, network usage, contention, etc., so that we can make an informed decision when things go bad.

In this series of articles, I aim to capture some of the most common reasons why a Spark application fails or slows down. The first and most common is memory management.

If we were to get all Spark developers to vote, out-of-memory (OOM) conditions would surely be the number one problem everyone has faced.  This comes as no big surprise as Spark’s architecture is memory-centric. Some of the most common causes of OOM are:

• Incorrect usage of Spark
• High concurrency
• Inefficient queries
• Incorrect configuration

To avoid these problems, we need to have a basic understanding of Spark and our data. There are certain things that can be done that will either prevent OOM or rectify an application which failed due to OOM.  Spark’s default configuration may or may not be sufficient or accurate for your applications. Sometimes even a well-tuned application may fail due to OOM as the underlying data has changed.

Out of memory issues can be observed for the driver node, executor nodes, and sometimes even for the node manager. Let’s take a look at each case.

Out of memory at the driver level

A driver in Spark is the JVM where the application’s main control flow runs. More often than not, the driver fails with an OutOfMemory error due to incorrect usage of Spark. Spark is an engine to distribute workload among worker machines. The driver should only be considered as an orchestrator. In typical deployments, a driver is provisioned less memory than executors. Hence we should be careful what we are doing on the driver.

Common causes which result in driver OOM are:

1. rdd.collect()
2. sparkContext.broadcast
3. Low driver memory configured as per the application requirements
4. Misconfiguration of spark.sql.autoBroadcastJoinThreshold. Spark uses this limit to broadcast a relation to all the nodes in case of a join operation. At the very first usage, the whole relation is materialized at the driver node. Sometimes multiple tables are also broadcasted as part of the query execution.

Try to write your application in such a way that you can avoid all explicit result collection at the driver. You can very well delegate this task to one of the executors. E.g., if you want to save the results to a particular file, either you can collect it at the driver or assign an executor to do that for you.

If you are using Spark’s SQL and the driver is OOM due to broadcasting relations, then either you can increase the driver memory if possible; or else reduce the   “spark.sql.autoBroadcastJoinThreshold” value so that your join operations will use the more memory-friendly sort merge join.

Out of memory at the executor level

This is a very common issue with Spark applications which may be due to various reasons. Some of the most common reasons are high concurrency, inefficient queries, and incorrect configuration. Let’s look at each in turn.

High concurrency

Before understanding why high concurrency might be a cause of OOM, let’s try to understand how Spark executes a query or job and what are the components that contribute to memory consumption.

Spark jobs or queries are broken down into multiple stages, and each stage is further divided into tasks. The number of tasks depends on various factors like which stage is getting executed, which data source is getting read, etc. If it’s a map stage (Scan phase in SQL), typically the underlying data source partitions are honored.

For example, if a hive ORC table has 2000 partitions, then 2000 tasks get created for the map stage for reading the table assuming partition pruning did not come into play. If it’s a reduce stage (Shuffle stage), then spark will use either “spark.default.parallelism” setting for RDDs or “spark.sql.shuffle.partitions” for DataSets for determining the number of tasks. How many tasks are executed in parallel on each executor will depend on “spark.executor.cores” property. If this value is set to a higher value without due consideration to the memory,  executors may fail with OOM. Now let’s see what happens under the hood while a task is getting executed and some probable causes of OOM.

Let’s say we are executing a map task or the scanning phase of SQL from an HDFS file or a Parquet/ORC table. For HDFS files, each Spark task will read a 128 MB block of data.  So if 10 parallel tasks are running, then memory requirement is at least 128 *10 only for storing partitioned data. This is again ignoring any data compression which might cause data to blow up significantly depending on the compression algorithms.

Spark reads Parquet in a vectorized format. To put it simply, each task of Spark reads data from the Parquet file batch by batch. As Parquet is columnar, these batches are constructed for each of the columns.  It accumulates a certain amount of column data in memory before executing any operation on that column. This means Spark needs some data structures and bookkeeping to store that much data. Also, encoding techniques like dictionary encoding have some state saved in memory. All of them require memory.

Figure: Spark task and memory components while scanning a table

So with more concurrency, the overhead increases. Also, if there is a broadcast join involved, then the broadcast variables will also take some memory. The above diagram shows a simple case where each executor is executing two tasks in parallel.

Inefficient queries

While Spark’s Catalyst engine tries to optimize a query as much as possible, it can’t help if the query itself is badly written. E.g., selecting all the columns of a Parquet/ORC table. As seen in the previous section, each column needs some in-memory column batch state. If more columns are selected, then more will be the overhead.

Try to read as few columns as possible. Try to use filters wherever possible, so that less data is fetched to executors. Some of the data sources support partition pruning. If your query can be converted to use partition column(s), then it will reduce data movement to a large extent.

Incorrect configuration

Incorrect configuration of memory and caching can also cause failures and slowdowns in Spark applications. Let’s look at some examples.

Executor & Driver memory

Each application’s memory requirement is different. Depending on the requirement, each app has to be configured differently. You should ensure correct spark.executor.memory or spark.driver.memory values depending on the workload.  As obvious as it may seem, this is one of the hardest things to get right. We need the help of tools to monitor the actual memory usage of the application. Unravel does this pretty well.

Memory Overhead

Sometimes it’s not executor memory, rather its YARN container memory overhead that causes OOM or the node gets killed by YARN. “YARN kill” messages typically look like this:

YARN runs each Spark component like executors and drivers inside containers. Overhead memory is the off-heap memory used for JVM overheads, interned strings and other metadata of JVM. In this case, you need to configure spark.yarn.executor.memoryOverhead to a proper value. Typically 10% of total executor memory should be allocated for overhead.

Caching Memory

If your application uses Spark caching to store some datasets, then it’s worthwhile to consider Spark’s memory manager settings. Spark’s memory manager is written in a very generic fashion to cater to all workloads. Hence, there are several knobs to set it correctly for a particular workload.

Spark has defined memory requirements as two types: execution and storage. Storage memory is used for caching purposes and execution memory is acquired for temporary structures like hash tables for aggregation, joins etc.

Both execution & storage memory can be obtained from a configurable fraction of (total heap memory – 300MB). That setting is “spark.memory.fraction”. Default is 60%. Out of which, by default, 50% is assigned (configurable by “spark.memory.storageFraction”) to storage and rest assigned for execution.

There are situations where each of the above pools of memory, namely execution and storage, may borrow from each other if the other pool is free. Also, storage memory can be evicted to a limit if it has borrowed memory from execution. However, without going into those complexities, we can configure our program such that our cached data which fits in storage memory should not cause a problem for execution.

If we don’t want all our cached data to sit in memory, then we can configure “spark.memory.storageFraction” to a lower value so that extra data would get evicted and execution would not face memory pressure.

Out of memory at Node Manager

Spark applications which do data shuffling as part of group by or join like operations, incur significant overhead. Normally data shuffling process is done by the executor process. If the executor is busy or under heavy GC load, then it can’t cater to the shuffle requests. This problem is alleviated to some extent by using an external shuffle service.

External shuffle service runs on each worker node and handles shuffle requests from executors. Executors can read shuffle files from this service rather than reading from each other. This helps requesting executors to read shuffle files even if the producing executors are killed or slow. Also, when dynamic allocation is enabled, its mandatory to enable external shuffle service.

When Spark external shuffle service is configured with YARN, NodeManager starts an auxiliary service which acts as an External shuffle service provider. By default, NodeManager memory is around 1 GB. However, applications which do heavy data shuffling might fail due to NodeManager going out of memory. Its imperative to properly configure your NodeManager if your applications fall into the above category.

End of Part I – Thanks for the Memory

Spark’s in-memory processing is a key part of its power. Therefore, effective memory management is a critical factor to get the best performance, scalability, and stability from your Spark applications and data pipelines. However, the Spark defaults settings are often insufficient. Depending on the application and environment, certain key configuration parameters must be set correctly to meet your performance goals. Having a basic idea about them and how they can affect the overall application helps.

I have provided some insights into what to look for when considering Spark memory management. This is an area that the Unravel platform understands and optimizes very well, with little, if any, human intervention needed. See how Unravel can make Spark perform better and more reliably here.

Event better, schedule a demo to see Unravel in action. The performance speedups we are seeing for Spark apps are pretty significant.

In Part II of this series Why Your Spark Apps are Slow or Failing: Part II Data Skew and Garbage Collection, I will be discussing how data organization, data skew, and garbage collection impact Spark performance.

The post Why Memory Management is Causing Your Spark Apps To Be Slow or Fail appeared first on Unravel.

Data Actionability: Boost Productivity with Unravel’s New DataOps AI Agent

The post Data Actionability^TM DataOps Webinar appeared first on Unravel.

Data Actionability: Empower Your Team with Unravel’s New AI Agents

The post Data Actionability^TM Webinar appeared first on Unravel.

AI Agents for Data Teams

Data is the driving force of the world’s modern economies, but data teams are struggling to meet demand to support generative AI (GenAI), including rapid data volume growth and the increasing complexity of data pipelines. More than 88% of software engineers, data scientists, and SQL analysts surveyed say they are turning to AI for more effective bug-fixing and troubleshooting. And 84% of engineers who use AI said it frees up their time to focus on high-value activities.

AI Agents represent the next wave of AI innovation and have arrived just in time to help data teams make more efficient use of their limited bandwidth to build, operate, and optimize data pipelines and GenAI applications on modern data platforms.

Data Teams Grapple with High Demand for GenAI

A surge in adoption of new technologies such as GenAI is putting tremendous pressure on data teams, leading to broken apps and burnout. In order to support new GenAI products, data teams must deliver more production data pipelines and data apps, faster. The result is that data teams have too much on their plates, the pipelines are too complex, there is not enough time, and not everyone has the deep tech skills required. No surprise that 70% of organizations have difficulty integrating data into AI models and only 48% of AI projects get deployed into production.

Understanding AI Agents

Defining AI Agents

AI agents are software-based systems that gather information, provide recommended actions, initiate and complete tasks in collaboration with or on behalf of humans to achieve a goal. AI agents can act independently, utilizing components like perception and reasoning, provide step-by-step guidance to augment human abilities, or can provide supporting information to support complex human-led tasks. AI agents play a crucial role in automating tasks and simplifying data-driven decision-making, and achieving greater productivity and efficiency.

How AI Agents Work

AI agents operate by leveraging a wide range of data sources and signals, using algorithms and data processing to identify anomalies and actions, then interact with their environment and users to effectively achieve specific goals. AI agents can achieve >90% accuracy, primarily driven by the reliability, volume, and variety of input data and telemetry to which they have access.

Types of Intelligent Agents

• Reactive and proactive agents are two primary categories of intelligent agents.
• Some agents perform work for you, while others help complete tasks with you or provide information to support your work.
• Each type of intelligent agent has distinct characteristics and applications tailored to specific functions, enhancing productivity and efficiency.

AI for Data Driven Organizations

Enhancing Decision Making

AI agents empower teams by improving data support decision-making processes for you, with you, or by you. Examples of how AI agents act on your behalf include reducing toil and handling routine decisions based on AI insights. In various industries, AI agents optimize decision-making and provide recommendations to support your decisions. For complex tasks, AI agents provide supporting information needed to build data pipelines, write SQL queries, and partition data.

Benefits of broader telemetry sources for AI agents

Integrating telemetry from various platforms and systems enhances AI agents’ ability to provide accurate recommendations. Incorporating AI agents into root cause analysis (RCA) systems offers significant benefits. Meta’s AI-based root cause analysis system shows how AI agents enhance tools and applications.

Overcoming Challenges

Enterprises running modern data stacks face common challenges like high costs, slow performance, and impaired productivity. Leveraging AI agents can automate tasks for you, with you, and by you. Unravel customers such as Equifax, Maersk, and Novartis have successfully overcome these challenges using AI.

The Value of AI Agents for Data Teams

Reducing Costs

When implementing AI agents, businesses benefit from optimized data stacks, reducing operational costs significantly. These agents continuously analyze telemetry data, adapting to new information dynamically. Unravel customers have successfully leveraged AI to achieve operational efficiency and cost savings.

Accelerating Performance

Performance is crucial in data analytics, and AI agents play a vital role in enhancing it. By utilizing these agents, enterprise organizations can make well-informed decisions promptly. Unravel customers have experienced accelerated data analytics performance through the implementation of AI technologies.

Improving Productivity

AI agents are instrumental in streamlining processes within businesses, leading to increased productivity levels. By integrating these agents into workflows, companies witness substantial productivity gains. Automation of repetitive tasks by AI agents simplify troubleshooting to boost overall productivity and efficiency.

Future Trends in AI Agents for FinOps, DataOps, and Data Engineering

Faster Innovation with AI Agents

By 2026 conversational AI will reduce agent labor costs by $80 billion. AI agents are advancing, providing accurate recommendations to address more issues automatically. This allows your team to focus on innovation. For example, companies like Meta use AI agents to simplify root cause analysis (RCA) for complex applications.

Accelerated Data Pipelines with AI Agents

Data processing is shifting towards real-time analytics, enabling faster revenue growth. However, this places higher demands on data teams. Equifax leverages AI to serve over 12 million daily requests in near real time.

Improved Data Analytics Efficiency with AI Agents

Data management is the fastest-growing segment of cloud spending. In the cloud, time is money; faster data processing reduces costs. One of the word’s largest logistics companies improved efficiency by up to 70% in just 6 months using Unravel’s AI recommendations.

Empower Your Team with AI Agents

Harnessing the power of AI agents can revolutionize your business operations, enhancing efficiency, decision-making, and customer experiences. Embrace this technology to stay ahead in the competitive landscape and unlock new opportunities for growth and innovation.

Learn more about our FinOps Agent, DataOps Agent, and Data Engineering Agent.

The post AI Agents: Empower Data Teams With Actionability^TM for Transformative Results appeared first on Unravel.

UNRAVEL DATA SECURITY AND TRUST
ENABLE DATA ACTIONABILITY + FINOPS WITH CONFIDENCE

Privacy and security are top priorities for Unravel and our customers. At Unravel, we help organizations better understand and improve the performance, quality, and cost efficiency of their data and AI pipelines. As a data business, we appreciate the scope and implications of privacy and security threats.

This data sheet provides details to help information security (InfoSec) teams make informed decisions. Specifically, it includes:

• An overview of our approach to security and trust
• An architectural diagram with connectivity descriptions
• Details about Unravel compliance and certifications
• Common questions about Unravel privacy and security

For additional details, please reach out to our security experts.

The post Unravel Data Security and Trust appeared first on Unravel.

Unravel Data, the first AI-enabled data actionability; and FinOps platform built to address the speed and scale of modern data platforms, today announced it has been named in the 2024 Gartner® Market Guide for Data Observability Tools.

According to Gartner, “By 2026, 50% of enterprises implementing distributed data architectures will have adopted data observability tools to improve visibility over the state of the data landscape, up from less than 20% in 2024.”¹

Gartner also notes that “One of the leading causes for the high demand for data observability is the huge demand for D&A leaders to implement emerging technologies, particularly generative AI (GenAI), in their organization.”¹

Unravel’s Perspective

We believe that the recognition of Unravel as a Representative Vendor underscores our commitment to innovation and excellence in providing comprehensive data observability solutions that empower organizations to drive performance, efficiency, and cost-effectiveness in their data operations.

The Importance of Data Observability

Data observability is crucial in today’s data-driven world. It involves monitoring, tracking, and analyzing data to ensure the health and performance of data systems. Effective data observability helps organizations detect and resolve issues quickly, maintain service-level agreements (SLAs), resource efficiency, and speed time-to-market of AI and other data driven initiatives.

Demand is increasing for robust data observability tools to manage the growing complexity of modern data environments. As data volumes and velocities increase, so does the need for tools that can provide comprehensive visibility and actionable insights. Data observability platforms meet these needs by offering advanced features that support the governance and optimization of performance, costs, and productivity.

Figure 1. Options for data observability and FinOps. Source: Unravel Data.

Go Beyond Data Observability to Actionability^TM

Unravel’s unique approach to data observability enables organizations to take action, automate, and streamline toilsome cloud data analytics platform monitoring, troubleshooting, and cost management. Unravel helps data teams overcome key challenges to accelerate time to value.

Figure 2. The critical features of data observability. Source: Gartner¹.

Unravel’s Comprehensive Data Actionability^TM Platform

Unravel Data’s AI-powered actionability^TM platform is designed to provide deep visibility and enable teams to go beyond observability and take action to improve their modern data stacks, including platforms like Databricks, Snowflake, and BigQuery. AI-powered insights and automation help organizations optimize their data pipelines and infrastructure.

Accelerating Value Across the Organization

Although data observability is typically thought of as being synonymous with data quality, the role of data observability is expanding. As per Gartner, “The increasing complexity in data ecosystems will favor comprehensive data observability tools that can provide additional utilities beyond the monitoring and detection of data issues across platforms.”1 Data observability now includes a variety of new observation categories, including data content, data flow and pipelines, infrastructure and compute, users, usage, and utilization1.

With the introduction of its new AI agents—Unravel FinOps AI Agent, Unravel DataOps AI Agent, and Unravel Data Engineering AI Agent—Unravel enables each role within the organization to simplify and automate recommended actions.

These AI agents are built to tackle specific challenges faced by data teams:

Unravel FinOps AI Agent: This agent provides real-time insights into cloud expenditures, identifying cost-saving opportunities and ensuring budget adherence. It automates financial governance, allowing organizations to manage their data-related costs effectively.

Unravel DataOps AI Agent: By automating routine tasks such as data pipeline monitoring and anomaly detection, this agent frees up human experts to focus on strategic endeavors, enhancing overall productivity and efficiency.

Unravel Data Engineering AI Agent: This agent reduces the burden of mundane tasks for data engineers, enabling them to concentrate on high-value problem-solving. It enhances productivity and ensures precise and reliable data operations.

Figure 3. Unravel’s differentiated value. Source: Unravel Data.

Achieving Data Actionability^TM with Unravel

Unravel’s platform not only observes but also enables teams to take action. By leveraging AI-driven insights, it provides recommendations and automated solutions to optimize data operations. This proactive approach allows organizations to stay ahead of potential issues and continuously improve their data performance and efficiency.

The recent announcement of AI agents exemplifies how Unravel is pushing the boundaries of what data observability can achieve. These agents are designed to lower the barriers to expertise, enabling even under-resourced data teams to operate at peak efficiency.

Get Started Today

Experience the benefits of Unravel’s data actionability^TM platform with a free Health Check Report. This assessment provides insights into the current state of your data operations and quantifies potential improvements to boost performance, cost efficiency, and productivity.

Figure 4. Unravel’s Health Check Report. Source: Unravel Data.

¹Gartner, Market Guide for Data Observability Tools, By Melody Chien, Jason Medd, Lydia Ferguson, Michael Simone, 25 June 2024
GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The post Unravel a Representative Vendor in the 2024 Gartner® Market Guide for Data Observability Tools appeared first on Unravel.

In the data-driven world where real-time decision-making and innovation are not just goals but necessities, global data analytics and technology companies like Equifax must navigate a complex environment to achieve success. Equifax sets the standard for operational excellence by enabling real-time decision-making, accelerating innovation, scaling efficiently, consistently achieving service level agreements (SLAs), and building reliable data pipelines. Let’s delve into the strategies that lead to such a resounding transformation, with a spotlight on the pivotal role of data observability and FinOps.

When Speed Meets Precision: Revolutionizing Credit Scoring

Kumar Menon, the Equifax CTO for Data Mesh and Decision-Making Technologies, faced a formidable challenge: enabling sub-second credit scoring. And he is not alone. “Many organizations find it challenging to turn their data into insights quickly enough to generate real and timely results” (Deloitte). With Unravel’s data observability and FinOps platform, Kumar’s team overcame this hurdle to deliver faster insights and decisions while fortifying Equifax’s competitive edge.

Unifying Data with Vision and Strategy

The necessity to integrate data across six cloud regions cannot be overstated for a company operating at Equifax’s scale. Gartner discovered that many organizations struggle with “high-cost and low-value data integration cycles.” Kumar Menon and his team, leveraging the tools and methodologies of data observability and FinOps, streamlined this intricate process. The result? Faster and more economical products capable of satisfying the needs of a constantly shifting market.

Mastering the Data Deluge

Managing over 10 petabytes of data is no small feat. IDC noted enterprises’ need for “responsiveness, scalability, and resiliency of the digital infrastructure.” Equifax, with Kumar Menon’s foresight, embraced the power of Unravel’s data observability and FinOps frameworks to adapt and grow. By efficiently managing their cloud resource usage, the team was able to scale data processing with the appropriate cloud resources.

Delivering Real-Time Decisions

The Equifax team needed to support 12 million daily inquiries. Operating production services at this scale can be overwhelming for any system not prepared for such a deluge. In fact, Gartner uncovered a significant challenge around the ability to “detect errors and lower the cost to fix and shorten the resulting downtime.” Data observability and FinOps serve as Kumar Menon’s frameworks to not only confront these challenges but to ensure that Equifax can consistently deliver accurate, real-time decisions such as credit scores and employment verifications.

Streamlining Massive Data Ingestion

The Equifax data team’s colossal task of ingesting over 15 billion observations per day could potentially entangle any team in a web of complexity. Again, Gartner articulates the frustration many organizations face with “complex, convoluted data pipelines that require toilsome remediation to manage errors.” Unravel’s platform provided Kumar’s team the means to build and maintain reliable and robust data pipelines, assuring the integrity of Equifax’s reliable and responsive data products.

The Path Forward with Unravel

In a data-centric industry, Equifax exemplifies leadership through precision, agility, and efficiency. The company’s journey demonstrates their capacity to enable real-time decision-making, accelerate innovation, and ensure operational scale and reliability.

At Unravel, we understand and empathize with data teams facing substantial challenges like the ones described above. As your ally in data performance, productivity, and cost management, we’re committed to equipping you with tools that not only remove obstacles but also enhance your operational prowess. Harness the power of data observability and FinOps with Unravel Data through a self-guided tour that shows how you can:

• Deliver faster insights and decisions with Unravel’s pipeline bottleneck analysis.
• Bring faster and more efficient products to market by using Unravel’s speed, cost, and reliability optimizer.
• Scale data processing efficiently with Unravel’s correlated data observability.
• Achieve and exceed your SLAs using Unravel’s out-of-the-box reporting.
• Build performant data pipelines with Unravel’s pipeline bottleneck analysis.

Ready to unlock the full potential of your data strategies and operations? See how you can achieve more with data observability and FinOps with Unravel Data in a self-guided tour.

The post Empowering Data Agility: Equifax’s Journey to Operational Excellence appeared first on Unravel.

A Recap of the Unravel Data Customer Session at Gartner Data & Analytics Summit 2024

The Gartner Data & Analytics Summit 2024 (“D&A”) in London is a pivotal event for Chief Data Analytics Officers (CDAOs) and data and analytics leaders, drawing together a global community eager to dive deep into the insights, strategies, and frameworks that are shaping the future of data and analytics. With an unprecedented assembly of over 3,000 attendees, spanning 150+ knowledge-packed sessions and joined by 90+ innovative partners, the D&A Summit was designed to catalyze data transformation within organizations, offering attendees unique insights to think big and drive real change.

The Unravel Data customer session, titled “How to Stop Burning Money (or At Least Slow the Burn)”, emerged as a highlight of the D&A Summit, drawing attention to the pressing need for cost-efficient data processing in today’s rapid digital evolution. The session—presented by one of the largest logistics giants globally, including over 100,000 employees and a fleet of 700+ container vessels operating across 130 countries—captivated CDAOs & data and analytics leaders from over 150+ attendees. The audience was from 140+ companies across 30+ industries such as banking, retail, and pharma, spanning 110+ cities across 20+ countries. This compelling turnout underscored the universal relevance and urgency of cost-optimization strategies in data engineering. Watch the highlight reel here.

The session was presented by Peter Rees, Director of GenAI, AI, Data and Integration at Maersk, who garnered unprecedented accolades, including a 170% higher-than-average attendance. Peter Rees’ session was the third highest-rated of all 40 partner sessions. These results reflect the session’s relevance and the invaluable insights shared on revolutionizing the efficiency of data processing pipelines, cost allocation, and optimization techniques.
The Gartner Data & Analytics Summit 2024, and particularly the Unravel Data customer session, brought together organizations striving to align their data engineering costs with the value of their data and analytics. Unravel Data’s innovative approach, showcased through the success of a world-leading logistics company, provides a blueprint for organizations across industries looking to dramatically enhance the speed, productivity, and efficiency of their data processing and AI investments.

We invite you to explore how your organization can benefit from Unravel Data’s groundbreaking platform. Take the first step towards transforming your data processing strategy by scheduling an Unravel Data Health Check. Embark on your journey towards optimal efficiency and cost savings today.

The post How to Stop Burning Money (or at Least Slow the Burn) appeared first on Unravel.

In a recently published report, “Beyond FinOps: Optimizing Your Public Cloud Costs”, Gartner shares a graphic which is adapted from Unravel.

Unravel’s Perspective

How FinOps Helps

Unravel helps organizations adopt a FinOps approach to improve cloud data spending efficiency. FinOps helps organizations address overspend, including under- and over-provisioned cloud services, suboptimal architectures, and inefficient pricing strategies. Infrastructure and operations (I&O) leaders and practitioners can use FinOps principles to optimize cloud service design, configuration, and spending commitments to reduce costs. Data and analytics (D&A) leaders and teams are using FinOps to achieve predictable spend for their cloud data platforms.

Introducing Cloud Financial Management and Governance

Cloud Financial Management (CFM) and Governance empowers organizations to quickly adapt in today’s dynamic landscape, ensuring agility and competitiveness. CFM principles help organizations take advantage of the cloud’s variable consumption model through purpose-built tools tailored for your modern cloud finance needs. A well-defined cloud cost governance model lets cloud users monitor and manage their cloud costs and balance cloud spending vs. performance and end-user experience.

Three Keys to Optimizing Your Modern Data Stack

1. Cloud data platform usage analysis plays a crucial role in cloud financial management by providing insights into usage patterns, cost allocation, and optimization opportunities. By automatically gathering, analyzing, and correlating data from various sources, such as traces, logs, metrics, source code, and configuration files, organizations can identify areas for cost savings and efficiency improvements. Unravel’s purpose-built AI reduces the toil required to manually examine metrics, such as resource utilization, spending trends, and performance benchmarks for modern data stacks such as Databricks, Snowflake, and BigQuery.
2. Cost allocation, showback, and chargeback are essential for effective cloud cost optimization. Organizations need to accurately assign costs to different departments or projects based on their actual resource consumption. This ensures accountability and helps in identifying areas of overspending or underutilization. Automated tools and platforms like Unravel can streamline the cost allocation process for cloud data platforms such as Databricks, Snowflake, and BigQuery, making it easier to track expenses and optimize resource usage.
3. Budget forecasting and management is another critical aspect of cloud financial management. By analyzing historical data and usage trends, organizations can predict future expenses more accurately. This enables them to plan budgets effectively, avoid unexpected costs, and allocate resources efficiently. Implementing robust budget forecasting processes can help organizations achieve greater financial control and optimize their cloud spending.

Next Steps

You now grasp the essence of cloud financial management and governance to optimize cloud spending and your cloud data platform. Start your journey and embrace these concepts to enhance your cloud strategy and drive success. Take charge of your Databricks, Snowflake, and BigQuery optimization today with a free health check.

Gartner, Beyond FinOps: Optimizing Your Public Cloud Costs, By Marco Meinardi, 21 March 2024

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The post Unravel Data was Mentioned in a Graphic source in the 2024 Gartner® Report appeared first on Unravel.

Harnessing Data Observability for AI-Driven Innovation

Organizations are now embarking on a journey to harness AI for significant business advancements, from new revenue streams to productivity gains. However, the complexity of delivering AI-powered software efficiently and reliably remains a challenge. With AI investments expected to surge beyond $520 billion by 2027, this brief underscores the necessity for a robust intelligence architecture, scalable digital operations, and specialized skills. Learn how AI-driven data observability can be leveraged as a strategic asset for businesses aiming to lead in innovation and operational excellence.

Get a copy of the IDC Analyst Brief by Research Director Nancy Gohring.

The post IDC Analyst Brief: The Role of Data Observability and Optimization in Enabling AI-Driven Innovation appeared first on Unravel.

Business leaders from every industry now find themselves under the gun to somehow, someway leverage AI into an actual product that companies (and individuals) can use. Yet, an estimated 70%-85% of artificial intelligence (AI) and machine learning (ML) projects fail.

In our thought-leadership white paper Winning the AI Innovation Race: How AI Helps Optimize Speed to Market and Cost Inefficiencies of AI Innovation, you will learn:

• Top pitfalls that impede speed and ROI for running AI and data pipelines in the cloud

• How the answers to addressing these impediments can be found at the code level

• How AI is paramount for optimization of cloud data workloads

• How Unravel helps

The post Winning the AI Innovation Race appeared first on Unravel.

AI-DRIVEN DATA OBSERVABILITY + FINOPS FOR SNOWFLAKE DATA ENGINEERS

Snowflake data engineers are under enormous pressure to deliver results. This data sheet provides more context about the challenges data engineers face and how Unravel helps them address these challenges.

Specifically, it discusses:

• Key Snowflake data engineering roadblocks
• Unravel’s purpose-built AI for Snowflake
• Data engineering benefits

With Unravel, Snowflake data engineers can speed data pipeline development and analytics initiatives with granular and real-time cost visibility, predictive, predictive spend forecasting, and performance insights for their data cloud.

To see Unravel Data for Snowflake in action contact: Data experts

The post Data Observability + FinOps for Snowflake Engineers appeared first on Unravel.

5 key differences between Overwatch and Unravel’s free observability for Databricks.

Before investing significant time and effort to configure Overwatch, compare how Unravel’s free data observability solution gives you end-to-end, real-time granular visibility into your Databricks Lakehouse platform out of the box.

The post Open Source Overwatch VS Unravel Free Comparison Infographic appeared first on Unravel.

In today’s data-driven world, there is a huge amount of data flowing into the business. Engineers spend a large part of their time in building pipelines—to collect the data from different sources, process it, and transform it to useful datasets that can be sent to business intelligence applications or machine learning models. Tools like Airflow are used to orchestrate complex data pipelines by programmatically authoring, scheduling, and monitoring the workflow pipelines. From a 10,000-foot view, Airflow may look like just a powerful cron, but it has additional capabilities like monitoring, generating logs, retrying jobs, adding dependencies between tasks, enabling a huge number of operators to run different types of tasks, etc. 

Pipelines become increasingly complex because of the interdependence of several tools needed to complete a workflow. Common tasks done in day-to-day pipeline activities call for a number of different technologies to process data for different purposes: running a Spark job, executing a Snowflake query, executing a data transfer across different platforms (like from a GCS bucket to an AWS S3 bucket), and pushing data into a queue system like SQS. 

Airflow orchestrates pipelines running several third-party tools and supports about 80 different providers.  But you need to monitor what’s going on underneath pipeline operations across all those tools and get deeper insights about pipeline execution. Unravel supports this by bringing in all the information monitored from Airflow via API, StatsD, and notifications. Unravel also collects metrics from Spark, BigQuery, Snowflake to connect the dots to find out how the pipeline execution in Airflow impacts other systems.

Let’s examine two common use cases to illustrate how Unravel stitches together all pipeline details to solve problems that you can’t address with Airflow alone.

View DAG Run Information

Let’s say a financial services company is using Airflow to run Spark jobs on their instances and notices that a few of the DAG runs are taking longer to complete. Airflow UI shows information about the duration of each task run and the status of the task run, but that’s about it. For deeper insight into why the runs are taking longer, you’d need to gather detailed information about the Spark ID that is created via this task run. 

By tightly integrating with the Spark ecosystem, Unravel fills that gap and gives you deeper insights about the Spark job run, its duration, and the resource utilized. These details, along with information collected from Airflow, can help you see the bigger picture of what’s happening underneath, inside your data pipelines. 

This is an example test_spark DAG that runs the Spark job via the SparkSubmitOperator in the company’s environment. The SparkSubmitOperator task is taking longer to execute, and the screen shot below shows the long duration of the Spark submit task.This information flows into Unravel and is visualized in the Pipeline Observer UI.  A summary list of failed/degraded pipeline runs helps to filter DAG runs with problems to debug. Clicking on a specific pipeline run ID drills down into more information about the pipeline, e.g., detailed information about the task, duration of each task, total duration, baseline compared to previous runs, and stack traces about the failure. Unravel also shows a list of all pipeline runs.Clicking on one of the Spark DAG runs with a long duration reveals more detailed information about the DAG and allows comparison with previous DAG runs. You can also find information about the Spark ID that is run for this particular DAG.

Clicking on the Spark app ID takes you to the Unravel UI with more information about the Spark job that is collected from Spark: the resources used to run the Spark job, any error logs, configurations made to run that Spark job, duration of the Spark jobs, and more.

Here we have more detail about the Spark job and the Spark ID created during the DAG run that helps verify the list of Spark jobs, understand which DAG run created the specific Spark job, see the resources consumed by that Spark job, or find out the configuration of the Spark job. 

Tracking down a specific Spark job from a DAG run is difficult since although Airflow initiates the task run, Spark takes care of running the task. Unravel can monitor and collect details from Spark, identify the Spark ID, and correlate the Spark job to the Airflow DAG run that initiated the job.

Delay in Scheduling Pipeline Run 

Unravel helps pinpoint the delay in scheduling DAG runs.

The same financial services company runs multiple DAGs as part of their pipeline system. While the DAG runs are working fine, the team stumbles upon a different problem: the DAG that is expected to run at the scheduled time is running at a delayed time. This delay affects running subsequent DAG runs, resulting in tasks not getting completed on time.

Airflow has the capability to alert if a Task/DAG that’s running misses its SLA. But there are cases when the DAG run is not initiated at all in the first place—e.g., no slot is available or the number of DAGs that can run in parallel exceeds the configured maximum number.  

Unravel helps bring the problem to the surface by rendering underlying details about the DAG run delay right in the UI. For a DAG run ID, Unravel shows the useful information collected from StatsD like the total parse time for processing the DAG, DAG run schedule delay, etc. This information provides pipeline insights that help to identify why there is delay in scheduling the DAG run around the DAG execution time.

The Pipeline Details page presents DAG run ID information, with pipeline insights about the schedule delay in the right-hand pane. It clearly shows that the DAG run was delayed well beyond the acceptable threshold. 

Unravel helps keep the pipeline running in check and automatically delivers insights about what’s happening inside Airflow. Unravel monitors systems like Spark, BigQuery, and Airflow and has the granular data about each of the systems, connecting that information and rendering the right insights to make it a powerful tool for monitoring cloud data systems.

The post How Unravel Enhances Airflow appeared first on Unravel.

In this fast-paced era of artificial intelligence (AI), the need for data is multiplying. The demand for faster data life cycles has skyrocketed, thanks to AI’s insatiable appetite for knowledge. According to a recent McKinsey survey, 75% expect generative AI (GenAI) to “cause significant or disruptive change in the nature of their industry’s competition in the next three years.”

Next-gen AI craves unstructured, streaming, industry-specific data. Although the pace of innovation is relentless, “when it comes to generative AI, data really is your moat.”

But here’s the twist: efficiency is now the new cool kid in town. Data product profitability hinges on optimizing every step of the data life cycle—from ingestion and transformation, to processing, curating, and refining. It’s no longer just about gathering mountains of information; it’s about collecting the right data efficiently.

As new, industry-specific GenAI use cases emerge, there is an urgent need for large data sets for training, validation, verification, and drift analysis. GenAI requires flexible, scalable, and efficient data architecture, infrastructure, code, and operating models to achieve success.

Leverage a Scalable Operating Model to Accelerate Your Data Life Cycle Velocity

To optimize your data life cycle, it’s crucial to leverage a scalable operating model that can accelerate the velocity of your data processes. By following a systematic approach and implementing efficient strategies, you can effectively manage your data from start to finish.

Databricks recently introduced a scalable operating model for data and AI to help customers achieve a positive Return on Data Assets (RODA).

Databricks’ iterative end-to-end operating pipeline

Define Use Cases and Business Requirements

Before diving into the data life cycle, it’s essential to clearly define your use cases and business requirements. This involves understanding what specific problems or goals you plan to address with your data. By identifying these use cases and related business requirements, you can determine the necessary steps and actions needed throughout the entire process.

Build, Test, and Iterate the Solution

Once you have defined your use cases and business requirements, it’s time to build, test, and iterate the solution. This involves developing the necessary infrastructure, tools, and processes required for managing your data effectively. It’s important to continuously test and iterate on your solution to ensure that it meets your desired outcomes.

During this phase, consider using agile methodologies that allow for quick iterations and feedback loops. This will enable you to make adjustments as needed based on real-world usage and feedback from stakeholders.

Scale Efficiently

As your data needs grow over time, it’s crucial to scale efficiently. This means ensuring that your architecture can handle increased volumes of data without sacrificing performance or reliability.

Consider leveraging cloud-based technologies that offer scalability on-demand. Cloud platforms provide flexible resources that can be easily scaled up or down based on your needs. Employing automation techniques such as machine learning algorithms or artificial intelligence can help streamline processes and improve efficiency.

By scaling efficiently, you can accommodate growing datasets while maintaining high-quality standards throughout the entire data life cycle.

Elements of the Business Use Cases and Requirements Phase

In the data life cycle, the business requirements phase plays a crucial role in setting the foundation for successful data management. This phase involves several key elements that contribute to defining a solution and ensuring measurable outcomes. Let’s take a closer look at these elements:

• Leverage design thinking to define a solution for each problem statement: Design thinking is an approach that focuses on understanding user needs, challenging assumptions, and exploring innovative solutions. In this phase, it is essential to apply design thinking principles to identify and define a single problem statement that aligns with business objectives.
• Validate the business case and define measurable outcomes: Before proceeding further, it is crucial to validate the business case for the proposed solution. This involves assessing its feasibility, potential benefits, and alignment with strategic goals. Defining clear and measurable outcomes helps in evaluating project success.
• Map out the MVP end user experiences: To ensure user satisfaction and engagement, mapping out Minimum Viable Product (MVP) end-user experiences is essential. This involves identifying key touchpoints and interactions throughout the data life cycle stages. By considering user perspectives early on, organizations can create intuitive and effective solutions.
• Understand the data requirements: A thorough understanding of data requirements is vital for successful implementation. It includes identifying what types of data are needed, their sources, formats, quality standards, security considerations, and any specific regulations or compliance requirements.
• Gather required capabilities with platform architects: Collaborating with platform architects helps gather insights into available capabilities within existing infrastructure or technology platforms. This step ensures compatibility between business requirements and technical capabilities while minimizing redundancies or unnecessary investments.
• Establish data management roles, responsibilities, and procedures: Defining clear roles and responsibilities within the organization’s data management team is critical for effective execution. Establishing procedures for data observability, stewardship practices, access controls, privacy policies ensures consistency in managing data throughout its life cycle.

By following these elements in the business requirements phase, organizations can lay a solid foundation for successful data management and optimize the overall data life cycle. It sets the stage for subsequent phases, including data acquisition, storage, processing, analysis, and utilization.

Build, Test, and Iterate the Solution

To successfully implement a data life cycle, it is crucial to focus on building, testing, and iterating the solution. This phase involves several key steps that ensure the development and deployment of a robust and efficient system.

• Plan development and deployment: The first step in this phase is to carefully plan the development and deployment process. This includes identifying the goals and objectives of the project, defining timelines and milestones, and allocating resources effectively. By having a clear plan in place, the data team can streamline their efforts towards achieving desired outcomes.
• Gather end-user feedback at every stage: Throughout the development process, it is essential to gather feedback from end users at every stage. This allows for iterative improvements based on real-world usage scenarios. By actively involving end users in providing feedback, the data team can identify areas for enhancement or potential issues that need to be addressed.
• Define CI/CD pipelines for fast testing and iteration: Implementing Continuous Integration (CI) and Continuous Deployment (CD) pipelines enables fast testing and iteration of the solution. These pipelines automate various stages of software development such as code integration, testing, deployment, and monitoring. By automating these processes, any changes or updates can be quickly tested and deployed without manual intervention.
• Data preparation, cleaning, and processing: Before training machine learning models or conducting experiments with datasets, it is crucial to prepare, clean, and process the data appropriately. This involves tasks such as removing outliers or missing values from datasets to ensure accurate results during model training.
• Feature engineering: Feature engineering plays a vital role in enhancing model performance by selecting relevant features from raw data or creating new features based on domain knowledge. It involves transforming raw data into meaningful representations that capture essential patterns or characteristics.
• Training and ML experiments: In this stage of the data life cycle, machine learning models are trained using appropriate algorithms on prepared datasets. Multiple experiments may be conducted, testing different algorithms or hyperparameters to find the best-performing model.
• Model deployment: Once a satisfactory model is obtained, it needs to be deployed in a production environment. This involves integrating the model into existing systems or creating new APIs for real-time predictions.
• Model monitoring and scoring: After deployment, continuous monitoring of the model’s performance is essential. Tracking key metrics and scoring the model’s outputs against ground truth data helps identify any degradation in performance or potential issues that require attention.

By following these steps and iterating on the solution based on user feedback, data teams can ensure an efficient and effective data life cycle from development to deployment and beyond.

Efficiently Scale and Drive Adoption with Your Operating Model

To efficiently scale your data life cycle and drive adoption, you need to focus on several key areas. Let’s dive into each of them:

• Deploy into production: Once you have built and tested your solution, it’s time to deploy it into production. This step involves moving your solution from a development environment to a live environment where end users can access and utilize it.
• Evaluate production results: After deploying your solution, it is crucial to evaluate its performance in the production environment. Monitor key metrics and gather feedback from users to identify any issues or areas for improvement.
• Socialize data observability and FinOps best practices: To ensure the success of your operating model, it is essential to socialize data observability and FinOps best practices among your team. This involves promoting transparency, accountability, and efficiency in managing data operations.
• Acknowledge engineers who “shift left” performance and efficiency: Recognize and reward engineers who prioritize performance and efficiency early in the development process. Encourage a culture of proactive optimization by acknowledging those who contribute to improving the overall effectiveness of the data life cycle.
• Manage access, incidents, support, and feature requests: Efficiently scaling your operating model requires effective management of access permissions, incident handling processes, support systems, and feature requests. Streamline these processes to ensure smooth operations while accommodating user needs.
• Track progress towards business outcomes by measuring and sharing KPIs: Measuring key performance indicators (KPIs) is vital for tracking progress towards business outcomes. Regularly measure relevant metrics related to adoption rates, user satisfaction levels, cost savings achieved through efficiency improvements, etc., then share this information across teams for increased visibility.

By implementing these strategies within your operating model, you can efficiently scale your data life cycle while driving adoption among users. Remember that continuous evaluation and improvement are critical for optimizing performance throughout the life cycle.

Drive for Performance with Purpose-Built AI

Unravel helps with many elements of the Databricks operating model:

• Quickly identify failed and inefficient Databricks jobs: One of the key challenges is identifying failed and inefficient Databricks jobs. However, with AI purpose-built for Databricks, this task becomes much easier. By leveraging advanced analytics and monitoring capabilities, you can quickly pinpoint any issues in your job executions.
• Creating ML models vs deploying them into production: Creating machine learning models is undoubtedly challenging, but deploying them into production is even harder. It requires careful consideration of factors like scalability, performance, and reliability. With purpose-built AI tools, you can streamline the deployment process by automating various tasks such as model versioning, containerization, and orchestration.
• Leverage Unravel’s Analysis tab for insights: To gain deeper insights into your application’s performance during job execution, leverage the analysis tab provided by purpose-built AI solutions. This feature allows you to examine critical details like memory usage errors or other bottlenecks that may be impacting job efficiency.
  
  

  Unravel’s AI-powered analysis automatically provides deep, actionable insights.

• Share links for collaboration: Collaboration plays a crucial role in data management and infrastructure optimization. Unravel enables you to share links with data scientists, developers, and data engineers to provide detailed information about specific test runs or failed Databricks jobs. This promotes collaboration and facilitates a better understanding of why certain jobs may have failed.
• Cloud data cost management made easy: Cloud cost management, also known as FinOps, is another essential aspect of data life cycle management. Purpose-built AI solutions simplify this process by providing comprehensive insights into cost drivers within your Databricks environment. You can identify the biggest cost drivers such as users, clusters, jobs, and code segments that contribute significantly to cloud costs.
• AI recommendations for optimization: To optimize your data infrastructure further, purpose-built AI platforms offer valuable recommendations across various aspects, including infrastructure configuration, parallelism settings, handling data skewness issues, optimizing Python/SQL/Scala/Java code snippets, and more. These AI-driven recommendations help you make informed decisions to enhance performance and efficiency.

Learn More & Next Steps

Unravel hosted a virtual roundtable, Accelerate the Data Analytics Life Cycle, with a panel of Unravel and Databricks experts. Unravel VP Clinton Ford moderated the discussion with Sanjeev Mohan, principal at SanjMo and former VP at Gartner, Subramanian Iyer, Unravel training and enablement leader and Databricks SME, and Don Hilborn, Unravel Field CTO and former Databricks lead strategic solutions architect.

FAQs

How can I implement a scalable operating model for my data life cycle?

To implement a scalable operating model for your data life cycle, start by clearly defining roles and responsibilities within your organization. Establish efficient processes and workflows that enable seamless collaboration between different teams involved in managing the data life cycle. Leverage automation tools and technologies to streamline repetitive tasks and ensure consistency in data management practices.

What are some key considerations during the Business Requirements Phase?

During the Business Requirements Phase, it is crucial to engage stakeholders from various departments to gather comprehensive requirements. Clearly define project objectives, deliverables, timelines, and success criteria. Conduct thorough analysis of existing systems and processes to identify gaps or areas for improvement.

How can I drive adoption of my data life cycle operational model?

To drive adoption of your data management solution, focus on effective change management strategies. Communicate the benefits of the solution to all stakeholders involved and provide training programs or resources to help them understand its value. Encourage feedback from users throughout the implementation process and incorporate their suggestions to enhance usability and address any concerns.

What role does AI play in optimizing the data life cycle?

AI can play a significant role in optimizing the data life cycle by automating repetitive tasks, improving data quality through advanced analytics and machine learning algorithms, and providing valuable insights for decision-making. AI-powered tools can help identify patterns, trends, and anomalies in large datasets, enabling organizations to make data-driven decisions with greater accuracy and efficiency.

How do I ensure performance while implementing purpose-built AI?

To ensure performance while implementing purpose-built AI, it is essential to have a well-defined strategy. Start by clearly defining the problem you want to solve with AI and set measurable goals for success. Invest in high-quality training data to train your AI models effectively. Continuously monitor and evaluate the performance of your AI system, making necessary adjustments as needed.

The post Rev Up Your Lakehouse: Lap the Field with a Databricks Operating Model appeared first on Unravel.

Overview

In today’s digital landscape, data-driven decisions form the crux of successful business strategies. However, the path to harnessing data’s full potential is strewn with challenges. Let’s delve into the hurdles organizations face and how Unravel is the key to unlocking seamless data operations.

The Roadblocks in the Fast Lane of Data Operations

In today’s data-driven landscape, organizations grapple with erratic spending, cloud constraints, AI complexities, and prolonged MTTR, urgently seeking solutions to navigate these challenges efficiently. The four most common roadblocks are:

• Data Spend Forecasting: Imagine a roller coaster with unpredictable highs and lows. That’s how most organizations view their data spend forecasting. Such unpredictability wreaks havoc on financial planning, making operational consistency a challenge.
• Constraints in Adding Data Workloads: Imagine tying an anchor to a speedboat. That’s how the constraints feel when trying to adopt cloud data solutions, holding back progress and limiting agility.
• Surge in AI Model Complexity: AI’s evolutionary pace is exponential. As it grows, so do the intricacies surrounding data volume and pipelines, which strain budgetary limitations.
• The MTTR Bottleneck: The multifaceted nature of modern tech stacks means longer Mean Time to Repair (MTTR). This slows down processes, consumes valuable resources, and stalls innovation.

By acting as a comprehensive data observability and FinOps solution, Unravel Data empowers businesses to move past the challenges and frictions that typically hinder data operations, ensuring smoother, more efficient data-driven processes. Here’s how Unravel Data aids in navigating the roadblocks in the high-speed lane of data operations:

• Predictive Data Spend Forecasting: With its advanced analytics, Unravel Data can provide insights into data consumption patterns, helping businesses forecast their data spending more accurately. This eliminates the roller coaster of unpredictable costs.
• Simplifying Data Workloads: Unravel Data optimizes and automates workload management. Instead of being anchored down by the weight of complex data tasks, businesses can efficiently run and scale their data processes in the cloud.
• Managing AI Model Complexity: Unravel offers visibility and insights into AI data pipelines. Analyzing and optimizing these pipelines ensure that growing intricacies do not overwhelm resources or budgets.
• Reducing MTTR: By providing a clear view of the entire tech stack and pinpointing bottlenecks or issues, Unravel Data significantly reduces Mean Time to Repair. With its actionable insights, teams can address problems faster, reducing downtime and ensuring smooth data operations.
• Streamlining Data Pipelines: Unravel Data offers tools to diagnose and improve data pipeline performance. This ensures that even as data grows in volume and complexity, pipelines remain efficient and agile.
• Efficiency and ROI: With its clear insights into resource consumption and requirements, Unravel Data helps businesses run 50% more workloads in their existing Databricks environments, ensuring they only pay for what they need, reducing wastage and excess expenditure.

The Skyrocketing Growth of Cloud Data Management

As the digital realm expands, cloud data management usage is soaring, with data services accounting for a significant chunk. According to the IDC, the public cloud IaaS and PaaS market is projected to reach $400 billion by 2025, growing at a 28.8% CAGR from 2021 to 2025. Some highlights are:

• Data management and application development account for 39% and 20% of the market, respectively, and are the main workloads backed by PaaS solutions, capturing a major share of its revenue.
• In IaaS revenue, IT infrastructure leads with 25%, trailed by business applications (21%) and data management (20%).
• Unstructured data analytics and media streaming are predicted to be the top-growing segments with CAGRs of 41.9% and 41.2%, respectively.

Unravel provides a comprehensive solution to address the growth associated with cloud data management. Here’s how:

• Visibility and Transparency: Unravel offers in-depth insights into your cloud operations, allowing you to understand where and how costs are accruing, ensuring no hidden fees or unnoticed inefficiencies.
• Optimization Tools: Through its suite of analytics and AI-driven tools, Unravel pinpoints inefficiencies, recommends optimizations, and automates the scaling of resources to ensure you’re only using (and paying for) what you need.
• Forecasting: With predictive analytics, Unravel provides forecasts of data usage and associated costs, enabling proactive budgeting and financial planning.
• Workload Management: Unravel ensures that data workloads run efficiently and without wastage, reducing both computational costs and storage overhead.
• Performance Tuning: By optimizing query performance and data storage strategies, Unravel ensures faster results using fewer resources, translating to 50% more workloads.
• Monitoring and Alerts: Real-time monitoring paired with intelligent alerts ensures that any resource-intensive operations or anomalies are flagged immediately, allowing for quick intervention and rectification.

By employing these strategies and tools, Unravel acts as a financial safeguard for businesses, ensuring that the ever-growing cloud data bill remains predictable, manageable, and optimized for efficiency.

The Tightrope Walk of Efficiency Tuning and Talent

Modern enterprises hinge on data and AI, but shrinking budgets and talent gaps threaten them. Gartner pinpoints overprovisioning and skills shortages as major roadblocks, while Google and IDC underscore the high demand for data analytics skills and the untapped potential of unstructured data. Here are some of the problems modern organizations face:

• Production environments are statically overprovisioned and therefore underutilized. On-premises, 30% utilization is common, but it’s all capital expenditures (capex), and as long as it’s within budget, no one has traditionally cared about the waste. However, in the cloud, you pay for that excess resource monthly, forcing you to confront the ongoing cost of the waste. – Gartner
• The cloud skills gap has reached a crisis level in many organizations – Gartner
• Revenue creation through digital transformation requires talent engagement that is currently scarce and difficult to acquire and maintain. – Gartner
• Lack of skills remains the biggest barrier to infrastructure modernization initiatives, with many organizations finding they cannot hire outside talent to fill these skills gaps. IT organizations will not succeed unless they prioritize organic skills growth. – Gartner
• Data analytics skills are in demand across industries as businesses of all types around the world recognize that strong analytics improve business performance.- Google via Coursera

Unravel Data addresses the delicate balancing act of budget and talent in several strategic ways:

• Operational Efficiency: Purpose-built AI provides actionable insights into data operations across Databricks, Spark, EMR, BigQuery, Snowflake, etc. Unravel Data reduces the need for trial-and-error and time-consuming manual interventions. At the core of Unravel’s data observability platform is our AI-powered Insights Engine. This sophisticated Artificial Intelligence engine incorporates AI techniques, algorithms, and tools to process and analyze vast amounts of data, learn from patterns, and make predictions or decisions based on that learning. This not only improves operational efficiency but also ensures that talented personnel spend their time innovating rather than on routine tasks.
• Skills Gap Bridging: The platform’s intuitive interface and AI-driven insights mean that even those without deep expertise in specific data technologies can navigate, understand, and optimize complex data ecosystems. This eases the pressure to hire or train ultra-specialized talent.
• Predictive Analysis: With Unravel’s ability to predict potential bottlenecks or inefficiencies, teams can proactively address issues, leading to more efficient budget allocation and resource utilization.
• Cost Insights: Unravel provides detailed insights into the efficiency of various data operations, allowing organizations to make informed decisions on where to invest and where to cut back.
• Automated Optimization: By automating many of the tasks traditionally performed by data engineers, like performance tuning or troubleshooting, Unravel ensures teams can do more with less, optimizing both budget and talent.
• Talent Focus Shift: With mundane tasks automated and insights available at a glance, skilled personnel can focus on higher-value activities, like data innovation, analytics, and strategic projects.

By enhancing efficiency, providing clarity, and streamlining operations, Unravel Data ensures that organizations can get more from their existing budgets while maximizing the potential of their talent, turning the tightrope walk into a more stable journey.

The Intricacies of Data-Centric Organizations

Data-centric organizations grapple with the complexities of managing vast and fast-moving data in the digital age. Ensuring data accuracy, security, and compliance, while integrating varied sources, is challenging. They must balance data accessibility with protecting sensitive information, all while adapting to evolving technologies, addressing talent gaps, and extracting actionable insights from their data reservoirs. Here is some relevant research on the topic:

• “Data is foundational to AI” yet “unstructured data remains largely untapped.” – IDC
• Even as organizations rush to adopt data-centric operations, challenges persist. For instance, manufacturing data projects often hit roadblocks due to outdated legacy technology, as observed by the World Economic Forum.
• Generative AI is supported by large language models (LLMs), which require powerful and highly scalable computing capabilities to process data in real-time. – Gartner

Unravel Data provides a beacon for data-centric organizations amid complex challenges. Offering a holistic view of data operations, it simplifies management using AI-driven tools. It ensures data security, accessibility, and optimized performance. With its intuitive interface, Unravel bridges talent gaps and navigates the data maze, turning complexities into actionable insights.

Embarking on the Unravel Journey: Your Step-By-Step Guide

• Beginning your data journey with Unravel is as easy as 1-2-3. We guide you through the sign-up process, ensuring a smooth and hassle-free setup.
• Unravel for Databricks page
  

• Unravel for Databricks plans & pricing
  

• Unravel for Databricks free account
  

Level Up with Unravel Premium

Ready for an enhanced data experience? Unravel’s premium account offers a plethora of advanced features that the free version can’t match. Investing in this upgrade isn’t just about more tools; it’s about supercharging your data operations and ROI.

Wrap-Up

Although rising demands on the modern data landscape are challenging, they are not insurmountable. With tools like Unravel, organizations can navigate these complexities, ensuring that data remains a catalyst for growth, not a hurdle. Dive into the Unravel experience and redefine your data journey today.

Unravel is a business’s performance sentinel in the cloud realm, proactively ensuring that burgeoning cloud data expenses are not only predictable and manageable but also primed for significant cost savings. Unravel Data transforms the precarious balance of budget and talent into a streamlined, efficient journey for organizations. Unravel Data illuminates the path for data-centric organizations, streamlining operations with AI tools, ensuring data security, and optimizing performance. Its intuitive interface simplifies complex data landscapes, bridging talent gaps and converting challenges into actionable insights.

The post Overcoming Friction & Harnessing the Power of Unravel: Try It for Free appeared first on Unravel.

I was an Enterprise Data Architect at Boeing NASA Systems the day of the Columbia Shuttle disaster. The tragedy has had a profound impact in my career on how I look at data. Recently I have been struck by how, had it been available back then, purpose-built AI observability might have altered the course of events. 

The Day of the Disaster

With complete reverence and respect for the Columbia disaster, I can remember that day with incredible amplification. I was standing in front of my TV watching the Shuttle Columbia disintegrate on reentry. As the Enterprise Data Architect at Boeing NASA Systems, I received a call from my boss: “We have to pull and preserve all the Orbitor data. I will meet you at the office.” Internally, Boeing referred to the Shuttles as Oribitors. It was only months earlier I had the opportunity to share punch and cookies with some of the crew. To say I was being flooded with emotions would be a tremendous understatement.

When I arrived at Boeing NASA Systems, near Johnson Space Center, the mood in the hallowed aeronautical halls was somber. I sat intently, with eyes scanning lines of data from decades of space missions. The world had just witnessed a tragedy — the Columbia Shuttle disintegrated on reentry, taking the lives of seven astronauts. The world looked on in shock and sought answers. As a major contractor for NASA, Boeing was at the forefront of this investigation.

As the Enterprise Data Architect, I was one of an army helping dissect the colossal troves of data associated with the shuttle, looking for anomalies, deviations — anything that could give a clue as to what had gone wrong. Days turned into nights, and nights turned into weeks and months as we tirelessly pieced together Columbia’s final moments. But as we delved deeper, a haunting reality began to emerge.

Every tiny detail of the shuttle was monitored, from the heat patterns of its engines to the radio signals it emitted. But there was a blind spot, an oversight that no one had foreseen. In the myriad of data sets, there was nothing that indicated the effects of a shuttle’s insulation tiles colliding with a piece of Styrofoam, especially at speeds exceeding 500 miles per hour.

The actual incident was seemingly insignificant — a piece of foam insulation had broken off and struck the shuttle’s left wing. But in the vast expanse of space and the brutal conditions of reentry, even minor damage could prove catastrophic.

Video footage confirmed: the foam had struck the shuttle. But without concrete data on what such an impact would do, the team was left to speculate and reconstruct potential scenarios. The lack of this specific data had turned into a gaping void in the investigation.

As a seasoned Enterprise Data Architect, I always believed in the power of information. I absolutely believed that in the numbers, in the bytes and bits, we find the stories that the universe whispers to us. But this time, the universe had whispered a story that we didn’t have the data to understand fully.

Key Findings

After the accident, NASA formed the Columbia Accident Investigation Board (CAIB) to investigate the disaster. The board consisted of experts from various fields outside of NASA to ensure impartiality.

1. Physical Cause: The CAIB identified the direct cause of the accident as a piece of foam insulation from the shuttle’s external fuel tank that broke off during launch. This foam struck the leading edge of the shuttle’s left wing. Although foam shedding was a known issue, it had been wrongly perceived as non-threatening because of prior flights where the foam was lost but didn’t lead to catastrophe.

2. Organizational Causes: Beyond the immediate physical cause, the CAIB highlighted deeper institutional issues within NASA. They found that there were organizational barriers preventing effective communication of safety concerns. Moreover, safety concerns had been normalized over time due to prior incidents that did not result in visible failure. Essentially, since nothing bad had happened in prior incidents where the foam was shed, the practice had been erroneously deemed “safe.”

3. Decision Making: The CAIB pointed out issues in decision-making processes that relied heavily on past success as a predictor of future success, rather than rigorous testing and validation.

4. Response to Concerns: There were engineers who were concerned about the foam strike shortly after Columbia’s launch, but their requests for satellite imagery to assess potential damage were denied. The reasons were multifaceted, ranging from beliefs that nothing could be done even if damage was found, to a misunderstanding of the severity of the situation.

The CAIB made a number of recommendations to NASA to improve safety for future shuttle flights. These included:

1. Physical Changes: Improving the way the shuttle’s external tank was manufactured to prevent foam shedding and enhancing on-orbit inspection and repair techniques to address potential damage.

2. Organizational Changes: Addressing the cultural issues and communication barriers within NASA that led to the accident, and ensuring that safety concerns were more rigorously addressed.

3. Continuous Evaluation: Establishing an independent Technical Engineering Authority responsible for technical requirements and all waivers to them, and building an independent safety program that oversees all areas of shuttle safety.

Could Purpose-built AI Observability Have Helped?

In the aftermath, NASA grounded the shuttle fleet for more than two years after the disaster. They then implemented the CAIB’s recommendations before resuming shuttle flights. Columbia’s disaster, along with the Challenger explosion in 1986, are stark reminders of the risks of space travel and the importance of a diligent and transparent safety culture. The lessons from Columbia shaped many of the safety practices NASA follows in its current human spaceflight programs.

The Columbia disaster led to profound changes in how space missions were approached, with a renewed emphasis on data collection and eliminating informational blind spots. But for me, it became a deeply personal mission. I realized that sometimes, the absence of data could speak louder than the most glaring of numbers. It was a lesson I would carry throughout my career, ensuring that no stone was left unturned, and no data point overlooked.

The Columbia disaster, at its core, was a result of both a physical failure (foam insulation striking the shuttle’s wing) and organizational oversights (inadequate recognition and response to potential risks). Purpose-built AI Observability, which involves leveraging artificial intelligence to gain insights into complex systems and predict failures, might have helped in several key ways:

1. Real-time Anomaly Detection: Modern AI systems can analyze vast amounts of data in real time to identify anomalies. If an AI-driven observability platform had been monitoring the shuttle’s various sensors and systems, it might have detected unexpected changes or abnormalities in the shuttle’s behavior after the foam strike, potentially even subtle ones that humans might overlook.

2. Historical Analysis: An AI system with access to all previous shuttle launch and flight data might have detected patterns or risks associated with foam-shedding incidents, even if they hadn’t previously resulted in a catastrophe. The system could then raise these as potential long-term risks.

3. Predictive Maintenance: AI-driven tools can predict when components of a system are likely to fail based on current and historical data. If applied to the shuttle program, such a system might have provided early warnings about potential vulnerabilities in the shuttle’s design or wear-and-tear.

4. Decision Support: AI systems could have aided human decision-makers in evaluating the potential risks of continuing the mission after the foam strike, providing simulations, probabilities of failure, or other key metrics to help guide decisions.

5. Enhanced Imaging and Diagnosis: If equipped with sophisticated imaging capabilities, AI could analyze images of the shuttle (from external cameras or satellites) to detect potential damage, even if it’s minor, and then assess the risks associated with such damage.

6. Overcoming Organizational Blind Spots: One of the major challenges in the Columbia disaster was the normalization of deviance, where foam shedding became an “accepted” risk because it hadn’t previously caused a disaster. An AI system, being objective, doesn’t suffer from these biases. It would consistently evaluate risks based on data, not on historical outcomes.

7. Alerts and Escalations: An AI system can be programmed to escalate potential risks to higher levels of authority, ensuring that crucial decisions don’t get caught in bureaucratic processes.

While AI Observability could have provided invaluable insights and might have changed the course of events leading to the Columbia disaster, it’s essential to note that the integration of such AI systems also requires organizational openness to technological solutions and a proactive attitude toward safety. The technology is only as effective as the organization’s willingness to act on its findings.

The tragedy served as a grim reminder for organizations worldwide: It’s not just about collecting data; it’s about understanding the significance of what isn’t there. Because in those blind spots, destiny can take a drastic turn.

In Memory and In Action

The Columbia crew and their families deserve our utmost respect and admiration for their unwavering commitment to space exploration and the betterment of humanity. 

• Rick D. Husband: As the Commander of the mission, Rick led with dedication, confidence, and unparalleled skill. His devotion to space exploration was evident in every decision he made. We remember him not just for his expertise, but for his warmth and his ability to inspire those around him. His family’s strength and grace, in the face of the deepest pain, serve as a testament to the love and support they provided him throughout his journey.
• William C. McCool: As the pilot of Columbia, William’s adeptness and unwavering focus were essential to the mission. His enthusiasm and dedication were contagious, elevating the spirits of everyone around him. His family’s resilience and pride in his achievements are a reflection of the man he was — passionate, driven, and deeply caring.
• Michael P. Anderson: As the payload commander, Michael’s role was vital, overseeing the myriad of experiments and research aboard the shuttle. His intellect was matched by his kindness, making him a cherished member of the team. His family’s courage and enduring love encapsulate the essence of Michael’s spirit — bright, optimistic, and ever-curious.
• Ilan Ramon: As the first Israeli astronaut, Ilan represented hope, unity, and the bridging of frontiers. His enthusiasm for life was infectious, and he inspired millions with his historic journey. His family’s grace in the face of the unthinkable tragedy is a testament to their shared dream and the values that Ilan stood for.
• Kalpana Chawla: Known affectionately as ‘KC’, Kalpana’s journey from a small town in India to becoming a space shuttle mission specialist stands as an inspiration to countless dreamers worldwide. Her determination, intellect, and humility made her a beacon of hope for many. Her family’s dignity and strength, holding onto her legacy, reminds us all of the power of dreams and the sacrifices made to realize them.
• David M. Brown: As a mission specialist, David brought with him a zest for life, a passion for learning, and an innate curiosity that epitomized the spirit of exploration. He ventured where few dared and achieved what many only dreamt of. His family’s enduring love and their commitment to preserving his memory exemplify the close bond they shared and the mutual respect they held for each other.
• Laurel B. Clark: As a mission specialist, Laurel’s dedication to scientific exploration and discovery was evident in every task she undertook. Her warmth, dedication, and infectious enthusiasm made her a beloved figure within her team and beyond. Her family’s enduring spirit, cherishing her memories and celebrating her achievements, is a tribute to the love and support that were foundational to her success.

To each of these remarkable individuals and their families, we extend our deepest respect and gratitude. Their sacrifices and contributions will forever remain etched in the annals of space exploration, reminding us of the human spirit’s resilience and indomitable will.

For those of us close to the Columbia disaster, it was more than a failure; it was a personal loss. Yet, in memory of those brave souls, we are compelled to look ahead. In the stories whispered to us by data, and in the painful lessons from their absence, we seek to ensure that such tragedies remain averted in the future.

While no technology can turn back time, the promise of AI Observability beckons a future where every anomaly is caught, every blind spot illuminated, and every astronaut returns home safely.

The above narrative seeks to respect the gravity of the Columbia disaster while emphasizing the potential of AI Observability. It underlines the importance of data, both in understanding tragedies and in preventing future ones.

The post Blind Spots in Your System: The Grave Risks of Overlooking Observability appeared first on Unravel.

DBS Bank leverages Unravel to identify inefficiencies across 10,000s of data applications/pipelines and guide individual developers and engineers on how, where, and what to improve—no matter what technology or platform they’re using.  

DBS Bank is one of the largest financial services institutions in Asia—the biggest in Southeast Asia—with a presence in 19 markets. Headquartered in Singapore, DBS is recognized globally for its technological innovation and leadership, having been named World’s Best Bank by both Global Finance and Euromoney, Global Bank of the Year by The Banker, World’s Best Digital Bank by Euromoney (three times), and one of the world’s top 10 business transformations of the decade by Harvard Business Review. In addition, DBS has been given Global Finance’s Safest Bank in Asia award for 14 consecutive years. 

DBS is, in its words, “redefining the future of banking as a technology company.” Says the bank’s Executive Director of Automation, Infrastructure for Big Data, AI, and Analytics Luis Carlos Cruz Huertas, “Technology is in our DNA, from the top down. Six years ago, when we started our data journey, the philosophy was that we’re going to be a data-driven organization, 100%. Everything we decide is through data.” 

Realizing innovation and efficiency at scale and speed

DBS backs up its commitment to being a leading data-forward company. Almost 40% of the bank’s employees—some 23,000—are native developers. The volume, variety, and velocity of DBS’ data products, initiatives, and analytics fuel an incredibly complex environment. The bank has developed more than 200 data-use cases to deliver business value: detecting fraud, providing value to customers via their DBS banking app, and providing hyper personalized “nudges” that guide customers in making more informed banking and investment decisions.

As with all financial services enterprises, the DBS data estate is a multi-everything mélange: on-prem, hybrid, and several cloud providers; all the principal platforms; open source, commercial, and proprietary solutions; and a wide variety of technologies, old and new. Adding to this complexity are the various country-specific requirements and restrictions throughout the bank’s markets. As Luis says, “There are many ways to do the cloud, but there’s also the banking way to do cloud. Finance is ring-fenced by regulation all over the world. To do cloud in Asia is quite complex. We don’t [always] get to choose the cloud, sometimes the cloud chooses us.” (For example, India now requires all data to run in India. Google was the only cloud available in Indonesia at the time. Data cannot leave China; there are no hyperscalers there other than Chinese.)

The pace of today’s innovation at a data-forward bank like DBS makes ensuring efficient, reliable performance an ever-growing challenge.

Today DBS runs more than 40,000 data jobs, with tens of thousands of ML models. “And that actually keeps growing,” says Luis. “We have exponentially grown—almost 100X what we used to run five years ago. There’s a lot of innovation we’re bringing, month to month.”

As Luis points out, “Sometimes innovation is a disruptor of stability.” With thousands  of developers all using whatever technologies are best-suited for their particular workload, DBS has to contend with the triple-headed issues of exponentially increasing data volumes (think: OpenAI); increasingly more technologies, platforms, and cloud providers; and the ever-present challenge of having enough skilled people with enough time to ensure that everything is running reliably and efficiently on time, every time—at population scale.  

DBS empowers self-service optimization

One of DBS’ lessons learned in its modern data journey is that, as Luis says, “a lot of the time the efficiencies relied on the engineering team to fix the problems. We didn’t really have a viable mechanism to put into the [business] users’ hands for them to analyze and understand their code, their deficiencies, and how to improve.”

That’s where Unravel comes in. The Unravel platform harnesses deep full-stack visibility, contextual awareness, AI-powered intelligence, and automation to not only quickly show what’s going on and why, but provide crisp, prescriptive recommendations on exactly how to make things better and then keep them that way proactively.

“In Unravel, [we] have a source to identify areas of improvement for our current operational jobs. By leveraging Unravel, we’ve been able to put an analytical tool in the hands of business users to evaluate themselves before they actually do a code release. [They now get] a lot of additional insights they can use to learn, and it puts a lot more responsibility into how they’re doing their job.”

To learn more about how Unravel puts self-service performance and cost optimization capabilities into the hands of individual engineers, see the Unravel Platform Overview.

Marrying data engineering and data science

DBS is perpetually innovating with data—40% of the bank’s workforce are data science developers—yet the scale and speed of DBS’ data operations bring the data engineering side of things to the forefront. As Luis points out, there are thousands of developers committing code but only a handful of data engineers to make sure everything runs reliably and efficiently.

While few organizations have such a high percentage of developers, virtually everyone has the same lopsided developer : engineer ratio. Tackling this reality is at the heart of DataOps and means new roles, where data science and data engineering meet. Unravel has helped facilitate this new intersection of roles and responsibilities by making it easier to pinpoint code inefficiencies and provide insights into what, where, and how to optimize code — without having to “throw it over the fence” to already overburdened engineers.

Luis discusses how DBS addresses the issue: “In hardware, you have something called system performance engineers. They’re dedicated to optimizing and tuning how a processor operates, until it’s pristine. I said, ‘Why don’t we apply that concept and bring it over to data?’ What we need is a very good data scientist who wants to learn data engineering. And a data engineer who wants to become a data scientist. Because I need to connect and marry both worlds. It’s the only way a person can actually say, ‘Oh, this is why pandas doesn’t work in Spark.’ For a data engineer, it’s very clear. For a data scientist, it’s not. Because the first thing they learned in Python is pandas. And they love it. And Spark hates it. It’s a perfect divorce. So we need to teach data scientists how to break their pandas into Spark. Our system performance engineers became masters at doing this. They use Unravel to highlight all the data points that we need to attack in the code, so let’s just go see it. 

“We call data jobs ‘mutants,’ because they mutate from the hands of one data scientist to another. You can literally see the differences on how they write the code. Some of it might be perfect, with Markdown files, explainability, and then there’s an entire chunk of code that doesn’t have that. So we tune and optimize the code. Unravel helps us facilitate the journey on deriving what to do first, what to attack first, from the optimization perspective.”

DBS usually builds their own products—why did they buy Unravel?

DBS has made the bold decision to develop its own proprietary technology stack as wrappers for its governance, control plane, data plane, etc. “We ring-fence all compute, storage, services—every resource the cloud provides. The reason we create our own products is that we’ve been let down way too many times. We don’t really control what happens in the market. It’s the world we live in. We accept that. Our data protection mechanism used to be BlueTalon, which was then acquired by Microsoft. And Microsoft decided to dispose of [BlueTalon] and use their own product. Our entire security framework depended on BlueTalon. . . . We decided to just build our own [framework].

“In a way DBS protects itself from being forced to just do what the providers want us to do. And there’s a resistance from us—a big resistance—to oblige that.” Luis uses a cooking analogy to describe the DBS approach. At one extreme is home cooking, where you go buy all the ingredients and make meals yourself. At the other end of the spectrum is going out to restaurants, where you choose items from the menu and you get a meal the way they prepare it. The DBS data platform is more like a cooking studio—Luis  provides the kitchen, developers pick the ingredients they want, and then cook the recipe with their own particular flavor. “That’s what we provide to our lines of business,” says Luis. “The good thing is, we can plug in anything new at any given point in time. The downside is that you need very, very good technical people. How do we sustain the pace of rebuilding new products, and keep up with the open source side, which moves astronomically fast—we’re very connected to the open source mission, close to 50% of our developers contribute to the open source world—and at the same time keep our [internal and external] customers happy?”

Luis explains the build vs. buy decision boils down to whether the product “drives the core of what we do, whether it’s the ‘spinal cord’ of what we do. We’ve had Unravel for a long, long time. It’s one of those products where we said, ‘Should we do it [i.e., build it ourselves]? Or should we find something on the market?’ With Unravel, it was an easy choice to say, ‘If they can do what they claim they can do, bring ’em in.’ We don’t need to develop our own [data observability solution]. Unravel has demonstrated value in [eliminating] hours of toil—nondeterministic tasks performed by engineers because of repetitive incidents. So, long story short: that [Unravel] is still with us demonstrates that there is value from the lines of business.”

Luis emphasizes his data engineering team are not the users of Unravel. “The users are the business units that are creating their jobs. We just drive adoption to make sure everyone in the bank uses it. Because ultimately our data engineers cannot ‘overrun’ a troop of 3,000 data scientists. So we gave [Unravel] to them, they can use it, they can optimize, and control their own fate.”

DBS realizes high ROI from Unravel

Unravel’s granular data observability, automation, and AI enable DBS to control its cloud data usage at the speed and scale demanded by today’s FinServ environment. “Unravel has saved potentially several $100,000s in [unnecessary] compute capacity,” says Luis. “This is a very big transformational year for us. We’re moving more and more into some sort of data mesh and to a more containerized world. Unravel is part of that journey and we expect that we’ll get the same level of return we’ve gotten so far—or even better.” 

The post DBS Bank Uplevels Individual Engineers at Scale appeared first on Unravel.

Unravel Data Observability Platform earns prestigious industry recognition for Best Data Tool & Platform

Palo Alto, CA — June 21, 2023 —Unravel Data, the first data observability platform built to meet the needs of modern data teams, was named Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards. The prestigious CODiE Awards recognize the companies producing the most innovative Business Technology products across the country and around the world.

“We are deeply honored to win a CODiE Award for Best Data Tool & Platform. Today, as companies put data products and AI/ML innovation front and center of their growth and customer service strategies, the volume of derailed projects and the costs associated are ascending astronomically. Companies need a way to increase performance of their data pipelines and a way to manage costs for effective ROI,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “The Unravel Data Platform brings pipeline performance management and FinOps to the modern data stack. Our AI-driven Insights Engine provides recommendations that allow data teams to make smarter decisions that optimize pipeline performance along with the associated cloud data spend, making innovation more efficient for organizations.”

Cloud-first companies are seeing cloud data costs exceed 40% of their total cloud spend. These organizations lack the visibility into queries, code, configurations, and infrastructure required to manage data workloads effectively, which in turn, leads to over-provisioned capacity for data jobs, an inability to quickly detect pipeline failures and slowdowns, and wasted cloud data spend.

“The 2023 Business Technology CODiE Award Winners maintain the vital legacy of the CODiEs in spotlighting the best and most impactful apps, services and products serving the business tech market,” said SIIA President Chris Mohr. “We are so proud to recognize this year’s honorees – the best of the best! Congratulations to all of this year’s CODiE Award winners!”

The Software & Information Industry Association (SIIA), the principal trade association for the software and digital content industries, announced the full slate of CODiE winners during a virtual winner announcement. Awards were given for products and services deployed specifically for education and learning professionals, including the top honor of the Best Overall Business Technology Solution.

A SIIA CODiE Award win is a prestigious honor, following rigorous reviews by expert judges whose evaluations determined the finalists. SIIA members then vote on the finalist products, and the scores from both rounds are tabulated to select the winners.

Details about the winning products can be found at https://siia.net/codie/2023-codie-business-technology-winners/.

Learn more about Unravel’s award-winning data observability platform.

About the CODiE Awards

The SIIA CODiE Awards is the only peer-reviewed program to showcase business and education technology’s finest products and services. Since 1986, thousands of products, services and solutions have been recognized for achieving excellence. For more information, visit siia.net/CODiE.

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact

Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data Recognized by SIIA as Best Data Tool & Platform at 2023 CODiE Awards appeared first on Unravel.

Check out the 2023 Intellyx Analyst Guide for Unravel, Demystifying Data Observability, for an independent discussion on the specific requirements and bottlenecks of data-dependent applications/pipelines that are addressed by data observability.

Discover:

• Why DataOps needs its own observability
• How DevOps and DataOps are similar–and how they’re very different
• How the emerging discipline of DataFinOps is more than cost governance
• Unique considerations of DataFinOps
• DataOps resiliency and tracking down toxic workloads

The post Demystifying Data Observability appeared first on Unravel.

Overview: The Right Instance Type

This is the first blog in a five-blog series. For an overview of this blog series please review my post All Data Ecosystems Are Real-Time, It Is Just a Matter of Time.

Choosing the right instance type in the cloud is an important decision that can have a significant impact on the performance, cost, and scalability of your applications. Here are some steps to help you choose the right instance types:

1. Determine your application requirements. Start by identifying the resource requirements of your application, such as CPU, memory, storage, and network bandwidth. You should also consider any special hardware or software requirements, such as GPUs for machine learning workloads.
2. Evaluate the available instance types. Each cloud provider offers a range of instance types with varying amounts of resources, performance characteristics, and pricing. Evaluate the available instance types and their specifications to determine which ones best match your application requirements.
3. Consider the cost. The cost of different instance types can vary significantly, so it’s important to consider the cost implications of your choices. You should consider not only the hourly rate but also the overall cost over time, taking into account any discounts or usage commitments.
4. Optimize for scalability. As your application grows, you may need to scale up or out by adding more instances. Choose instance types that can easily scale horizontally (i.e., adding more instances) or vertically (i.e., upgrading to a larger instance type).
5. Test and optimize. Once you have chosen your instance types, test your application on them to ensure that it meets your performance requirements. Monitor the performance of your application and optimize your instance types as necessary to achieve the best balance of performance and cost.

Choosing the right instance types in the cloud requires careful consideration of your application requirements, available instance types, cost, scalability, and performance. By following these steps, you can make informed decisions and optimize your cloud infrastructure to meet your business needs.

Determine Application Requirements

Determining your application’s CPU, memory, storage, and network requirements is a crucial step in choosing the right instance types in the cloud. Here are some steps to help you determine these requirements:

1. CPU requirements Start by identifying the CPU-intensive tasks in your application, such as video encoding, machine learning, or complex calculations. Determine the number of cores and clock speed required for these tasks, as well as any requirements for CPU affinity or hyperthreading.
2. Memory requirements Identify the memory-intensive tasks in your application, such as caching, database operations, or in-memory processing. Determine the amount of RAM required for these tasks, as well as any requirements for memory bandwidth or latency.
3. Storage requirements Determine the amount of storage required for your application data, as well as any requirements for storage performance (e.g., IOPS, throughput) or durability (e.g., replication, backup).
4. Network requirements Identify the network-intensive tasks in your application, such as data transfer, web traffic, or distributed computing. Determine the required network bandwidth, latency, and throughput, as well as any requirements for network security (e.g., VPN, encryption).

To determine these requirements, you can use various monitoring and profiling tools to analyze your application’s resource usage, such as CPU and memory utilization, network traffic, and storage I/O. You can also use benchmarks and performance tests to simulate different workloads and measure the performance of different instance types.

By understanding your application’s resource requirements, you can choose the right instance types in the cloud that provide the necessary CPU, memory, storage, and network resources to meet your application’s performance and scalability needs.

Evaluate the Available Instance Types

Evaluating the available instance types in a cloud provider requires careful consideration of several factors, such as the workload requirements, the performance characteristics of the instances, and the cost. Here are some steps you can take to evaluate the available instance types in a cloud provider:

1. Identify your workload requirements. Before evaluating instance types, you should have a clear understanding of your workload requirements. For example, you should know the amount of CPU, memory, and storage your application needs, as well as any specific networking or GPU requirements.
2. Review the instance types available. Cloud providers offer a range of instance types with varying configurations and performance characteristics. You should review the available instance types and select the ones that are suitable for your workload requirements.
3. Evaluate performance characteristics. Each instance type has its own performance characteristics, such as CPU speed, memory bandwidth, and network throughput. You should evaluate the performance characteristics of each instance type to determine if they meet your workload requirements.
4. Consider cost. The cost of each instance type varies based on the configuration and performance characteristics. You should evaluate the cost of each instance type and select the ones that are within your budget.
5. Conduct benchmarks. Once you have selected a few instance types that meet your workload requirements and budget, you should conduct benchmarks to determine which instance type provides the best performance for your workload.
6. Consider other factors. Apart from performance and cost, you should also consider other factors such as availability, reliability, and support when evaluating instance types.

The best way to evaluate the available instance types in a cloud provider is to carefully consider your workload requirements, performance characteristics, and cost, and conduct benchmarks to determine the best instance type for your specific use case.

Consider the Cost

When evaluating cloud instance types, it is important to consider both the hourly rate and the overall cost over time, as these factors can vary significantly depending on the provider and the specific instance type. Here are some steps you can take to determine the hourly rate and overall cost over time for cloud instance types:

1. Identify the instance types you are interested in. Before you can determine the hourly rate and overall cost over time, you need to identify the specific instance types you are interested in.
2. Check the hourly rate. Most cloud providers offer a pricing calculator that allows you to check the hourly rate for a specific instance type. You can use this calculator to compare the hourly rate for different instance types and providers.
3. Consider the length of time you will use the instance. While the hourly rate is an important consideration, it is also important to consider the length of time you will use the instance. If you plan to use the instance for a long period of time, it may be more cost-effective to choose an instance type with a higher hourly rate but lower overall cost over time.
4. Look for cost-saving options. Many cloud providers offer cost-saving options such as reserved instances or spot instances. These options can help reduce the overall cost over time, but may require a longer commitment or be subject to availability limitations.
5. Factor in any additional costs. In addition to the hourly rate, there may be additional costs such as data transfer fees or storage costs that can significantly impact the overall cost over time.
6. Consider the potential for scaling. If you anticipate the need for scaling in the future, you should also consider the potential cost implications of adding additional instances over time.

By carefully considering the hourly rate, overall cost over time, and any additional costs or cost-saving options, you can make an informed decision about the most cost-effective instance type for your specific use case.

Optimize for Scalability

To optimize resources for scalability in the cloud, you can follow these best practices:

1. Design for scalability. When designing your architecture, consider the needs of your application and design it to scale horizontally. This means adding more resources, such as servers or containers, to handle an increase in demand.
2. Use auto-scaling. Auto-scaling allows you to automatically increase or decrease the number of resources based on the demand for your application. This helps ensure that you are using only the necessary resources at any given time, and can also save costs by reducing resources during low demand periods.
3. Use load balancing. Load balancing distributes incoming traffic across multiple resources, which helps prevent any one resource from being overloaded. This also helps with failover and disaster recovery.
4. Use caching. Caching can help reduce the load on your servers by storing frequently accessed data in a cache. This reduces the number of requests that need to be processed by your servers, which can improve performance and reduce costs.
5. Use cloud monitoring. Cloud monitoring tools can help you identify potential performance issues and bottlenecks before they become problems. This can help you optimize your resources more effectively and improve the overall performance of your application.
6. Use serverless architecture. With serverless architecture, you don’t need to manage servers or resources. Instead, you pay only for the resources that your application uses, which can help you optimize your resources and reduce costs.

By following the above best practices, you can optimize your resources for scalability in the cloud and ensure that your application can handle an increase in demand.

Test & Optimize

Testing and optimizing your cloud environment is a critical aspect of ensuring that your applications and services are performing optimally and that you’re not overspending on cloud resources. Here are some steps you can take to test and optimize your cloud environment:

1. Set up monitoring. Before you can start testing and optimizing, you need to have visibility into your cloud environment. Set up monitoring tools that can give you insights into key metrics such as CPU utilization, network traffic, and storage usage.
2. Conduct load testing. Conduct load testing to determine how your applications and services perform under different levels of traffic. This can help you identify bottlenecks and performance issues, and make optimizations to improve performance.
3. Optimize resource allocation. Make sure that you’re not overspending on cloud resources by optimizing resource allocation. This includes things like resizing virtual machines, choosing the right storage options, and using auto-scaling to automatically adjust resource allocation based on demand.
4. Implement security measures. Make sure that your cloud environment is secure by implementing appropriate security measures such as firewalls, access controls, and encryption.
5. Use automation. Automating routine tasks can help you save time and reduce the risk of errors. This includes things like automating deployments, backups, and resource provisioning.
6. Review cost optimization options. Consider reviewing your cloud provider’s cost optimization options, such as reserved instances or spot instances. These can help you save money on your cloud bill while still maintaining performance.
7. Continuously monitor and optimize. Continuous monitoring and optimization is key to ensuring that your cloud environment is performing optimally. Set up regular reviews to identify opportunities for optimization and ensure that your cloud environment is meeting your business needs.

By following these steps, you can test and optimize your cloud environment to ensure that it’s secure, cost-effective, and performing optimally.

Recap

Following the above steps will help to make informed decisions and optimize your cloud infrastructure to meet your business needs. Carefully consider your workload requirements. Evaluating performance characteristics, cost, and conducting benchmarks requires understanding your application’s resource requirements. With resource requirements you can choose the right instance types in the cloud that provide the necessary CPU, memory, storage, and network resources to meet your application’s performance and scalability needs.

The post The Modern Data Ecosystem: Choose the Right Instance appeared first on Unravel.

Monitor Cloud Resources

When monitoring cloud resources, there are several factors to consider:

1. Performance It is essential to monitor the performance of your cloud resources, including their availability, latency, and throughput. You can use metrics such as CPU usage, network traffic, and memory usage to measure the performance of your resources.
2. Scalability You should monitor the scalability of your cloud resources to ensure that they can handle changes in demand. You can use tools such as auto-scaling to automatically adjust the resources based on demand.
3. Security You must monitor the security of your cloud resources to ensure that they are protected from unauthorized access or attacks. You can use tools such as intrusion detection systems and firewalls to monitor and protect your resources.
4. Cost It is important to monitor the cost of your cloud resources to ensure that you are not overspending on resources that are not being used. You can use tools such as cost optimization and billing alerts to manage your costs.
5. Compliance If you are subject to regulatory compliance requirements, you should monitor your cloud resources to ensure that you are meeting those requirements. You can use tools such as audit logs and compliance reports to monitor and maintain compliance.
6. Availability It is important to monitor the availability of your cloud resources to ensure that they are up and running when needed. You can use tools such as load balancing and failover to ensure high availability.
7. User Experience You should also monitor the user experience of your cloud resources to ensure that they are meeting the needs of your users. You can use tools such as user monitoring and feedback to measure user satisfaction and identify areas for improvement.

Performance Monitoring

Here are some best practices for performance monitoring in the cloud:

1. Establish baselines. Establish baseline performance metrics for your applications and services. This will allow you to identify and troubleshoot performance issues more quickly.
2. Monitor resource utilization. Monitor resource utilization such as CPU usage, memory usage, network bandwidth, and disk I/O. This will help you identify resource bottlenecks and optimize resource allocation.
3. Use automated monitoring tools. Use automated monitoring tools such as CloudWatch, DataDog, and New Relic to collect performance metrics and analyze them in real time. This will allow you to identify and address performance issues as they arise.
4. Set alerts. Set up alerts for critical performance metrics such as CPU utilization, memory utilization, and network bandwidth. This will allow you to proactively address performance issues before they impact end users.
5. Use load testing. Use load testing tools to simulate heavy loads on your applications and services. This will help you identify performance bottlenecks and optimize resource allocation before going live.
6. Monitor end-user experience. Monitor end-user experience using tools such as synthetic monitoring and real user monitoring (RUM) . This will allow you to identify and address performance issues that impact end users.
7. Analyze logs. Analyze logs to identify potential performance issues. This will help you identify the root cause of performance issues and optimize resource allocation.
8. Continuously optimize. Continuously optimize your resources based on performance metrics and end-user experience. This will help you ensure that your applications and services perform at their best at all times.

Scalability Monitoring

Here are some best practices for scalability monitoring in the cloud:

1. Establish baselines. Establish baseline performance metrics for your applications and services. This will allow you to identify and troubleshoot scalability issues more quickly.
2. Monitor auto-scaling. Monitor auto-scaling metrics to ensure that your resources are scaling up or down according to demand. This will help you ensure that you have the right amount of resources available to meet demand.
3. Use load testing. Use load testing tools to simulate heavy loads on your applications and services. This will help you identify scalability bottlenecks and optimize resource allocation before going live.
4. Set alerts. Set up alerts for critical scalability metrics such as CPU utilization, memory utilization, and network bandwidth. This will allow you to proactively address scalability issues before they impact end users.
5. Use horizontal scaling. Use horizontal scaling to add more instances of your application or service to handle increased traffic. This will allow you to scale quickly and efficiently.
6. Use vertical scaling. Use vertical scaling to increase the size of your resources to handle increased traffic. This will allow you to scale quickly and efficiently.
7. Analyze logs. Analyze logs to identify potential scalability issues. This will help you identify the root cause of scalability issues and optimize resource allocation.
8. Continuously optimize. Continuously optimize your resources based on scalability metrics and end-user experience. This will help you ensure that your applications and services can handle any level of demand.

Security Monitoring

Here are some best practices for handling security monitoring in the cloud:

1. Use security services. Use cloud-based security services such as AWS Security Hub, Azure Security Center, and Google Cloud Security Command Center to centralize and automate security monitoring across your cloud environment.
2. Monitor user activity. Monitor user activity across your cloud environment, including login attempts, resource access, and changes to security policies. This will help you identify potential security threats and ensure that access is granted only to authorized users.
3. Use encryption. Use encryption to protect data at rest and in transit. This will help you protect sensitive data from unauthorized access.
4. Set up alerts. Set up alerts for critical security events such as failed login attempts, unauthorized access, and changes to security policies. This will allow you to respond quickly to security threats.
5. Use multi-factor authentication. Use multi-factor authentication to add an extra layer of security to user accounts. This will help prevent unauthorized access even if a user’s password is compromised.
6. Use firewalls. Use firewalls to control network traffic to and from your cloud resources. This will help you prevent unauthorized access and ensure that only authorized traffic is allowed.
7. Implement access controls. Implement access controls to ensure that only authorized users have access to your cloud resources. This will help you prevent unauthorized access and ensure that access is granted only to those who need it.
8. Perform regular security audits. Perform regular security audits to identify potential security threats and ensure that your cloud environment is secure. This will help you identify and address security issues before they become major problems.

Cost Monitoring

Here are some best practices for monitoring cost in the cloud:

1. Use cost management tools. Use cloud-based cost management tools such as AWS Cost Explorer, Azure Cost Management, and Google Cloud Billing to monitor and optimize your cloud costs.
2. Set budgets. Set budgets for your cloud spending to help you stay within your financial limits. This will help you avoid unexpected charges and ensure that you are using your cloud resources efficiently.
3. Monitor usage. Monitor your cloud resource usage to identify any unnecessary or underutilized resources. This will help you identify opportunities for optimization and cost savings.
4. Analyze cost data. Analyze your cost data to identify trends and areas of high spending. This will help you identify opportunities for optimization and cost savings.
5. Use cost allocation. Use cost allocation to assign costs to individual users, teams, or projects. This will help you identify which resources are being used most and which users or teams are driving up costs.
6. Use reserved instances. Use reserved instances to save money on long-term cloud usage. This will help you save money on your cloud costs over time.
7. Optimize resource allocation. Optimize your resource allocation to ensure that you are using the right amount of resources for your needs. This will help you avoid over-provisioning and under-provisioning.
8. Implement cost optimization strategies. Implement cost optimization strategies such as using spot instances, turning off non-critical resources when not in use, and using serverless architectures. This will help you save money on your cloud costs without sacrificing performance or reliability.

Compliance Monitoring

Here are some best practices for monitoring compliance in the cloud:

1. Understand compliance requirements. Understand the compliance requirements that apply to your organization and your cloud environment, such as HIPAA, PCI-DSS, or GDPR.
2. Use compliance services. Use cloud-based compliance services such as AWS Artifact, Azure Compliance Manager, and Google Cloud Compliance to streamline compliance management and ensure that you are meeting your regulatory requirements.
3. Conduct regular audits. Conduct regular audits to ensure that your cloud environment is in compliance with regulatory requirements. This will help you identify and address compliance issues before they become major problems.
4. Implement security controls. Implement security controls such as access controls, encryption, and multi-factor authentication to protect sensitive data and ensure compliance with regulatory requirements.
5. Monitor activity logs. Monitor activity logs across your cloud environment to identify potential compliance issues, such as unauthorized access or data breaches. This will help you ensure that you are meeting your regulatory requirements and protect sensitive data.
6. Use automation. Use automation tools to help you enforce compliance policies and ensure that your cloud environment is compliant with regulatory requirements.
7. Establish incident response plans. Establish incident response plans to help you respond quickly to compliance issues or data breaches. This will help you minimize the impact of any incidents and ensure that you are meeting your regulatory requirements.
8. Train your employees. Train your employees on compliance policies and procedures to ensure that they understand their roles and responsibilities in maintaining compliance with regulatory requirements. This will help you ensure that everyone in your organization is working together to maintain compliance in the cloud.

Monitor Availability

Here are some best practices for monitoring resource availability in the cloud:

1. Use monitoring services. Use cloud-based monitoring services such as AWS CloudWatch, Azure Monitor, and Google Cloud Monitoring to monitor the availability of your cloud resources.
2. Set up alerts. Set up alerts to notify you when there are issues with resource availability, such as when a server goes down or a service becomes unresponsive. This will help you respond quickly to issues and minimize downtime.
3. Monitor performance metrics. Monitor performance metrics such as CPU usage, memory usage, and network latency to identify potential issues before they become major problems. This will help you ensure that your resources are performing optimally and prevent performance issues from affecting availability.
4. Conduct regular load testing. Conduct regular load testing to ensure that your resources can handle the expected levels of traffic and usage. This will help you identify potential performance bottlenecks and ensure that your resources are available when you need them.
5. Use high availability architectures. Use high availability architectures such as load balancing, auto-scaling, and multi-region deployments to ensure that your resources are available even in the event of a failure. This will help you minimize downtime and ensure that your resources are always available.
6. Monitor service-level agreements (SLAs). Monitor SLAs to ensure that your cloud providers are meeting their service-level commitments. This will help you hold your providers accountable and ensure that your resources are available as expected.
7. Conduct disaster recovery drills. Conduct disaster recovery drills to ensure that you can recover from major outages or disasters. This will help you minimize downtime and ensure that your resources are available even in the event of a major failure.
8. Implement redundancy. Implement redundancy for critical resources to ensure that they are always available. This can include redundant servers, databases, or storage systems. This will help you ensure that your critical resources are always available and minimize downtime.

Monitor User Experience

Here are some best practices for monitoring user experience in the cloud:

1. Define user experience metrics. Define user experience metrics that are important to your business, such as page load times, error rates, and response times. This will help you track user experience and identify areas for improvement.
2. Use synthetic transactions. Use synthetic transactions to simulate user interactions with your applications and services. This will help you identify performance issues and ensure that your applications and services are delivering a good user experience.
3. Monitor real user traffic. Monitor real user traffic to identify issues that may not be apparent in synthetic transactions. This will help you understand how your users are actually using your applications and services and identify any performance issues that may be impacting the user experience.
4. Monitor third-party services. Monitor third-party services that your applications and services rely on, such as payment gateways and content delivery networks. This will help you identify any issues that may be impacting the user experience and ensure that your users have a seamless experience.
5. Use application performance management (APM) tools. Use APM tools to monitor application performance and identify potential issues that may be impacting the user experience. This will help you quickly identify and resolve issues that may be impacting your users.
6. Monitor network latency. Monitor network latency to ensure that your applications and services are delivering a good user experience. This will help you identify any network-related issues that may be impacting the user experience.
7. Set up alerts. Set up alerts to notify you when user experience metrics fall below acceptable levels. This will help you respond quickly to issues and ensure that your users have a good experience.
8. Continuously test and optimize. Continuously test and optimize your applications and services to ensure that they are delivering a good user experience. This will help you identify and fix issues before they impact your users and ensure that your applications and services are always performing optimally.

Recap

When monitoring cloud resources, there are several factors to consider. First, performance. It is essential to monitor the performance of your cloud resources, including their availability, latency, and throughput. This will allow you to identify and address performance issues that impact end users. You can use tools such as cost optimization and billing alerts to manage your costs. This will help you avoid unexpected charges and ensure that you are using your cloud resources efficiently. Conduct regular load testing to ensure that your resources can handle the expected levels of traffic and usage. Define user experience metrics that are important to your business, such as page load times, error rates, and response times.

The post The Modern Data Ecosystem: Monitor Cloud Resources appeared first on Unravel.

Use Managed Services

Using managed services in the cloud can help you reduce your operational burden, increase efficiency, and improve scalability. However, to fully realize the benefits of managed services, you need to follow some best practices. Here are some best practices to consider when using managed services in the cloud:

1. Understand your service-level agreements (SLAs). Before using any managed service, you should understand the SLAs offered by your cloud provider. This will help you understand the availability and reliability of the service, as well as any limitations or restrictions that may impact your use of the service.
2. Choose the right service. You should choose the right managed service for your needs. This means selecting a service that meets your requirements and offers the features and functionality you need. You should also consider the cost of the service and how it will impact your budget.
3. Plan for scalability. Managed services in the cloud are designed to be highly scalable, so you should plan for scalability when using them. This means understanding how the service will scale as your needs change and ensuring that you can easily scale the service up or down as required.
4. Monitor service performance. You should monitor the performance of your managed services to ensure that they are meeting your expectations. This may involve setting up monitoring tools to track service usage, performance, and availability. You should also define appropriate thresholds and alerts to notify you when issues arise.
5. Secure your services. Security is critical when using managed services in the cloud. You should ensure that your services are secured according to best practices, such as using strong passwords, encrypting data in transit and at rest, and implementing access controls. You should also regularly audit your services to ensure that they remain secure.
6. Stay up-to-date. Managed services in the cloud are continually evolving, so you should stay up-to-date with the latest features, updates, and releases. This will help you take advantage of new features and ensure that your services are up-to-date and secure.

By following these best practices, you can ensure that your managed services in the cloud are efficient, reliable, and secure.

Understand Your Service-Level Agreements (SLAs)

Understanding cloud service-level agreements (SLAs) is crucial when you use cloud services. SLAs define the level of service you can expect from a cloud provider and outline the terms and conditions of the service. Here are some ways to help you understand cloud SLAs:

1. Read the SLA. The best way to understand cloud SLAs is to read the SLA itself. The SLA provides details on what services are offered, how they are delivered, and what level of availability you can expect. It also outlines the terms and conditions of the service and what you can expect in the event of an outage or other issues.
2. Understand the metrics. Cloud SLAs typically include metrics that define the level of service you can expect. These metrics may include availability, performance, and response time. You should understand how these metrics are measured and what level of service is guaranteed for each metric.
3. Know the guarantees. The SLA also specifies the guarantees that the cloud provider offers for each metric. You should understand what happens if the provider fails to meet these guarantees, such as compensation or penalties.
4. Identify exclusions. The SLA may also include exclusions or limitations to the service, such as scheduled maintenance, force majeure events, or issues caused by your own actions. You should understand what these exclusions are and how they may impact your use of the service.
5. Ask questions. If you are unsure about any aspect of the SLA, you should ask questions. The cloud provider should be able to provide clarification on any issues and help you understand the SLA better.
6. Get expert help. If you are still unsure about the SLA or need help negotiating SLAs with multiple providers, consider getting expert help. Cloud consultants or legal advisors can help you understand the SLA better and ensure that you get the best possible terms for your needs.

By following these steps, you can better understand cloud SLAs and make informed decisions about the cloud services you use.

Choose the Right Service

Choosing the right cloud service is a critical decision that can have a significant impact on your organization. Here are some factors to consider when choosing a cloud service:

1. Business needs The first step in choosing a cloud service is to understand your business needs. What are your specific requirements? What do you need the cloud service to do? Consider factors such as scalability, security, compliance, and cost when evaluating your options.
2. Reliability and availability Reliability and availability are critical when choosing a cloud service. Look for a provider with a strong track record of uptime and availability. Also, ensure that the provider has a robust disaster recovery plan in place in case of service disruptions or outages.
3. Security Security is a top priority when using cloud services. Choose a provider that has robust security measures in place, such as encryption, access controls, and multi-factor authentication. Also, consider whether the provider meets any relevant compliance requirements, such as HIPAA or GDPR.
4. Cost Cost is another critical factor to consider when choosing a cloud service. Look for a provider that offers transparent pricing and provides a clear breakdown of costs. Also, consider any hidden fees or charges, such as data transfer costs or support fees.
5. Support Choose a cloud service provider that offers robust support options, such as 24/7 customer support or online resources. Ensure that the provider has a reputation for providing excellent support and responding quickly to issues.
6. Integration Ensure that the cloud service provider integrates with any existing systems or applications that your organization uses. This can help reduce the complexity of your IT environment and improve productivity.
7. Scalability Choose a cloud service provider that can scale as your needs change. Ensure that the provider can handle your growth and provides flexibility in terms of scaling up or down.

By considering these factors, you can choose the right cloud service that meets your business needs, is secure, reliable, and scalable, and provides good value.

Plan for Scalability

Scalability is a key advantage of cloud computing, allowing organizations to quickly and easily increase or decrease resources as needed. Here are some best practices for planning for scalability in the cloud:

1. Start with a solid architecture. A solid architecture is essential for building scalable cloud applications. Ensure that your architecture is designed to support scalability from the beginning, by leveraging horizontal scaling, load balancing, and other cloud-native capabilities.
2. Automate everything. Automation is critical for scaling in the cloud. Automate deployment, configuration, and management tasks to reduce manual effort and increase efficiency. Use tools like cloud orchestration or infrastructure-as-code (IAC) to automate the provisioning and configuration of resources.
3. Use elasticity. Elasticity is the ability to automatically adjust resource capacity to meet changes in demand. Use auto-scaling to automatically increase or decrease resources based on utilization or other metrics. This can help ensure that you always have the right amount of resources to handle traffic spikes or fluctuations.
4. Monitor and optimize. Monitoring is critical for maintaining scalability. Use monitoring tools to track application performance and identify potential bottlenecks or areas for optimization. Optimize your applications, infrastructure, and processes to ensure that you can scale as needed without encountering issues.
5. Plan for failure. Scalability also means being prepared for failure. Ensure that your application is designed to handle failures and that you have a plan in place for dealing with outages or other issues. Use fault tolerance and high availability to ensure that your application can continue running even if a component fails.

By following these best practices, you can plan for scalability in the cloud and ensure that your applications can handle changes in demand without encountering issues.

Monitor Service Performance

Monitoring service performance is essential to ensure that your cloud applications are running smoothly and meeting service-level agreements (SLAs). Here are some best practices for monitoring service performance in the cloud:

1. Define key performance indicators (KPIs). Define KPIs that are relevant to your business needs, such as response time, throughput, and error rates. These KPIs will help you track how well your applications are performing and whether they are meeting your SLAs.
2. Use monitoring tools. Use monitoring tools to collect and analyze data on your application’s performance. These tools can help you identify issues before they become critical and track how well your application is meeting your KPIs.
3. Set alerts. Set up alerts based on your KPIs to notify you when something goes wrong. This can help you quickly identify and resolve issues before they impact your application’s performance.
4. Monitor end-to-end performance. Monitor end-to-end performance, including network latency, database performance, and third-party services, to identify any potential bottlenecks or issues that could impact your application’s performance.
5. Analyze and optimize. Analyze performance data to identify patterns and trends. Use this information to optimize your application’s performance and identify areas for improvement. Optimize your application code, database queries, and network configurations to improve performance.
6. Use machine learning. Leverage machine learning to analyze performance data and identify anomalies or issues automatically. This can help you identify issues before they become critical and take proactive steps to resolve them.

By following these best practices, you can monitor service performance in the cloud effectively and ensure that your applications are meeting your SLAs and delivering the best possible user experience.

Secure Your Services

Securing your services in the cloud is critical to protect your data and applications from cyber threats. Here are some best practices for securing your services in the cloud:

1. Implement strong access control. Implement strong access control mechanisms to restrict access to your cloud resources. Use least privilege principles to ensure that users only have access to the resources they need. Implement multi-factor authentication (MFA) and use strong passwords to protect against unauthorized access.
2. Encrypt your data. Encrypt your data both at rest and in transit to protect against data breaches. Use SSL/TLS protocols for data in transit and encryption mechanisms like AES or RSA for data at rest. Additionally, consider encrypting data before it is stored in the cloud to provide an additional layer of protection.
3. Implement network security. Implement network security measures to protect against network-based attacks, such as DDoS attacks, by using firewalls, intrusion detection/prevention systems (IDS/IPS), and VPNs. Segregate your network into logical segments to reduce the risk of lateral movement by attackers.
4. Use security groups and network ACLs. Use security groups and network ACLs to control inbound and outbound traffic to your resources. Implement granular rules to restrict traffic to only what is necessary, and consider using security groups and network ACLs together to provide layered security.
5. Implement logging and monitoring. Implement logging and monitoring to detect and respond to security incidents. Use cloud-native tools like CloudTrail or CloudWatch to monitor activity in your environment and alert you to any unusual behavior.
6. Perform regular security audits. Perform regular security audits to identify potential vulnerabilities and ensure that your security controls are effective. Conduct regular penetration testing and vulnerability assessments to identify and remediate any weaknesses in your environment.

By following these best practices, you can secure your services in the cloud and protect your applications and data from cyber threats.

Stay Up-to-Date on Managed Services

Staying up to date on managed services in the cloud is essential to ensure that you are using the latest features and capabilities and making the most out of your cloud investment. Here are some ways to stay up-to-date on managed services in the cloud:

1. Subscribe to cloud service providers’ blogs. Cloud service providers like AWS, Google Cloud, and Microsoft Azure regularly publish blog posts announcing new features and services. By subscribing to these blogs, you can stay up-to-date on the latest developments and updates.
2. Attend cloud conferences. Attending cloud conferences like AWS re:Invent, Google Cloud Next, and Microsoft Ignite is an excellent way to learn about new and upcoming managed services in the cloud. These events feature keynote speeches, technical sessions, and hands-on workshops that can help you stay up-to-date with the latest trends and technologies.
3. Join cloud user groups. Joining cloud user groups like AWS User Group, Google Cloud User Group, and Azure User Group is a great way to connect with other cloud professionals and learn about new managed services in the cloud. These groups often hold meetings and events where members can share their experiences and discuss the latest developments.
4. Participate in online communities. Participating in online communities like Reddit, Stack Overflow, and LinkedIn Groups is an excellent way to stay up-to-date on managed services in the cloud. These communities often have active discussions about new features and services, and members can share their experiences and insights.
5. Follow industry experts. Following industry experts on social media platforms like Twitter, LinkedIn, and Medium is an excellent way to stay up-to-date on managed services in the cloud. These experts often share their thoughts and insights on the latest developments and can provide valuable guidance and advice.

By following these methods, you can stay up-to-date on managed services in the cloud and ensure that you are using the latest features and capabilities to achieve your business goals.

Recap

Understand your cloud SLAs and make informed decisions about the cloud services you use. By following these best practices, you can ensure that your managed services in the cloud are efficient, reliable, and secure. Leveraging scalability in the cloud ensures that your applications can handle changes in demand without encountering issues. Ensuring that your applications are meeting your SLAs and delivering the best possible user experience requires constant vigilance. Make sure you can secure your services in the cloud and protect your applications and data from cyber threats. By considering the right factors, you can choose the right cloud service that meets your business needs, is secure, reliable, and scalable, and provides good value. Ultimately this helps achieve key business objectives for every customer using the cloud.

The post The Modern Data Ecosystem: Use Managed Services appeared first on Unravel.

Optimize Storage

There are several ways to optimize cloud storage, depending on your specific needs and circumstances. Here are some general tips that can help:

1. Understand your data. Before you start optimizing, it’s important to understand what data you have and how it’s being used. This can help you identify which files or folders are taking up the most space, and which ones are being accessed the most frequently.
2. Use storage compression. Compression can reduce the size of your files, which can save you storage space and reduce the amount of data you need to transfer over the network. However, keep in mind that compressed files may take longer to access and may not be suitable for all types of data.
3. Use deduplication. Deduplication can identify and eliminate duplicate data, which can save you storage space and reduce the amount of data you need to transfer over the network. However, keep in mind that deduplication may increase the amount of CPU and memory resources required to manage your data.
4. Choose the right storage class. Most cloud storage providers offer different storage classes that vary in performance, availability, and cost. Choose the storage class that best meets your needs and budget.
5. Set up retention policies. Retention policies can help you automatically delete old or outdated data, which can free up storage space and reduce your storage costs. However, be careful not to delete data that you may need later.
6. Monitor your usage. Regularly monitor your cloud storage usage to ensure that you’re not exceeding your storage limits or paying for more storage than you need. You can use cloud storage monitoring tools or third-party services to help you with this.
7. Consider a multi-cloud strategy. If you have very large amounts of data, you may want to consider using multiple cloud storage providers to spread your data across multiple locations. This can help you optimize performance, availability, and cost, while also reducing the risk of data loss.

Overall, optimizing cloud storage requires careful planning, monitoring, and management. By following these tips, you can reduce your storage costs, improve your data management, and get the most out of your cloud storage investment.

Understand Your Data

Analyzing data in the cloud can be a powerful way to gain insights and extract value from large datasets. Here are some best practices for analyzing data in the cloud:

1. Choose the right cloud platform. There are several cloud platforms available, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform. Choose the one that suits your needs and budget.
2. Store data in a scalable, secure, and cost-effective way. You can store data in cloud-based databases, data lakes, or data warehouses. Make sure that you choose a storage solution that is scalable, secure, and cost-effective.
3. Choose the right data analysis tool. There are several cloud-based data analysis tools available, such as Amazon SageMaker, Microsoft Azure Machine Learning, and Google Cloud AI Platform. Choose the one that suits your needs and budget.
4. Prepare data for analysis. Data preparation involves cleaning, transforming, and structuring the data for analysis. This step is crucial for accurate analysis results.
5. Choose the right analysis technique. Depending on the nature of the data and the business problem you are trying to solve, you may choose from various analysis techniques such as descriptive, diagnostic, predictive, or prescriptive.
6. Visualize data. Visualization helps to communicate insights effectively. Choose a visualization tool that suits your needs and budget.
7. Monitor and optimize performance. Monitor the performance of your data analysis system and optimize it as necessary. This step helps to ensure that you get accurate and timely insights from your data.

Overall, analyzing data in the cloud can be a powerful way to gain insights and extract value from large datasets. By following these best practices, you can ensure that you get the most out of your cloud-based data analysis system.

Use Storage Compression

Storage compression is a useful technique for reducing storage costs and improving performance in the cloud. Here are some best practices for using storage compression in the cloud:

1. Choose the right compression algorithm. There are several compression algorithms available, such as gzip, bzip2, and LZ4. Choose the algorithm that suits your needs and budget. Consider factors such as compression ratio, speed, and memory usage.
2. Compress data at the right time. Compress data when it is written to storage or when it is not frequently accessed. Avoid compressing data that is frequently accessed, as this can slow down performance.
3. Monitor compression performance. Monitor the performance of your compression system to ensure that it is not slowing down performance. Use tools such as monitoring dashboards to track the performance of your system.
4. Test compression performance. Test the performance of your compression system with different types of data to ensure that it is effective. Consider testing with data that has varying levels of redundancy, such as log files or images.
5. Use compression in conjunction with other techniques. Consider using compression in conjunction with other storage optimization techniques such as deduplication, tiering, and archiving. This can further reduce storage costs and improve performance.
6. Consider the cost of decompression. Decompressing data can be a resource-intensive process. Consider the cost of decompression when choosing a compression algorithm and when designing your storage architecture.

Overall, using storage compression in the cloud can be an effective way to reduce storage costs and improve performance. By following these best practices, you can ensure that you get the most out of your storage compression system.

Data Deduplication 

Deduplication is a technique used to reduce the amount of data stored in the cloud by identifying and removing duplicate data. Here are some best practices for deduplicating cloud data:

1. Choose the right deduplication algorithm. There are several deduplication algorithms available, such as content-defined chunking and fixed-size chunking. Choose the algorithm that suits your needs and budget. Consider factors such as data type, deduplication ratio, and resource usage.
2. Deduplicate data at the right time. Deduplicate data when it is written to storage or when it is not frequently accessed. Avoid deduplicating data that is frequently accessed, as this can slow down performance.
3. Monitor deduplication performance. Monitor the performance of your deduplication system to ensure that it is not slowing down performance. Use tools such as monitoring dashboards to track the performance of your system.
4. Test deduplication performance. Test the performance of your deduplication system with different types of data to ensure that it is effective. Consider testing with data that has varying levels of redundancy, such as log files or images.
5. Consider the tradeoff between storage cost and compute cost. Deduplicating data can be a resource-intensive process. Consider the tradeoff between storage cost and compute cost when choosing a deduplication algorithm and when designing your storage architecture.
6. Use deduplication in conjunction with other techniques. Consider using deduplication in conjunction with other storage optimization techniques such as compression, tiering, and archiving. This can further reduce storage costs and improve performance.

Overall, deduplicating cloud data can be an effective way to reduce storage costs and improve performance. By following these best practices, you can ensure that you get the most out of your deduplication system.

Use the Right Storage Class

Choosing the right storage class for data in the cloud involves considering factors such as access frequency, durability, availability, and cost. Here are some steps to follow when choosing the right storage class for your data in the cloud:

1. Determine your access needs. Consider how frequently you need to access your data. If you need to access your data frequently, you should choose a storage class that provides low latency and high throughput. If you don’t need to access your data frequently, you can choose a storage class that provides lower performance and lower cost.
2. Consider your durability needs. Durability refers to the probability of losing data due to hardware failure. If your data is critical and needs high durability, you should choose a storage class that provides high durability, such as Amazon S3 Standard or Google Cloud Storage Nearline.
3. Evaluate your availability needs. Availability refers to the ability to access your data when you need it. If your data is critical and needs high availability, you should choose a storage class that provides high availability, such as Amazon S3 Standard or Google Cloud Storage Nearline.
4. Determine your cost needs. Cost is also an important factor when choosing a storage class. If you have a limited budget, you should choose a storage class that provides lower cost, such as Amazon S3 Infrequent Access or Google Cloud Storage Coldline.
5. Consider any compliance requirements. Some industries have compliance requirements that dictate how data must be stored. If you have compliance requirements, you should choose a storage class that meets those requirements.
6. Consider data lifecycle management. Depending on the type of data, you may need to store it for a certain period of time before deleting it. Some storage classes may provide lifecycle management features to help you manage your data more efficiently.

By considering these factors, you can choose the right storage class for your data in the cloud that meets your needs and helps you save costs.

Set Data Retention Policies

Setting up retention policies for your cloud data is an important step in managing your data and ensuring that you are in compliance with regulatory requirements. Here are some steps you can follow to set up retention policies for your cloud data:

1. Identify the types of data you need to retain. The first step in setting up retention policies is to identify the types of data that you need to retain. This could include data related to financial transactions, employee records, customer information, and other types of data that are important for your business.
2. Determine the retention periods. Next, you will need to determine how long each type of data needs to be retained. This will depend on the regulatory requirements for your industry as well as your own internal policies. 
3. Decide on the retention strategy. There are several different retention strategies you can use for your cloud data. For example, you could choose to retain all data for a certain period of time, or you could choose to delete data after a certain period of time has elapsed. You could also choose to retain data based on certain triggers, such as when a legal or regulatory inquiry is initiated.
4. Implement the retention policies. Once you have determined the types of data you need to retain, the retention periods, and the retention strategy, you can implement your retention policies in your cloud storage provider. Most cloud storage providers have built-in tools for setting up retention policies.
5. Monitor the retention policies. It’s important to regularly monitor your retention policies to ensure that they are working as intended. You should periodically review the types of data being retained, the retention periods, and the retention strategy to ensure that they are still appropriate. You should also regularly audit your retention policies to ensure that they are in compliance with any changes in regulatory requirements.

By following these steps, you can set up retention policies for your cloud data that will help you manage your data effectively, ensure compliance with regulatory requirements, and reduce your risk of data breaches or loss.

Monitor Usage 

Monitoring your cloud usage is essential for managing your costs, optimizing your resources, and ensuring the security of your data. Here are some of the best ways to monitor your cloud usage:

1. Cloud provider monitoring tools Most cloud providers offer built-in monitoring tools that allow you to track your usage, monitor your costs, and receive alerts when you approach your resource limits. These tools typically provide real-time insights into your cloud usage and can help you identify areas where you can optimize your resources.
2. Third-party monitoring tools There are many third-party monitoring tools available that can help you monitor your cloud usage across multiple cloud providers. These tools offer more advanced features and can help you identify usage patterns, forecast future usage, and detect anomalies that may indicate security threats or performance issues.
3. Cost optimization tools Cost optimization tools can help you identify areas where you can reduce your costs, such as by using more efficient resource configurations or by identifying idle resources that can be decommissioned. These tools typically integrate with your cloud provider’s monitoring tools to provide a comprehensive view of your usage and costs.
4. Security and compliance tools Security and compliance tools can help you monitor your cloud usage for security threats and compliance violations. These tools typically monitor your cloud resources for suspicious activity, such as unauthorized access attempts, and can help you stay in compliance with regulatory requirements.
5. Regular audits Regular audits of your cloud usage can help you identify areas where you can optimize your resources, reduce costs, and improve security. You should periodically review your cloud usage and costs, and adjust your resources and policies as necessary to ensure that you are getting the most value from your cloud investment.

By using these monitoring tools and strategies, you can gain better visibility into your cloud usage, optimize your resources, reduce costs, and ensure the security and compliance of your cloud resources.

Pursue a Multi-Cloud Strategy

Pursuing a multi-cloud strategy can offer several benefits, such as increased resilience, reduced vendor lock-in, and improved performance. However, there are several considerations you should keep in mind before pursuing a multi-cloud strategy. Here are some of the key considerations:

1. Business objectives The first consideration is your business objectives. You need to determine why you want to pursue a multi-cloud strategy and what you hope to achieve. For example, you may be looking to improve the resilience of your applications or reduce vendor lock-in.
2. Compatibility The next consideration is the compatibility of your applications and workloads across different cloud providers. You need to ensure that your applications and workloads are compatible with the different cloud providers you plan to use. You may need to modify your applications and workloads to ensure they can run on multiple cloud platforms.
3. Data management Another important consideration is data management. You need to ensure that your data is managed securely and efficiently across all the cloud providers you use. This may involve implementing data management policies and tools to ensure that your data is always available and protected.
4. Cost management Managing costs is also a critical consideration. You need to ensure that you can manage costs effectively across all the cloud providers you use. This may involve using cost management tools and monitoring usage and costs to identify areas where you can optimize spending.
5. Security Security is always a key consideration, but it becomes even more important when using multiple cloud providers. You need to ensure that your applications and data are secure across all the cloud providers you use. This may involve implementing security policies and using security tools to detect and respond to security threats.
6. Skills and resources Finally, you need to consider the skills and resources required to manage a multi-cloud environment. This may involve hiring additional staff or up-skilling existing staff to ensure that they have the necessary expertise to manage a multi-cloud environment.

By considering these key factors, you can develop a successful multi-cloud strategy that meets your business objectives and helps you achieve your goals.

Recap

Optimizing cloud storage requires careful planning, monitoring, and management. Analyzing data in the cloud can be a powerful way to gain insights and extract value. Using storage compression is an effective way to reduce storage costs and improve performance. Using monitoring tools and strategies, you can gain better visibility into your cloud usage, optimize your resources, reduce costs, and ensure the security and compliance of your cloud resources. By considering these key factors, you can develop a successful multi-cloud strategy that meets your business objectives and helps you achieve your goals. By following these tips, you can reduce your storage costs, improve your data management, and get the most out of your cloud storage investment.

The post The Modern Data Ecosystem: Optimize Your Storage appeared first on Unravel.

Auto-Scaling Overview

This is the second blog in a five-blog series. For an overview of this blog series, please review my post All Data Ecosystems Are Real-Time, It Is Just a Matter of Time. The series should be read in order.

Auto-scaling is a powerful feature of cloud computing that allows you to automatically adjust the resources allocated to your applications based on changes in demand. Here are some best practices for using auto-scaling in the cloud:

1. Set up appropriate triggers. Set up triggers based on metrics such as CPU utilization, network traffic, or memory usage to ensure that your application scales up or down when needed.
2. Use multiple availability zones. Deploy your application across multiple availability zones to ensure high availability and reliability. This will also help you to avoid any potential single points of failure.
3. Start with conservative settings. Start with conservative settings for scaling policies to avoid over-provisioning or under-provisioning resources. You can gradually increase the thresholds as you gain more experience with your application.
4. Monitor your auto-scaling. Regularly monitor the performance of your auto-scaling policies to ensure that they are working as expected. You can use monitoring tools such as CloudWatch to track metrics and troubleshoot any issues.
5. Use automated configuration management. Use tools such as Chef, Puppet, or Ansible to automate the configuration management of your application. This will make it easier to deploy and scale your application across multiple instances.
6. Test your auto-scaling policies. Test your auto-scaling policies under different load scenarios to ensure that they can handle sudden spikes in traffic. You can use load testing tools such as JMeter or Gatling to simulate realistic load scenarios.

By following these best practices, you can use auto-scaling in the cloud to improve the availability, reliability, and scalability of your applications.

Set Up Appropriate Triggers

Setting up appropriate triggers is an essential step when using auto-scaling in the cloud. Here are some best practices for setting up triggers:

1. Identify the right metrics. Start by identifying the metrics that are most relevant to your application. Common metrics include CPU utilization, network traffic, and memory usage. You can also use custom metrics based on your application’s specific requirements.
2. Determine threshold values. Once you have identified the relevant metrics, determine the threshold values that will trigger scaling. For example, you might set a threshold of 70% CPU utilization to trigger scaling up, and 30% CPU utilization to trigger scaling down.
3. Set up alarms. Set up CloudWatch alarms to monitor the relevant metrics and trigger scaling based on the threshold values you have set. For example, you might set up an alarm to trigger scaling up when CPU utilization exceeds 70% for a sustained period of time.
4. Use hysteresis. To avoid triggering scaling up and down repeatedly in response to minor fluctuations in metrics, use hysteresis. Hysteresis introduces a delay before triggering scaling in either direction, helping to ensure that scaling events are only triggered when they are really needed.
5. Consider cooldown periods. Cooldown periods introduce a delay between scaling events, helping to prevent over-provisioning or under-provisioning of resources. When a scaling event is triggered, a cooldown period is started during which no further scaling events will be triggered. This helps to ensure that the system stabilizes before further scaling events are triggered.

By following these best practices, you can set up appropriate triggers for scaling in the cloud, ensuring that your application can scale automatically in response to changes in demand.

Use Multiple Availability Zones

Using multiple availability zones is a best practice in the cloud to improve the availability and reliability of your application. Here are some best practices for using multiple availability zones:

1. Choose an appropriate region. Start by choosing a region that is geographically close to your users to minimize latency. Consider the regulatory requirements, cost, and availability of resources when choosing a region.
2. Deploy across multiple availability zones. Deploy your application across multiple availability zones within the same region to ensure high availability and fault tolerance. Availability zones are isolated data centers within a region that are designed to be independent of each other.
3. Use load balancers. Use load balancers to distribute traffic across multiple instances in different availability zones. This helps to ensure that if one availability zone goes down, traffic can be automatically redirected to other availability zones.
4. Use cross-zone load balancing. Enable cross-zone load balancing to distribute traffic evenly across all available instances, regardless of which availability zone they are in. This helps to ensure that instances in all availability zones are being utilized evenly.
5. Monitor availability zones. Regularly monitor the availability and performance of instances in different availability zones. You can use CloudWatch to monitor metrics such as latency, network traffic, and error rates, and to set up alarms to alert you to any issues.
6. Use automatic failover. Configure automatic failover for your database and other critical services to ensure that if one availability zone goes down, traffic can be automatically redirected to a standby instance in another availability zone.

By following these best practices, you can use multiple availability zones in the cloud to improve the availability and reliability of your application, and to minimize the impact of any potential disruptions.

Start with Conservative Settings

Over-provisioning or under-provisioning resources can lead to wasted resources or poor application performance, respectively. Here are some best practices to avoid these issues:

1. Monitor resource usage. Regularly monitor the resource usage of your application, including CPU, memory, storage, and network usage. Use monitoring tools such as CloudWatch to collect and analyze metrics, and set up alarms to alert you to any resource constraints.
2. Set appropriate thresholds. Set appropriate thresholds for scaling based on your application’s resource usage. Start with conservative thresholds, and adjust them as needed based on your monitoring data.
3. Use automation. Use automation tools such as AWS Auto Scaling to automatically adjust resource provisioning based on demand. This can help to ensure that resources are provisioned efficiently and that you are not over-provisioning or under-provisioning.
4. Use load testing. Use load testing tools such as JMeter or Gatling to simulate realistic traffic loads and test your application’s performance. This can help you to identify any performance issues before they occur in production.
5. Optimize application architecture. Optimize your application architecture to reduce resource usage, such as by using caching, minimizing database queries, and using efficient algorithms.
6. Use multiple availability zones. Deploy your application across multiple availability zones to ensure high availability and fault tolerance, and to minimize the impact of any potential disruptions.

By following these best practices, you can ensure that you are not over-provisioning or under-provisioning resources in your cloud infrastructure, and that your application is running efficiently and reliably.

Monitor and Auto-Scale Your Cloud

The best way to monitor and auto-scale your cloud applications is by using a combination of monitoring tools, scaling policies, and automation tools. Here are some best practices for monitoring and auto-scaling your cloud apps:

1. Monitor application performance. Use monitoring tools such as AWS CloudWatch to monitor the performance of your application. Collect metrics such as CPU utilization, memory usage, and network traffic, and set up alarms to notify you of any performance issues.
2. Define scaling policies. Define scaling policies for each resource type based on the performance metrics you are monitoring. This can include policies for scaling based on CPU utilization, network traffic, or other metrics.
3. Set scaling thresholds. Set conservative thresholds for scaling based on your initial analysis of resource usage, and adjust them as needed based on your monitoring data.
4. Use automation tools. Use automation tools to automatically adjust resource provisioning based on demand. This can help to ensure that resources are provisioned efficiently and that you are not over-provisioning or under-provisioning.
5. Use load testing. Use load testing tools such as JMeter or Gatling to simulate realistic traffic loads and test your application’s performance. This can help you to identify any performance issues before they occur in production.
6. Use multiple availability zones. Deploy your application across multiple availability zones to ensure high availability and fault tolerance, and to minimize the impact of any potential disruptions.
7. Monitor and optimize. Regularly monitor the performance of your application and optimize your scaling policies based on the data you collect. This will help you to ensure that your application is running efficiently and reliably.

By following these best practices, you can ensure that your cloud applications are monitored and auto-scaled effectively, helping you to optimize performance and minimize the risk of downtime.

Use Automated Configuration Management

Automated configuration management in the cloud can help you manage and provision your infrastructure efficiently and consistently. Here are some best practices for using automated configuration management in the cloud:

1. Use infrastructure as code. Use infrastructure as code tools such as AWS CloudFormation or Terraform to define your infrastructure as code. This can help to ensure that your infrastructure is consistent across different environments and can be easily reproduced.
2. Use configuration management tools. Use configuration management tools such as Chef, Puppet, or Ansible to automate the configuration of your servers and applications. These tools can help you ensure that your infrastructure is configured consistently and can be easily scaled.
3. Use version control. Use version control tools such as Git to manage your infrastructure code and configuration files. This can help you to track changes to your infrastructure and roll back changes if necessary.
4. Use testing and validation. Use testing and validation tools to ensure that your infrastructure code and configuration files are valid and that your infrastructure is properly configured. This can help you to avoid configuration errors and reduce downtime.
5. Use monitoring and logging. Use monitoring and logging tools to track changes to your infrastructure and to troubleshoot any issues that arise. This can help you to identify problems quickly and resolve them before they impact your users.
6. Use automation. Use automation tools such as AWS OpsWorks or AWS CodeDeploy to automate the deployment and configuration of your infrastructure. This can help you to deploy changes quickly and efficiently.

By following these best practices, you can use automated configuration management in the cloud to manage your infrastructure efficiently and consistently, reducing the risk of configuration errors and downtime, and enabling you to scale your infrastructure easily as your needs change.

Testing Auto-Scaling Policies

Testing your auto-scaling policies is an important step in ensuring that your cloud infrastructure can handle changes in demand effectively. Here are some best practices for testing your auto-scaling policies:

1. Use realistic test scenarios. Use realistic test scenarios to simulate the traffic patterns and demand that your application may experience in production. This can help you to identify any potential issues and ensure that your auto-scaling policies can handle changes in demand effectively.
2. Test different scenarios. Test your auto-scaling policies under different scenarios, such as high traffic loads or sudden spikes in demand. This can help you to ensure that your policies are effective in a variety of situations.
3. Monitor performance. Monitor the performance of your application during testing to identify any performance issues or bottlenecks. This can help you to optimize your infrastructure and ensure that your application is running efficiently.
4. Validate results. Validate the results of your testing to ensure that your auto-scaling policies are working as expected. This can help you to identify any issues or areas for improvement.
5. Use automation tools. Use automation tools such as AWS CloudFormation or Terraform to automate the testing process and ensure that your tests are consistent and reproducible.
6. Use load testing tools. Use load testing tools such as JMeter or Gatling to simulate realistic traffic loads and test your auto-scaling policies under different scenarios.

By following these best practices, you can ensure that your auto-scaling policies are effective and can handle changes in demand effectively, reducing the risk of downtime and ensuring that your application is running efficiently.

Recap

Auto-scaling can be leveraged to improve the availability, reliability, and scalability of your applications. Multiple availability zones in the cloud improve the availability and reliability of cloud applications, and minimizes the impact of any potential disruptions. Make changes conservatively. Increase resources incrementally. This makes sure you don’t oversize. We must manage our infrastructure efficiently and consistently to reduce the risk of configuration errors and downtime. Conservative scaling enables us to scale our infrastructure easily as our needs change. Handle changes in demand effectively and reduce the risk of downtime and ensure our application is running efficiently.

The post The Modern Data Ecosystem: Use Auto-Scaling appeared first on Unravel.

Overview: Six-Part Blog

In this six-part blog I will demonstrate why what I call Services Oriented Data Architecture (SΘDΔ^®) is the right data architecture for now and the foreseeable future. I will drill into specific examples of how to build the most optimal cloud data architecture regardless of your cloud provider. This will lay the foundation for SΘDΔ^®. We will also define the Data Asset Management System(DΔḾṢ)^®. DΔḾṢ is the modern data management system approach for advanced data ecosystems. The modern data ecosystem must focus on interchangeable interoperable services and let the system focus on optimally storing, retrieving, and processing data. DΔḾṢ takes care of this for the modern data ecosystem.

We will drill into the exercises necessary to optimize the full stack of your cloud data ecosystem. These exercises will work regardless of the cloud provider. We will look at the best ways to store data regardless of type. Then we will drill into how to optimize your compute in the cloud. The compute is generally the most expensive of all cloud assets. We will also drill into how to optimize memory use. Finally, we will wrap up with examples of SΘDΔ^®.

Modern data architecture is a framework for designing, building, and managing data systems that can effectively support modern data-driven business needs. It is focused on achieving scalability, flexibility, reliability, and cost-effectiveness, while also addressing modern data requirements such as real-time data processing, machine learning, and analytics.

Some of the key components of modern data architecture include:

1. Data ingestion and integration This involves collecting and integrating data from various sources, including structured and unstructured data, and ingesting it into the data system.
2. Data storage and processing This involves storing and processing data in a scalable, distributed, and fault-tolerant manner using technologies such as cloud storage, data lakes, and data warehouses.
3. Data management and governance This involves ensuring that data is properly managed, secured, and governed, including policies around data access, privacy, and compliance.
4. Data analysis and visualization This involves leveraging advanced analytics tools and techniques to extract insights from data and present them in a way that is understandable and actionable.
5. Machine learning and artificial intelligence This involves leveraging machine learning and AI technologies to build predictive models, automate decision-making, and enable advanced analytics.
6. Data streaming and real-time processing This involves processing and analyzing data in real time, allowing organizations to respond quickly to changing business needs.

Overall, modern data architecture is designed to help organizations leverage data as a strategic asset and gain a competitive advantage by making better data-driven decisions.

Cloud Optimization Best Practices

Running efficiently on the large cloud providers requires careful consideration of various factors, including your application’s requirements, the size and type of instances needed, and the selected services to leverage.

Here are some general tips to help you run efficiently on the large cloud providers’ cloud:

1. Choose the right instance types. The large cloud providers offer a wide range of instance types optimized for different workloads. Choose the instance type that best fits your application’s requirements to avoid over-provisioning or under-provisioning.
2. Use auto-scaling. Auto-scaling allows you to scale your infrastructure up or down based on demand. This ensures that you have enough capacity to handle traffic spikes while minimizing costs during periods of low usage.
3. Optimize your storage. The large cloud providers offer various storage options, each with its own performance characteristics and costs. Select the storage type that best fits your application’s needs.
4. Use managed services. The large cloud providers provide various managed services, These services allow you to focus on your application’s business logic while the large cloud providers take care of the underlying infrastructure. SaaS vendors manage the software, and PaaS vendors manage the platform.
5. Monitor your resources. The major cloud providers provide various monitoring and logging tools that allow you to track your application’s performance and troubleshoot issues quickly. Use these tools to identify bottlenecks and optimize your infrastructure.
6. Use a content delivery network (CDN). If your application serves static content, consider using a CDN to cache content closer to your users, reducing latency and improving performance.

By following these best practices, you can ensure that your application runs efficiently on the large cloud providers, providing a great user experience while minimizing costs.

The Optimized Way to Store Data in the Cloud

The best structure for storing data for reporting depends on various factors, including the type and volume of data, the reporting requirements, and the performance considerations. Here are some general guidelines for choosing a suitable structure for storing data for reporting:

1. Use a dimensional modeling approach. Dimensional modeling is a database design technique that organizes data into dimensions and facts. It is optimized for reporting and analysis and can help simplify complex queries and improve performance. The star schema and snowflake schema are popular dimensional modeling approaches.
2. Choose a suitable database type. Depending on the size and type of data, you can choose a suitable database type for storing data for reporting. Relational databases are the most common type of database used for reporting, but NoSQL databases can also be used for certain reporting scenarios.
3. Normalize data appropriately. Normalization is the process of organizing data in a database to minimize data redundancy and improve data integrity. However, over-normalization can make querying complex and slow down reporting. Therefore, it is important to normalize data appropriately based on the reporting requirements.
4. Use indexes to improve query performance. Indexes can help improve query performance by allowing the database to quickly find the data required for a report. Choose appropriate indexes based on the reporting requirements and the size of the data.
5. Consider partitioning. Partitioning involves splitting large tables into smaller, more manageable pieces. It can improve query performance by allowing the database to quickly access the required data.
6. Consider data compression. Data compression can help reduce the storage requirements of data and improve query performance by reducing the amount of data that needs to be read from disk.

Overall, the best structure for storing data for reporting depends on various factors, and it is important to carefully consider the reporting requirements and performance considerations when choosing a suitable structure.

Optimal Processing of Data in the Cloud

The best way to process data in the cloud depends on various factors, including the type and volume of data, the processing requirements, and the performance considerations. Here are some general guidelines for processing data in the cloud:

1. Use cloud-native data processing services. Cloud providers offer a wide range of data processing services, such as AWS Lambda, GCP Cloud Functions, and Azure Functions, which allow you to process data without managing the underlying infrastructure. These services are highly scalable and can be cost-effective for small- to medium-sized workloads.
2. Use serverless computing. Serverless computing is a cloud computing model in which the cloud provider manages the infrastructure and automatically scales the resources based on the workload. Serverless computing can be a cost-effective and scalable solution for processing data, especially for sporadic or bursty workloads.
3. Use containerization. Containerization allows you to package your data processing code and dependencies into a container image and deploy it to a container orchestration platform, such as Kubernetes or Docker Swarm. This approach can help you achieve faster deployment, better resource utilization, and improved scalability.
4. Use distributed computing frameworks. Distributed computing frameworks, such as Apache Hadoop, Spark, and Flink, allow you to process large volumes of data in a distributed manner across multiple nodes. These frameworks can be used for batch processing, real-time processing, and machine learning workloads.
5. Use data streaming platforms. Data streaming platforms, such as Apache Kafka and GCP Pub/Sub, allow you to process data in real time and respond quickly to changing business needs. These platforms can be used for real-time processing, data ingestion, and event-driven architectures.
6. Use machine learning and AI services. Cloud providers offer a wide range of machine learning and AI services, such as AWS SageMaker, GCP AI Platform, and Azure Machine Learning, which allow you to build, train, and deploy machine learning models in the cloud. These services can be used for predictive analytics, natural language processing, computer vision, and other machine learning workloads.

Overall, the best way to process data in the cloud depends on various factors, and it is important to carefully consider the processing requirements and performance considerations when choosing a suitable approach.

Optimize Memory

The best memory size for processing 1 terabyte of data depends on the specific processing requirements and the type of processing being performed. In general, the memory size required for processing 1 terabyte of data can vary widely depending on the data format, processing algorithms, and performance requirements. For example, if you are processing structured data in a relational database, the memory size required will depend on the specific SQL query being executed and the size of the result set. In this case, the memory size required may range from a few gigabytes to several hundred gigabytes or more, depending on the complexity of the query and the number of concurrent queries being executed.

On the other hand, if you are processing unstructured data, such as images or videos, the memory size required will depend on the specific processing algorithm being used and the size of the data being processed. In this case, the memory size required may range from a few gigabytes to several terabytes or more, depending on the complexity of the algorithm and the size of the input data.

Therefore, it is not possible to give a specific memory size recommendation for processing 1 terabyte of data without knowing more about the specific processing requirements and the type of data being processed. It is important to carefully consider the memory requirements when designing the processing system and to allocate sufficient memory resources to ensure optimal performance.

Service Oriented Data Architecture Is the Future for Data Ecosystems 

A Services Oriented Data Architecture (SΘDΔ^®) is an architectural approach used in cloud computing that focuses on creating and deploying software systems as a set of interconnected services. Each service performs a specific business function, and communication between services occurs over a network, typically using web-based protocols such as RESTful APIs.

In the cloud, SΘDΔ can be implemented using a variety of cloud computing technologies, including infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). In an SΘDΔ-based cloud architecture, services are hosted on cloud infrastructure, such as virtual machines or containers, and can be dynamically scaled up or down based on demand.

One of the key benefits of SΘDΔ in the cloud is its ability to enable greater agility and flexibility in software development and deployment. By breaking down a complex software system into smaller, more manageable services, SΘDΔ makes it easier to build, test, and deploy new features and updates. It also allows for more granular control over resource allocation, making it easier to optimize performance and cost.

Overall, service-based architecture is a powerful tool for building scalable, flexible, and resilient software systems in the cloud, especially data ecosystems.

Recap

In this blog we began a conversation about the modern data ecosystem. By following best practices, we can ensure that our cloud applications run efficiently, on the large cloud providers, providing a great user experience while minimizing costs. We covered the following:

1. The modern data architecture is designed to help organizations leverage data as a strategic asset and gain a competitive advantage by making better data-driven decisions.
2. The best way to process data in the cloud depends on various factors, and it is important to carefully consider the processing requirements and performance considerations when choosing a suitable approach.
3. Overall, service-based architecture is a powerful tool for building scalable, flexible, and resilient software systems in the cloud, especially data ecosystems.

The post All Data Ecosystems Are Real Time, It Is Just a Matter of Time appeared first on Unravel.

Leverage Content Delivery Network

Here are some best practices when using a Content Delivery Network (CDN):

1. Choose the right CDN provider. Choose a CDN provider that is reliable, scalable, and has a global network of servers. Look for providers that offer features such as caching, load balancing, and DDoS protection.
2. Configure your CDN properly. Configure your CDN properly to ensure that it is delivering content efficiently and securely. This may include setting up caching rules, configuring SSL/TLS encryption, and configuring firewall rules.
3. Monitor your CDN. Monitor your CDN to ensure that it is performing optimally and delivering content efficiently. This may include monitoring CDN usage, network latency, and caching effectiveness.
4. Optimize your CDN. Optimize your CDN to ensure that it is delivering content as efficiently as possible. This may include using compression, optimizing images, and minifying JavaScript and CSS files.
5. Use multiple CDNs. Consider using multiple CDNs to ensure that your content is always available and is delivered quickly. This may include using multiple providers or using multiple CDNs from the same provider.
6. Test your CDN. Test your CDN to ensure that it is delivering content as expected. This may include conducting load testing to ensure that your CDN can handle expected levels of traffic and testing performance from different geographical locations.
7. Use CDN analytics. Use CDN analytics to track CDN usage and monitor performance. This will help you identify any issues and optimize your CDN for better performance.
8. Implement security measures. Implement security measures to protect your content and your users. This may include configuring SSL/TLS encryption, setting up firewalls, and using DDoS protection.

By following these best practices, you can ensure that your CDN is delivering content efficiently and securely, and is providing a positive user experience.

Choosing the Right CDN Provider

Here are some best practices when choosing the right Content Delivery Provider (CDP) in the cloud:

1. Understand your requirements. Before selecting a CDP, determine your requirements for content delivery. This includes identifying the types of content you want to deliver, the expected traffic volume, and the geographical locations of your users.
2. Research multiple providers. Research multiple CDP providers to determine which one best meets your requirements. Look for providers with a global network of servers, high reliability, and good performance.
3. Evaluate performance. Evaluate the performance of each provider by conducting tests from different geographical locations. This will help you determine which provider delivers content quickly and efficiently to your users.
4. Consider cost. Consider the cost of each provider, including the cost of data transfer, storage, and other associated fees. Look for providers that offer flexible pricing models and transparent pricing structures.
5. Evaluate security features. Evaluate the security features of each provider, including DDoS protection, SSL/TLS encryption, and firewalls. Look for providers that offer comprehensive security features to protect your content and your users.
6. Check for integration. Check if the provider integrates well with your existing infrastructure and tools, such as your content management system, analytics tools, and monitoring tools.
7. Look for analytics. Look for providers that offer analytics and reporting tools to help you track content delivery performance and optimize your content delivery.
8. Check for support. Check the level of support offered by each provider, including support for troubleshooting, upgrades, and maintenance. Look for providers with a responsive support team that can help you resolve issues quickly.

By following these best practices, you can select the right CDP that meets your requirements and delivers content efficiently and securely to your users.

Configure Your CDN Properly

Here are some best practices when configuring your Content Delivery Network (CDN) properly in the cloud:

1. Configure caching. Caching is a key feature of CDNs that enables content to be delivered quickly and efficiently. Configure caching rules to ensure that frequently accessed content is cached and delivered quickly.
2. Configure content compression. Configure content compression to reduce the size of your files and improve performance. Gzip compression is a popular option that can be configured at the CDN level.
3. Configure SSL/TLS encryption. Configure SSL/TLS encryption to ensure that content is delivered securely. Look for CDNs that offer free SSL/TLS certificates or have the option to use your own certificate.
4. Configure firewall rules. Configure firewall rules to protect your content and your users. This includes setting up rules to block traffic from malicious sources and to restrict access to your content.
5. Use multiple CDNs. Consider using multiple CDNs to improve performance and ensure availability. Use a multi-CDN strategy to distribute traffic across multiple CDNs, which can reduce latency and increase reliability.
6. Configure DNS settings. Configure DNS settings to ensure that traffic is directed to your CDN. This includes configuring CNAME records or using a DNS provider that integrates with your CDN.
7. Test your CDN configuration. Test your CDN configuration to ensure that it is working properly. This includes testing performance, testing from different geographical locations, and testing content delivery from different devices.
8. Monitor your CDN configuration. Monitor your CDN configuration to ensure that it is delivering content efficiently and securely. This includes monitoring CDN usage, network latency, caching effectiveness, and security events.

By following these best practices, you can configure your CDN properly to ensure that content is delivered quickly, efficiently, and securely to your users.

Monitor Your CDN

Here are some best practices when monitoring your Content Delivery Network (CDN) in the cloud:

1. Monitor CDN usage. Monitor CDN usage to track how much content is being delivered and where it is being delivered. This can help you identify potential issues and optimize content delivery.
2. Monitor network latency. Monitor network latency to ensure that content is being delivered quickly and efficiently. Use tools like Pingdom or KeyCDN’s real-time monitoring to identify network latency issues.
3. Monitor caching effectiveness. Monitor caching effectiveness to ensure that frequently accessed content is being cached and delivered quickly. Use CDN analytics and monitoring tools to track caching effectiveness.
4. Monitor security events. Monitor security events to ensure that your content and your users are protected from security threats. This includes monitoring for DDoS attacks, intrusion attempts, and other security events.
5. Monitor CDN performance. Monitor CDN performance to ensure that content is being delivered efficiently and that the CDN is performing optimally. This includes monitoring server response times, cache hit rates, and other performance metrics.
6. Monitor user experience. Monitor user experience to ensure that users are able to access content quickly and efficiently. Use tools like Google Analytics or Pingdom to monitor user experience and identify issues.
7. Monitor CDN costs. Monitor CDN costs to ensure that you are staying within your budget. Use CDN cost calculators to estimate costs and monitor usage to identify potential cost savings.
8. Set up alerts. Set up alerts to notify you when issues arise, such as network latency spikes, security events, or server downtime. Use CDN monitoring tools to set up alerts and notifications.

By following these best practices, you can monitor your CDN effectively to ensure that content is being delivered quickly, efficiently, and securely to your users, while staying within your budget.

Optimize Your Content Delivery Network

Optimizing your Content Delivery Network (CDN) in the cloud is crucial to ensure fast and reliable content delivery to your users. Here are some best practices to follow:

1. Choose the right CDN provider. Research different CDN providers and choose the one that meets your needs in terms of cost, performance, and geographical coverage.
2. Use a multi-CDN approach. Consider using multiple CDN providers to improve your content delivery performance and reliability.
3. Optimize your CDN configuration. Configure your CDN to serve static assets, such as images, videos, and files, directly from the CDN cache, while serving dynamic content, such as HTML pages, from your origin server.
4. Use caching effectively. Set appropriate cache control headers for your content to ensure that it is cached by the CDN and served quickly to your users.
5. Monitor your CDN performance. Monitor your CDN performance regularly and identify any issues or bottlenecks that may be affecting your content delivery performance.
6. Use compression. Use compression techniques, such as gzip compression, to reduce the size of your content and improve its delivery speed.
7. Optimize DNS resolution. Use a global DNS service to optimize DNS resolution and reduce the time it takes for your users to access your content.
8. Implement HTTPS. Implement HTTPS to ensure secure content delivery and improve your search engine ranking.
9. Consider using edge computing. Consider using edge computing to offload some of the processing and caching tasks to the CDN edge servers, which can help reduce the load on your origin servers and improve content delivery performance.

By following these best practices, you can optimize your CDN in the cloud and ensure fast, reliable, and secure content delivery to your users.

Use Multiple Content Delivery Networks

Using multiple Content Delivery Networks (CDNs) in the cloud can improve the performance and reliability of your content delivery, but it also requires careful management to ensure optimal results. Here are some best practices to follow when using multiple CDNs:

1. Use a multi-CDN management platform. Consider using a multi-CDN management platform to manage your CDNs and monitor their performance. These platforms allow you to configure your CDNs and dynamically route traffic to the optimal CDN based on real-time performance metrics.
2. Define a clear CDN selection strategy. Develop a clear strategy for selecting the CDN to use for each request. Consider factors such as geographical proximity, latency, and availability when selecting a CDN.
3. Test your CDNs. Regularly test the performance of your CDNs using real-world scenarios. This will help you identify any issues and ensure that your CDNs are performing optimally.
4. Implement consistent caching policies. Ensure that your caching policies are consistent across all your CDNs. This will help avoid cache misses and improve content delivery performance.
5. Implement a failover strategy. Define a failover strategy to ensure that your content is always available even if one of your CDNs experiences an outage. This may involve dynamically routing traffic to a backup CDN or using your origin server as a fallback.
6. Monitor and optimize costs. Monitor your CDN costs and optimize your usage to ensure that you are getting the best value for your investment.
7. Ensure security. Implement appropriate security measures, such as SSL encryption and DDoS protection, to ensure that your content is delivered securely and your CDNs are protected from attacks.

By following these best practices, you can successfully use multiple CDNs in the cloud to improve your content delivery performance and reliability.

Test Your Content Delivery Network

Testing your Content Delivery Network (CDN) in the cloud is crucial to ensure that it is performing optimally and delivering content quickly and reliably to your users. Here are some best practices to follow when testing your CDN:

1. Define clear testing objectives. Define clear objectives for your CDN testing, such as identifying bottlenecks or measuring performance metrics. This will help you design your test scenarios and evaluate the results effectively.
2. Use real-world testing scenarios. Use real-world scenarios to test your CDN, such as simulating traffic from different geographic locations, devices, and network conditions. This will help you identify any issues that may affect your users’ experience.
3. Measure key performance metrics. Measure key performance metrics such as response time, latency, throughput, and cache hit rate. This will help you identify any areas of improvement and optimize your CDN performance.
4. Test caching behavior. Test the caching behavior of your CDN by measuring cache hit and miss rates for different types of content. This will help you optimize your caching policies and reduce your origin server load.
5. Test CDN failover and disaster recovery. Test your CDN failover and disaster recovery mechanisms to ensure that your content is always available even if one or more CDNs experience an outage.
6. Use automated testing tools. Use automated testing tools to simulate traffic and measure performance metrics. This will help you perform tests more efficiently and accurately.
7. Test regularly. Regularly test your CDN to ensure that it is performing optimally and identify any issues as soon as possible.

By following these best practices, you can effectively test your CDN in the cloud and ensure fast and reliable content delivery to your users.

Use CDN Analytics

Analytics play a critical role in optimizing your Content Delivery Network (CDN) in the cloud, as they provide insights into your CDN performance and user behavior. Here are some best practices to follow when using CDN analytics in the cloud:

1. Define clear goals. Define clear goals for your CDN analytics, such as identifying areas of improvement or measuring user engagement. This will help you select the appropriate analytics tools and collect the right data.
2. Use a multi-CDN analytics platform. If you are using multiple CDNs, consider using a multi-CDN analytics platform to consolidate data from multiple CDNs and provide a unified view of your content delivery performance.
3. Monitor key performance metrics. Monitor key performance metrics such as page load time, bounce rate, and conversion rate. This will help you identify any performance issues and optimize your content delivery.
4. Use real-time analytics. Use real-time analytics to monitor your CDN performance and user behavior in real time. This will help you identify any issues as soon as they occur and take immediate action.
5. Use A/B testing. Use A/B testing to test different CDN configurations or content variations and measure the impact on user engagement and performance.
6. Use data visualization tools. Use data visualization tools to help you visualize and analyze your CDN performance data effectively. This will help you identify trends and patterns that may be difficult to detect using raw data.
7. Optimize CDN configuration based on analytics insights. Use insights from your CDN analytics to optimize your CDN configuration, such as adjusting caching policies, optimizing content delivery routes, or reducing page load time.

By following these best practices, you can use CDN analytics in the cloud to optimize your content delivery performance, improve user engagement, and achieve your business goals.

Implement Security Measures

Implementing security measures in the cloud is essential to protect your applications, data, and infrastructure from cyber threats. Here are some best practices to follow when implementing security measures in the cloud:

1. Use strong authentication and access controls. Implement strong authentication mechanisms such as multi-factor authentication and use access controls to restrict access to your cloud resources based on the principle of least privilege.
2. Implement network security controls. Use network security controls such as firewalls, intrusion detection and prevention systems (IDPS), and virtual private networks (VPNs) to protect your cloud infrastructure from network-based attacks.
3. Implement encryption. Use encryption to protect sensitive data, both in transit and at rest. Implement encryption for data stored in the cloud, and use secure protocols such as HTTPS and TLS for data in transit.
4. Regularly apply security patches and updates. Regularly apply security patches and updates to your cloud infrastructure and applications to ensure that you are protected against known vulnerabilities.
5. Implement security monitoring and logging. Implement security monitoring and logging to detect and respond to security events. Use tools that provide visibility into your cloud environment and alert you to potential security threats.
6. Use cloud-native security tools. Use cloud-native security tools and services such as security groups, network ACLs, and security information and event management (SIEM) to secure your cloud environment.
7. Develop an incident response plan. Develop an incident response plan that outlines how you will respond to security incidents, including containment, investigation, and remediation.

By following these best practices, you can effectively implement security measures in the cloud and protect your applications, data, and infrastructure from cyber threats.

The post The Modern Data Ecosystem: Leverage Content Delivery Network (CDN) appeared first on Unravel.

The post DBS Empowers Self-Service Engineering with Unravel appeared first on Unravel.

The post DBS Discusses Data+FinOps for Banking appeared first on Unravel.

By Jason Bloomberg, Managing Partner, Intellyx
Part 4 of the Demystifying Data Observability Series for Unravel Data

In the first three articles in this four-post series, my colleague Jason English and I explored DataOps observability, the connection between DevOps and DataOps, and data-centric FinOps best practices.

In this concluding article in the series, I’ll explore DataOps resiliency – not simply how to prevent data-related problems, but also how to recover from them quickly, ideally without impacting the business and its customers.

Observability is essential for any kind of IT resiliency – you can’t fix what you can’t see – and DataOps is no exception. Failures can occur anywhere in the stack, from the applications on down to the hardware. Understanding the root causes of such failures is the first step to fixing, or ideally preventing, them.

The same sorts of resiliency problems that impact the IT environment at large can certainly impact the data estate. Even so, traditional observability and incident management tools don’t address specific problems unique to the world of data processing.

In particular, DataOps resiliency must address the problem of toxic workloads.

Understanding Toxic Workloads

Toxic data workloads are as old as relational database management systems (RDBMSs), if not older. Anyone who works with SQL on large databases knows there are some queries that will cause the RDBMS to slow dramatically or completely grind to a halt.

The simplest example: SELECT * FROM TRANSACTIONS where the TRANSACTIONS table has millions of rows. Oops! Your resultset also has millions of rows!

JOINs, of course, are more problematic, because they are difficult to construct, and it’s even more difficult to predict their behavior in databases with complex structures.

Such toxic workloads caused problems in the days of single on-premises databases. As organizations implemented data warehouses, the risks compounded, requiring increasing expertise from a scarce cadre of query-building experts.

Today we have data lakes as well as data warehouses, often running in the cloud where the meter is running all the time. Organizations also leverage streaming data, as well as complex data pipelines that mix different types of data in real time.

With all this innovation and complexity, the toxic workload problem hasn’t gone away. In fact, it has gotten worse, as the nuances of such workloads have expanded.

Breaking Down the Toxic Workload

Poorly constructed queries are only one of the causes of a modern toxic workload. Other root causes include:

• Poor quality data – one table with NULL values, for example, can throw a wrench into seemingly simple queries. Expand that problem to other problematic data types and values across various cloud-based data services and streaming data sources, and small data quality problems can easily explode into big ones.
• Coding issues – Data engineers must create data pipelines following traditional coding practices – and whenever there’s coding, there are software bugs. In the data warehouse days, tracking down toxic workloads usually revealed problematic queries. Today, coding issues are just as likely to be the root cause.
• Infrastructure issues – Tracking down the root causes of toxic workloads means looking everywhere – including middleware, container infrastructure, networks, hypervisors, operating systems, and even the hardware. Just because a workload runs too slow doesn’t mean it’s a data issue. You have to eliminate as many possible root causes as you can – and quickly.
• Human issues – Human error may be the root cause of any of the issues above – but there is more to this story. In many cases, root causes of toxic workloads boil down to a shortage of appropriate skills among the data team or a lack of effective collaboration within the team. Human error will always crop up on occasion, but a skills or collaboration issue will potentially cause many toxic workloads over time.

The bottom line: DataOps resiliency includes traditional resiliency challenges but extends to data-centric issues that require data observability to address.

Data Resiliency at Mastercard

Mastercard recently addressed its toxic workload problem on Hadoop, as well as Impala, Spark, and Hive.

The payment processor has petabytes of data across hundreds of nodes, as well as thousands of users who access the data in an ad hoc fashion – that is, they build their own queries.

Mastercard’s primary issue was poorly constructed queries, a combination of users’ inexperience as well as the complexity of the required queries.

In addition, the company faced various infrastructure issues, from overburdened data pipelines to maxed-out storage and disabled daemons.

All these problems led to application failures, system slowdowns and crashes, and resource bottlenecks of various types.

To address these issues, Mastercard brought in Unravel Data. Unravel quickly identified hundreds of unused data tables. Freeing up the associated resources improved query performance dramatically.

Mastercard also uses Unravel to help users tune their own query workloads as well as automate the monitoring of toxic workloads in progress, preventing the most dangerous ones from running in the first place.

Overall, Unravel helped Mastercard improve its mean time to recover (MTTR) – the best indicator of DataOps Resiliency.

The Intellyx Take

The biggest mistake an organization can make around DataOps observability and resiliency is to assume these topics are special cases of the broader discussion of IT observability and resiliency.

In truth, the areas overlap – after all, infrastructure issues are often the root causes of data-related problems – but without the particular focus on DataOps, many problems would fall through the cracks.

The need for this focus is why tools like Unravel’s are so important. Unravel adds AI optimization and automated governance to its core data observability capabilities, helping organizations optimize the cost, performance, and quality of their data estates.

DataOps resiliency is one of the important benefits of Unravel’s approach – not in isolation, but within the overall context for resiliency that is so essential to modern IT.

Copyright © Intellyx LLC. Unravel Data is an Intellyx customer. None of the other organizations mentioned in this article is an Intellyx customer. Intellyx retains final editorial control of this article. No AI was used in the production of this article.

The post DataOps Resiliency: Tracking Down Toxic Workloads appeared first on Unravel.

Check out Sanjeev Mohan’s, Principal, SanjMo & Former Gartner Research VP, Big Data and Advanced Analytics, chapter on “Data observability’s newest frontiers: DataFinOps and DataBizOps” in the book, Data Observability, The Reality.

What you’ll learn from the former VP of Research at Gartner:

• DataFinOps defined
• Why you need DataFinOps
• The challenges of DataFinOps
• DataFinOps case studies & more!

Don’t miss out on your chance to read this chapter and gain valuable insights from a top industry leader.

The post Data observability’s newest frontiers: DataFinOps and DataBizOps appeared first on Unravel.

Thrilled to announce that Unravel is contributing a chapter in Ravit Jain’s new ebook, Data Observability, The Reality.

What you’ll learn from reading this ebook:

• What data observability is and why it’s important
• Identify key components of an observability framework
• Learn how to design and implement a data observability strategy
• Explore real-world use cases and best practices for data observability
• Discover tools and techniques for monitoring and troubleshooting data pipelines

Don’t miss out on your chance to read our chapter, “Automation and AI Are a Must for Data Observability,” and gain valuable insights from top industry leaders such as Sanjeev Mohan and others.

The post Data Observability, The Reality eBook appeared first on Unravel.

By Craig Wiley, Senior Director of Product Management, Databricks and Clinton Ford, Director of Product Marketing, Unravel Data

Introduction

Machine learning (ML) enables organizations to extract more value from their data than ever before. Companies who successfully deploy ML models into production are able to leverage that data value at a faster pace than ever before. But deploying ML models requires a number of key steps, each fraught with challenges:

• Data preparation, cleaning, and processing
• Feature engineering
• Training and ML experiments
• Model deployment
• Model monitoring and scoring

Figure 1. Phases of the ML lifecycle with Databricks Machine Learning and Unravel Data

Challenges at each phase

Data preparation and processing

Data preparation is a data scientist’s most time-consuming task. While there are many phases in the data science lifecycle, an ML model can only be as good as the data that feeds it. Reliable and consistent data is essential for training and machine learning (ML) experiments. Despite advances in data processing, a significant amount of effort is required to load and prepare data for training and ML experimentation. Unreliable data pipelines slow the process of developing new ML models.

Training and ML experiments

Once data is collected, cleansed, and refined, it is ready for feature engineering, model training, and ML experiments. The process is often tedious and error-prone, yet machine learning teams also need a way to reproduce and explain their results for debugging, regulatory reporting, or other purposes. Recording all of the necessary information about data lineage, source code revisions, and experiment results can be time-consuming and burdensome. Before a model can be deployed into production, it must have all of the detailed information for audits and reproducibility, including hyperparameters and performance metrics.

Model deployment and monitoring

While building ML models is hard, deploying them into production is even more difficult. For example, data quality must be continuously validated and model results must be scored for accuracy to detect model drift. What makes this challenge even more daunting is the breadth of ML frameworks and the required handoffs between teams throughout the ML model lifecycle– from data preparation and training to experimentation and production deployment. Model experiments are difficult to reproduce as the code, library dependencies, and source data change, evolve, and grow over time.

The solution

The ultimate hack to productionize ML is data observability combined with scalable, serverless, and automated ML model serving. Unravel’s AI-powered data observability for Databricks on AWS and Azure Databricks simplifies the challenges of data operations, improves performance, saves critical engineering time, and optimizes resource utilization.

Databricks Model Serving deploys machine learning models as a REST API, enabling you to build real-time ML applications like personalized recommendations, customer service chatbots, fraud detection, and more – all without the hassle of managing serving infrastructure.

Databricks + data observability

Whether you are building a lakehouse with Databricks for ML model serving, ETL, streaming data pipelines, BI dashboards, or data science, Unravel’s AI-powered data observability for Databricks on AWS and Azure Databricks simplifies operations, increases efficiency, and boosts productivity. Unravel provides AI insights to proactively pinpoint and resolve data pipeline performance issues, ensure data quality, and define automated guardrails for predictability.

Scalable training and ML experiments with Databricks

Databricks uses pre-installed, optimized libraries to build and train machine learning models. With Databricks, data science teams can build and train machine learning models. Databricks provides pre-installed, optimized libraries. Examples include scikit-learn, TensorFlow, PyTorch, and XGBoost. MLflow integration with Databricks on AWS and Azure Databricks makes it easy to track experiments and store models in repositories.

MLflow monitors machine learning model training and running. Information about the source code, data, configuration information, and results are stored in a single location for quick and easy reference. MLflow also stores models and loads them in production. Because MLflow is built on open frameworks, many different services, applications, frameworks, and tools can access and consume the models and related details.

Serverless ML model deployment and serving

Databricks Serverless Model Serving accelerates data science teams’ path to production by simplifying deployments and reducing mistakes through integrated tools. With the new model serving service, you can do the following:

• Deploy a model as an API with one click in a serverless environment.
• Serve models with high availability and low latency using endpoints that can automatically scale up and down based on incoming workload.
• Safely deploy the model using flexible deployment patterns such as progressive rollout or perform online experimentation using A/B testing.
• Seamlessly integrate model serving with online feature store (hosted on Azure Cosmos DB), MLflow Model Registry, and monitoring, allowing for faster and error-free deployment.

Conclusion

You can now train, deploy, monitor, and retrain machine learning models, all on the same platform with Databricks Model Serving. Integrating the feature store with model serving and monitoring helps ensure that production models are leveraging the latest data to produce accurate results. The end result is increased availability and simplified operations for greater AI velocity and positive business impact.

Ready to get started and try it out for yourself? Watch this Databricks event to see it in action. You can read more about Databricks Model Serving and how to use it in the Databricks on AWS documentation and the Azure Databricks documentation. Learn more about data observability in the Unravel documentation.

The post Solving key challenges in the ML lifecycle with Unravel and Databricks Model Serving appeared first on Unravel.

Thank you for your interest in the Data Teams Summit 2023 Survey Infographic.

Download the full pdf.

Take a look at last year’s survey results here.

The post Data Teams Summit Survey Infographic appeared first on Unravel.

By Jason English, Principal Analyst, Intellyx
Part 3 in the Demystifying Data Observability Series, by Intellyx for Unravel Data

IT and data executives find themselves in a quandary about deciding how to wrangle an exponentially increasing volume of data to support their business requirements – without breaking an increasingly finite IT budget.

Like an overeager diner at a buffet who’s already loaded their plate with the cheap carbs of potatoes and noodles before they reach the protein-packed entrees, they need to survey all of the data options on the menu before formulating their plans for this trip.

In our previous chapters of this series, we discussed why DataOps needs its own kind of observability, and then how DataOps is a natural evolution of DevOps practices. Now there’s a whole new set of options in the data observability menu to help DataOps teams track the intersection of value and cost.

From ROI to FinOps

Executives can never seem to get their fill of ROI insights from IT projects, so they can measure bottom-line results or increase top-line revenue associated with each budget line item. After all, predictions about ROI can shape the perception of a company for its investors and customers.

Unfortunately, ROI metrics are often discussed at the start of a major technology product or services contract – and then forgotten as soon as the next initiative gets underway.

The discipline of FinOps burst onto the scene over the last few years, as a strategy to address the see-saw problem of balancing the CFO’s budget constraints with the CIO’s technology delivery requirements to best meet the current and future needs of customers and employees.

FinOps focuses on improving technology spending decisions of an enterprise using measurements that go beyond ROI, to assess the value of business outcomes generated through technology investments.

Some considerations frequently seen on the FinOps menu include:

• Based on customer demand or volatility in our consumption patterns, should we buy capacity on-demand or reserve more cloud capacity?
• Which FinOps tools should we buy, and what functionality should we build ourselves, to deliver this important new capability?
• Which cloud cost models are preferred for capital expenditures (capex) projects and operational expenditures (opex)?
• What is the potential risk and cost of known and unknown usage spikes, and how much should we reasonably invest in analysts and tools for preventative purposes?

As a discipline, FinOps has come a long way, building communities of interest among expert practitioners, product, business, and finance teams as well as solution providers through its own FinOps Foundation and instructional courses on the topic.

FinOps + DataOps = DataFinOps?

Real-time analytics and AI-based operational intelligence are enabling revolutionary business capabilities, enterprise-wide awareness, and innovative machine learning-driven services. All of this is possible thanks to a smorgasbord of cloud data storage and processing, cloud data lakes, cloud data warehouse, and cloud lakehouse options.

Unfortunately, the rich streams of data required for such sophisticated functionality bring along the unwanted side effect of elastically expanding budgetary waistbands, due to ungoverned cloud storage and compute consumption costs. Nearly a third of all data science projects go more than 40% over budget on cloud data, according to a recent survey–a huge delta between cost expectations and reality.

How can better observability into data costs help the organization wring more value from data assets without cutting into results, or risking cost surprises?

As it turns out, data has its own unique costs, benefits, and value considerations. Combining the disciplines of FinOps and DataOps – which I’ll dub DataFinOps just for convenience here – can yield a unique new set of efficiencies and benefits for the enterprise’s data estate.

Some of unique considerations of DataFinOps:

• Which groups within our company are the top spenders on cloud data analytics, and is anything anomalous about their spending patterns versus the expected budgets?
• What is the value of improving data performance or decreasing the latency of our data estate by region or geography, in order to improve local accuracy, reduce customer and employee attrition and improve retention?
• If we are moving to a multi-cloud, hybrid approach, what is an appropriate and realistic mix of reserved instances and spot resources for processing data of different service level agreements (SLAs)?
• Where are we paying excessive ingress / egress fees within our data estate? Would it be more cost effective to process data near the data or move our data elsewhere?
• How much labor do our teams spend building and maintaining data pipelines, and what is that time worth?
• Are cloud instances being intelligently right-sized and auto-scaled to meet demand?

Systems-oriented observability platforms such as DataDog and Dynatrace can measure system or service level telemetry, which is useful for a DevOps team looking at application-level cloud capacity and cost/performance ratios. Unfortunately these tools do not dig into enough detail to answer data analytics-specific FinOps questions.

Taming a market of data options

Leading American grocery chain Kroger launched its 84.51° customer experience and data analytics startup to provide predictive data insights and precision marketing for its parent company and other retailers, across multiple cloud data warehouses such as Snowflake and Databricks, using data storage in multiple clouds such as Azure and GCP.

Using the Unravel platform for data observability, they were able to get a grip on data costs and value across multiple data platforms and clouds without having to train up more experts on the gritty details of data job optimization within each system.

It’s still early days for this transformation, but a data cost reduction of up to 50% would go a long way toward extracting value from deep customer analytics, as transaction data volumes continue to increase by 2x or 3x a year as more sources come online.

The Intellyx Take

It would be nice if CFOs could just tell CIOs and CDOs to simply stop consuming and storing so much data, and have that reduce their data spend. But just like in real life, crash diets will never produce long-term results, if the ‘all-you-can-eat’ data consumption pattern isn’t changed.

The hybrid IT underpinnings of advanced data-driven applications evolves almost every day. To achieve sustainable improvements in cost/benefit returns on data, analysts and data scientists would have to become experts on the inner workings of each public cloud and data warehousing vendor.

DataFinOps practices should encourage data team accountability for value improvements, but more importantly, it should give them the data observability, AI-driven recommendations, and governance controls necessary to both contain costs, and stay ahead of the organization’s growing business demand for data across hybrid IT data resources and clouds.

©2023 Intellyx LLC. Intellyx is editorially responsible for the content of this document. At the time of writing, Unravel is an Intellyx customer. Image source: crayion.ai

The post DataFinOps: More on the menu than data cost governance appeared first on Unravel.

By Jason Bloomberg, President, Intellyx
Part 2 of the Demystifying Data Observability Series for Unravel Data

In part one of this series, fellow Intellyx analyst Jason English explained the differences between DevOps and DataOps, drilling down into the importance of DataOps observability.

The question he left open for this article: how did we get here? How did DevOps evolve to what it is today, and what parallels or differences can we find in the growth of DataOps?

DevOps Precursors

The traditional, pre-cloud approach to building custom software in large organizations separated the application development (‘dev’) teams from the IT operations (‘ops’) personnel responsible for running software in the corporate production environment.

In between these two teams, organizations would implement a plethora of processes and gates to ensure the quality of the code and that it would work properly in production before handing it to the ops folks to deploy and manage.

Such ‘throw it over the wall’ processes were slow and laborious, leading to deployment cycles many months long. The importance of having software that worked properly, so the reasoning went, was sufficient reason for such onerous delays.

Then came the Web. And the cloud. And enterprise digital transformation initiatives. All of these macro-trends forced enterprises to rethink their plodding software lifecycles.

Not only were they too slow to deliver increasingly important software capabilities, but business requirements would evolve far too quickly for the deployed software to keep up.

Such pressures led to the rise of agile software methodologies, cloud native computing, and DevOps.

Finding the Essence of DevOps

The original vision of DevOps was to pull together the dev and ops teams to foster greater collaboration, in hopes that software deployment cadences would accelerate while maintaining or improving the quality of the resulting software.

Over time, this ‘kumbaya’ vision of seamless collaboration itself evolved. Today, we can distill the essence of modern DevOps into these five elements:

• A cultural and organizational shift away from the ‘throw it over the wall’ mentality to greater collaboration across the software lifecycle
• A well-integrated, comprehensive automation suite that supports CI/CD activities, along with specialists who manage and configure such technologies, i.e., DevOps engineers
• A proactive, shift-left mentality that seeks to represent production behavior declaratively early in the lifecycle for better quality control and rapid deployment
• Full-lifecycle observability that shifts problem resolution to the left via better prediction of problematic behavior and preemptive mitigation of issues in production
• Lean practices to deliver value and improve efficiency throughout the software development lifecycle

Furthermore, DevOps doesn’t live in a vacuum. Rather, it is consistent with and supports other modern software best practices, including infrastructure-as-code, GitOps, and the ‘cattle not pets’ approach to supporting the production environment via metadata representations that drive deployment.

The Evolution of DataOps

Before information technology (IT), organizations had management of information systems (MIS). And before MIS, at the dawn of corporate computing, enterprises implemented data processing (DP).

The mainframes at the heart of enterprise technology as far back as the 1960s were all about processing data – crunching numbers in batch jobs that yielded arcane business results, typically dot-matrix printed on green and white striped paper.

Today, IT covers a vast landscape of technology infrastructure, applications, and hybrid on-premises and cloud environments – but data processing remains at the heart of what IT is all about.

Early in the evolution of DP, it became clear that the technologies necessary for processing transactions were different from the technology the organization required to provide business intelligence to line-of-business (LoB) professionals.

Enterprises required parallel investments in online transaction processing (OLTP) and online analytical processing (OLAP), respectively. OLAP proved the tougher nut to crack, because enterprises generated voluminous quantities of transactional data, while LoB executives required complex insights that would vary over time – thus taxing the ability of the data infrastructure to respond to the business need for information.

To address this need, data warehouses exploded onto the scene, separating the work of OLAP into two phases: transforming and loading data into the warehouses and supporting business intelligence via queries of the data in them.

Operating these early OLAP systems was relatively straightforward, centering on administering the data warehouses. In contrast, today’s data estate – the sum total of all the data infrastructure in a modern enterprise – is far more varied than in the early data warehousing days.

Motivations for DataOps

Operating this data estate has also become increasingly complex, as the practice of DataOps rises in today’s organizations.

Complexity, however, is only one motivation for DataOps. There are more reasons why today’s data estate requires it:

• Increased mission-criticality of data, as digital transformations rework organizations into digital enterprises
• Increased importance of real-time data, a capability that data warehouses never delivered
• Greater diversity of data-centric use cases beyond basic business intelligence
• Increased need for dynamic applications of data, as different LoBs need an ever-growing variety of data-centric solutions
• Growing need for operational cost predictability, optimization, and governance

Driving these motivations is the rise of AI, as it drives the shift from code-based to data-based software behavior. In other words, AI is more than just another data-centric use case. It repositions data as the central driver of software functionality for the enterprise.

The Intellyx Take

For all these reasons, DataOps can no longer follow the simplistic data warehouse administration pattern of the past. Today’s data estate is dynamic, diverse, and increasingly important, requiring organizations to take a full-lifecycle approach to collecting, transforming, storing, querying, managing, and consuming data.

As a result, DataOps requires the application of core DevOps practices along the data lifecycle. DataOps requires the cross-lifecycle collaboration, full-lifecycle automation and observability, and the shift-left mentality that DevOps brings to the table – only now applied to the enterprise data estate.

Thinking of DataOps as ‘DevOps for data’ may be too simplistic an explanation of the role DataOps should play. Instead, it might be more accurate to say that as data increasingly becomes the driver of software behavior, DataOps becomes the new DevOps.

Next up in part 3 of this series: DataFinOps: More on the menu than data cost governance

Copyright © Intellyx LLC. Unravel Data is an Intellyx customer. Intellyx retains final editorial control of this article.

The post The Evolution from DevOps to DataOps appeared first on Unravel.

By Jason English, Principal Analyst, Intellyx
Part 1 of the Demystifying Data Observability Series for Unravel Data

Don’t we already have DevOps?

DevOps was started more than a decade ago as a movement, not a product or solution category.

DevOps offered us a way of collaborating between development and operations teams, using automation and optimization practices to continually accelerate the release of code, measure everything, lower costs, and improve the quality of application delivery to meet customer needs.

Today, almost every application delivery shop naturally aspires to take flight with DevOps practices, and operate with more shared empathy and a shared commitment to progress through faster feature releases and feedback cycles.

DevOps practices also include better management practices such as self-service environments, test and release automation, monitoring, and cost optimization.

On the journey toward DevOps, teams who apply this methodology deliver software more quickly, securely, reliably, and with less burnout.

For dynamic applications to deliver a successful user experience at scale, we still need DevOps to keep delivery flowing. But as organizations increasingly view data as a primary source of business value, data teams are tasked with building and delivering reliable data products and data applications. Just as DevOps principles emerged to enable efficient and reliable delivery of applications by software development teams, DataOps best practices are helping data teams solve a new set of data challenges.

What is DataOps?

If “data is the new oil,” as pundits like to say, then data is also the most valuable resource in today’s modern data-driven application world.

The combination of commodity hardware, ubiquitous high-bandwidth networking, cloud data warehouses, and infrastructure abstraction methods like containers and Kubernetes creates an exponential rise in our ability to use data itself to dynamically compose functionality such as running analytics and informing machine learning-based inference within applications.

Enterprises recognized data as a valuable asset, welcoming the newly minted CDO (chief data officer) role to the E-suite, with responsibility for data and data quality across the organization. While leading-edge companies like Google, Uber and Apple increased their return on data investment by mastering DataOps, many leaders struggled to staff up with enough data scientists, data analysts, and data engineers to properly capitalize on this trend.

Progressive DataOps companies began to drain data swamps by pouring massive amounts of data (and investment) into a new modern ecosystem of cloud data warehouses and data lakes from open source Hadoop and Kafka clusters to vendor-managed services like Databricks, Snowflake, Amazon EMR, BigQuery, and others.

The elastic capacity and scalability of cloud resources allowed new kinds of structured, semi-structured, and unstructured data to be stored, processed and analyzed, including streaming data for real-time applications.

As these cloud resources quickly grew and scaled, they became a complex tangle of data sources, pipelines, dashboards, and machine learning models, with a variety of interdependencies, owners, stakeholders, and products with SLAs. Getting additional cloud resources and launching new data pipelines was easy, but operating them well required a lot of effort, and making sense of the business value of any specific component to prioritize data engineering efforts became a huge challenge.

Software teams went through the DevOps revolution more than a decade ago, and even before that, there were well-understood software engineering disciplines for design/build/deploy/change, as well as monitoring and observability. Before DataOps, data teams didn’t typically think about test and release cycles, or misconfiguration of the underlying infrastructure itself.

Where DevOps optimized the lifecycle of software from coding to release, DataOps is about the flow of data, breaking data out of work silos to collaborate on the movement of data from its inception, to its arrival, processing, and use within modern data architectures to feed production BI and machine learning applications.

DataOps jobs, especially in a cloudy, distributed data estate, aren’t the same as DevOps jobs. For instance, if a cloud application becomes unavailable, DevOps teams might need to reboot the server, adjust an API, or restart the K8s cluster.

If a DataOps-led application starts failing, it may show incorrect results instead of simply crashing, and cause leaders informed by faulty analytics and AI inferences to make disastrous business decisions. Figuring out the source of data errors and configuration problems can be maddeningly difficult, and DataOps teams may even need to restore the whole data estate – including values moving through ephemeral containers and pipelines – back to a valid, stable state for that point in time.

Why does DataOps need its own observability?

Once software observability started finding its renaissance within DevOps practices and early microservices architectures five years ago, we also started seeing some data management vendors pivoting to offer ‘data observability’ solutions.

The original concept of data observability was concerned with database testing, properly modeling, addressing and scaling databases, and optimizing the read/write performance and security of both relational and cloud back ends.

In much the same way that the velocity and automated release cadence of DevOps meant dev and ops teams needed to shift component and integration testing left, data teams need to tackle data application performance and data quality earlier in the DataOps lifecycle.

In essence, DataOps teams are using agile and other methodologies to develop and deliver analytics and machine learning at scale. Therefore they need DataOps observability to clarify the complex inner plumbing of apps, pipelines and clusters handling that moving data. Savvy DataOps teams must monitor ever-increasing numbers of unique data objects moving through data pipelines.

The KPIs for measuring success in DataOps observability include metrics and metadata that standard observability tools would never see: differences or anomalies in data layout, table partitioning, data source lineages, degrees of parallelism, data job and subroutine runtimes and resource utilization, interdependencies and relationships between data sets and cloud infrastructures – and the business tradeoffs between speed, performance and cost (or FinOps) of implementing recommended changes.

The Intellyx Take

A recent survey noted that 97 percent of data engineers ‘feel burned out’ at their current jobs, and 70 percent say they are likely to quit within a year! That’s a wakeup call for why DataOps observability matters now more than ever.

We must maintain the morale of understaffed and overworked data teams, when these experts take a long time to train and are almost impossible to replace in today’s tight technical recruiting market.

Any enterprise that intends to deliver modern DataOps should first consider equipping data teams with DataOps observability capabilities. Observability should go beyond the traditional metrics and telemetry of application code and infrastructure, empowering DataOps teams to govern data and the resources used to refine and convert raw data into business value as it flows through their cloud application estates.

Next up in part 2 of this series: Jason Bloomberg on the transformation from DevOps to DataOps!

©2022 Intellyx LLC. Intellyx retains editorial control of this document. At the time of writing, Unravel Data is an Intellyx customer. Image credit: Phil O’Driscoll, flickr CC2.0 license, compositing pins by author.

The post Why do we need DataOps Observability? appeared first on Unravel.

As DataOps moves along the maturity curve, many organizations are deciphering how to best balance the success of running critical jobs with optimized time and cost governance.

Watch the fireside chat from Data Teams Summit where Mark Sear, Head of Data Platform Optimization for Maersk, shares how his team is driving towards enabling strong engineering practices, design tenets, and culture at one of the largest shipping and logistics companies in the world. Transcript below.

Transcript

Kunal Agarwal:

Very excited to have a fireside chat here with Mark Sear. Mark, you’re the director of data integration, AI, machine learning, and analytics at Maersk. And Maersk is one of the largest shipping line and logistics companies in the world. Based out of Copenhagen, but with subsidiaries and offices across 130 countries with about 83,000 employees worldwide. We know that we always think about logistics and shipping as something just working harmoniously, transparently in the background, but in the recent past, given all of the supply chain pressures that have happened with the pandemic and beyond, and even that ship getting stuck in the Suez Canal, I think a lot more people are paying attention to this industry as well. So I was super excited to have you here, Mark, to hear more about yourself, you as the leader of data teams, and about what Maersk is doing with data analytics. Thank you so much for joining us.

Mark Sear:

It’s an absolute pleasure. You’ve just illustrated the perils of Wikipedia. Maersk is not just one of the largest shipping companies in the world, but we’re also actually one of the largest logistics companies in the world. We have our own airline. We’ve got hundreds of warehouses globally. We’re expanding massively, so we are there and of course we are a leader in decarbonation. We’ve got a pledge to be carbon-neutral way before just about anybody else. So it’s a fantastic company to work at. Often I say to my kids, we don’t just deliver stuff, we are doing something to help the planet. It’s a bigger mission than just delivering things, so it’s a pleasure to be here.

Kunal Agarwal:

That’s great. Mark, before we get into Maersk, we’d love to learn about you. So you have an amazing background and accumulation of all of these different experiences. Would you help the audience to understand some of your interests and how you got to be in the role that you currently are at? And what does your role comprise inside of Maersk?

Mark Sear:

Wow. It’s a long story. I’m an old guy, so I’m just couple of years over 60 now, which you could say you don’t look it, but don’t worry about it.

Kunal Agarwal:

You don’t look it at all, only 40.

Mark Sear:

I’m a generation that didn’t, not many of us went to university, so let me start there. So I left school at 18, did a bit of time in the basic military before going to what you would call, I suppose fundamentally, a crypto analyst school. They would detect how smart you were, whether you had a particular thing for patents, and they sent me there. Did that, and then since then I’ve worked in banking, in trading in particular. I ran a big trading group for a major bank, which was great fun, so we were using data all the time to look for both, not just arbitrage, but other things. Fundamentally, my life has been about data.

Kunal Agarwal:

Right.

Mark Sear:

Even as a kid, my dad had a very small business and because he didn’t know anything about computers, I would do the computing for him and work out the miles per gallon that his trucks were getting and what the trade-in was.

Kunal Agarwal:

Sure.

Mark Sear:

And things like that. So data’s been part of my life and I love everything about data and what it can do for people, companies, everything. Yeah, that’s it. Data.

Kunal Agarwal:

That’s great, Mark. Obviously this is a conference spot, a data team, so it’s great to hear from the data guy who’s been doing it for a really long time. So, Mark, to begin, Maersk, as you said, is one of the largest shipping and logistics companies in the world. How has data transformed your company?

Mark Sear:

One thing, this is a great question. How has it transformed and how will it transform?

Kunal Agarwal:

Yes.

Mark Sear:

I think that for the first time in the last couple of years, and I’ve been very lucky, I’ve only been with the company three years, but shortly after I joined, we had a new tech leader, a gentleman called Navneet Kapoor. The guy is a visionary. If you imagine shipping was seen for many years, there’s a bit of a backwater really. You move containers from one country to another country on ships, that was it. Navneet has changed the game for us all and made people realize that data is pervasive in logistics. It’s literally everywhere. If you think about our biggest ship, ship class, for example, it’s called an E-Class. That can take over 18,000 shipping containers on one journey from China to Europe, 18,000.

Kunal Agarwal:

Oh wow.

Mark Sear:

Think about that. So that’s absolutely huge. Now, to put that into context, in one journey, one of those ships will move more goods than was moved in the entire 19th century between continents, one journey. And we got six of them and they’re going backwards and forwards all the time. So the data has risen exponentially and what you can do with it, we are now just starting to get to grips with it, that’s what so exciting. Consider, we have companies that do want to know how much carbon is being produced as part of their products. We have things like that. We just have an incredibly diverse set of products.

To give you an example, I worked on a project about 18 months ago where we worked out, working in tandem with a couple of nature organizations, that if a ship hits a whale at 12 knots and above, that whale will largely die. If you hit it below 12 knots, it will live. It’s a bit like hitting an adult at 30 miles an hour versus 20. The company puts some money in so we could use the data for where the whales were to slow the ships down. So this is an example of where this company doesn’t just think about what can we do to make money. This is a company that thinks about how can we use data to better the company, make us more profitable, and at the same time, put back into the planet that gave us the ability to have this business.

Kunal Agarwal:

Let’s not forget that we’re human, most importantly.

Mark Sear:

Yeah, it’s super exciting, right? You can make all the money in the world. If you trash the planet, there’s not a lot left to enjoy as part of it. And I love that about this company.

Kunal Agarwal:

Absolutely. And I’m guessing with the pandemic and post-pandemic, and all of the other data sets that you guys are gathering anyways from sensors or from the shipping lines or from all the efficiencies, with all the proliferation of all this data inside your organization, what challenges has your team faced or does the Maersk data team face?

Mark Sear:

Well, my team is in the enterprise architecture team. We therefore deal with all the other teams that are dealing with data, and I think we got the same challenges as everybody. We’ve got the data quality right? Do we know where that data comes from? Are we processing it efficiently? Do we have the right ideas to work on the right insights to get value out of that data? I think they’re common industry things, and as with everything, it’s a learning process. So one man’s high-quality data is another woman’s low-quality data.

And depending on who you are and what you want to do with that data, people have to understand how that quality affects other people downstream. And of course, because you’re quite right, we did have a pandemic, and in the pandemic shipping rates went a little bit nuts and they’re normalizing now. But, of course, if you think about introducing predictive algorithms where the price is going vertically and the algorithm may not know that there’s a pandemic on, it just sees price. So I think what we find is challenging, same as everybody else, is how do you put that human edge around data? Very challenging. How do you build really high-performing teams? How do you get teams to truly work together and develop that esprit de corps? So there are a lot of human problems that go alongside the data problems.

Kunal Agarwal:

Yeah. Mark, give us a sense of your size. In terms of teams, applications, whatever would help us understand what you guys were, where you guys are, and where you guys headed.

Mark Sear:

Three years ago when I joined there were 1,900 people in tech; we’ve now got nearly 6,000. We had a huge amount of outsourcing; now we’re insourcing, we’re moving to an open source first event-based company. We’ve been very inquisitive. We’ve bought some logistics companies, so we’ve gone on the end-to-end journey now with the logistics integrator of choice globally. We’ve got our own airline. So you have to think about a lot of things that play together.

My team is a relatively tiny team. We’ve got about 12, but we liaise with, for example, our global data and analytics team that has got 600 people in it. We then organized into platforms, which are vertically problem solving, but fully horizontally integrated passing events between them. And each one of those has their own data team in it as well. So overall, I would guess we’ve got 3,000 people working directly with data in IT and then of course many thousands more.

Kunal Agarwal:

Wow.

Mark Sear:

Out in the organization. So it’s big organizations. Super exciting. Should say, now I’m going to get a quick commercial in. If you are watching this and you are a top data talent, please do hit me up with your resume.

Kunal Agarwal:

There’s a couple of thousand people watching this live, so you’ll definitely.

Mark Sear:

Hey, there you go, man. So listen, as long as they’re quality, I don’t care.

Kunal Agarwal:

From Mark, he’s a great boss as well. So when you think about the maturity curve of data operations, where do you think Maersk is at and what stands in your way to be fully matured?

Mark Sear:

Okay, so let’s analyze that. I think the biggest problem in any maturity curve is not defining the curve. It’s not producing a pyramid to say we are here and a dial to say, well, you rank as a one, you want to rank as a five.

Kunal Agarwal:

Sure.

Mark Sear:

The biggest problem to me is the people that actually formulate that curve. Now everyone’s got staff turnover and everyone or the majority of people know that they’re part of a team. But the question is how do you get that team to work with other teams and how do you disseminate that knowledge and get that group think of what is best practice for DataOps? What is best practice for dealing with these problems?

Kunal Agarwal:

It’s almost a spectrum on the talent side, isn’t it?

Mark Sear:

It’s a spectrum on the talent side, there’s a high turnover because certainly in the last 12 to 18 months, salaries have been going crazy, so you’ve had crazy turnover rates in some areas, not so much in other areas. So the human side of this is one part of the problem, and it’s not just the human side to how do you keep them engaged, it’s how do you share that knowledge and how do you get that exponential learning organization going?

And perhaps when we get into how we’ve arrived at tools like Unravel, I’ll explain to you what my theory is on that, but it’s almost a swarm learning that you need here, an ants style learning of how to solve problems. And that’s the hardest thing, is getting everybody in that boat swimming in the same direction before you can apply best practices because everybody says this is best practice. Sure, but if it was as simple as looking at a Gartner or whoever thing and saying, “Oh, there are the five lines we need to do,” everybody would do it. There’d be no need for anybody to innovate because we could do it; human beings aren’t very good at following rules, right?

Kunal Agarwal:

Yeah. So what kind of shifts and changes did you have to make in your big data operations and tools that you had to put into place for getting that maturity to where you expected it to be?

Mark Sear:

I think the first thing we’ve got to do, we’ve got to get people thinking slightly shorter timeframe. So everybody talks about Agile, Agile, Agile.

Kunal Agarwal:

Right.

Mark Sear:

Agile means different things to different people. We had some people who thought that Agile was, “Well, you’re going to get a fresh data set at the end of the day, so what the heck are you complaining about? When I started 15 years ago, you got it weekly.” That’s not agile. Equally, you’ve got people who say, I need real-time data. Well, do you really need real-time data if you’re actually dealing with an expense account? You probably don’t.

Kunal Agarwal:

Right.

Mark Sear:

Okay, so the first thing we’ve got to do is level set expectations of our users and then we’ve got to dovetail what we can deliver into those. You’ve got to be business focused, you’ve got to bring value. And that’s a journey. It’s a journey for the business users.

Kunal Agarwal:

Sure.

Mark Sear:

It’s a journey for our users. It’s about learning. So that’s what we’re doing. It’s taking time. Yeah, it’s taking time, but it’s like a snowball. It is rolling and it is getting bigger and it’s getting better, getting faster.

Kunal Agarwal:

And then when you think about the tools, Mark, are there any that you have to put into place to accelerate this?

Mark Sear:

I mean, we’ve probably got one of everything to start and now we’re shrinking. If I take . . . am I allowed to talk about Unravel?

Kunal Agarwal:

Sure.

Mark Sear:

So I’ll talk about–

Kunal Agarwal:

As much as you would.

Mark Sear:

–Unravel for a few seconds. So if you think about what we’ve got, let’s say we’ve got 3,000 people, primarily relatively young, inexperienced people churning out Spark code, let’s say Spark Databricks code, and they all sit writing it. And of course if you are in a normal environment, you can ask the person next to you, how would you do this? You ask the person over there, how would you do this? We’ve had 3,000 engineers working from home for two years, even now, they don’t want to come into the office per se, because it’s inconvenient, number one, because you might be journeying in an hour in and an hour home, and also it’s not actually, truly is productive. So the question is how do you harvest that group knowledge and how do people learn?

So for us, we put Unravel in to look at and analyze every single line of code we write and come up with those micro suggestions and indeed macro suggestions that you would miss. And believe me, we’ve been through everything like code walkthroughs, code dives, all those things. They’re all standard practice. If you’ve got 2,000 people and they write, let’s say, 10 lines of code a day each, 20,000 lines of code, you are never going to walk through all of that code. You are never going to be able to level set expectations. And this is key to me, be able to go back to an individual data engineer and say, “Hey, dude, listen, about these couple of lines of code. Did you realize if you did it like this, you could be 10 times as efficient?” And it’s about giving that feedback in a way that allows them to learn themselves.

And that’s what I love about Unravel: you can get the feedback, but it’s not like when you get that feedback, nobody says, “Come into my office, let’s have a chat about these lines of code.” You go into your private workspace, it gives you the suggestions, you deal with the suggestions, you learn, you move on, you don’t make the mistakes again. And they may not even be mistakes. They might just be things you didn’t know about.

Kunal Agarwal:

Right.

Mark Sear:

And so because Unravel takes data from lots of other organizations as well, as I see it, we’re in effect, harvesting the benefits of hundreds of thousands of coders globally, of data engineers globally. And we are gaining the insights that we couldn’t possibly gain by being even the best self-analysis on the planet, you couldn’t do it without that. And that to me is the advantage of it. It’s like that swarm mentality. If you’ve ever, anybody watching this, had a look at swarm AI, which is to predict, you can use it to predict events. It’s like if you take a soccer game, and I’ve worked in gambling, if you take a soccer match and you take a hundred people, I’ll call it soccer, even though the real name for is football, you Americans.

Kunal Agarwal:

It’s football, I agree too.

Mark Sear:

It’s football, so we’re going to call it football, association football to give you it’s full name. If you ask a hundred football fans to predict a score, you’ll get a curve, and you’ll generally, from that predictor, get a good result. Way more accurate than asking 10 so-called experts, such as with code. And that’s what we’re finding with Unravel is that sometimes it’s the little nuances that just pop up that are giving us more benefits.

Kunal Agarwal:

Right.

Mark Sear:

So it’s pivotal to how we are going to get benefits out over the longer term of what we’re doing.

Kunal Agarwal:

That’s great. And we always see a spectrum of skills inside an organization. So our mission is trying to level the playing field so anybody, even a business user, can log in without knowing the internals of all of these complex data technologies. So it’s great to hear the way Maersk is actually using it. We spoke a little bit about making these changes. We’d love to double click on some of these hurdles, right? Because you said it was a journey to get to people to this mature or fast-moving data operations, if you may, or more agile data operations if you may. If we can double click for a second, what has been the biggest hurdle? Is it the mindset? Is it managing the feedback loop? Is it changing the practices? Is it getting new types of people on board? What has been the biggest hurdle?

Mark Sear:

Tick all of the above.

Kunal Agarwal:

Okay.

Mark Sear:

But I think–

Kunal Agarwal:

Pick for option E.

Mark Sear:

Yeah, so let me give you an example. There are several I’ve had with people that have said to me, “I’ve been doing this 25 years. There’s nothing, I’ve been doing it 25 years.” That presupposes that 25 years of knowledge and experience is better than 10 minutes with a tool that’s got 100,000 years of learning.

Kunal Agarwal:

Right.

Mark Sear:

Over a 12-month period. So that I classify that as the ego problem. Sometimes people need their ego brushing, sometimes they need their ego crushing. It’s about looking the person in the eye, working out what’s the best strategy of dealing with them and saying to them, “Look, get on board.” This isn’t about saying you are garbage or anything else. This is about saying to you, learn and keep mentoring other people as you learn.

Kunal Agarwal:

Yeah.

Mark Sear:

I remember another person said to me, “Oh my god, I’ve seen what this tool can do. Is AI going to take my job?” And I said to them, no, AI isn’t going to take your job, but if you’re not careful, somebody, a human being that is using AI will take it, and that doesn’t apply to me. That applies just in general to the world. You cannot be a Luddite, you cannot fight progress. And as we’ve seen with Chat GPT and things like that recently, the power of the mass of having hundreds and thousands and millions of nodes analyzing stuff is precisely what will bring that. For example, my son who’s 23, smart kid, well, so he tells me. Smart kid, good uni, good university, blah blah blah. He said to me, “Oh Tesla, they make amazing cars.” And I said to him, Tesla isn’t even a car company. Tesla is a data company that happens to build a fairly average electric car.

Kunal Agarwal:

Absolutely.

Mark Sear:

That’s it. It’s all about data. And I keep saying to my data engineers, to be the best version of you at work and even outside work, keep picking up data about everything, about your life, about your girlfriend, the way she feels. About your boyfriend, the way he feels. About your wife, your mother. Everything is data. And that’s the mindset. And the biggest thing for me, the biggest issue has been getting everybody to think and recognize how vital data is in their life, and to be open to change. And we all know throughout go through this cycle of humanity, a lack of openness to change is what’s held humanity back. I seek to break that as well.

Kunal Agarwal:

I love that Mark. Switching gears, we spoke a little bit about developer productivity. We spoke about agility and data operations. Maersk obviously runs, like you were explaining, a lot of their data operations on the cloud. And as we see a lot of organizations when they start to get bigger and bigger and bigger in use cases on the cloud, cost becomes a front and center or a first-class citizen conversation to have. Shed some light on that for us. What is that maturity inside of Maersk, or how do you think about managing costs and budgets and forecast on the cloud, and what’s the consequence of not doing that correctly?

Mark Sear:

Well, there are some things that I can’t discuss because they’re obviously internal, but I think, let’s say I speak to a lot of people in a lot of companies, and there seem to be some themes that run everywhere, which is there’s a rush towards certain technologies, and people, they test it out on something tiny and say, “Hey, isn’t this amazing? Look how productive I am.” Then they get into production and somebody else says, “That’s really amazing. You were very productive. But have you seen what comes out the other end? It’s a cost, a bazillion dollars an hour to run it.” Then you’ve got this, I think they called it the Steve Jobs reality distortion field, where both sets of people go into this weird thing of, “Well, I’m producing value because I’m generating code and isn’t it amazing?” And the other side is saying, “Yeah, but physically the company’s going to spend all its money on the cloud. We won’t be able to do any other business.”

Kunal Agarwal:

Yeah.

Mark Sear:

So we are now getting to a stage where we have some really nice cost control mechanisms coming in. For me, it’s all in the audit. And crucial to this is do it upfront. Do it in your dev environment. Don’t go into production, get a giant bill and then say, how do I cut that bill? Which is again, where we’ve put Unravel now, right in the front of our development environment. So nothing even goes into production unless we know it’s going to work at the right cost price. Because otherwise, you’ve just invented the world’s best mechanism for closing the stable door after the cost horse has bolted, right?

Kunal Agarwal:

Right.

Mark Sear:

And then that’s always a pain because post-giant bill examinations are really paying, it’s a bit like medicine. I don’t know if you know, but in China, you only pay a doctor when you are well. As soon as you are sick, you stop paying bills and they have to take care of you. So that to me is how we need to look at cost.

Kunal Agarwal:

I love that. Love that analogy.

Mark Sear:

Do it upfront. Keep people well, don’t ever end up with a cost problem. So that’s again, part of the mindset. Get your data early, deal with it quickly. And that’s the level of maturity we are getting to now. It’s taking time to get there. We’re not the only people, I know it’s everywhere. But I would say to anybody, I was going to say lucky enough to be watching this, but that’s a little bit cocky, isn’t it? Anybody watching this? Whatever you do, get in there early, get your best practice in as early as possible. Go live with fully costed jobs. Don’t go live, work out what the job cost is and then go, how the hell do I cut it?

Kunal Agarwal:

Yeah.

Mark Sear:

Go live with fully costed jobs and work out well, if it costs this much in dev test, what’s it going to cost in prod? Then check it as soon as it goes live and say, yeah, okay, the delta’s right, game on. That’s it.

Kunal Agarwal:

So measure twice, cut once, and then you’re almost shifting left. So you’re leaving it for the data engineers to go and figure this out. So there’s a practice that’s emerging called FinOps, which is really a lot of these different groups of teams getting together to exactly solve this problem of understand what the cost is, optimize what the cost is, and then govern what the cost is as well. So who within your team does what I’m sure the audience would love to hear that a little bit.

Mark Sear:

Pretty much everybody will do everything, every individual data engineer, man, woman, child, whatever will be, but we’re not using child labor incidentally, that was.

Kunal Agarwal:

Yeah, let’s clarify that one for the audience.

Mark Sear:

That’s a joke. Edit that out. Every person will take it on themselves to do that because ultimately, I have a wider belief that every human being wants to do the right thing, given everything else being equal, they want to do the right thing. So I will say to the people that I speak to as data engineers, as new data engineers, I will say to them, we will empower you to create the best systems in the world. Only you can empower yourself to make them the most efficient systems in the world.

Kunal Agarwal:

Interesting.

Mark Sear:

And by giving it to them and saying, “This is a matter of personal pride, guys,” at the end of the day, am I going to look at every line of your code and say, “You wouldn’t have got away with that in my day.” Of course not. When I started in it, this is how depressingly sad it is. We had 16K of main memory on the main computer for a bank in an IBM mainframe, and you had to write out a form if you wanted 1K of disk. So I was in a similar program in those days. Now I’ve got a phone with God knows how much RAM on it.

Kunal Agarwal:

Right, and anybody can spin up a cloud environment.

Mark Sear:

Absolutely. I can push a button, spin up whatever I want.

Kunal Agarwal:

Right.

Mark Sear:

But I think the way to deal with this problem is to, again, push it left. Don’t have somebody charging in from finance waiving a giant bill saying, “Guys, you are costing a fortune.” Say to people, let’s just keep that finance dude or lady out of the picture. Take it on yourself, yourself. Show a bit of pride, develop this esprit de corps, and let’s do it together.

Kunal Agarwal:

Love it. Mark, last question. This is a fun one and I know you’re definitely going to have some fun answer over here. So what are your predictions for this data industry for this year and beyond? What are we going to see?

Mark Sear:

Wow, what do I think? Basically–

Kunal Agarwal:

Since you’ve got such a pulse on the overall industry and market.

Mark Sear:

So to me, the data industry, obviously it’ll continue to grow. I don’t believe that technology in many levels, I’ll give you over a couple of years, technology in many levels, we’re actually a fashion industry. If the fashion is to outsource, everybody outsource. So the fashion is to in-source, everybody does. Women’s skirts go up, fashion changes, they come down. Guys wear flared trousers, guy wears wear narrow trousers and nobody wants to be out of fashion. What I think’s going to happen is data is going to continue to scale, quantum computing will take off within a few years. What’s going to happen is your CEO is going to say, “Why have I got my data in the cloud and in really expensive data centers when someone has just said that I can put the whole of our organization on this and keep it in the top drawer of my desk?”

And you will have petabyte, zettabyte scale in something that can fit in a shoebox. And at that point it’ll change everything. I will probably either be dead, or at least hopefully retired and doing something by then. But I think it is for those people that are new to this industry, this is an industry that’s going to go forever. I personally hope I get to have an implant in my head at some point from Elon. I will be going for, I’m only going to go for version two. I’m not going for version one and hopefully–

Kunal Agarwal:

Yeah, you never want to go for V1.

Mark Sear:

Exactly, absolutely right. But, guys, ladies, everybody watching this, you are in the most exciting part, not just of technology, of humanity itself. I really believe that, of humanity itself, you can make a difference that very few people on the planet get to make.

Kunal Agarwal:

And on that note, I think the big theme that we have going on this series, we strongly feel that data teams are running the world and will continue to run the world. Mark, thank you so much for sharing this, exciting insights, and it’s always fun having you. Thanks you for making the time.

Mark Sear:

Complete pleasure.

The post Enabling Strong Engineering Practices at Maersk appeared first on Unravel.

The 2023 Data Teams Summit (formerly DataOps Unleashed) was a smashing success, with over 2,000 participants from 1,600 organizations attending 23 expert-led breakout sessions, panel discussions, case studies, and keynote presentations covering a wide range of thought leadership and best practices.

There were a lot of sessions devoted to different strategies and considerations when building a high-performing data team, how to become a data team leader, where data engineering is heading, and emerging trends in DataOps (asset-based orchestration, data contracts, data mesh, digital twins, data centers center of excellence). And winding as a common theme throughout almost every presentation was that top-of-mind topic: FinOps and how to get control over galloping cloud data costs.

Some of the highlight sessions are available now on demand (no form to fill out) on our Data Teams Summit 2023 page. More are coming soon. 

There was a lot to see (and full disclosure: I didn’t get to a couple of sessions), but here are 3 sessions that I found particularly interesting.

Enabling strong engineering practices at Maersk

The fireside chat between Unravel CEO and Co-founder Kunal Agarwal and Mark Sear, Head of Data Platform Optimization at Maersk, one of the world’s largest logistics companies, is entertaining and informative. Kunal and Mark cut through the hype to simplify complex issues in commonsensical, no-nonsense language about:

• The “people problem” that nobody’s talking about
• How Maersk was able to upskill its data teams at scale
• Maersk’s approach to rising cloud data costs
• Best practices for implementing FinOps for data teams

Check out their talk here

Maximize business results with FinOps

Unravel DataOps Champion and FinOps Certified Practitioner Clinton Ford and FinOps Foundation Ambassador Thiago Gil explain how and why the emerging cloud financial management discipline of FinOps is particularly relevant—and challenging—for data teams. They cover:

• The hidden costs of cloud adoption
• Why observability matters
• How FinOps empowers data teams
• How to maximize business results 
• The state of production ML

See their session here

Situational awareness in a technology ecosystem

Charles Boicey, Chief Innovation Officer and Co-founder of Clearsense, a healthcare data platform company, explores the various components of a healthcare-centric data ecosystem and how situational awareness in the clinical environment has been transferred to the technical realm. He discusses:

• What clinical situational awareness looks like
• The concept of human- and technology-assisted observability
• The challenges of getting “focused observability” in a complex hybrid, multi-cloud, multi-platform modern data architecture for healthcare
• How Clearsense leverages observability in practice
• Observability at the edge

Watch his presentation here

How to get more session recordings on demand

1. To see other session recordings without any registration, visit the Unravel Data Teams Summit 2023 page. 
2. To see all Data Teams Summit 2023 recordings, register for access here.

And please share your favorite takeaways, see what resonated with your peers, and join the discussion on LinkedIn.  

The post 3 Takeaways from the 2023 Data Teams Summit appeared first on Unravel.

Palo Alto, CA – Jan. 31, 2023 – Unravel Data, the first data observability platform built to meet the needs of modern data teams, announced today that it was recognized as a Representative Vendor in the DataOps Market in the 2022 Gartner Market Guide for DataOps Tools.

“In recent years, organizations have witnessed significant increases in the number of data sources and data consumers as they deploy more applications and use cases. DataOps has risen as a set of best practices and technologies to streamline and efficiently handle this onslaught of new upcoming workloads,” said Sanjeev Mohan, principal with SanjMo. “Just as DevOps helped streamline web application development and make software teams more productive, DataOps aims to do the same thing for data applications.”

According to a Gartner strategic planning assumption from this Market Guide, “by 2025, a data engineering team guided by DataOps practices and tools will be 10 times more productive than teams that do not use DataOps.” In this Market Guide for DataOps Tools, Gartner examines the DataOps market and explains the various capabilities of DataOps tools, paints a picture of the DataOps tool landscape, and offers recommendations.

“Data teams are struggling to keep up with the increased volume, velocity, variety, and complexity of their data applications and data pipelines. These teams are facing many of the same generic challenges that software teams did 10-plus years ago,” said Kunal Agarwal, CEO of Unravel Data. “We’re proud to be recognized by Gartner in the DataOps Market. Unravel Data provides unprecedented visibility across users’ data stacks, proactively troubleshooting and optimizing data workloads, and defining guardrails to govern costs and improve predictability.”

Today’s modern data stack is a complex collection of different systems, platforms, technologies, and environments. Enterprises need a DataOps solution that works with every type of workload and addresses the performance, cost, and quality issues facing data teams today. Founded by data pioneers Kunal Agarwal and Dr. Shivnath Babu, Unravel Data was created on the notion that the exponential growth of data combined with the broad adoption of the public cloud requires an entirely new way to manage and optimize the data pipelines that support the real-time analytics needs of data-driven enterprises.

Customers who have deployed the Unravel platform have been able to double the productivity of data teams and ensure that data applications run on time, while scaling costs efficiently in the cloud. To learn more about how Unravel is leading the DataOps observability space, visit the new library of demonstration videos.

*Gartner, “Market Guide for DataOps Tools”, December 5, 2022.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Gartner^® Market Guide for Dataops Tools… Unravel Data Highlighted appeared first on Unravel.

Analyst Deep Dive into Unravel

This Eckerson Group report gives you a good understanding of how the Unravel platform addresses multiple categories of data observability—application/pipeline performance, cluster/platform performance, data quality, and, most significant, FinOps cost governance—with automation and AI-driven recommendations.

Eckerson Group, a leading global research, consulting, and advisory firm that focuses solely on data and analytics, profiles various elements of Unravel Data:

• Target customers and use cases
• Product functionality
• Differentiators
• Product architecture
• Pricing

You’ll walk away with a clear picture of what Unravel does, how it does it, and why its features and capabilities are crucial for today’s DataOps and FinOps teams.

Eckerson Group profiles provide independent and objective research on products they believe offer exceptional value to customers.

The post Eckerson Report: Data Observability for Modern Digital Enterprises appeared first on Unravel.

Observability designed for data teams

The Unravel platform harnesses full-stack visibility, contextual awareness, AI-powered intelligence, and automation to go “beyond observability”— to not only show you what’s going on, but why and exactly how to make things better and then keep them that way proactively. Unravel is designed for every member of your data team.

Key Use Cases

• Advanced cost governance
• AI-enabled optimization
• Automated troubleshooting
• Flexible data quality
• Cloud migration

Read this datasheet to learn more.

Get the help you need to simplify your modern dataops.

The post Data Observability Datasheet appeared first on Unravel.

Data teams and their business-side colleagues now expect—and need—more from their observability solutions than ever before. Modern data stacks create new challenges for performance, reliability, data quality, and, increasingly, cost. And the challenges faced by operations engineers are going to be different from those for data analysts, which are different from those people on the business side care about. That’s where DataOps observability comes in.

But what is DataOps observability, exactly? And what does it look like in a practical sense for the day-to-day life of data application developers or data engineers or data team business leaders? 

In the Unravel virtual panel discussion What Is Data Observability? Sanjeev Mohan, principal with SanjMo and former Research Vice President at Gartner, lays out the business context and driving forces behind DataOps observability, and Chris Santiago, Unravel Vice President of Solutions Engineering, shows how different roles use the Unravel DataOps observability platform to address performance, cost, and quality challenges. 

Why (and what is) DataOps observability?

Sanjeev opens the conversation by discussing the top three driving trends he’s seeing from talking with data-driven organizations, analysts, vendors, and fellow leaders in the data space. Specifically, how current economic headwinds are causing reduced IT spend—except in cloud computing and, in particular, data and analytics. Second, with the explosion of innovative new tools and technologies, companies are having difficulty in finding people who can tie together all of these moving pieces and are looking to simplify this increasing complexity. Finally, more global data privacy regulations are coming into force while data breaches continue unabated. Because of these market forces, Sanjeev says, “We are seeing a huge emphasis on integrating, simplifying, and understanding what happens between a data producer and a data consumer. What happens between these two is data management, but how well we do the data management is what we call DataOps.”

Sanjeev presents his definition of DataOps, why it has matured more slowly than its cousin DevOps, and the kind(s) of observability that is critical to DataOps—data pipeline reliability and trust in data quality—and how his point of view has evolved to now include a third dimension: demonstrating the business value of data through BizOps and FinOps. Taken together, these three aspects (performance, cost, quality) give all the different personas within a data team the observability they need. This is what Unravel calls DataOps observability.

DataOps observability in practice with Unravel

Chris Santiago walks through how the various players on a data team—business leadership, application/pipeline developers and engineers, data analysts—use Unravel across the three critical vectors of DataOps observability: performance (reliability of data applications/pipelines), quality (trust in the data), and cost (value/ROI modern data stack investments).

Cost (value, ROI)

First up is how Unravel DataOps observability provides deep visibility and actionable intelligence into cost governance. As opposed to the kind of dashboards that cloud providers themselves offer—which are batch-processed aggregated summaries—Unravel lets you drill down from that 10,000-foot view into granular details to see exactly where the money is going. Chris uses a Databricks chargeback report example, but it would be similar for Snowflake, EMR, or GCP. He shows how with just a click, you can filter all the information collected by Unravel to see with granular precision which workspaces, teams, projects, even individual users, are consuming how many resources across your entire data estate—in real time. 

From there, Chris demonstrates how Unravel can easily set up budget tracking and automated guardrails for, say, a user (or team or project or any other tagging category that makes sense to your particular business). Say you want to track usage by the metric of DBUs; you set the budget/guardrail at a predefined threshold and get real-time status insight into whether that user is on track or is in danger of exceeding the DBU budget. You can set up alerts to get ahead of usage instead of getting notified only after you’ve blown a monthly budget.

See more self-paced interactive product tours here

Performance (reliability)

Chris then showed how Unravel DataOps observability helps the people who are actually on the hook for data pipeline reliability, making sure everything is working as expected. When applications or pipelines fail, it’s a cumbersome task to hunt down and cobble together all the information from disparate systems (Databricks, Airflow, etc.) to understand the root cause and figure out the next steps. Chris shows how Unravel collects and correlates all the granular details about what’s going on and why from a pipeline view. And then how you can drill down into specific Spark jobs. From a single pane of glass, you have all the pertinent logs, errors, metrics, configuration settings, etc., from Airflow or Spark or any other component. All the heavy lifting has been done for you automatically.

But where Unravel stands head and shoulders above everyone else is its AI-powered analysis. Automated root cause analysis diagnoses why jobs failed in seconds, and pinpoints exactly where in a pipeline something went wrong. For a whole class of problems, Unravel goes a step further and provides crisp, prescriptive recommendations for changing configurations or code to improve performance and meet SLAs. 

See more self-paced interactive product tours here

Data quality (trust)

Chris then pivots away from the processing side of things to look at observability of the data itself—especially how Unravel provides visibility into the characteristics of data tables and helps prevent bad data from wreaking havoc downstream. From a single view, you can understand how large tables are partitioned, size of the data, who’s using the data tables (and how frequently), and a lot more information. But what may be most valuable is the automated analysis of the data tables. Unravel integrates external data quality checks (starting with the open source Great Expectations) so that if certain expectations are not met—null values, ranges, number of final columns—Unravel flags the failures and can automatically take user-defined actions, like alert the pipeline owner or even kill a job that fails a data quality check. At the very least, Unravel’s lineage capability enables you to track down where the rogue data got introduced and what dependencies are affected.

Whether it’s engineering teams supporting data pipelines, developers themselves making sure they hit their SLAs, budget owners looking to control costs, or business leaders who funded the technology looking to realize full value—everyone who constitutes a “data team”—DataOps observability is crucial to ensuring that data products get delivered reliably on time, every time, in the most cost-efficient manner.

The Q&A session

As anybody who’s ever attended a virtual panel discussion knows, sometimes the Q&A session is a snoozefest, sometimes it’s great. This one is the latter. Some of the questions that Sanjeev and Chris field:

• If I’m migrating to a modern data stack, do I still need DataOps observability, or is it baked in?
• How is DataOps observability different from application observability?
• When should we implement DataOps observability?
• You talked about FinOps and pipeline reliability. What other use cases come up for DataOps observability?
• Does Unravel give us a 360 degree view of all of the tools in the ecosystem, or does it only focus on data warehouses like Snowflake?

To jump directly to the Q&A portion of the event, click on the video image below.

The post Panel recap: What Is DataOps observability? appeared first on Unravel.

The Data Teams Summit 2023 is just around the corner!

This year, on January 25, 2023, we’re taking the peer-to-peer empowerment of data teams one step further, transforming DatOps Unleashed into Data Teams Summit to better reflect our focus on the people—data teams—who unlock the value of data.

Data Teams Summit is an annual, full-day virtual conference, led by data rockstars at future-forward organizations about how they’re establishing predictability, increasing reliability, and creating economic efficiencies with their data pipelines.

Join us for sessions on:

• DataOps best practices
• Data team productivity and self-service
• DataOps observability
• FinOps for data teams
• Data quality and governance
• Data modernizations and infrastructure

The peer-built agenda is packed, with over 20 panel discussions and breakout sessions. Here’s a sneak peek at some of the most highly anticipated presentations:

Keynote Panel: Winning strategies to unleash your data team

Great data outcomes depend on successful data teams. Every single day, data teams deal with hundreds of different problems arising from the volume, velocity, variety—and complexity—of the modern data stack.

Learn best practices and winning strategies about what works (and what doesn’t) to help data teams tackle the top day-to-day challenges and unleash innovation.

Breakout Session: Maximize business results with FinOps

As organizations run more data applications and pipelines in the cloud, they look for ways to avoid the hidden costs of cloud adoption and migration. Teams seek to maximize business results through cost visibility, forecast accuracy, and financial predictability.

In this session, learn why observability matters and how a FinOps approach empowers DataOps and business teams to collaboratively achieve shared business goals. This approach uses the FinOps Framework, taking advantage of the cloud’s variable cost model, and distributing ownership and decision-making through shared visibility to get the biggest return on their modern data stack investments.

See how organizations apply agile and lean principles using the FinOps framework to boost efficiency, productivity, and innovation.

Breakout Session: Going from DevOps to DataOps

DevOps has had a massive impact on the web services world. Learn how to leverage those lessons and take them further to improve the quality and speed of delivery for analytics solutions.

Ali’s talk will serve as a blueprint for the fundamentals of implementing DataOps, laying out some principles to follow from the DevOps world and, importantly, adding subject areas required to get to DataOps—which participants can take back and apply to their teams.

Breakout Session: Becoming a data engineering team leader

As you progress up the career ladder for data engineering, responsibilities shift as you start to become more hands-off and look at the overall picture rather than a project in particular.

How do you ensure your team’s success? It starts with focusing on the team members themselves.

In this talk, Matt Weingarten, a lead Data Engineer at Disney Streaming, will walk through some of his suggestions and best practices for how to be a leader in the data engineering world.

Attendance is free! Sign up here for a free ticket

The post Sneak Peek into Data Teams Summit 2023 Agenda appeared first on Unravel.

Unravel is recognized as a Representative Vendor in the DataOps Market in the 2022 Gartner Market Guide for DataOps Tools.

Data teams are struggling to keep pace with the increased volume, velocity, variety—and complexity—of their data applications/pipelines. They are facing many of the same (generic) challenges that software teams did 10+ years ago. Just as DevOps helped streamline web application development and make software teams more productive, DataOps aims to do the same thing for data applications.

According to a Gartner strategic planning assumption from this Market Guide, “by 2025, a data engineering team guided by DataOps practices and tools will be 10 times more productive than teams that do not use DataOps.”

The report states that “the DataOps market is highly fragmented.” This Gartner Market Guide analyzes the DataOps market and explains the various capabilities of DataOps tools, paints a picture of the DataOps tool landscape, and offers recommendations.

Our understanding on DataOps and some of the key points we took away from the Gartner Market Guide:

The way data teams are doing things today isn’t working.

One of the key findings in the Gartner report analysis reveals that “a DataOps tool is a necessity to reduce the use of custom solutions and manual efforts around data pipeline operations. Buyers seek DataOps tools to streamline their data operations.” In our opinion, manual effort/custom solutions require prohibitively vast amounts of time and expertise—both of which are already in short supply within enterprise data teams. 

No single DataOps tool does everything.

DataOps covers a lot of ground. Gartner defines the core capabilities of DataOps tools as orchestration (including connectivity, workflow automation, data lineage, scheduling, logging, troubleshooting and alerting), observability (monitoring live/historic workflows, insights into workflow performance and cost metrics, impact analysis), environment management (infrastructure as code, resource provisioning, environment repository templates, credentials management), deployment automation, and test automation. We think it’s clear that no one tool does it all. The report recommends, “Follow the decision guidelines and avoid multibuy circumstances by understanding the diverse market landscape and focusing on a desired set of core capabilities.”

DataOps tools break down silos.

A consequence of modern data stack complexity is that disparate pockets of experts all run their own particular tools of choice in silos. Communication and collaboration break down, and you wind up with an operational Tower of Babel that leads to missed SLAs, friction and finger-pointing, and everybody spending way too much time firefighting. DataOps tools are designed specifically to avoid this unsustainable situation.

Choose DataOps tools that give a single pane of glass view.

The Gartner report recommends to “prioritize DataOps tools that give you a ‘single pane of glass’ for diverse data workloads across heterogeneous technologies with orchestration, lineage, and automation capabilities.” The modern data stack is a complex collection of different systems, platforms, technologies, and environment. Most enterprises use a combination of them all. You need a DataOps tool that works with all kinds of workloads in this heterogenous stack for different capabilities and different reasons—which can be boiled down to the three dimensions of performance, cost, and quality.

We are excited that Gartner recognized Unravel as a Representative Vendor in the DataOps market, and we couldn’t agree more with their Market Guide for DataOps Tools recommendations.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The post Our 4 key takeaways from the 2022 Gartner^® Market Guide for DataOps Tools appeared first on Unravel.

Unravel Data Accelerates DataOps Time to Value with Launch of its 2022 Fall Release

Latest Edition of the Unravel Platform Includes BigQuery Support and Cost 360 for Amazon EMR to Help Customers Optimize the Cost, Efficiency, and Performance of their Data Applications

Palo Alto, CA – Nov 9, 2022 – Unravel Data, the first DataOps observability platform built to meet the needs of modern data teams, today announced the general availability of its 2022 Fall Release of the Unravel Platform. With this new release, users of the Unravel Platform are now able to leverage several new capabilities including support for Google Cloud BigQuery and Cost 360 for Amazon EMR. These new capabilities are designed to help users boost the efficiency of their public cloud spend, simplify troubleshooting across their big data ecosystem, and improve the overall performance of their business-critical data applications.

“Data teams have a clear mandate to ensure that the data pipelines that support their data analytics programs are fully optimized, running efficiently and staying within budget. However, given the complexity of their data ecosystem, getting answers about the health and performance of their data pipelines is harder than ever,” said Kunal Agarwal, founder and CEO of Unravel Data. “Whether they’re migrating more workloads to platforms like BigQuery or Amazon EMR or already running it as part of their data ecosystem, enterprise data teams are struggling to control costs and accurately forecast their resource requirements that are ultimately impacting their ability to execute on their strategic data analytics initiatives. With this latest edition, Unravel customers will be better able to gain the full-stack observability they need to optimize performance and manage their costs according to budget.”

Some of the new capabilities in the Fall Edition of the Unravel Platform include:

• Full Cost Governance for Amazon EMR: With Unravel’s new ‘Cost 360 for Amazon EMR’ capability, customers can enjoy full visibility and gain critical insights into their spending on Amazon EMR. A new cost page has been added that enables customers to observe their EMR cost and usage trends and identify anomalies, slice and dice chargeback views with a variety of filters, and actively monitor costs against user-defined budgets that can automatically trigger warnings if costs are projected to exceed predefined thresholds.
• Advanced EMR Cluster Management: To streamline the management of EMR clusters, a new Clusters page is available inside the Unravel UI for Amazon EMR, allowing users to actively monitor all their EMR clusters and cost data from a single location. The Clusters page also helps users quickly visualize, debug, and troubleshoot issues at both the cluster and application levels.
• Unified View into BigQuery Deployments: Unravel enables customers to now view all information about any number of Google Cloud Platform projects and all BigQuery queries in these respective projects from a single interface. Key performance indicators and metadata are collected for every query. The new edition now supports advanced relevance-based ranked search, faceted search based on metadata and performance indicators, time-based search, as well as drill downs into the individual query level.
• Enhancements for Databricks: When scheduling a job to extract metadata from Databricks, Unravel users can now specify multiple database names and can extract up to 25,000 Delta tables in a single job. Other Databricks enhancements include custom pricing for workload and tier combinations, APIs to detect anomalies and metrics, as well as the ability to encrypt passwords and tokens to harden application security.

More than 20 enterprise companies in the Fortune 100, including two of the top five global pharmaceutical companies and three of the top 10 financial institutions in the world, today rely on Unravel Data to gain unprecedented visibility across their data stacks, proactively troubleshoot and optimize their data workloads, and define guardrails to govern costs and improve predictability.

Data teams are invited to sign up for a free trial of the Unravel Platform at: https://www.unraveldata.com/create-account/

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading DataOps observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51 (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data Accelerates DataOps Time to Value appeared first on Unravel.

Unravel Data to Host India’s First DataOps Observability Conference in Bangalore

The day-long conference on Friday, October 14, will decode DataOps observability and showcase the latest trends and developments in observability

October 12, 2022, Bangalore: Unravel Data, the first DataOps observability platform built to meet the needs of modern data teams, today announced that it is hosting India’s first DataOps Observability Conference on Friday, October 14, 2022 in Bangalore. The day-long conference is hosted in partnership with JetBrains and The Fifth Elephant. Hundreds of data engineers, data leaders, architects, and operations engineers from leading technology companies will be attending the conference.

DataOps and observability are two of the biggest trends in modern data management. This first-of-its-kind conference dedicated to these trends will explain why DataOps observability has become a mission-critical requirement for managing modern data stacks.

The conference will feature leading data professionals from companies like Uber, Dell Technologies, Infosys and Intuit as speakers. The agenda for the event includes panel discussions and talks on managing the modern data stack and applying DataOps observability to all data and analytics use cases.

Dr. Shivnath Babu – co-founder and CTO of Unravel Data will deliver the keynote address at the event. Industry experts who will be speaking on DataOps are Anuj Gupta – chief AI officer at Vahan Inc, Atri Sharma – Senior Staff Engineer at Uber, Sona Samad, Staff Engineer at Intuit and Pallavi Angraje – Software Engineer at Intuit. A Panel discussion on “DataOps Observability: Old Wine in a New Bottle or the Next Big Thing?” will be hosted by Dr. Shivnath Babu. The panelists include Govinda Sambamurthy – Head Of Engineering, Enterprise Data Portability at Atlassian, Akhila Prabhakaran – Consultant Data Scientist at Dell and Giriraj Bagdi – VP Engineering Operations, Data/AI Platform at Unravel.

“We are excited to host the first ever DataOps Observability Conference for data leaders and professionals in India and demonstrate the different aspects of DataOps observability, including application and pipeline performance, operational observability into how the entire platform or system is running end-to-end, and business observability aspects such as ROI and—most significantly—FinOps insights to govern and control escalating cloud costs,” said Shivnath.

Unravel Data estimates that data engineers and operations teams spend around 70-75% of their time tracking down and resolving problems through manual detective work and time-consuming trial and error. Further, as more workloads migrate to the cloud, team leaders are finding that costs are getting out of control—often leading to migration initiatives stalling out completely because of budget overruns.

“Data is eating the world, but complexity is eating data teams who are struggling with cloud costs, pipeline performance, and data quality issues. True DataOps observability can empower these teams to easily tackle even the most complex modern data stack challenges, so they can spend more time on innovation and less time troubleshooting issues,” Shivnath added.

Unravel’s DataOps Observability Platform extracts, correlates, and analyzes information from every system in the modern data stack, within and across some of the most popular data ecosystems. The platform is designed to show data application and pipeline performance, cloud costs, and quality; and proactively suggests precise, prescriptive fixes to quickly and efficiently solve problems. The AI-powered solution helps enterprises realize greater return on their investment in the modern data stack by delivering faster troubleshooting, better performance to meet service level agreements, self-service features that allow applications to get out of development and into production faster and more reliably, and reduced cloud costs.

Data teams can preview how Unravel supports a wide variety of daily observability challenges by viewing the new library of quick demonstration videos on YouTube here.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading DataOps Observability Platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51 (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit www.unraveldata.com.

Media Contact
Bhawana Singh
+917204277001
bhawanas@evoc.in

The post India’s First DataOps Observability Conference in Bangalore appeared first on Unravel.

Unravel now pulls in data quality checks from external tools into its single-pane-of-glass full-stack observability view.

All data details in one place

Have all the granular data details at your fingertips, correlated in context. One-click drill down into lineage, schema, timelines, partitions, size, usage, and more.

Understand the impact in seconds

Automated insights show exactly what’s affected downstream, where bad data got introduced upstream, and how to fix the issue.

Automated circuit-breaker guardrails

Automated alerts let you know when—and where—quality checks fail. Customizable system- and user-defined, policy-based guardrails prevent bad data from seeing the light of day.

Download our Data Quality Datasheet here.

The post Unravel Automated Data Quality Datasheet appeared first on Unravel.

DBTA recently hosted a roundtable webinar with four industry experts on “Unlocking the Value of Cloud Data and Analytics.” Moderated by Stephen Faig, Research Director, Unisphere Research and DBTA, the webinar featured presentations from Progress, Ahana, Reltio, and Unravel.

You can see the full 1-hour webinar “Unlocking the Value of Cloud Data and Analytics” below.

Here’s a quick recap of what each presentation covered.

Todd Wright, Global Product Marketing Manager at Progress, in his talk “More Data Makes for Better Analytics” showed how Progress DataDirect connectors let users get to their data from more sources securely without adding a complex software stack in between. He quoted former Google research director Peter Norvig (now fellow at the Stanford Institute for Human-Centered Artificial Intelligence) about how more data beats clever algorithms: “Simple models and a lot of data trump more elaborate models based on less data.” He then outlined how Progress DataDirect and its Hybrid Data Pipeline platform uses standard-based connectors to expand connectivity options of BI and analytics tools, access all your data using a single connector, and make on-prem data available to cloud BI and analytics tools without exposing ports or implementing costly and complex VPN tunnels. He also addressed how to secure all data sources behind your corporate authentication/identity to mitigate the risk of exposing sensitive private information (e.g., which tables and columns to expose to which people) and keep tabs on data usage for auditing and compliance.

Rachel Pedreschi, VP of Technical Services at Ahana, presented “4 Easy Tricks to Save Big Money on Big Data in the Cloud.” She first simplified the cloud data warehouse into its component parts (data on disk + some kind of metadata + a query layer + system that authenticates users and allows them to do stuff). Then she broke it down to see where you could save some money. Starting at the bottom, the storage layer, she said data lakes are a more cost-effective way of providing data to users throughout the organization. At the metadata level, Hive Metastore or AWS Glue are less expensive options. For authentication, she mentioned Apache Ranger or AWS Lake Formation. But what about SQL? For that she has Presto, an open source project that came out of Facebook as a successor to Hive. Presto is a massively scalable distributed in-memory system that allows you write queries not just against data lake files but also other databases as well. She calls this collectively the Open SQL Data Lakehouse. Ahana is an AWS-based service that gets you up and running with this Open Data Lakehouse in 30 minutes. 

Mike Frasca, Field Chief Field Technology Officer at Reltio, discussed the value and benefits of a modern master data management platform. He echoed previous presenters’ points about how the volume, velocity, and variety of data have become almost overwhelming, especially given how highly siloed and fragmented data is today. Data teams spend most of their time getting the data ready for insights—data consolidation, aggregation, and cleansing; synchronizing and standardizing data, ensuring data quality, timeliness, and accuracy, etc.—rather than actually delivering insights from analytics. He outlined the critical functions a master data management (MDM) platform should provide to deliver precision data. Entity management automatically unifies data into a dynamic enterprise source of truth, including context-aware master records. Data quality management continuously validates, cleans, and standardizes the data via custom validation rules. Data integration receives input from and distributes mastered data to any application or data store in real time and at high volumes. He emphasized that what characterizes a “modern” MDM is the ability to access data in real time—so you’re not making decisions based on data that’s a week or a month old. Cloud MDM extends the capabilities to relationship management, data governance, and reference data management. He wound up his presentation with compelling real-life examples of customer efficiency gains and effectiveness, including Forrester TEI (total economic impact) results on ROI.

Chris Santaigo, VP of Solutions Engineering at Unravel, presented how AI-enabled optimization and automatic insights help unlock the value of cloud data. He noted how nowadays every company is a data company. If they’re not leveraging data analytics to create strategic business advantages, they’re falling behind. But he illustrated how the complexity of the modern data stack is slowing companies down. He broke down the challenges into three C’s: cost, where enterprises are experiencing significant budget overruns; resource constraints—not infrastructure resources, but human resources—the talent gap and mismatch between supply and demand for expertise; and complexity of the stack, where 87% of data science projects never make it into production because it’s so challenging to implement them.

But it all comes down to people—all the different roles on data teams. Analysts, data scientists, engineers on the application side; architects, operations teams, FinOps, and business stakeholders on the operations side. Everybody needs to be working off a “single source of truth” to break down silos, enable collaboration, eliminate finger-pointing, and empower more self-service. Software teams have APM to untangle this complexity—for web apps. But data apps are a totally different beast. You need observability designed specifically for data teams. You could try to stitch together the details you need for performance, cost, data quality from a smorgasbord of point tools or solutions that do some of what you need (but not all), but that’s time-consuming and misses the holistic view of how everything works together so necessary to connect the dots when you’re looking to troubleshoot faster, control spiraling cloud costs, automate AI-driven optimization (for performance and cost), or migrate to the cloud on budget and on time. That’s exactly where Unravel comes in.

Check out the full webinar, including attendee Q&A here!

The post 3-Minute Recap: Unlocking the Value of Cloud Data and Analytics appeared first on Unravel.

This blog was originally published by Unravel CEO Kunal Agarwal on LinkedIn in September 2022.

I was chatting with Sanjeev Mohan, Principal and Founder of SanjMo Consulting and former Research Vice President at Gartner, about how the emergence of DataOps is changing people’s idea of what “data observability” means. Not in any semantic sense or a definitional war of words, but in terms of what data teams need to stay on top of an increasingly complex modern data stack. While much ink has been spilled over how data observability is much more than just data profiling and quality monitoring, until only very recently the term has pretty much been restricted to mean observing the condition of the data itself. 

But now DataOps teams are thinking about data observability more comprehensively as embracing other “flavors” of observability like application and pipeline performance, operational observability into how the entire platform or system is running end-to-end, and business observability aspects such as ROI and—most significantly—FinOps insights to govern and control escalating cloud costs.

That’s what we at Unravel call DataOps observability.

Data teams are getting bogged down

Data teams are struggling, overwhelmed by the increased volume, velocity, variety, and complexity of today’s data workloads. These data applications are simultaneously becoming ever more difficult to manage and ever more business-critical. And as more workloads migrate to the cloud, team leaders are finding that costs are getting out of control—often leading to migration initiatives stalling out completely because of budget overruns.

The way data teams are doing things today isn’t working.

Data engineers and operations teams spend way too much time firefighting “by hand.” Something like 70-75% of their time is spent tracking down and resolving problems through manual detective work and a lot of trial and error. And with 20x more people creating data applications than fixing them when something goes wrong, the backlog of trouble tickets gets longer, SLAs get missed, friction among teams creeps in, and the finger-pointing and blame game begins.

This less-than-ideal situation is a natural consequence of inherent process bottlenecks and working in silos. There are only a handful of experts who can untangle the wires to figure out what’s going on, so invariably problems get thrown “over the wall” to them. Self-service remediation and optimization is just a pipe dream. Different team members each use their own point tools, seeing only part of the overall picture, and everybody gets a different answer to the same problem. Communication and collaboration among the team breaks down, and you’re left operating in a Tower of Babel.

Accelerating next-gen DataOps observability

These problems aren’t new. DataOps teams are facing some of the same general challenges as their DevOps counterparts did a decade ago. Just as DevOps united the practice of software development and operations and transformed the application lifecycle, today’s data teams need the same observability but tailored to their unique needs. And while application performance management (APM) vendors have done a good job of collecting, extracting, and correlating details into a single pane of glass for web applications, they’re designed for web applications and give data teams only a fraction of what they need.

System point tools and cloud provider tools all provide some of the information data teams need, but not all. Most of this information is hidden in plain sight—it just hasn’t been extracted, correlated, and analyzed by a single system designed specifically for data teams.

That’s where Unravel comes in.

Data teams need what Unravel delivers—observability designed to show data application/pipeline performance, cost, and quality coupled with precise, prescriptive fixes that will allow you to quickly and efficiently solve the problem and get on to the real business of analyzing data. Our AI-powered solution helps enterprises realize greater return on their investment in the modern data stack by delivering faster troubleshooting, better performance to meet service level agreements, self-service features that allow applications to get out of development and into production faster and more reliably, and reduced cloud costs.

I’m excited, therefore, to share that earlier this week, we closed a $50 million Series D round of funding that will allow us to take DataOps observability to the next level and extend the Unravel platform to help connect the dots from every system in the modern data stack—within and across some of the most popular data ecosystems. 

Unlocking the door to success

By empowering data teams to spend more time on innovation and less time firefighting, Unravel helps data teams take a page out of their software counterparts’ playbook and tackle their problems with a solution that goes beyond observability—to not just show you what’s going on and why, but actually tell you exactly what to do about it. It’s time for true DataOps observability.

To learn more about how Unravel Data is helping data teams tackle some of today’s most complex modern data stack challenges, visit: www.unraveldata.com. 

The post Get Ready for the Next Generation of DataOps Observability appeared first on Unravel.

Chief Data & Analytics Officer UK (CDAO UK) is the United Kingdom’s premier event for senior data and analytics executives. The three-day event, with more than 200 attendees and 50+ industry-leading speakers, was packed with case studies, thought leadership, and practical advice around data culture, data quality and governance, building a data workforce, data strategy, metadata management, AI/MLOps, self-service strategies, and more.

Catherine King of Corinium Global Intelligence interviews Chris Santiago, Unravel Data VP of Solutions Engineering

Chris Santiago, Unravel VP of Solutions Engineering, sat down with Catherine King from event host Corinium Global Intelligence to discuss what’s top-of-mind for data and analytics leaders.

Here are the highlights from their 1-on-1 interview.

Catherine: What are you hearing in the market at the moment? What are people coming up to you and having a chat with you about today?

Chris: I think that, big picture-wise, there are a lot of things being talked about that are very insightful. But some of the stuff that hasn’t really been talked about are the things that people don’t want to talk about. They have this grand vision, but how are they going to get there? How are they going to execute? How are they going to have the processes in place, the right people hired—that sort of thing—to take advantage of data and all the great technology that’s out there and actually execute on the vision?

Catherine: I think you’ve hit the nail on the head there. I think the technology piece, which was so prevalent a few years ago—do we have the right tech, the right tools, to go out and do these things?—that’s almost been ticked off and done. Now it’s actually, do you have the processes in place to achieve it. So from your perspective, what would you like to see businesses do differently?

Chris: The technology has made leaps and bounds over the last few years. If you think about big data people who came from the Hadoop world, one of the challenges of that stack was that it did require expertise. Companies back then struggled to get the ROI on performance, on getting some sort of business value at the end of the day. Fast-forward to today, especially with the advancements in the cloud, they have solved a lot of the challenges. Ease of use? I can just literally just log in, click a button, and have an environment. Storage is cheaper. A lot of the problems back then have been solved with today’s technology.

I do think that the one thing that hasn’t been 100% solved yet, though, is the people, the skills. Right now, the things technology is not necessarily addressing directly are: Do we have the right skill set? Do we have the right people? As more folks are using these newer technologies, the gap in skills to do things the right way and best practices are not being directly addressed. The way that most customers are handling it right now is that they’ll bring in the experts, consultants like Accenture, Avanade, Deloitte, etc.

In order to achieve the true ROI in these data stacks, it’s adjusting the people-problem.

Catherine: What do you see coming in the next 12 months?

Chris: I think that if we look at the industry as a whole, a lot of the technology is still considered new(ish). You have a lot of folks who are still using DevOps tools. So they’re trying to work with observability tools that are focused on the issues that aren’t necessarily what data teams want. So I think in the next six to twelve months we’re going to have a proliferation of observability trying to solve this problem specifically for data teams. Because they’re different problems [than software application issues]. You can’t use the same [APM] tooling for folks who are running Databricks or Snowflake. There are going to be different problems, different challenges.

Obviously, Unravel is in that space now, but I do think the industry will recognize more and more that this is actually not just a small problem, but a major problem. Companies are starting to realize that [observability designed for data teams] is not a nice-to-have anymore, it’s a must-have—and needs to be addressed right now.

Everybody has a vision. Everybody has an idea of what they want to do with data, whether it’s having that strategic business advantage or getting insights that they didn’t know about—everybody trying to do really cool stuff—but everybody always seems to forget about how to execute. There’s lots of interesting talks about how we’re going to measure things, what KPIs we’re going to be tracking, what methodologies we should have in place—all great stuff—but if you truly want to be successful, it’s all about execution and [doing] the stuff people don’t want to talk about. That’s what will set up companies to be successful or not with their data initiatives, getting into the weeds and solving these hard challenges.

Check out the full interview with Chris from CDAO UK

The post The Data Challenge Nobody’s Talking About: An Interview from CDAO UK appeared first on Unravel.

The post DataOps Observability Designed for Data Teams appeared first on Unravel.

Join us at an event near you or attend virtually to discover our DataOps observability platform, discuss your challenges with one of our DataOps experts, go under the hood to check out platform capabilities, and see what your peers have been able to accomplish with Unravel.

September 21-22: Big Data LDN (London) 

Big Data LDN is the UK’s leading free-to-attend data & analytics conference and exhibition, hosting leading data and analytics experts, ready to arm you with the tools to deliver your most effective data-driven strategy. Stop by the Unravel booth (stand #724) to see how Unravel is observability designed specifically for the unique needs of today’s data teams. 

Register for Big Data LDN here

And be sure to stop by the Unravel booth at 5PM on Day 1 for the Data Drinks Happy Hour for drinks and snacks (while supplies last!)

October 5-6: AI & Big Data Expo – North America (Santa Clara) 

The world’s leading AI & Big Data event returns to Santa Clara as a hybrid in-person and virtual event, with more than 5,000 attendees expected to join from across the globe. The expo will showcase the most cutting-edge technologies from 250+ speakers sharing their unparalleled industry knowledge and real-life experiences, in the forms of solo presentations, expert panel discussions and in-depth fireside chats.

Register for AI & Big Data Expo here

And don’t miss Unravel Co-Founder and CEO Kunal Agarwal’s feature presentation on the different challenges facing different AI & Big Data team members and how multidimensional observability (performance, cost, quality) designed specifically for the modern data stack can help.

October 10-12: Chief Data & Analytics Officers (CDAO) Fall (Boston)

The premier in-person gathering for data & analytics leaders in North America, CDAO Fall offers focus tracks on data infrastructure, data governance, data protection & privacy; analytics, insights, and business intelligence; and data science, artificial intelligence, and machine learning. Exclusive industry summit days for data and analytics professionals in Financial Services, Insurance, Healthcare, and Retail/CPG.

Register for CDAO Fall here

October 14: DataOps Observability Conf India 2022 (Bengaluru)

India’s first DataOps observability conference, this event brings together data professionals to collaborate and discuss best practices and trends in the modern data stack, analytics, AI, and observability.

Join leading DataOps observability experts to:

• Understand what DataOps is and why it’s important
• Learn why DataOps observability has become a mission-critical need in the modern data stack
• Discover how AI is transforming DataOps and observability

Register for DataOps Observability Conf India 2022 here

November 1-3: ODSC West (San Francisco)

The Open Data Science Conference (ODSC) is essential for anyone who wants to connect to the data science community and contribute to the open source applications it uses every day. A hybrid in-person/virtual event, ODSC West features 250 speakers, with 300 hours of content, including keynote presentations, breakout talk sessions, hands-on tutorials and workshops, partner demos, and more. 

Register for ODSC West here

Sneak peek into what you’ll see from Unravel 

Want a taste of what we’ll be showing? Check out our 2-minute guided-tour interactive demos of our unique capabilities. Explore features like:

• Full-stack data pipeline observability
• Automated root cause analysis, at both the job and pipeline level
• Job-level AI optimization recommendations
• Automated cloud cluster optimizations
• Pinpoint-precise chargeback reports
• Automated budget tracking
• Proactive cost governance alerts
• Cloud migration workload fit reports
• Automated cluster discovery

Explore all our interactive guided tours here.

The post Unravel Goes on the Road at These Upcoming Events appeared first on Unravel.

Modern data pipelines have become more business-critical than ever. Every company today is a data company, looking to leverage data analytics as a competitive advantage. But the complexity of the modern data stack imposes some significant challenges that are hindering organizations from realizing their goals and realizing the value of data.

TDWI recently hosted an expert panel on modern data pipelines, moderated by Fern Halper, VP and Senior Director of Advanced Analytics at TDWI, with guests Kunal Agarwal, Co-Founder and CEO of Unravel Data, and Krish Krishnan, Founder of Sixth Sense Advisors.

Dr. Halper opened the discussion with a quick overview of the trends she’s seeing in the changing data landscape, characteristics of the modern data pipeline (automation, universal connectivity, scalable/flexible/fast, comprehensive and cloud-native), and some of the challenges with pipeline processes. Data takes weeks or longer to access new data. Organizations want to enrich data with new data sources, but can’t. There aren’t enough data engineers. Pipeline growth causes problems such as errors and management. It’s hard for different personas to make use of pipelines for self-service. A quick poll of attendees showed a pretty even split among the different challenges.

The panel was asked how they define a modern data pipeline, and what challenges their customers have that modern pipelines help solve.

Kunal talked about the different use cases data-driven companies have for their data to help define what a pipeline is. Banks are using big data to detect and prevent fraud, retailers are running multidimensional recommendation engines (products, price). Software companies are measuring their SaaS subscriptions.

“All these decisions, and all these insights, are now gotten through data products. And data pipelines are the backbone of running any of these business-critical processes,” he said. “So, ultimately, a data pipeline is a sequence of actions that’s gathering, collecting, moving this data from all the different sources to a destination. And the destination could be a data product or a dashboard. And a pipeline is all the stages it takes to clean up the data, transform the data, connect it together, and give it to the data scientist or business analyst to be able to make use of it.”

He said that with the increased volume, velocity, and variety of data, modern data pipelines need to be scalable and extensible—to add new data sources, to move to a different processing paradigm, for example. “And there are tons of vendors who are trying to crack this problem in unique ways. But besides the tools, it should all go back to what you’re trying to achieve, and what you’re trying to drive from the data, that dictates what kind of data pipeline or architecture you should have,” Kunal said.

Krish sees the modern data pipeline as the integration point enabling the true multi-cloud vision. It’s no longer just one system or another, but a mix-and-match “system of systems.” And the challenges revolve around moving workloads to the cloud. “If a company is an on-premises shop and they’re going to the cloud, it’s a laborious exercise,” he said. “There is no universal lift-and-shift mechanism for going to the cloud. That’s where pipelines come into play.”

Observability and the Modern Data Pipeline

The panel discussed the components and capabilities of the modern data pipeline, again circling back to the challenges spawned by the increased complexity. Kunal noted that one common capability among organizations that are running modern data pipelines successfully is observability.

“One key component we see along all the different stages of a pipeline is observability. Observability helps you ultimately improve reliability of these pipelines—that they work on time, every time—and improve the productivity of all the data team members. The poll results show that there aren’t enough data engineers, the demand far exceeds the supply. And what we see is that the data engineers who are present are spending more than 50% of their time just firefighting issues. So observability can help you eliminate—or at least get ahead of—all these different issues. And last but not least, we see that businesses tame their data ambitions by looking at their ballooning cloud bills and cloud costs. And that also happens because of the complexity that these data technologies present. Observability can also help get control and governance around cloud costs so you can start to scale your data operations in a much more efficient manner.”

See the entire panel discussion on demand

Watch the entire conversation with Kunal, Krish, and Fern from TDWI’s Expert Panel: Modern Data Pipelines on-demand replay.

The post Expert Panel: Challenges with Modern Data Pipelines appeared first on Unravel.

The Eckerson Group recently presented a CDO TechVent that explored data observability, “Data Observability: Managing Data Quality and Pipelines for the Cloud Era.” Hosted by Wayne Eckerson, president of Eckerson Group, Dr. Laura Sebastian-Coleman, Data Quality Director at Prudential Financial, and Eckerson VP of Research Kevin Petrie, the virtual event kicked off with a keynote overview of data observability products and best practices, followed by a technology panel discussion, “How to Evaluate and Select a Data Observability Platform,” with four industry experts.

Here are some of the highlights and interesting insights from the roundtable discussion on the factors data leaders should consider when looking at data observability solutions.

Josh Benamram, CEO of Databand, said it really depends on what problem you’re facing, or which team is trying to solve which problem. It’s going to be different for different stakeholders. For example, if your challenge is maintaining SLAs, your evaluation criteria would lean more towards Ops-oriented solutions that cater to data platform teams that need to manage the overall system. On the other hand, if your observability requirements are more targeted towards analysts, your criteria would be more oriented towards understanding the health of data tables rather than the overall pipeline. 

This idea of identifying the problems you’re trying to solve first, or as Wayne put it, “not putting the tools before the horse,” was a consistent refrain among all panelists.

Seth Rao, CEO of FirstEigen, said he would start by asking three questions: Where do you want to observe? What do you want to observe? How much do you want to automate? If you’re running Snowflake, there are a bunch of vendor solutions; but if you’re talking about data lakes, there’s a different set of solutions. Pipelines are yet another group of solutions. If you’re looking to observe the data itself, that’s a different type of observability altogether. Different solutions automate different pieces of observability. He suggested not trying to “boil the ocean” with one product that tries to do everything. He feels that you’ll get only an average product for all functions. Rather, he said, get flexibility of tooling—like Lego blocks that connect with other Lego blocks in your ecosystem.

This point drew the biggest reaction from the attendees (at lest as evidenced by the Q&A chat). Who’s responsible for integrating all the different tools? We already don’t have enough time! A couple of panelists tackled the argument head-on, either in the panel discussion or in breakout sessions. 

Specifically, Rohit Choudhary, CEO of Acceldata, said that the purpose of observability is to simplify everything data teams have to do. You don’t have enough data engineers as it is, and now you’re asking data leaders to invest in a bunch of different data observability tools. Instead of actually help them solve problems, you’re handing them more problems. He said to look at two things when evaluating data observability solutions: what it is capable of today, and what its roadmap looks like and what use cases it will support moving forward. Observability means different things to different people, and it all depends on whether the offering fits your maturity model. Smaller organizations with analytics teams of 10-20 people are probably fine with point solutions. But large enterprises that are dealing with data pipelines at petabyte scale are dealing with much greater complexity. For them, it would be prohibitively expensive to build their own observability solution. 

Chris Santiago, Unravel Data VP of Solutions Engineering, was of the same opinion but looked at things from a different slant. He agreed that different tools—system-specific point tools, native cloud vendor capabilities, various data quality monitoring solutions—all have strengths and weaknesses, with insight into different “pieces of the puzzle.” But rather than connect them all together as discrete building blocks, observability would be better realized by extracting all the relevant granular details, correlating them into a holistic context, and analyzing them with ML and other analytical algorithms so that data teams get the intelligence they need in one place. The problems data teams face—around performance, quality, reliability, cost—are all interconnected, so you’re saving a lot of valuable time and reducing manual effort to have as much insight as possible in a single pane of glass. He refers to such comprehensive capabilities as DataOps observability.

The dimension of cost was something Eckerson analyst Kevin Petrie highlighted in the wrap-up as a key emerging factor. He’s seeing an increased focus on FinOps capabilities, which Chris called out specifically: it’s not just making sure pipelines are running smoothly, but understanding where the spend is going and who the “big spenders” are, so that observability can uncover opportunities to optimize for cost and control/govern the cloud spend.

That’s the cost side for the business, but he said it’s also crucial to understand the profit side. Companies are investing millions of dollars in their modern data stack, but how are we measuring whether they’re getting the value they expected from their investment? Can the observability platform help make sense of all the business metrics in some way? Because at the end of the day, all these data projects have to deliver value. 

Check out Unravel’s breakout presentation, A DataOps Observability Dialogue: Empowering DevOps for Data Teams.

The post Takeaways from CDO TechVent on Data Observability appeared first on Unravel.

Swing by the Unravel Data booth at the AI & Big Data Expo in Santa Clara on October 5-6. The world’s leading AI & Big Data event returns as a hybrid in-person and virtual event, with more than 5,000 attendees expected to join from across the globe. 

And don’t miss Unravel Co-Founder and CEO Kunal Agarwal’s feature presentation on DataOps Observability. He’ll explain why AI & Big Data organizations need an observability platform designed specifically for data teams and their unique challenges, the limitations of trying to “borrow” other observability (like APM) or relying on a bunch of different point tools, and how DataOps observability cuts across and incorporates cross-sections of multiple observability domains (data applications/pipelines/model observability, operations observability, business and FinOps observability, data observability).

Stop by our booth and you’ll be able to: 

• Go under the hood with Unravel’s DataOps observability platform
• Deep-dive into features and capabilities with our experts 
• Learn what your peers have been able to accomplish with Unravel

Our experts will run demos and be available for 1-on-1 conversations throughout the conference. 

Want a taste of what we’ll be showing? Check out our 2-minute guided-tour interactive demos of our unique capabilities. Explore features like:

• Full-stack data pipeline observability
• Automated root cause analysis, at both the job and pipeline level
• Job-level AI optimization recommendations
• Automated cloud cluster optimizations
• Pinpoint-precise chargeback reports
• Automated budget tracking
• Proactive cost governance alerts
• Cloud migration workload fit reports
• Automated cluster discovery

Explore all our interactive guided tours here.

The expo will showcase the most cutting-edge technologies from 250+ speakers sharing their unparalleled industry knowledge and real-life experiences, in the forms of solo presentations, expert panel discussions and in-depth fireside chats.

You can register for the event here.

The post Join Unravel at the AI & Big Data Expo appeared first on Unravel.

Learn more about Apache Spark drivers and how to tune spark application quickly.

During the lifespan of a Spark application, the driver should distribute most of the work to the executors, instead of doing the work itself. This is one of the advantages of using python with Spark over Pandas. The Contented Driver event is detected when a Spark application spends way more time on the driver than the time spent in the Spark jobs on the executors. Leaving the executors idle could waste lots of time and money, specifically in the Cloud environment.

Here is a sample application shown in Unravel for Azure Databricks. Note in the Gantt Chart there was a huge gap of about 2 hours 40 minutes between Job-1 and Job-2, and the job duration for most of the Spark jobs was under 1 minute. Based on the data Unravel collected, Unravel detected the bottleneck which was the contended driver event.

Further digging into the Python code itself, we found that it actually tried to ingest data from the MongoDB server on the driver node alone. This left all the executors idle while the meter was still ticking.

There was some network issue that caused the MongoDB injection slowing down from 15 minutes to 2 plus hours. Once this issue was resolved, there was about 93% reduction in cost. The alternative solution is to move the MongoDB ingestion out of the Spark application. If there is no dependency on previous Spark jobs, we can do it before the Spark application.

If there is a dependency, we can split the Spark application into two. Unravel also collects all the job run status such as Duration and IO as shown below and we can easily see the history of all the job runs and monitor the job.

In conclusion, we must pay attention to the Contended Driver event in a Spark application, so we can save money and time without leaving the executors IDLE for a long time.

Next steps

• Check out what are the biggest spark troubleshooting challenges in 2022 and how to fix them.
• Learn ways to troubleshoot Apache Spark issues.
• Create your free account today.
• Book a demo to see how Unravel simplifies modern data stack management.

The post Tuning Spark applications: Detect and fix common issues with Spark driver appeared first on Unravel.

AI-DRIVEN DATA OBSERVABILITY + FINOPS FOR GOOGLE BIGQUERY

Performance. Reliability. Cost-effectiveness.

Unravel’s automated, AI-powered data observability + FinOps platform for Google Cloud BigQuery and other modern data stacks provides 360° visibility to allocate costs with granular precision, accurately predict spend, run 50% more workloads at the same budget, launch new apps 3X faster, and reliably hit greater than 99% of SLAs.

Unravel Data Observability + FinOps for BigQuery you can:

• Launch new apps 3X faster: End-to-end observability of data-native applications and pipelines. Automatic improvement of performance, cost efficiency, and reliability.
• Run 50% more workloads for same budget: Break down spend and forecast accurately. Optimize apps and platforms by eliminating inefficiencies. Set guardrails and automate governance. Unravel’s AI helps you implement observability and FinOps to ensure you achieve efficiency goals.
• Reduce firefighting time by 99% using AI-enabled troubleshooting: Detect anomalies, drift, skew, missing and incomplete data end-to-end. Integrate with multiple data quality solutions. All in one place.
• Forecast budget with ⨦ 10% accuracy: Accurately anticipate cloud data spending to for more predictable ROI. Unravel helps you accurately forecast spending with granular cost allocation. Purpose-built AI, at job, user and workgroup levels, enables real-time visibility of ongoing usage.

To see Unravel Data for BigQuery in action contact: Data experts  | 650 741-3442

The post Unravel for Google BigQuery Datasheet appeared first on Unravel.

At the recent Data Summit 2022 conference in Boston, Unravel presented “The DataOps Journey: Beyond Observability”—the most highly attended session within the conference’s DataOps Boot Camp track. In a nutshell, Unravel VPs Keith Alsheimer and Chris Santiago discussed:

• The top challenges facing different types of data teams today
• Why web APM and point tools leave fundamentally big observability gaps
• How observability purpose-built for the modern data stack closes these gaps
• What going “beyond observability” looks like

Data teams get caught in a 4-way crossfire 

It’s been said many times, many ways, but it’s true: nowadays every company has become a data company. Data pipelines are creating strategic business advantages, and companies are depending on their data pipelines more than ever before. And everybody wants in—every department in every business unit in every kind of company wants to leverage the insights and opportunities of big data.

But data teams are finding themselves caught in a 4-way crossfire: not only are data pipelines more important than ever, but more people are demanding more output from them, there’s more data to process, people want their data faster, and the modern data stack has gotten ever more complex.

The increased volume, velocity, and variety of today’s data pipelines coupled with their increasing complexity is leaving battle scars across all teams. Data engineers find it harder to troubleshoot apps/pipelines, meet SLAs, and get quality right. Operations teams have too many issues to solve and not enough time—they’re drowning in a flood of service tickets. Data architects have to figure out how to scale these complex pipelines efficiently. Business unit leaders are seeing cloud costs skyrocket and are under the gun to deliver better ROI.

It’s becoming increasingly harder for data teams to keep pace, much less do everything faster, better, cheaper. All this complexity is taking its toll. Data teams are bogged down and burning out. A recent survey shows that 70% of data engineers are likely to quit in the next 12 months. Demand for talent has always exceeded supply, and it’s getting worse. McKinsey research reveals that data analytics has the greatest need to address potential skills gaps, with 15 open jobs for every active job seeker. The latest LinkedIn Emerging Jobs Report says that 5 of the top 10 jobs most in demand are data-related.

And complexity continues in the cloud. First off, the variety of choices is huge—with more coming every day. Too often everybody is doing their own thing in the cloud, with no clear sense of what’s going on. This makes things hard to rein in and scale out, and most companies find their cloud costs are getting out of control.

DataOps is the answer, but  . . .

Adopting a DataOps approach can help solve these challenges. There are varying viewpoints on exactly what DataOps is (and isn’t), but we can certainly learn from how software developers and operations teams tackled a similar issue via DevOps—specifically, leveraging and enhancing collaboration, automation, and continuous improvement. 

One of the key foundational needs for this approach is observability. You absolutely must have a good handle on what’s going on in your system, and why. After all, you can’t improve things—much less simplify and streamline—if you don’t really know everything that’s happening. And you can’t break down silos and embrace collaboration without the ability to share information.

At DataOps Unleashed 2022 we surveyed some 2,000 participants about their top challenges, and by far the #1 issue was the lack of visibility across the environment (followed by lack of proactive alerts, the expense of runaway jobs/pipelines, lack of experts, and no visibility into cloud cost/usage).

One of the reasons teams have so much difficulty gaining visibility into their data estate is that the kind of observability that works great for DevOps doesn’t cut it for DataOps. You need observability purpose-built for the modern data stack.

Get the right kind of observability

When it comes to the modern data stack, traditional observability tools leave a “capabilities gap.” Some of the details you need live in places like the Spark UI, other information can be found in native cloud tools or other point solutions, but all too often you have to jump from tool to tool, stitching together disparate data by yourself manually. Application performance monitoring (APM) does this correlation for web applications, but falls far short for data applications/pipelines. 

First and foremost, the whole computing framework is different for data applications. Data workloads get broken down into multiple, smaller, often similar parts each processed concurrently on a separate node, with the results re-combined upon completion — parallel processing. In contrast, web applications are a tangle of discrete request-response services processed individually.

Consequently, there’s a totally different class of problems, root causes, and remediation for data apps vs. web apps. When doing your detective work into a slow or failed app, you’re looking at a different kind of culprit for a different type of crime, and need different clues. You need a whole new set of data points, different KPIs, from distinct technologies, visualized in another way, and correlated in a uniquely modern data stack–specific context.

You need to see details at a highly granular level — for each sub-task within each sub-part of each job — and then marry them together into a single pane of glass that comprises the bigger picture at the application, pipeline, platform, and cluster levels.

That’s where Unravel comes in.

Unravel extracts all the relevant raw data from all the various components of your data stack and correlates it to paint a picture of what’s happening. All the information captured by telemetry data (logs, metrics, events, traces) is pulled together in context and in the “language you’re speaking”—the language of data applications and pipelines—within the context of your workloads. 

click to enlarge

Going beyond observability

Full-stack observability tells you what’s going on. But to understand why, you need to go beyond observability and apply AI/ML to correlate patterns, identify anomalies, derive meaningful insights, and perform automated root cause analysis. And for that, you need lots of granular data feeding ML models and AI algorithms purpose-built specifically for modern data apps/pipelines.

The size and complexity of modern data pipelines—with hundreds (even thousands) of intricate interdependencies, jobs broken down into stages processing in parallel—could lead to a lot of trial-and-error resolution effort even if you know what’s happening and why. What you really need to know is what to do.

That’s where Unravel goes where other observability solutions can only dream about.

Unravel’s AI recommendation engine tells you exactly what your next step is in crisp, precise detail. For example, if there’s one container in one part of one job that’s improperly sized and so causing the entire pipeline to fail, Unravel not only pinpoints the guilty party but tells you what the proper configuration settings would be. Or another example is that Unravel can tell you exactly why a pipeline is slow and how you can speed it up.

AI recommendations tell you exactly what to do to optimize for performance.

Observability is all about context. Traditional APM provides observability in context for web applications, but not for data applications and pipelines. But Unravel does. Its telemetry, correlation, anomaly detection, root cause analysis capabilities,  and AI-powered remediation recommendations are all by design built specifically to understand, troubleshoot, and optimize modern data workloads.

The post Unravel at Data Summit 2022 Recap: Beyond Observability appeared first on Unravel.

Whether they know it or not, every company has become a data company. Data is no longer just a transactional byproduct, but a transformative enabler of business decision-making. In just a few years, modern data analytics has gone from being a science project to becoming the backbone of business operations to generate insights, fuel innovation, improve customer satisfaction, and drive revenue growth. But none of that can happen if data applications and pipelines aren’t running well.

Yet data-driven organizations find themselves caught in a crossfire: their data applications/pipelines are more important than ever, but managing them is more difficult than ever. As more data is generated, processed, and analyzed in an increasingly complex environment, businesses are finding the tools that served them well in the past or in other parts of their technology stack simply aren’t up to the task.

Modern data stacks are a different animal

Would you want an auto mechanic (no matter how excellent) to diagnose and fix a jet engine problem before you took flight? Of course not. You’d want an aviation mechanic working on it. Even though the basic mechanical principles and symptoms — engine trouble — are similar, automobiles and airplanes are very different under the hood. The same is true with observability for your data application stacks and your web application stacks. The process is similar, but they are totally different animals.

At first glance, it may seem that the leading APM monitoring and observability tools like Datadog, New Relic, Dynatrace, AppDynamics, etc., do the same thing as a modern data stack observability platform like Unravel. And in the sense that both capture and correlate telemetry data to help you understand issues, that’s true. But one is designed for web apps, while the other for modern data pipelines and applications.

Observability for the modern data stack is indeed completely different from observability for web (or mobile) applications. They are built and behave differently, face different types of issues for different reasons, requiring different analyses to resolve problems. To fully understand, troubleshoot, and optimize (for both performance and cost) data applications and pipelines, you need an observability platform that’s built from the ground up to tackle the specific complexities of the modern data stack. Here’s why.

What’s different about modern data applications?

First and foremost, the whole computing framework is different for data applications. Data workloads get broken down into multiple, smaller, often similar parts each processed concurrently on a separate node, with the results re-combined upon completion — parallel processing. And this happens at each successive stage of the workflow as a whole. Dependencies within data applications/pipelines are deep and layered. It’s crucial that everything — execution time, scheduling and orchestration, data lineage, and layout — be in sync.

In contrast, web applications are a tangle of discrete request-response services processed individually. Each service does its own thing and operates relatively independently. What’s most important is the response time of each service request and how that contributes to the overall response time of a user transaction. Dependencies within web apps are not especially deep but are extremely broad.

Web apps are request-response; data apps process in parallel.

Consequently, there’s a totally different class of problems, root causes, and remediation for data apps vs. web apps. When doing your detective work into a slow or failed app, you’re looking at a different kind of culprit for a different type of crime, and need different clues (observability data). You need a whole new set of data points, different KPIs, from distinct technologies, visualized in another way, and correlated in a uniquely modern data stack–specific context.

The flaw with using traditional APM tools for modern data stacks

What organizations who try to use traditional APM for the modern data stack find is that they wind up getting only a tiny fraction of the information they need from a solution like Dynatrace or Datadog or AppDynamics, such as infrastructure and services-level metrics. But over 90% of the information data teams need is buried in places where web APM simply doesn’t go — you need an observability platform specifically designed to dig through all the systems to get this data and then stitch it together into a unified context.

This is where the complexity of modern data applications and pipelines rears its head. The modern data stack is not a single system, but a collection of systems. You might have Kafka or Kinesis or Data Factory for data ingestion, some sort of data lake to store it, then possibly dozens of different components for different types of processing: Druid for real-time processing, Databricks for AI/ML processing, BigQuery or Snowflake for data warehouse processing, another technology for batch processing — the list goes on. So you need to capture deep information horizontally across all the various systems that make up your data stack. But you also need to capture deep information vertically, from the application down to infrastructure and everything in between (pipeline definitions, data usage/lineage, data types and layout, job-level details, execution settings, container-level information, resource allocation, etc.).

Cobbling it together manually via “swivel chair whiplash,” jumping from screen to screen, is a time-consuming, labor-intensive effort that can take hours — even days — for a single problem. And still there’s a high risk that it won’t be completely accurate. There is simply too much data to make sense of, in too many places. Trying to correlate everything on your own, whether by hand or with a homegrown jury-rigged solution, requires two things that are always in short supply: time and expertise. Even if you know what you’re looking for, trolling through hundreds of log files is like looking for a needle in a stack of needles.

An observability platform purpose-built for the modern data stack can do all that for you automatically. Trying to make traditional APM observe data stacks is simply using the wrong tool for the job at hand.

DevOps APM vs. DataOps observability in practice

With the growing complexity in today’s modern data systems, any enterprise-grade observability solution should do 13 things:

1. Capture full-stack data — both horizontally and vertically — from the various systems that make up the stack, including engines, schedulers, services, and cloud provider
2. Capture information about all data applications: pipelines, warehousing, ETL/ELT, machine learning models, etc.
3. Capture information about datasets, lineage, users, business units, computing resources, infrastructure, and more
4. Correlate, not just aggregate, data collected into meaningful context
5. Understand all application/pipeline dependencies on data, resources, and other apps
6. Visualize data pipelines end to end from source to output
7. Provide a centralized view of your entire data stack, for governance, usage analytics, etc.
8. Identify baseline patterns and detect anomalies
9. Automatically analyze and pinpoint root causes
10. Proactively alert to prevent problems
11. Provide recommendations and remedies to solve issues
12. Automate resolution or self-healing
13. Show the business impact

While the principles are the same for data app and web app observability, how to go about this and what it looks like are markedly dissimilar.

Everything starts with the data — and correlating it

If you don’t capture the right kind of telemetry data, nothing else matters.

APM solutions inject agents that run 24×7 to monitor the runtime and behavior of applications written in .NET, Java, Node.js, PHP, Ruby, Go, and dozens of other languages. These agents collect data on all the individual services as they snake through the application ecosystem. Then APM stitches together all the data to understand which services the application calls and how the performance of each discrete service call impacts performance of the overall transaction. The various KPIs revolve around response times, availability (up/down, green/red), and the app users’ digital experience. The volume of data to be captured is incredibly broad, but not especially deep.

APM is primarily concerned with response times and availability. Here, Datadog shows red/green status and aggregated metrics.

Here, AppDynamics shows the individual response times for various interconnected services.

The telemetry details to be captured and correlated for data applications/pipelines, on the other hand, need to be both broad and extremely deep. A modern data workload comprises hundreds of jobs, each broken down into parallel-processing parts, with each part executing various tasks. And each job feeds into hundreds of other jobs and applications not only in this particular pipeline but all the other pipelines in the system.

Today’s pipelines are built on an assortment of distributed processing engines; each might be able to monitor its own application’s jobs but not show you how everything works as a whole. You need to see details at a highly granular level — for each sub-task within each sub-part of each job — and then marry them together into a single pane of glass that comprises the bigger picture at the application, pipeline, platform, and cluster levels.

DataOps observability (here, Unravel) looks at completely different metrics at the app level . .

. . . as well as the pipeline level.

Let’s take troubleshooting a slow Spark application as an example. The information you need to investigate the problem lives in a bunch of different places, and the various tools for getting this data give you only some of what you need, not all.

The Spark UI can tell you about the status of individual jobs but lacks infrastructure and configuration details and other information to connect together a full pipeline view. Spark logs help you retrospectively find out what happened to a given job (and even what was going on with other jobs at the same time) but don’t have complete information about resource usage, data partitioning, container configuration settings, and a host of other factors that can affect performance. And, of course, Spark tools are limited to Spark. But Spark jobs might have data coming in from, say, Kafka and run alongside a dozen other technologies.

Conversely, platform-specific interfaces (Databricks, Amazon EMR, Dataproc, BigQuery, Snowflake) have the information about resource usage and the status of various services at the cluster level, but not the granular details at the application or job level.Having all the information specific for data apps is a great start, but it isn’t especially helpful if it’s not all put into context. The data needs to be correlated, visualized, and analyzed in a purposeful way that lets you get to the information you need easily and immediately.

Then there’s how data is visualized and analyzed

Even the way you need to look at a data application environment is different. A topology map for a web application shows dependencies like a complex spoke-and-wheel diagram. When visualizing web app environments, you need to see the service-to-service interrelationships in a map like this:

How Dynatrace visualizes service dependencies in a topology map.

With drill-down details on service flows and response metrics:

Dynatrace drill-down details

For a modern data environment, you need to see how all the pipelines are interdependent and in what order. The view is more like a complex system of integrated funnels:

A modern data estate involves many interrelated application and pipeline dependencies (Source: Sandeep Uttamchandani)

You need full observability into not only how all the pipelines relate to one another, but all the dependencies of multiple applications within each pipeline . . .

An observability platform purpose-built for modern data stacks provides visibility into all the individual applications within a particular pipeline

. . . with granular drill-down details into the various jobs within each application. . .

 . . and the sub-parts of each job processing in parallel . . .

How things get fixed

Monitoring and observability tell you what’s going on. But to understand why, you need to go beyond observability and apply AI/ML to correlate patterns, identify anomalies, derive meaningful insights, and perform automated root cause analysis. “Beyond observability” is a continuous and incremental spectrum, from understanding why something happened to knowing what to do about it to automatically fixing the issue. But to make that leap from good to great, you need ML models and AI algorithms purpose-built for the task at hand. And that means you need complete data about everything in the environment.

The best APM tools have some sort of AI/ML-based engine (some are more sophisticated than others) to analyze millions of data points and dependencies, spot anomalies, and alert on them.

For data applications/pipelines, the type of problems and their root causes are completely different than web apps. The data points and dependencies needing to be analyzed are completely different. The patterns, anomalous behavior, and root causes are different. Consequently, the ML models and AI algorithms need to be different.

In fact, DataOps observability needs to go even further than APM. The size of modern data pipelines and the complexities of multi-layered dependencies — from clusters to platforms and frameworks to applications to jobs within applications to sub-parts of those jobs to the various tasks within each sub-part — could lead to a lot of trial-and-error resolution effort even if you know what’s happening and why. What you really need to know is what to do.

An AI-driven recommendation engine like Unravel goes beyond the standard idea of observability to tell you how to fix a problem. For example, if there’s one container in one part of one job that’s improperly sized and so causing the entire pipeline to fail, Unravel not only pinpoints the guilty party but tells you what the proper configuration settings would be. Or another example is that Unravel can tell you exactly why a pipeline is slow and how you can speed it up. This is because Unravel’s AI has been trained over many years to understand the specific intricacies and dependencies of modern data stacks.

AI recommendations tell you exactly what to do to optimize for performance.

Business impact

Sluggish or broken web applications cost organizations money in terms of lost revenue and customer dissatisfaction. Good APM tools are able to put the problem into a business context by providing a lot of details about how many customer transactions were affected by an app problem.

As more and more of an organization’s operations and decision-making revolve around data analytics, data pipelines that miss SLAs or fail outright have an increasingly significant (negative) impact on the company’s revenue, productivity, and agility. Businesses must be able to depend on their data applications, so their applications need to have predictable, reliable behavior.

For example: If a fraud prevention data pipeline stops working for a bank, it can cause billions of dollars in fraudulent transactions going undetected. Or a slow healthcare analysis pipeline may increase risk for patients by failing to provide timely responses. Measuring and optimizing performance of data applications and pipelines correlates directly to how well the business is performing.

Businesses need proactive alerts when pipelines deviate from their normal behavior. But going “beyond observability” would tell them automatically why this is happening and what they can do to get the application back on track. This allows businesses to have reliable and predictable performance.

There’s also an immediate bottom-line impact that businesses need to consider: maximizing their return on investment and controlling/optimizing cloud spend. Modern data applications process a lot of data, which usually consumes a large amount of resources — and the meter is always running. This means the cloud bills can rack up fast.

To keep costs from spiraling out of control, businesses need actionable intelligence on how best to optimize their data pipelines. An AI recommendations engine can take all the profile and other key information it has about applications and pipelines and identify where jobs are overprovisioned or could be tuned for improvement. For example: optimizing code to remove inefficiencies, right-sizing containers to avoid wastage, providing the best data partitioning based on goals, and much more.

AI recommendations pinpoint exactly where and how to optimize for cost.

AI recommendations and deep insights lay the groundwork for putting in place some automated cost guardrails for governance. Governance is really all about converting the AI recommendations and insights into impact. Automated guardrails (per user, app, business unit, project) would alert operations teams about unapproved spend, potential budget overruns, jobs that run over a certain time/cost threshold, and the like. You can then proactively manage your budget, rather than getting hit with sticker shock after the fact.

In a nutshell

Application monitoring and observability solutions like Datadog, Dynatrace, and AppDynamics are excellent tools for web applications. Their telemetry, correlation, anomaly detection, and root cause analysis capabilities do a good job of helping you understand, troubleshoot, and optimize most areas of your digital ecosystem — one exception being the modern data stack. They are by design built for general-purpose observability of user interactions.

In contrast, an observability platform for the modern data stack like Unravel is more specialized. Its telemetry, correlation, anomaly detection, root cause analysis capabilities — and in the case of Unravel uniquely, its AI-powered remediation recommendations, automated guardrails, and automated remediation — is by design built specifically to understand, troubleshoot, and optimize modern data workloads.

Observability is all about context. Traditional APM provides observability in context for web applications, but not for data applications and pipelines. That’s not a knock on these APM solutions. Far from it. They do an excellent job at what they were designed for. They just weren’t built for observability of the modern data stack. That requires another kind of solution designed specifically for a different kind of animal.

The post Why Legacy Observability Tools Don’t Work for Modern Data Stacks appeared first on Unravel.

Get a quick Unravel product demo at Data Summit 2022 Tuesday & Wednesday, May 17-18, at the Hyatt Regency Boston.

Presented by our friends from DBTA (Database Trends and Applications), Data Summit 2022 is a unique conference that brings together IT practitioners and business stakeholders from all types of organizations to learn, share, and celebrate the trends and technologies shaping the future of data. See where the world of Big Data and data science is going, and how to get there fast. 

Featuring workshops, panel discussions, and provocative talks, Data Summit 2022 provides a comprehensive educational experience designed to guide you through all of today’s key issues in data management and analysis. Whether your interests lie in the technical possibilities and challenges of new and emerging technologies or using Big Data for business intelligence, analytics, and other business strategies, we have something for you!

And Unravel is there as a sponsor, so be sure to stop by and check out how we’re going beyond observability to solve some of today’s biggest challenges with:

• Full-stack data pipeline observability: Accelerate troubleshooting with observability purpose-built for the modern data stack
• AI-enabled optimization: Get automated AI-driven recommendations to improve performance and cost
• Advanced cost governance: Pinpoint exactly where cloud spend is going, track actual and projected costs against budgets, and put in automated guardrails
• Cloud migration IQ: Make accurate data-driven decisions before, during, and after your migration

 And we’re running a raffle – stop by our booth for a chance to an Oculus VR headset.

The conference has different talk tracks over the two days—Modern Data Strategy Essentials Today, Emerging Technologies & Trends in Data & Analytics, What’s Next in Data & Analytics Architecture, The Future of Data Warehouses, Data Lakes & Data Hubs—as well as special day-long programs: DataOps Boot Camp, Database & DevOps Boot Camp, and AI & Machine Learning Summit.

Doug Laney, Data & Strategy Innovation Fellow at West Monroe and author of Infonomics, is the keynote speaker with his presentation Data Is Not the New Oil.

On Tuesday, May 17, don’t miss Unravel Co-Founder & CEO Kunal Agarwal’s feature presentation, Exploiting the Multi-Cloud Opportunity With DataOps, in which he

• details the common obstacles that data teams encounter in data migration
• explains why next-generation data tools must evolve beyond simple observability to provide prescriptive insights
• shares best practices for optimizing big data costs
• demonstrates through real-world case studies how a mature DataOps practice can accelerate even the most complex cloud migration projects.

So stop by the Unravel booth to go beyond observability, win an Oculus VR headset, and see how Unravel’s AI-enabled purpose-built observability for modern data stack can help your data team monitor, observe, manage, troubleshoot, and optimize the performance and cost of large-scale modern data applications. 

Register here

The post Visit Unravel at Data Summit 2022 appeared first on Unravel.

The term DataOps (like its older cousin, DevOps) means different things to different people. But one thing that everyone agrees on is its objective: accelerate the delivery of data-driven insights to the business quickly and reliably.

As the business now expects—or more accurately, demands—increasingly more insights from data, on increasingly shorter timelines, with increasingly more data being generated, in increasingly more complex data estates, it becomes ever more difficult to keep pace with the new levels of speed and scale.

Eliminating as much manual effort as possible is key. In a recent Database Trends and Applications (DBTA)-hosted roundtable webinar, Succeeding with DataOps: Implementing, Managing, and Scaling, three engineering leaders with practical experience in enterprise-level DataOps present solutions that simplify and accelerate:

• Security. Satori Chief Scientist Ben Herzberg discusses how you can meet data security, privacy, and compliance requirements faster with a single universal approach.
• Analytics. Incorta VP of Product Management Ashwin Warrier shows how you can make more data available to the business faster and more frequently.
• Observability. Unravel VP of Solutions Engineering Chris Santiago explains how AI-enabled full-stack observability accelerates data application/pipeline optimization and troubleshooting, automates cost governance, and delivers the cloud migration intelligence you need before, during, and after your move to the cloud.

Universal data access and security

Businesses today are sitting on a treasure trove of data for analytics. But within that gold mine is a lot of personally identifiable information (PII) that must be safeguarded. That brings up an inherent conflict: making the data available to the company’s data teams in a timely manner without exposing the company to risk. The accelerated volume, velocity, and variety of data operations only makes the challenge greater. 

Satori’s Ben Herzberg shared that recently one CISO said he has visibility over his network and his endpoints, but data was like a black box—it was hard for the Security team to understand what was going on. Security policies usually had to be implemented by other teams (data engineering or DataOps, database administrators). 

The 5 capabilities of Satori’s DataSecOps

So what Satori does is provide a universal data access and security platform where organizations can manage all their security policies across their entire data environment from one single place in a simplified way. Ben outlined the five capabilities that Satori enables to realize DataSecOps:

1. Continuous data discovery and classification
   Data keeps on changing, with more people touching it every day. Auditing or mapping your sensitive data once a year, even once a quarter or once a month, may not be sufficient.
2. Dynamic data masking
   Different data consumers need different levels of data anonymization. Usually to dynamically mask data, you have to rely on the different data platforms themselves.
3. Granular data access policies
   A key way to safeguard data is via Attribute-based Access Control (ABAC) policies. For example, you may want data analysts to be able access data stores using Tableau or Looker but not with any other technologies.
4. Access control and workflows
   Satori automates and simplifies the way access is granted or revoked. A common integration is with Slack, so Security team can approve or deny access requests quickly.
5. Monitoring, audit & compliance Satori sits between the data consumers and the data that’s being accessed, so monitoring/auditing for compliance is transparent without ever touching the data itself.

Keeping data safe is always a top-of-mind concern, but no organization can afford to delay projects for months in order to apply security.

Scaling analytics with DataOps

When you think of the complexity of the modern data stack, you’re looking at a proliferation of cloud-based applications but also many legacy on-prem applications that continue to be the backbone of a large enterprise (SAP, Oracle). You have all these sources rich with operational data that business users need to build analytics on top of. So the first thing you see in a modern data stack is the data being moved to a data lake or data warehouse, to make it readily accessible along with making refinements along the way.

But to Incorta’s Ashwin Warrier, this presents two big challenges when it comes to enterprise analytics and operational reporting. First, getting through this entire process takes weeks, if not months. Second, once you finally get the data into the hands of business users, it’s aggregated. Much of your low-fidelity data is gone because of summarizations and aggregations. Building a bridge between the analytics plane and the operations plane requires some work.

Incorta’s Direct Data Platform approach

Where Incorta is different is that it does enrichments but doesn’t try to create transformations to take your source third normal form data and convert that into star schemas. It takes an exact copy with minimal amount of change. Having the same fidelity means that now your business users can get access to their data at scale in just a matter of hours or days instead of weeks or months.

Ashwin related the example of a Fortune 50 customer with about 2,000 users developing over 5,000 unique reports. It had daily refreshes, with an average query time of over 2 minutes, and took over 8 days to deliver insights. With Incorta, it reduced average query time to just a few seconds, now refreshes data 96 times every day, and can deliver insights in less than 1 day.

Taming the beast of modern data stacks

Unravel’s Chris Santiago opened with an overview of how the complexity compounded by complexity of modern data stacks causes problems for different groups of data teams. Data engineers’ apps aren’t hitting their SLAs; how can they make them more reliable? Operations teams are getting swamped with tickets; how can they troubleshoot faster? Data architects are falling behind schedule and going over budget; how can they speed migrations without more risk? Business leaders are seeing skyrocketing cloud bills; how can they better govern costs?

Then he went into the challenges of complexity. A single modern data pipeline is complex enough on its own, with dozens or even hundreds of interdependent jobs on different technologies (Kafka, Airflow, Spark, Hive, all the barn animals in Hadoop). But most enterprises run large volumes of these complex pipelines continuously—now we’re talking about the number of jobs getting into the thousands. Plus you’ve got multiple instances and data stored in different geographical locations around the globe. Already it’s a daunting task to figure out where and why something went wrong. And now there’s multi-cloud.

Managing this environment with point tools is going to be a challenge: they are very specific to the service they’re running, and the crucial information you need is spread across dozens of different systems and tools. Chris points out four key reasons why managing modern data stacks with point tools falls flat:

4 ways point tools fall flat for managing the modern data stack

1. You get only a partial picture.
   The Spark UI, for instance, has granular details about individual jobs but not at the cluster level. Public cloud vendors’ tools have some of this information, but not nothing at the job level.
2. You don’t get to the root cause.
   You’ll get a lot of graphs and charts, but what you really need to know is what went wrong, why—and how to fix it.
3. You’re blind to where you’re overspending.
   You need to know exactly which jobs are using how much resources, whether that’s appropriate for the job at hand, and how you can optimize for performance and cost.
4. You can’t understand your cloud migration needs.
   Things are always changing, and changing fast. You need to always be thinking about the next move.

That’s where Unravel comes in.

Unravel is purpose-built to collect granular data app/pipeline-specific details from every system in your data estate, correlate it all in a “workload-aware” context automatically, analyze everything for you, and provide actionable insights and precise AI recommendations on what to do next.

Unravel is purpose-built for the modern data stack

1. Get single-pane-of-glass visibility across all environments.
   You’ll have complete visibility to understand what’s happening at the job level up to the cluster level. 
2. Optimize automatically.
   Unravel’s AI engine is like having a Spark or Databricks or Amazon EMR telling you exactly what you need to do to optimize performance to meet SLAs or change instance configurations to control cloud costs.
3. Fine-grained insight into cloud costs.
   See at a granular level exactly where the money is going, set some budgets, track spend month over month—by project, team, even individual job or user—and have AI uncover opportunities to save.
4. Migrate on time, on budget.
   Move to the cloud with confidence, knowing that you have complete and accurate insight into how long migration will take, the level of effort involved, and what it’ll cost once you migrate.

As businesses become ever more data-driven, build out more modern data stack workloads, and adopt newer cloud technologies, it will become ever more important to be able to see everything in context, let AI take much of the heavy lifting and manual investigation off the shoulders of data teams already stretched too thin, and manage, troubleshoot, and optimize data applications/pipelines at scale.

Check out the full webinar here.

The post Webinar Recap: Succeeding with DataOps appeared first on Unravel.

The term “observability” means many things to many people. A lot of energy has been spent—particularly among vendors offering an observability solution—in trying to define what the term means in one context or another.

But instead of getting bogged down in the “what” of observability, I think it’s more valuable to address the “why.” What are we trying to accomplish with observability? What is the end goal? 

At Unravel, I’m not only the co-founder and CTO but also the head of our Customer Success team, so when thinking about modern data stack observability, I look at the issue through the lens of customer problems and what will alleviate their pain and solve their challenges. 

I start by considering the ultimate end goal or Holy Grail: autonomous systems. Data teams just want things to work; they want everything to be taken care of, automatically. They want the system itself to be watching over issues, resolving problems without any human intervention at all. It’s the true spirit of AI: all they have to do is tell the system what they want to achieve. It’s going from reactive to proactive. No one ever has to troubleshoot the problem, because it was “fixed” before it ever saw the light of day. The system recognizes that there will be a problem and auto-corrects the issue invisibly.

As an industry, we’re not completely at that point yet for the modern data stack. But we’re getting closer.

We are on a continuous and incremental spectrum of progressively less toil and more automation: going from the old ways of manually stitching together logs, metrics, traces, and events from disparate systems; to accurate extraction and correlation of all that data in context; to automatic insights and identification of significant patterns; to AI-driven actionable recommendations; to autonomous governance.

“Beyond observability” is on a spectrum from manual to autonomous

Beyond observability: to optimization and on to governance

Let’s say you have a data pipeline from Marketing Analytics called ML_Model_Update_and_Scoring that’s missing its SLA. The SLA specifies that the pipeline must run in less than 20 minutes, but it’s taking 25 minutes. What happened? Why is the data pipeline now too slow? And how can you tune things to meet the SLA? This particular application is pretty complex, with multiple jobs processing in parallel and several components (orchestration, compute, auto-scaling clusters, streaming platforms, dashboarding), so the problem could be anywhere along the line.

It’s virtually impossible to manually pore through the thousands of logs that are generated at each stage of the pipeline from the various tools—Spark and Kafka and Airflow logs, Databricks cluster logs, etc.—to “untangle the wires” and figure out where the slowdown could be. But where should you even start? Even if you have an idea of what you’re looking for, it can take hours—even days or weeks for highly complex workflows—to stitch together all the raw metrics/events/logs/traces data to figure out what’s meaningful to why your data pipeline is missing its SLA. Just a single app can run 10,000 containers on 10,000 nodes, with 10,000 logs. 

Modern data stacks are simply too complex for humans to manage by hand

That’s where observability comes in.

Observability tells you “what”

Instead of having to sift through reams of logs and cobble together everything manually, full-stack observability extracts all the relevant raw data from all the various components of your data stack and correlates it to paint a picture of what’s going on. All the information captured by telemetry data (logs, metrics, events, traces) is pulled together in context and in the “language you’re speaking”—the language of data applications and pipelines.

Full-stack observability correlates data from all systems to provide a clear picture of what happened

In this case, observability shows you that while the application completed successfully, it took almost 23 minutes (22m 57s)—violating the SLA of 20 minutes. Here, Unravel takes you to the exact spot in the complex pipeline shown earlier and, on the left, has pulled together granular details about the various jobs processing in parallel. You can toggle on a Gantt chart view to get a better view of the degree of parallelism:

A Gantt chart view breaks down all the individual jobs and sub-tasks processing in parallel

So now you know what caused the pipeline to miss the SLA and where it happened (jobs #0 and #3), but you don’t know why. You’ve saved a lot of investigation time—you get the relevant data in minutes, not hours—and can be confident that you haven’t missed anything, but you still need to do some detective work to analyze the information and figure out what went wrong.

Optimization tells you “why”—and what to do about it

The better observability tools also point out, based on patterns, things you should pay attention to—or areas you don’t need to investigate. By applying machine learning and statistical algorithms, it essentially throws some math at the correlated data to identify significant patterns—what’s changed, what hasn’t, what’s different. This pipeline runs regularly; why was it slow this time? It’s the same kind of analysis a human expert would do, only done automatically with the help of ML and statistical algorithms.

While it would certainly be helpful to get some generalized insight into why the pipeline slowed—it’s a memory issue—what you really need to know is what to do about it. 

AI-enabled observability goes beyond telling you what happened and why, to pinpointing the root cause and providing recommendations on exactly what you need to do to fix it.

AI-driven recommendations pinpoint exactly where—and how—to optimize (click on image to expand)

AI-driven recommendations provide specific configuration parameters that need to be applied in order for the pipeline to run faster and meet the 20-minute SLA. After the AI recommendations are implemented, we can see that the pipeline now runs in under 11 minutes—a far cry from the almost 23 minutes before.

AI recommendations improved pipeline performance from 23m to <11m, now meeting its SLA

Too often getting to the point of actually fixing the problem is another time-consuming effort of trial and error. Actionable AI recommendations won’t fix things automatically for you—taking the action still requires human intervention—but all the Sherlock Holmes work is done for you. The fundamental questions of what went wrong, why, and what to do about it are answered automatically.

Beyond optimizing performance, AI recommendations can also identify where cost could be improved. Say your pipeline is completing within your SLA commitments, but you’re spending much more on cloud resources than you need to. An AI engine can determine how many or what size containers you actually need to run each individual component of the job—vs. what you currently have configured. Most enterprises soon realize that they’re overspending by as much as 50%. 

Governance helps avoid problems in the first place

These capabilities save a ton of time and money, but they’re still relatively reactive. What every enterprise wants is a more proactive approach to making their data applications and pipelines run better, faster, cheaper. Spend less time firefighting because there was no fire to put out. The system itself would understand that the memory configurations were insufficient and automatically take action so that pipeline performance would meet the SLA.

For a whole class of data application problems, this is already happening. AI-powered recommendations and insights lay the groundwork for putting in place some automated governance policies that take action on your behalf.

Governance is really all about converting the AI recommendations and insights into impact. In other words, have the system run automatic responses that implement fixes and remediations for you. No human intervention is needed. Instead of reviewing the AI recommendation and then pulling the trigger, have the system apply the recommendation automatically. 

Automated alerts proactively identify SLA violations

Policy-based governance rules could be as benign as sending an alert to the user if the data table exceeds a certain size threshold or as aggressive as automatically requesting a configuration change for a container with more memory. 

This is true AIOps. You don’t have to wait until after the fact to address an issue or perpetually scan the horizon for emerging problems. The system applies AI/ML to all the telemetry data extracted and correlated from everywhere in your modern data stack to not only tell you what went wrong, why, and what to do about it, but it predicts and prevents problems altogether without any human having to touch a thing.

The post Beyond Observability for the Modern Data Stack appeared first on Unravel.

One of the highlights of the DataOps Unleashed 2022 virtual conference was a roundtable panel discussion on building versus buying when it comes to your data stack. Build versus buy is a question for all layers of the enterprise infrastructure stack. But in the last five years — even in just the last year alone — it’s hard to think of a part of IT that has seen more dramatic change than that of the modern data stack. 

These transformations shape how today’s businesses engage and work with data. Moderated by Lightspeed Venture Partners’ Nnamdi Iregbulem, the panel’s three conversation partners — Andrei Lopatenko, VP of Engineering at Zillow; Gokul Prabagaren, Software Engineering Manager at Capital One; and Aaron Richter, Data Engineer at Squarespace — weighed in on the build versus buy question and walked us through their thoughts:

• What motivates companies to build instead of buy?
• How do particular technologies and/or goals affect their decision?

These issues and other considerations were discussed. A few of the highlights follow, but the entire session is available on demand here.

What are the key variables to consider when deciding whether to build or buy in the data stack? 

Gokul: I think the things which we probably consider most are what kind of customization a particular product offers or what we uniquely need. Then there are the cases in which we may need unique data schemas and formats to ingest the data. We must consider how much control we have of the product and also our processing and regulatory needs. We have to ask how we will be able to answer those kinds of questions if we are building in-house or choosing to adopt an outsourced product.

Aaron: Thinking from the organizational perspective, there are a few factors that come from just purchasing or choosing to invest in something. Money is always a factor. It’s going to depend on the organization and how much you’re willing to invest. 

Beyond that a key factor is the expertise of the organization or the team. If a company has only a handful of analysts doing the heavy-lifting data work, to go in and build an orchestration tool would take them away from their focus and their expertise of providing insights to the business. 

Andrei: Another important thing to consider is the quality of the solution. Not all the data products on the market have high quality from different points of view. So sometimes it makes sense to build something, to narrow the focus of the product. Compatibility with your operations environment is another crucial consideration when choosing build versus buy.

What’s the more compelling consideration: saving headcount or increasing productivity of the existing headcount? 

Aaron: In general, everybody’s oversubscribed, right? Everybody always has too much work to do. And we don’t have enough people to accomplish that work. From my perspective, the compelling part is, we’re going to make you more efficient, we’re going to give you fewer headaches, and you’ll have fewer things to manage. 

Gokul: I probably feel the same. It depends more on where we want to invest and if we’re ready to change where we’re investing: upfront costs or running costs. 

Andrei: And development costs: do we want to buy this, or invest in building? And again, consider the human equation. It’s not just the number of people in your headcount. Maybe you have a small number of engineers, but then you have to invest more of their time into data science or data engineering or analytics. Saving time is a significant factor when making these choices.

How does the decision matrix change when the cloud becomes part of the consideration set in terms of build versus buy? 

Gokul: I feel like it’s trending towards a place where it’s more managed. That may not be the same question as build or buy. But it skews more towards the manage option, because of that compatibility, where all these things are available within the same ecosystem. 

Aaron: I think about it in terms of a cloud data warehouse: some kind of processing tool, like dbt; and then some kind of orchestration tool, like Airflow or Prefect; and there’s probably one pillar on that side, where you would never think to build it yourself. And that’s the cloud data warehouse. So you’re now kind of always going to be paying for a cloud vendor, whether it’s Snowflake or BigQuery or something of that nature.

So you already have your foot in the door there, and you’re already buying, right? So then that opens the door now to buying more things, adding things on that integrate really easily. This approach helps the culture shift. If a culture is very build-oriented, this allows them to be more okay with buying things. 

Andrei: Theoretically you want to have your infrastructure independent on cloud, but it never happens, for multiple reasons. Firstly, cloud company tools make integration work much easier. Second, of course, once you have to think about multi-cloud, you must address privacy and security concerns. In principle, it’s possible to be independent, but you’ll often run into a lot of technical problems. There are multiple different factors when cloud becomes key in deciding what you will make and what tools to use.

The post Building vs. Buying Your Modern Data Stack: A Panel Discussion appeared first on Unravel.

Thank you for your interest in the DataOps Unleashed 2022 Survey Infographic.

You can download your copy here.

Looking for the 2023 results. Find the infographic here.

The post DataOps Unleashed Survey Infographic appeared first on Unravel.

At the recent DataOps Unleashed 2022 virtual conference, we surveyed several hundred leading DataOps professionals across multiple industries in North America and Europe on a range of issues, including:

• Current adoption of DataOps approaches
• Top challenges in operating/managing their data stack
• How long they estimate cloud migration will take
• Where they are prioritizing automation
• On which tasks they spend their time

Download our infographic here.

Highlights and takeaways

DataOps as a practice is hitting an inflection point

DataOps has gained momentum over the past 12 months. While a DataOps approach is still in the early innings for most companies—almost 4 out of 5 of respondents said they are “having active discussions” or “progressing”—this is an 80% jump from last year.

Visibility into data pipelines remains the top challenge

For the second consecutive year, “lack of visibility across the environment” was the #1 challenge. Interestingly, “lack of proactive alerts” leapfrogged “expensive runaway jobs/pipelines” as the second biggest challenge, followed by “lack of experts” and “no visibility into cost/usage.”

Projected cloud migration schedules are longer

When we asked respondents to estimate how long their cloud migration will take, only 28% said just one year—with the vast majority saying 2 years or more. This is a 150% jump from last year, reflecting the growing realization of the complexity in moving from on-prem to cloud.

Automation continues to be a key driver

Over 75% of survey respondents said that being able to “automatically test and verify before moving jobs/pipelines to production” was their top automation priority. Right behind was automating the troubleshooting of pipeline issues (65%), followed by a three-way tie (about 33% each) among automatically troubleshooting platform issues, troubleshooting jobs, and automatically reducing usage costs.

Teams spend more time building pipelines than maintaining/deploying them

Like last year, we saw that data teams spend more time building pipelines (43% in 2022, up from 39% in 2021) than maintaining/troubleshooting them (30% in 2022 vs. 34% in 2021) or deploying pipelines (holding steady at 27%).

Interested in more DataOps trends? 

Check out the summary recap of the DataOps Unleashed 2022 keynote session, Three Venture Capitalists Weigh In on the State of DataOps 2022, or watch the full roundtable discussion on demand.

The post Key Findings of the 2022 DataOps Unleashed Survey appeared first on Unravel.

Benchmarked Survey from 2022 DataOps Unleashed Event Highlights the Key Priorities and Pressing Challenges Facing the Burgeoning DataOps Market

Palo Alto, CA – April 14, 2022 – Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today released key findings from a survey administered to several hundred attendees at the most recent DataOps Unleashed event in order to gauge the priorities, challenges, and progress of leading data teams as they seek to modernize their big data management and analytics capabilities so they can fully realize their real-time data analytics ambitions. To view and download an infographic of the key findings from this survey, click here.

“Data is the lifeblood of the modern enterprise and those organizations who have dedicated the resources and budget to modernizing their data stacks are the ones who will be best positioned to drive innovations in the coming decade,” said Kunal Agarwal, co-founder and CEO of Unravel Data. “The results of this latest survey show just how complex the modern data stack has become and illustrates the many unanticipated challenges that come with efficiently managing and optimizing data pipelines across multiple public cloud providers and platforms. It also serves to validate that there is an obvious demand for a purpose-built solution that can help these teams gain the critical visibility they need to drive the most value from their data operations.”

Some of the key findings collected from the most recent survey, which benchmarks responses from the year prior, include:

• DataOps as a practice is hitting an inflection point: There was an almost 80% increase from the year prior of respondents who said they are in the active stage of adopting a formal DataOps approach to manage and optimize their data pipelines. This year, more than 41% of attendees reported they are actively employing DataOps methodologies, compared to just less than a quarter (24%) in 2021.
• Visibility into data pipelines remains the top challenge: For the second year in a row, when participants were asked what they viewed as the top challenge with operating their data stack, respondents cited the lack of visibility across their environment as their most significant obstacle. Whereas in the previous year respondents reported that “controlling runaway costs” was the second biggest, this year the “lack of proactive alerts” was noted as the second most challenging aspect.
• Complexity of cloud migrations is more time consuming than previously thought: Sixty percent of respondents from this year’s event estimated that their cloud migration project would take between 12-24 months, representing a 150% increase over the prior year’s projection. The challenge of forecasting the duration of these cloud migration initiatives reveals the vast amount of complexity and uncertainty that data teams face when attempting to map out these critical projects.
• Automation continues to be a key driver: When asked about the role of automation in managing their DataOps pipelines, three in four DataOps professionals in both years reported that the ability to “automatically test and verify before moving jobs/pipelines to production” was the most important automation priority when compared to other aspects such as automating troubleshooting of platform or pipeline issues.
• Data teams spend more time building than deploying/managing pipelines: For both years, data professionals reported that they spent the majority of their day building their data pipelines (39% in 2021 and 43% in 2022). In 2022, respondents reported spending slightly less time maintaining or troubleshooting their pipelines (30%) than the year prior (34%) while the time spent deploying data pipelines remained the same at 27% for both years.

Now in its second year, DataOps Unleashed is a one-day virtual conference underwritten by Unravel Data, that seeks to build a vibrant peer-based community for DataOps professionals to share best practices. Data professionals can productively collaborate with one another and deliver on the promise of what it means to be a data-led company. This past year’s event attracted more than 2,300 data professionals and featured presentations from data leaders at some of the most data-forward enterprise organizations in the world, including AWS, DBS, Google, Johnson & Johnson, and many others. Recordings of all 20 sessions are available to watch for free and on-demand on the DataOps Unleashed site at: www.dataopsunleashed.com.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post New Survey Reveals Poor Visibility and Lack of Prescriptive Insights as Top Challenges for DataOps in 2022 appeared first on Unravel.

At the DataOps Unleashed 2022 conference, Luis Carlos Cruz Huertas, Head of Technology Infrastructure & Automation at DBS Bank, discussed how the bank has developed a framework whereby they translate millions of telemetry data points into actionable recommendations and even self-healing capabilities.

The session, Beyond observability and how to get there, opens with Dr. Shivnath Babu, Unravel Co-Founder and CTO, setting the context for observability challenges in the modern data stack—how simple applications grow to become so complex—and walking through the successive stages of what’s “beyond observability.” How can we go from extracting and correlating events, logs, metrics, and traces from applications, datasets, infrastructure, and tenants to get to self-healing systems?

The evolution of “beyond observability” goes from correlation to causation.

Then Luis shows how DBS is doing just that—how they leverage what he calls “cognitive capabilities” to deliver faster root cause analysis insights and automated corrective actions.

Why DBS went beyond observability

“When you have an always-on environment where banking applications are fully needed, you come to a point where observability [by itself] doesn’t cut it. You come to a point where your NCI [negative customer impact] truly becomes a key valuable indicator on how your systems are relating. We’re no longer in the game of measuring systems to be able to monitor, we’re measuring systems with the intent to provide a better customer experience,” says Luis.

Given the complexity of the bank’s IT ecosystem (not just its data stack), DBS made a strategic decision to not focus on tools developed by third-party vendors but rather build an “overarching brain” that could collect and understand the metrics from the diversity of tools in place without forcing the organization to rip and replace for something new. The objective was to speed root cause analysis across the board, provide less “noise,” reduce manual effort (toil), and get to proactive, predictive alerting on emerging issues.

How DBS built its cognitive capability platform

“You have applications that are collecting different telemetry through different systems or different log collectors—node exporters, metric beats, file beats, you can have an ELK stack. But ultimately what you want to do is create an open platform that you can ingest all this data,” Luis says. And for that you need three elements, he explains:

• a historical repository, where you can collect and cross-check data
• a real-time series database, because time becomes the de facto metadata to identify a critical incident and its correlations
• a log aggregator

Luis notes that one of the things he gets asked constantly is, How do you define the ingestion? 

“We do it all based on metadata. We define the metadata before it actually gets ingested. And then we park that into the [system] data lake. On top of that we provide an [ML-driven] analytical engine. Ultimately, what our system does is basically provide a recommendation to our site reliability engineers. And it gives them a list of elements, saying I’ve identified this set of errors or incidents that have happened over the last month and are repetitive and continuous. And then our site reliability engineers need to marry that with our automation engine. You build the right scripting—the right automation—to properly fix and remediate the problem. So that every time an incident is identified, it maps Incident A to Automation B to get Outcome C.”

DBS turns diverse telemetry data into auto-corrective actions.

Luis adds that with Unravel, the telemetry data has already been correlated. He says, “Unravel is huge for us. I don’t need to worry about marrying the correlation. I can just consume it right away.”

Luis concludes: “So in the end, we’re not changing tools, we’re collecting the metrics from all of the tools. We are providing a higher overarching mapping of the data being collected across all the tools in our environment, mapping them through metadata, and leveraging that to provide the right ML.”

The bottom line? DBS Bank is able to go “beyond observability” and leverage machine learning to get closer to the ultimate goal of a self-healing system that doesn’t require any human intervention at all.

Check out the full presentation from DataOps Unleashed on demand here.

See the DBS schematics for its Cognitive Technology Services Platform, tools mapping to the architectural components, solution data flows, overview of data sources, and more.

The post DBS Bank Goes “Beyond Observability” for DataOps appeared first on Unravel.

The keynote presentation at DataOps Unleashed 2022 featured a roundtable panel discussion on the State of DataOps and the Modern Data Stack. Moderated by Unravel Data Co-Founder and CEO Kunal Agarwal, this session features insights from three investors who have a unique vantage point on what’s changing and emerging in the modern data world, the effects of these changes, and the opportunities being created.

The panel’s three venture capitalists—Matt Turck, Managing Director at FirstMark, Venky Ganesan, Partner at Menlo Ventures, and Glenn Solomon, Managing Partner at GGV Capital—all invest in companies that are both users and creators of the modern data stack. They are either crunching massive amounts of data and converting it into insights or are helping other companies do that at scale.

Today every company is a data company. And these data pipelines, AI, and machine learning models are creating tremendous strategic value for companies. And these companies are depending on them more than ever before. Now, while the advantages of becoming data-driven are clear, what’s often hard to grasp is what’s changing in this landscape.

 The discussion covered a broad range of topics, loosely revolving around a couple of key areas. Here are just a handful of the interesting observations and insights they shared.

What are the top data industry macro-trends?

Glenn Solomon: I don’t think companies are nearly as advanced as you’re likely to believe. Most companies are in the early innings of trying to figure out how to manage data. We see that even with born-digital companies. And the complexity is compounded by the fact that there is a lot of noise in the startup universe. Figuring out the decisions you make as a company is difficult and challenging. So I think this best-of-breed vs. platform, balancing act that companies had to go through in software is also going to happen in the data stack.

Matt Turck: The big driver is the rise of modern cloud data warehouses, and the lake houses as well. So for me, that’s been the big unlock in the space. We finally have that base level in the data hierarchy of needs, where we can take all this data, put it somewhere, and actually process it at scale. Now this whole thing is becoming very real, no longer experimental. Now the whole thing needs to work.

Venky Ganesan: The digital transformation that was happening just got super accelerated by the pandemic. All these analog business processes were digitized. And now that they are digitized, they can be tracked, stored, analyzed, evaluated and acted upon. I think the data stack has got to be one of the most important stacks in a company because your success long term is going to be based on how good is your data stack? How good is your DataOps? And then how do you build the analytics on top of it?

What trends are you seeing within DataOps specifically?

Venky: I would say the biggest trend I’m seeing is pushing these data workloads to the cloud. And I think it’s a really interesting game-changer. One of the things we are seeing now as we move to the cloud is suddenly you can separate out the storage and compute, have the infrastructure handle it, and then have the data warehouses such as Snowflake, Databricks. Now there are new sets of problems that come into play around DataOps when you move the data to Snowflake or Databricks or any other cloud providers, which is that you need to still understand the workloads, still need to optimize them. I think there’s going to be a whole DataOps category that helps you both migrate workloads to the cloud and also monitor them, because you can’t have the fox guarding the henhouse, you need some third party there to help you make sure you’re optimizing the workload, because the cloud provider is not interested in optimizing the workload for efficiency.

Glenn: A driver that I’m seeing accelerate, and gain momentum, is quite simply just the need for speed. In organizations there’s a tremendous amount of momentum around real-time streaming, real time analytics. Companies are growing the number of business processes for which they want real-time data to make decisions. And that is having a big impact on this whole world.

Another trend I’d point out is the rise of open source and the impact open source is having on many, many areas within the DataOps world. It looks like Kafka has had a massive impact on streaming as a result. That shows me that open source can really standardize markets. It can standardize technologies and standardize workflows. We’ll have to see how this all plays out, but I think open source—when it works well—is a very, very powerful trend.

Moving data to the cloud—why or why not?

Venky: I think if you are a company that has data on premises, you just have numerous issues. On prem is very heterogeneous: heterogeneous in hardware, heterogeneous in environment. And in a world of labor shortages, what happens if you don’t have the people? If you have the kind of turnover you’re seeing, can you get the people to manage it? So if you don’t move to the cloud, you’re going to be trapped on an island with fewer and fewer resources that cost more and more.

But I actually think the most important part of moving into the cloud is that it gives you an opportunity to standardize data, think about the data you want. And then once you move it to the cloud, you can unlock new generations of AI technologies that come into the cloud and allow you to get more insights from data. And so to me, eventually, data is worthless if it doesn’t translate to insights. Your best way of getting that insight is to figure out a scalable way of moving to the cloud, cheaper, and also unlock a lot of the new AI techniques to get insight from it.

Glenn: I think we’re on a continuum where there are still lots of companies who are reticent to move all their data to the cloud. I think the view is, hey, we have regulatory obligations and there’s risk if we don’t manage things ourselves. For data that is viewed as too sensitive, too risky, too valuable to move to the cloud—it’s just a matter of time. The value that can be driven from having data up in the cloud is just too great. But how do you safely move data into the cloud? And then once it’s there, how do you manage the applications that consume that data in a way that is rational? And if you want to use [cloud services] rationally, and use them the right way, and in a cost-efficient way, then you really do need other tools to make sure that things don’t run away from you. 

Matt: For many years, there was an almost cognitive dissonance, where everything I read, and all the conversations I was having with like execs and people in the industry, was all cloud, cloud, cloud. But our customers all wanted to be on prem—actually, zero people want to be in the cloud. It feels like in the last year and a half that cognitive dissonance has disappeared, and suddenly I starting seeing all these customers, almost all at once—and the pandemic certainly accelerated all this—saying, okay, now is the time I will move those workloads and the data to the cloud. So it feels like there’s been an inflection point of some sort. And it is very anecdotal, I realize, but it is very, very clear.

The only nuance to all of this is I think there’s a little bit of a growing realization and concern around the cost of being in the cloud. When you start in the cloud, you actually save a lot of money. Once you’ve configured your organization to actually run in the cloud, you save a lot of money for a while. But then there’s a moment when it starts actually being pretty expensive. And I think that’s a problem that’s starting to come to the fore. 

What’s the impact of the talent shortage?

Glenn: It’s very difficult to amass the kind of talent you need to really effectively both manage the data and then ultimately evaluate and analyze it for good purpose in your business. If you split the world into managing data—DataOps and the data engineer and all the challenges and complexities there, where there’s definitely a labor shortage—and then analyzing the data, getting value from it (data scientists up through business analysts), where there’s also a shortage—we have a human problem in both. One of my colleagues used the term “unbundling the data engineer.” If you look at all the tasks a data engineer would need to do to get a well-functioning data stack in place, there just aren’t enough of those people. Companies are picking off and automating various aspects of that workflow.

But on the other side, on analyzing the data, I think there are a lot of interesting things to be done there. How do you make data scientists, because there aren’t enough of them, more efficient? What tools and technologies do they need? I think we’ll see more solutions on the analysis side because we have that same human capital problem there too.

Matt: It’s an obvious problem that is only going to get worse, because the rate that we as a society produce technical people—data engineers, ML engineers data scientists—is nowhere near the pace we need to meet the demand. Again, every company is becoming not just a software company but a data company. That has two consequences. One, we need products and platforms that abstract away the complexity. That’s empowering people who are somewhat technical but not engineers to do more and more of what’s needed to make the whole machinery work. And the second, related consequence is the rise of automation—making a lot of those technologies just work in a way where no human is required. There’s plenty of opportunity there, especially AI-driven information of system optimization, tuning, anomaly detection, auto-repair, and the like.

Venky: Whether we’re talking about DataOps or security, these are things that will get automated at scale. It won’t replace humans. They will be complemented by technology that does most of the mundane stuff, and humans will deal with the exceptions. The mundane stuff gets done automatically, the exceptions get kicked to humans.That’s the only way forward. There’s no way to build the human capital required.

Watch the entire panel discussion on demand here. Hear more war stories, anecdotes, and expert insights, including predictions for they coming year.

The post Three Venture Capitalists Weigh In on the State of DataOps 2022 appeared first on Unravel.

DataOps Unleashed is back! Founded and hosted by Unravel Data, DataOps Unleashed is a free, full-day virtual event, that took place on Wednesday, February 2, that brings together DataOps, CloudOps, AIOps, MLOps, and other professionals to connect and collaborate on trends and best practices for running, managing, and monitoring data pipelines and data-intensive workloads. Once again, thousands of your peers signed up for what turned out to be the most high-impact big data summit of the year.

A combination of industry thought leadership and practical guidance, sessions included talks by DataOps professionals at such leading organizations as Slack, Zillow, Cisco, and IBM (and dozens of others, detailing how they’re establishing data predictability, increasing reliability, and reducing costs. 

Here’s just a taste of what DataOps Unleashed 2022 covered.

• In the keynote address, the state of DataOps and the modern data stack, Unravel CEO Kunal Agarwal moderates a roundtable panel discussion with three prominent venture capitalists (Matt Turck from FirstMark, Glenn Solomon from GGV Capital, Venky Ganesan from Menlo Ventures) to cut through the hype and talk about what they’re seeing in reality within the DataOps world. They talk about macro-trends, how DataOps is emerging, the pros and cons of moving data workloads to the cloud, how the industry id=s dealing with the talent shortage, and more.
• Ryan Kinney, Senior Data Engineer at Slack, shares how his team has streamlined its DataOps stack with Slack to not only collaborate, but to observe their data pipelines and orchestration, enforce data quality and governance, manage their CloudOps, and unlock the entire data science and analytics platform for their customers and stakeholders.
• See how Torsten Steinbach, Cloud Data Architect at IBM, has incorporated different open tech into a state-of-the-art cloud-native data lakehouse platform. Torsten shares practical tips on table formats for consistency, metastores and catalogs for usability, encryption for data protection, data skipping indexes for performance, and more.
• Michael DePrizio, Principal Architect at Akamai, and Jordan Tigani, Chief Product Officer at SingleStore, discuss How Akamai handled 13x data growth and moved from batch to near-real-time visibility and analytics.
• Shivnath Babu, Co-Founder, Chief Technology Officer, and Head of Customer Success at Unravel, discusses what’s beyond observability and how to get there. Luis Carlos Cruz Huertas, Head of Technology Infrastructure and Automation at DBS Bank, then walks through how DBS has gone “beyond observability” to build a framework for automated root cause analysis and auto-corrections.

See the entire lineup and agenda here.

Check out the recorded DataOps Unleashed 2022 sessions today!

The post DataOps Unleashed Returns, Even Bigger and Better appeared first on Unravel.

Unravel Data CEO to Keynote Second “DataOps Unleashed” Virtual Event on February 2, 2022

Peer-to-Peer Event Dedicated to Helping Data Professionals Untangle the Complexity of Modern Data Clouds and Simplify their Data Operations

WHAT: Unravel Data announced that its CEO and Co-founder, Kunal Agarwal will deliver the keynote address at the “DataOps Unleashed” event, a free, full-day virtual summit will showcase some of the most prominent voices from across the burgeoning DataOps community that will be taking place on February 2nd, 2022.

The most successful enterprise organizations of tomorrow will be the ones who can effectively harness data from a broad array of sources and operating environments and rapidly transform it into actionable intelligence that supports data-driven decision making. DataOps Unleashed is a growing, cross-industry community where data professionals can productively collaborate with one another, share industry best practices, and deliver on the promise of what it means to be a data-led company.

The event will feature compelling presentations, conversation, and peer-sharing between technical practitioners, data scientists, and executive strategists from some of the most data-forward enterprise organizations in the world, including Slack, Zillow, Cisco, IBM, and many other recognized global brands. Over the course of the day, attendees will learn how a modernized approach to DataOps can transform their operations, improve data predictability, increase reliability, and create economic efficiencies with their data pipelines. Leading vendors from the DataOps market including Census, Metaplane, Airbyte, and Manta will be joining Unravel Data in supporting this community event.

Speakers at the DataOps Unleashed event to include:

• Kunal Agarwal, Co-founder & CEO & Shivnath Babu, Co-founder & CTO, Unravel Data
• Ryan Kinney, Senior Data Engineer, Slack
• Andrei Lopatenko, VP of Engineering, Zillow
• Disha Ahuja, Senior Manager, Cisco
• Torsten Steinbach, Lead Architect, IBM

More information about the event can be found at: https://dataopsunleashed.com/

WHEN: Wednesday, February 2nd beginning at 9AM PST

COST: Free

WHERE: Register at: https://dataopsunleashed.com/

WHO: Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Adobe and Deutsche Bank. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

The post DataOps Unleashed 2022 Keynote appeared first on Unravel.

Thank you for your interest in the Unravel Google Cloud Solution Brief.

You can download it here.

The post Unravel Google Cloud GCP appeared first on Unravel.

Thanks for your interest in the Spark Troubleshooting Guides.

You can download it here.

This 3 part series is your one-stop guide to all things Spark troubleshooting. In Part 1, we describe the ten biggest challenges for troubleshooting Spark jobs across levels. In Part 2, we describe the major categories of tools and types of solutions to solve challenges.

Lastly, Part 3 of the guide builds on the other two to show you how to address the problems we described, and more, with a single tool that does the best of what single-purpose tools do, and more – our DataOps platform, Unravel Data.

The post Spark Troubleshooting Guides appeared first on Unravel.

Thank you for your interest in the 451 Research Report, Data Science and analytics in the cloud set to grow three times faster than on-premises.

You can download it here.

451 Research: Data Science and analytics in the cloud set to grow three times faster than on-premises
Published: September, 28 2021

Introduction
Data science and analytics, as well as the data abstraction and acceleration offerings that underpin them, represented a $29bn market in 2020, according to 451 Research’s Data, AI & Analytics Market Monitor: Data Science & Analytics. Moreover, this market is growing thanks to the critical role of data, AI and analytics in speeding up enterprise data-driven decision-making for faster time to insight. Cloud services, in particular, are exhibiting strong growth – a trend that has been underway for some time and has been accelerated by the COVID-19 pandemic.

Get the 451 Take. Download Report.

The post Data Science and Analytics in the Cloud appeared first on Unravel.

Apache Spark is the leading technology for big data processing, on-premises and in the cloud. Spark powers advanced analytics, AI, machine learning, and more. Spark provides a unified infrastructure for all kinds of professionals to work together to achieve outstanding results.

Technologies such as Cloudera’s offerings, Amazon EMR, and Databricks are largely used to run Spark jobs. However, as Spark’s importance grows, so does the importance of Spark reliability – and troubleshooting Spark problems is hard. Information you need for troubleshooting is scattered across multiple, voluminous log files. The right log files can be hard to find, and even harder to understand. There are other tools, each providing part of the picture, leaving it to you to try to assemble the jigsaw puzzle yourself.

Would your organization benefit from rapid troubleshooting for your Spark workloads? If you’re running significant workloads on Spark, then you may be looking for ways to find and fix problems faster and better – and to find new approaches that steadily reduce your problems over time.

This blog post is adapted from the recent webinar, Troubleshooting Apache Spark, part of the Unravel Data “Optimize” Webinar Series. In the webinar, Unravel Data’s Global Director of Solutions Engineering, Chis Santiago, runs through common challenges faced when troubleshooting Spark and shows how Unravel Data can help.

Spark: The Good, the Bad & the Ugly

Chris has been at Unravel for almost four years. Throughout that time, when it comes to Spark, he has seen it all – the good, the bad, and the ugly. On one hand, Spark as a community has been growing exponentially. Millions of users are adopting Spark, with no end in sight. Cloud platforms such as Amazon EMR and Databricks are largely used to run Spark jobs.

Spark is here to stay, and use cases are rising. There are countless product innovations powered by Spark, such as Netflix recommendations, targeted ads on Facebook and Instagram, or the “Trending” feature on Twitter. On top of its great power, the barrier to entry for Spark is now lower than ever before. But unfortunately, with the good comes the bad, and the number one common issue is troubleshooting.

Troubleshooting Spark is complex for a multitude of different reasons. First off, there are multiple points of failure. A typical Spark data pipeline could be using orchestration tools, such as Airflow or Oozie, as well as built-in tools, such as Spark UI. You also may be using cluster management technologies, such as Cluster Manager or Ambari.

A failure may not always start on Spark; it could rather be a failure within a network layer on the cloud, for example.

Because Spark uses so many tools, not only does that introduce multiple points of failure, but there is also a lot of correlating information from various sources that you must carry out across these platforms. This requires expertise. You need experience in order to understand not only the basics of Spark, but all the other platforms that can support Spark as well.

Lastly, when running Spark workloads, the priority is often to meet SLAs at all costs. To meet SLAs you may, for example, double your resources, but there will always be a downstream effect. Determining what’s an appropriate action to take in order to make SLAs can be tricky.

The Three Levels of Spark Troubleshooting

There are multiple levels when it comes to troubleshooting Spark. First there is the Job Level, which deals with the inner workings of Spark itself, from executors to drivers to memory allocation to logs. The job level is about determining best practices for using the tools that we have today to make sure that Spark jobs are performing properly. Next is the Pipeline Level. Troubleshooting at the pipeline level is about managing multiple Spark runs and stages to make sure you’re getting in front of issues and using different tools to your advantage. Lasty, there is the Cluster Level, which deals with infrastructure. Troubleshooting at the cluster level is about understanding the platform in order to get an end-to-end view of troubleshooting Spark jobs.

Troubleshooting: Job Level

A Spark job refers to actions such as doing work in a workbook or analyzing a Spark SQL query. One tool used on the Spark job level is Spark UI. Spark UI can be described as an interface for understanding Spark at the job level.

Spark UI is useful in giving granular details about the breakdown of tasks, breakdown of stages, the amount of time it takes workers to complete tasks, etc. Spark UI is a powerful dataset that you can use to understand every detail about what happened to a particular Spark job.

Challenges people often face are manual correlation; understanding the overall architecture of Spark; and, more importantly, things such as understanding what logs you need to look into and how one job correlates with other jobs. While Spark UI is the default starting point to determine what is going on with Spark jobs, there is still a lot of interpretation that needs to be done, which takes experience.

Further expanding on Spark job-level challenges, one thing people often face difficulty with is diving into the logs. If you truly want to understand what caused a job to fail, you must get down to the log details. However, looking at logs is the most verbose way of troubleshooting, because every component in Spark produces logs. Therefore, you have to look at a lot of errors and timestamps across multiple servers. Looking at logs gives you the most information to help understand why a job has failed, but sifting through all that information is time-consuming.

It also may be challenging to determine where to start on the job level. Spark was born out of the Hadoop ecosystem, so it has a lot of big data concepts. If you’re not familiar with big data concepts, determining a starting point may be difficult. Understanding the concepts behind Spark takes time, effort, and experience.

Lastly, when it comes to Spark jobs, there are often multiple job runs that make up a data pipeline. Figuring out how one Spark job affects another is tricky and must be done manually. In his experience working with the best engineers at large organizations, Chris finds that even they sometimes decide not to finish troubleshooting, and instead just keep on running and re-running a job until it’s completed or meets the SLA. While troubleshooting is ideal, it is extremely time-consuming. Therefore, having a better tool for troubleshooting on the job level would be helpful.

Troubleshooting: Pipeline Level

In Chris’ experience, most organizations don’t have just one Spark job that does everything, but there are rather multiple stages and jobs that are needed to carry out Spark workloads. To manage all these steps and jobs you’d need an orchestration tool. One popular orchestration tool is Airflow, which allows you to sequence out specific jobs.

Orchestration tools like Airflow are useful in managing pipelines. But while these tools are helpful for creating complex pipelines and mapping where points of failure are, they are lacking when it comes to providing detailed information about why a specific step may have failed. Orchestration tools are more focused on the higher, orchestration level, rather than the Spark job level. Orchestration tools tell you where and when something has failed, so they are useful as a starting point to troubleshoot data pipeline jobs on Spark.

Those who are running Spark on Hadoop, however, often use Oozie. Similarly to Airflow, Oozie gives you a high level view, alerting you when a job has failed, but not providing the type of information needed to answer questions such as “Where is the bottleneck?” or “Why did the job break down?”. To answer these questions, it’s up to the user to manually correlate the information that orchestration tools provide with information from job-level tools, which again requires expertise. For example, you may have to determine which Spark run that you see in Spark UI correlates to a certain step in Airflow. This can be very time consuming and prone to errors.

Troubleshooting: Cluster Level

The cluster level for Spark is the level that refers to things such as infrastructure, VMs, allocated resources, and Hadoop. Hadoop’s ResourceManager is a great tool for managing applications as they come in. ResourceManage is also useful for determining what the resource usage is, and where a Spark job is in the queue.

However, one shortcoming of ResourceManager is that you don’t get historical data. You cannot view the past state of ResourceManager from twelve or twenty-four hours ago, for example. Everytime you open ResourceManager you have a view of how jobs are consuming resources at that specific time, as shown below.

Another challenge when troubleshooting Spark on the cluster level is that while tools such as Cluster Manager or Ambari give a holistic view of what’s going on with the entire estate, you cannot see how cluster-level information, such as CPU consumption, I/O consumption, or network I/O consumption, relate to Spark jobs.

Lastly, and similarly to the challenges faced when troubleshooting on the job and pipeline level, manual correlation is also a problem when it comes to troubleshooting on the cluster level. Manual correlation takes time and effort that a data science team could instead be putting towards product innovations.

But what if there was a tool that takes all these troubleshooting challenges, on the job, pipeline, and cluster level, into consideration? Well, luckily, Unravel Data does just that. Chris next gives examples of how Unravel can be used to mitigate Spark troubleshooting issues, which we’ll go over in the remainder of this blog post.

Demo: Unravel for Spark Troubleshooting

The beauty of Unravel is that it provides a single pane of glass where you can look at logs, the type of information provided by Spark UI, Oozie, and all the other tools mentioned throughout this blog, and data uniquely available through Unravel, all in one view. At this point in the webinar, Chris takes us through a demo to show how Unravel aids in troubleshooting at all Spark levels – job, pipeline, and cluster. For a deeper dive into the demo, view the webinar.

Job Level

At the job level, one area where Unravel can be leveraged is in determining why a job failed. The image below is a Spark run that is monitored by Unravel.

On the left hand side of the dashboard, you can see that Job 3 has failed, indicated by the orange bar. With Unravel, you can click on the failed job and see what errors occurred. On the right side of the dashboard, within the Errors tab, you can see why Job 3 failed, as highlighted in blue. Unravel is showing the Spark logs that give information on the failure.

Pipeline Level

Using Unravel for troubleshooting at the data pipeline level, you can look at a specific workflow, rather than looking at the specifics of one job. The image shows the dashboard when looking at data pipelines.

On the left, the blue lines represent instance runs. The black dot represents a job that ran for two minutes and eleven seconds. You could use information on run duration to determine if you meet your SLA. If your SLA is under two minutes, for example, the highlighted run would miss the SLA. With Unravel you can also look at changes in I/O, as well as dive deeper into specific jobs to determine why they lasted a certain amount of time. The information in the screenshot gives us insight into why the job mentioned prior ran for two minutes and eleven seconds.

The Unravel Analysis tab, shown above, carries out analysis to detect issues and make recommendations on how to mitigate those issues.

Cluster Level

Below is the view of Unravel when troubleshooting at the cluster level, specifically focusing on the same job mentioned previously. The job, which again lasted two minutes and eleven seconds, took place on July 5th at around 7PM. So what happened?

The image above shows information about the data security queue at the time when the job we’re interested in was running. The table at the bottom of the dashboard shows the state of the jobs that were running on July 5th at 7PM, allowing you to see which, if any, job was taking up too much resources. In this case, Chris’ job, highlighted in yellow, wasn’t using a large amount of resources. From there, Chris can then conclude that perhaps the issue is instead on the application side and something needs to be fixed within the code. The best way to determine what needs to be fixed is to use the Analysis tab mentioned previously.

Conclusion

There are many ways to troubleshoot Spark, whether it be on the job, pipeline, or cluster level. Unravel can be your one-stop shop to determine what is going on with your Spark jobs and data pipelines, as well as give you proactive intelligence that allows you to quickly troubleshoot your jobs. Unravel can help you meet your SLAs in a resourceful and efficient manner.

We hope you have enjoyed, and learned from, reading this blog post. If you think Unravel Data can help you troubleshoot Spark and would like to know more, you can create a free account or contact Unravel.

The post Troubleshooting Apache Spark – Job, Pipeline, & Cluster Level appeared first on Unravel.

Note: This guide applies to running Spark jobs on any platform, including Cloudera platforms; cloud vendor-specific platforms – Amazon EMR, Microsoft HDInsight, Microsoft Synapse, Google DataProc; Databricks, which is on all three major public cloud providers; and Apache Spark on Kubernetes, which runs on nearly all platforms, including on-premises and cloud.

Introduction

Spark is known for being extremely difficult to debug. But this is not all Spark’s fault. Problems in running a Spark job can be the result of problems with the infrastructure Spark is running on, an inappropriate configuration of Spark, Spark issues, the currently running Spark job, other Spark jobs running at the same time – or interactions among these layers. But Spark jobs are very important to the success of the business; when a job crashes, or runs slowly, or contributes to a big increase in the bill from your cloud provider, you have no choice but to fix the problem.

Widely used tools generally focus on part of the environment – the Spark job, infrastructure, the network layer, etc. These tools don’t present a holistic view. But that’s just what you need to truly solve problems. (You also need the holistic view when you’re creating the Spark job, and as a check before you start running it, to help you avoid having problems in the first place. But that’s another story.)

In this guide, Part 2 in a series, we’ll show ten major tools that people use for Spark troubleshooting. We’ll show what they do well, and where they fall short. In Part 3, the final piece, we’ll introduce Unravel Data, which makes solving many of these problems easier.

What’s the Problem(s)?

The problems we mentioned in Part 1 of this series have many potential solutions. The methods people usually use to try to solve them often derive from that person’s role on the data team. The person who gets called when a Spark job crashes, such as the job’s developer, is likely to look at the Spark job. The person who is responsible for making sure the cluster is healthy will look at that level. And so on.

In this guide, we highlight five types of solutions that people use – often in various combinations – to solve problems with Spark jobs

• Spark UI
• Spark logs
• Platform-level tools such as Cloudera Manager, the Amazon EMR UI, Cloudwatch, the Databricks UI, and Ganglia
• APM tools
• DataOps platforms such as Unravel Data

As an example of solving problems of this type, let’s look at the problem of an application that’s running too slowly – a very common Spark problem, that may be caused by one or more of the issues listed in the chart. Here. we’ll look at how existing tools might be used to try to solve it.

Note: Many of the observations and images in this guide have been drawn from the  presentation Beyond Observability: Accelerate Performance on Databricks, by Patrick Mawyer, Systems Engineer at Unravel Data. We recommend this webinar to anyone interested in Spark troubleshooting and Spark performance management, whether on Databricks or on other platforms.

Solving Problems Using Spark UI

Spark UI is the first tool most data team members use when there’s a problem with a Spark job. It shows a snapshot of currently running jobs, the stages jobs are in, storage usage, and more. It does a good job, but is seen as having some faults. It can be hard to use, with a low signal-to-noise ratio and a long learning curve. It doesn’t tell you things like which jobs are taking up more or less of a cluster’s resources, nor deliver critical observations such as CPU, memory, and I/O usage.

In the case of a slow Spark application, Spark UI will show you what the current status of that job is. You can also use Spark UI for past jobs, if the logs for those jobs still exist, and if they were configured to log events properly. Also, the Spark history server tends to crash. When this is all working, it can help you find out how long an application took to run in the past – you need to do this kind of investigative work just to determine what “slow” is.

The following screenshot is for a Spark 1.4.1 job with a two-node cluster. It shows a Spark Streaming job that steadily uses more memory over time, which might cause the job to slow down. And the job eventually – over a matter of days – runs out of memory.

(Source: Stack Overflow)

To solve this problem, you might do several things. Here’s a brief list of possible solutions, and the problems they might cause elsewhere:

• Increase available memory for each worker. You can increase the value of the spark.executor.memory variable to increase the memory for each worker. This will not necessarily speed the job up, but will defer the eventual crash. However, you are either taking memory away from other jobs in your cluster or, if you’re in the cloud, potentially running up the cost of the job.
• Increase the storage fraction. You can change the value of spark.storage.memoryFraction, which varies from 0 to 1, to a higher fraction. Since the Java virtual machine (JVM) uses memory for caching RDDs and for shuffle memory, you are increasing caching memory at the expense of shuffle memory. This will cause a different failure if, at some point, the job needs shuffle memory that you allocated away at this step.
• Increase the parallelism of the job. For a Spark Cassandra Connector job, for example, you can change spark.cassandra.input.split.size to a smaller value. (It’s a different variable for other RDD types.) Increasing parallelism decreases the data set size for each worker, requiring less memory per worker. But more workers means more resources used. In a fixed-resources environment, this takes resources away from other jobs; in a dynamic environment, such as a Databricks job cluster, it directly runs up your bill.

The point here is that everything you might do has a certain amount of guesswork to it, because you don’t have complete information. You have to use trial and error approaches to see what might work – both which approach to try/variable to change, and how much to change the variable(s) involved. And, whichever approach you choose, you are putting the job in line for other, different problems – including later failure, failure for other reasons, or increased cost. And, when your done, this specific job may be fine – but at the expense of other jobs that then fail. And those failures will also be hard to troubleshoot.

Here’s a look at the Stages section of Spark UI. It gives you a list of metrics across executors. However, there’s no overview or big picture view to help guide you in finding problems. And the tool doesn’t make recommendations to help you solve problems, or avoid them in the first place.

Spark UI is limited to Spark, but a Spark job may, for example, have data coming in from Kafka, and run alongside other technologies. Each of those has its own monitoring and management tools, or does without; Spark UI doesn’t work with those tools. It also lacks pro-active alerting, automatic actions, and AI-driven insights, all found in Unravel.

Spark UI is very useful for what it does, but its limitations – and the limitations of the other tool types described here – lead many organizations to build homegrown tools or toolsets, often built on Grafana. These solutions are resource-intensive, hard to extend, hard to support, and hard to keep up-to-date.

A few individuals and organizations even offer their homegrown tools as open source software for other organizations to use. However, support, documentation, and updates are limited to non-existent. Several such tools, such as Sparklint and DrElephant, do not support recent versions of Spark. At this writing, they have not had many, if any, fresh commits in recent months or even years.

Spark Logs

Spark logs are the underlying resource for troubleshooting Spark jobs. As mentioned above, Spark UI can even use Spark logs, if available, to rebuild a view of the Spark environment on an historical basis. You can use the logs related to the job’s driver and executors to retrospectively find out what happened to a given job, and even some information about what was going on with other jobs at the same time.

If you have a slow app, for instance, you can painstakingly assemble a picture to tell you if the slowness was in one task versus the other by scanning through multiple log files. But answering why and finding the root cause is hard. These logs don’t have complete information about resource usage, data partitioning, correct configuration setting and many other factors than can affect performance. There are also many potential issues that don’t show up in Spark logs, such as “noisy neighbor” or networking issues that sporadically reduce resource availability within your Spark environment.

Spark logs are a tremendous resource, and are always a go-to for solving problems with Spark jobs. However, if you depend on logs as a major component of your troubleshooting toolkit, several problems come up, including:

• Access and governance difficulties. In highly secure environments, it can take time to get permission to access logs, or you may need to ask someone with the proper permissions to access the file for you. In some highly regulated companies, such as financial institutions, it can take hours per log to get access.
• Multiple files. You may need to look at the logs for a driver and several executors, for instance, to solve job-level problems. And your brain is the comparison and integration engine that pulls the information together, makes sense of it, and develops insights into possible causes and solutions.
• Voluminous files. The file for one component of a job can be very large, and all the files for all the components of a job can be huge – especially for long-running jobs. Again, you are the one who has to find and retain each part of the information needed, develop a complete picture of the problem, and try different solutions until one works.
• Missing files. Governance rules and data storage considerations take files away, as files are moved to archival media or simply lost to deletion. More than one large organization deletes files every 90 days, which makes quarterly summaries very difficult. Comparisons to, say, the previous year’s holiday season or tax season become impossible.
• Only Spark-specific information. Spark logs are just that – logs from Spark. They don’t include much information about the infrastructure available, resource allocation, configuration specifics, etc. Yet this information is vital to solving a great many of the problems that hamper Spark jobs.

Because Spark logs don’t cover infrastructure and related information, it’s up to the operations person to find as much information they can on those other important areas, then try to integrate it all and determine the source of the problem. (Which may be the result of a complex interaction among different factors, with multiple changes needed to fix it.)

Platform-Level Solutions

There are platform-level solutions that work on a given Spark platform, such as Cloudera Manager, the Amazon EMR UI, and Databricks UI. In general, these interfaces allow you to work at the cluster level. They tell you information about resource usage and the status of various services.

If you have a slow app, for example, these tools will give you part of the picture you need to put together to determine the actual problem, or problems. But these tools do not have the detail-level information in the tools above, nor do they even have all the environmental information you need. So again, it’s up to you to decide how much time to spend researching, pulling all the information together, and trying to determine a solution. A quick fix might take a few hours; a comprehensive, long-term solution may take days of research and experimentation.

This screenshot shows Databricks UI. It gives you a solid overview of jobs and shows you status, cluster type, and so on. Like other platform-level solutions, it doesn’t help you much with historical runs, nor in working at the pipeline level, across the multiple jobs that make up the pipeline.

Another monitoring tool for Spark, which is included as open source within Databricks, is called Ganglia. It’s largely complementary to Databricks UI, though it also works at the cluster and, in particular, at the node level. You can see hostlevel metrics such as CPU consumption, memory consumption, disk usage, network-level IO – all host-level factors that can affect the stability and performance of your job.

This can allow you to see if your nodes are configured appropriately, to institute manual scaling or auto-scaling, or to change instance types. (Though someone trying to fix a specific job is not inclined to take on issues that affect other jobs, other users, resource availability, and cost.) Ganglia does not have job-specific insights, nor work with pipelines. And there are no good output options; you might be reduced to taking a screen snapshot to get a JPEG or PNG image of the current status.

Support from the open-source community is starting to shift toward more modern observability platforms like Prometheus, which works well with Kubernetes. And cloud providers offer their own solutions – AWS Cloudwatch, for example, and Azure Log Monitoring and Analytics. These tools are all oriented toward web applications; they lack modern data stack application and pipeline information which is essential to understand what’s happening to your job and how your job is affecting things on the cluster or workspace.

Platform-level solutions can be useful for solving the root causes of problems such as out-of-memory errors. However, they don’t go down to the job level, leaving that to resources such as Spark logs and tools such as Spark UI. Therefore, to solve a problem, you are often using platform-level solutions in combination with job-level tools – and again, it’s your brain that has to do the comparisons and data integration needed to get a complete picture and solve the problem.

Like job-level tools, these solutions are not comprehensive, nor integrated. They offer snapshots, but not history, and they don’t make proactive recommendations. And, to solve a problem on Databricks, for example, you may be using Spark logs, Spark UI, Databricks UI, and Ganglia, along with Cloudwatch on AWS, or Azure Log Monitoring and Analytics. None of these tools integrate with the others.

APM Tools

There is a wide range of monitoring tools, generally known as Application Performance Management (APM) tools. Many organizations have adopted one or more tools from this
category, though they can be expensive, and provide very limited metrics on Spark and other modern data technologies. Leading tools in this category include Datadog, Dynatrace, and Cisco AppDynamics.

For a slow app, for instance, an APM tool might tell you if the system as a whole is busy, slowing your app, or if there were networking issues, slowing down all the jobs. While helpful, they’re oriented toward monitoring and observability for Web applications and middleware, not data-intensive operations such as Spark jobs. They tend to lack information about pipelines, specific jobs, data usage, configuration setting, and much more, as they are not designed to deal with the complexity of modern data applications.

Correlation is the Issue

To sum up, there are several types of existing tools:

• DIY with Spark logs. Spark keeps a variety of types of logs, and you can parse them, in a do it yourself (DIY) fashion, to help solve problems. But this lacks critical infrastructure, container, and other metrics.
• Open source tools. Spark UI comes with Spark itself, and there are other Spark tools from the open source community. But these lack infrastructure, configuration and other information. They also do not help connect together a full pipeline view, as you need for Spark – and even more so if you are using technologies such as Kafka to bring data in.
• Platform-specific tools. The platforms that Spark runs on – notably Cloudera platforms, Amazon EMR, and Databricks – each have platform-specific tools that help with Spark troubleshooting. But these lack application-level information and are best used for troubleshooting platform services.
• Application performance monitoring (APM) tools. APM tools monitor the interactions of applications with their environment, and can help with troubleshooting and even with preventing problems. But the applications these APM tools are built for are technologies such as .NET, Java, and Ruby, not technologies that work with data-intensive applications such as Spark.
• DataOps platforms. DataOps – applying Agile principles to both writing and running Spark, and other big data jobs – is catching on, and new platforms are emerging to embody these principles.

Each tool in this plethora of tools takes in and processes different, but overlapping, data sets. No one tool provides full visibility, and even if you use one or more tools of each type, full visibility is still lacking.

You need expertise in each tool to get the most out of that tool. But the most important work takes place in the expert user’s head: spotting a clue in one tool, which sends you looking at specific log entries and firing up other tools, to come up with a hypothesis as to the problem. You then have to try out the potential solution, often through several iterations of trial and error, before arriving at a “good enough” answer to the problem.

Or, you might pursue two tried and trusted, but ineffective, “solutions”: ratcheting up resources and retrying the job until it works, either due to the increased resources or by luck; or simply giving up, which our customers tell us they often had to do before they started using Unravel Data.

The situation is much worse in the kind of hybrid data clouds that organizations use today. To troubleshoot on each platform, you need expertise in the toolset for that platform, and all the others. (Since jobs often have cross-platform interdependencies, and the same team has to support multiple platforms.) In addition, when you find a solution for a problem on one platform, you should apply what you’ve learned on all platforms, taking into account their differences. In addition, you have issues that are inherently multi-platform, such as moving jobs from one platform to a platform that is better, faster, or cheaper for a given job. Taking on all this with the current, fragmented, and incomplete toolsets available is a mind-numbing prospect.

The biggest need is for a platform that integrates the capabilities from several existing tools, performing a five-step process:

1. Ingest all of the data used by the tools above, plus additional, application-specific and pipeline data.
2. Integrate all of this data into an internal model of the current environment, including pipelines.
3. Provide live access to the model.
4. Continually store model data in an internally maintained history.
5. Correlate information across the ingested data sets, the current, “live” model, and the stored historical background, to derive insights and make recommendations to the user.

This tool must also provide the user with the ability to put “triggers” onto current processes that can trigger either alerts or automatic actions. In essence, the tool’s inbuilt intelligence and the user are then working together to make the right things happen at the right time.

A simple example of how such a platform can help is by keeping information per pipeline, not just per job – then spotting, and automatically letting you know, when the pipeline suddenly starts running slower than it had previously. The platform will also make recommendations as to how you can solve the problem. All this lets you take any needed action before the job is delayed.

The post Spark Troubleshooting Solutions – DataOps, Spark UI or logs, Platform or APM Tools appeared first on Unravel.

The post Troubleshooting EMR appeared first on Unravel.

Data+AI Battlescars Takeaways: Driving Data Governance and Data Products at ING Bank France

In this episode of Data+AI Battlescars, Sandeep Uttamchandani, Unravel Data’s CDO, speaks with Samir Boualla, CDO at ING Bank France, one of the largest banks in the world. They cover his battlescars in Driving Data Governance Across Business Teams and Building Data Products.

At ING Bank France, Samir is the Chief Data Officer. He’s responsible for several teams that govern, develop, and manage data infrastructure and data assets to deliver value to the business. With over 20+ years of experience on various data topics, Samir shares interesting battle-tested techniques in this podcast, including using a process catalog, having a “data minimum standard,” and having a change management mindset. Here are the key takeaways from their chat.

Data Governance

Preparing for Data Governance

• The first step to preparing for data governance is defining data governance frameworks, guidelines, and principles as part of your organization’s data strategy and discussing them with various stakeholders.
• Next, you need to get approval and validation from your directors. It is important to have support and commitment from senior leaders since it will be impacting the whole organization.
• After that, you can identify the appropriate people that should be assigned to roles such as data steward, data owner, or process lead.
• It is important to implement data governance in parallel with building a data architecture. You can identify and close gaps quickly so you don’t develop something which later on is not compliant with a regulation or with other frameworks.

Having a Data and a Process Catalog

• In order to monitor compliance proactively, you must know your data. This is where a data catalog plays an important role.
• A data catalog allows you to have uniform definitions in place, have a data owner for each of the specific data categories, and have people who have more in-depth knowledge and the capability to manage data acting as data stewards.
• Having a process catalog in addition to a data catalog allows you to seamlessly track the data life cycle.
• In a process catalog, it’s documented, for each process, what sources are used, who is doing what, and which data is being used in those processes.
• The process catalog is linked with your data catalog and used to manage your life cycle and apply additional data capabilities, like retention and deletion, on the appropriate systems or appropriate process steps.

Data Minimum Standards

• Data minimum standards are key frameworks that can give you insight and input on the controls and checks that should be part of your minimum standards for governance.
• Having these data minimum standards, such as GDPR and BCBS 239, allows you to proactively monitor for compliance and apply governance.
• These data minimum standards are part of your compliance framework, so you are able to apply them to all kinds of different processes or departments.
• ING Bank is constantly assessing if the controls work, testing them for effectiveness, and auditing them every once in a while.

Data Products

Building External-Facing Data Products

• In preparation for building external-facing data products, you need to have a set of standardized APIs as a product, which you can deliver to third parties for external consumers.
• At the same time, you should also be using another product as your data catalog to make sure that the data that is being defined and flowing through those APIs are made unique.
• The biggest challenges ING faces when building external-facing data products is making sure they are acting more or less on the edge of technology and architecture, while also ensuring that they are working towards their goal of becoming a data-driven bank.
• They encounter several challenges in making sure that their platforms are compatible and can exchange data in the right formats and in the right structure, but also in a way that the infrastructure remains scalable.

Challenges of Building Data Products

• Sometimes when a product is based on data quality, while Samir hopes to identify upfront any data quality issues, somewhere down the line the consumer may identify issues or have questions regarding the quality of the data. This is where another data product, called remediation, can come into play.
• Remediation is when a consumer can address data quality issues directly to the appropriate data stewards in the organization. Using other complementary products, a consumer can look into certain data or to a certain report to identify which data point came from where and who’s the data steward or data owner of that specific data. They can then address it automatically in a workflow environment, and request remediation.
• When building a data product, you may run into manual, legacy processes that have not yet been redesigned.

Change Management

• Having a change management mindset means that you are willing to implement something new or change something from the legacy based on new data products.
• A standard data model and data catalog are essential when it comes to change management.
• By having a single data model and a data catalog, you have a decoupling layer which helps you and supports you in the exercise of identifying what that data point reflecting the truth exactly is.
• ING has a broad framework, which allows them to work in a similar, agile way across the organization, and across the globe, whenever something needs to be adjusted.
• Their data products also allow them to minimize the amount of effort that needs to be put into a change to make it available.

While we’ve highlighted the key talking points here, Sandeep and Samir talked about so much more in the full podcast. Be sure to check out the full episode and the rest of the Data+AI Battlescars podcast series!

The post Driving Data Governance and Data Products at ING Bank France appeared first on Unravel.

The post Troubleshooting Databricks – Challenges, Fixes, and Solutions appeared first on Unravel.

In the second of this two-part episode of Data+AI Battlescars, Sandeep Uttamchandani, Unravel Data’s CDO, speaks with Kapil Surlaker, VP of Engineering and Head of Data at LinkedIn. In part one, they covered LinkedIn’s challenges related to metadata management and data access APIs. This second part dives deep into data quality.

Kapil has 20+ years of experience in data and infrastructure, at large companies such as Oracle and at multiple startups. At LinkedIn, Kapil has been leading the next generation of big data infrastructure, platforms, tools, and applications to empower data scientists, AI engineers, app developers, to extract value from data. Kapil’s team has been at the forefront of innovation driving multiple open source initiatives such as Pinot, Gobblin, and DataHub. Here are some key talking points from the second part of their insightful chat.

What Does “Data Quality” Mean?

• Data quality spans many dimensions and can be really broad in its coverage. It answers the questions: Is your data accurate? Do you have data integrity, validity, security, timeliness, and completeness? Is the data consistent? Is it interpretable?
• LinkedIn measures tons of different metrics, aiming to understand the health of the business, products, and systems.
• When KPIs differ from the norm, it becomes a question of data quality.
• When determining the root cause of poor data quality, it can differ for each dimension of quality.
• For example, when there is metric inconsistency, you must ask yourself if you have an accurate source of truth for your metrics.
• Timeliness and completeness problems often happen as a result of infrastructure issues. In complex ecosystems you have a lot of moving parts, so a lot can go wrong that impacts timeliness and completeness.
• Data quality problems often don’t actually manifest themselves as data quality problems in obvious ways. It takes time to monitor, detect, and effectively analyze the root cause of these issues, and remediate the issues.
• When assessing and improving data quality, it often helps to categorize it in three buckets. There is (1) the monitoring and observability aspect, (2) anomaly detection and root cause analysis for anomalies, and (3) preventing data quality issues in the first place. The last bucket is the best-case scenario.
• In any complex ecosystem, when something goes wrong for any reason in a pipeline or in a single stage of a data set or a data flow, it can have real consequences on the entire downstream change. So your goal is to detect problems as close to the source as possible.

How Does LinkedIn Maintain Data Quality?

Unified Metrics Platform

• It is important to have an evolving schema of your datasets and signals for when something goes awry, to act as markers of data quality.
• At LinkedIn they built a platform for metric definition and the entire life cycle management of those metrics, called the Unified Metrics Platform. The platform processes all of LinkedIn’s critical metrics – to the point that if it’s not produced by the platform, it wouldn’t be considered a trusted metric. The Unified Metrics Platform defines their source of truth.
• The company turned to machine learning techniques to improve the detection of anomalies and alerting based on user feedback.

Data Sentinel

• You can have situations where the overall metric that you’re monitoring may not have a significant deviation, but when you look into the sub spaces within that metric, you find significant deviations. To solve this problem, LinkedIn leveraged algorithms to automatically build structures based on the nature of the data itself and build a multi-dimensional data cube.
• When you’re unable to pinpoint the root cause of a deviation, it becomes a matter of identifying the space of the key drivers which might impact the particular metric. You narrow that space, present it to users for their feedback, and then continuously refine the system.
• To detect issues based on the known properties of the data, Linkedin built a system called the Data Sentinel. This system has the ability to take the developer’s knowledge about the dataset and specify it as declarative roots. The Data Sentinel then takes on the responsibility of generating the code to perform data validations.
• Linkedin is considering making Data Sentinel open source in the future.

Building a Data Quality Team

• At LinkedIn, they make sure that team members take the time to treat event schemas as code. They have to be reviewed and checked on. The same goes for metrics. This requires collaboration between different teams. They are constantly talking to each other and coming together to improve not just tools, but also processes.
• What is accepted as state-of-the-art today is almost guaranteed not to be state-of-the-art tomorrow. So when hiring for a data quality or data management team, it is important to look for people who are naturally curious and have a growth mindset.

If you’re interested in any of the topics discussed here, Sandeep and Kapil talked about even more in the full podcast. Be sure to check out Part 1 of this blog post, the full episode and the rest of the Data+AI Battlescars podcast series!

The post Simplifying Data Management at LinkedIn – What is Data Quality appeared first on Unravel.

In the first of this two-part episode of Data+AI Battlescars, Sandeep Uttamchandani, Unravel Data’s CDO, speaks with Kapil Surlaker, VP of Engineering and Head of Data at LinkedIn. In this first part, they cover LinkedIn’s challenges related to Metadata Management and Data Access APIs. Part 2 will dive deep into data quality.

Kapil has 20+ years of experience in data and infrastructure, at large companies such as Oracle and at multiple startups. At LinkedIn, Kapil has been leading the next generation of big data infrastructure, platforms, tools, and applications to empower data scientists, AI engineers, app developers, to extract value from data. Kapil’s team has been at the forefront of innovation driving multiple open source initiatives such as Pinot, Gobblin, and DataHub. Here are some key talking points from the first part of their insightful chat.

Metadata Management

The Problem: So Much Data

• LinkedIn manages over a billion data points and over 50 billion relationships.
• As the number of datasets skyrocketed at LinkedIn, the company found that internal users were spending an inordinate amount of time searching for and trying to understand hundreds of thousands, if not millions, of datasets.
• LinkedIn could no longer rely on manually generated information about the datasets. The company needed a central metadata repository and a metadata management strategy.

Solution 1: WhereHows

• The company initiated an in-depth analysis of its Hadoop data lake and asked questions such as: What are the datasets? Where do they come from? Who produced the dataset? Who owns it? What are the associated SLAs? What other sets does a particular dataset depend on?
• Similar questions were asked about the jobs: What are the inputs and outputs? Who owns the jobs?.

• The first step was the development of an open source system called WhereHows, a central metadata repository to capture metadata across the diverse datasets, with a search engine.

Solution 2: Pegasus

• LinkedIn discovered that it was not enough just to focus on the datasets and the jobs. The human element had to be accounted for as well. A broader view was necessary, accommodating both static and dynamic metadata.
• In order to expand the capabilities of the metadata model, the company realized it needed to take a “push” approach rather than a metadata “scraping” approach.
• The company then built a library called Pegasus to create a push-based model that improved both efficiency and latency.

The Final Solution: DataHub

• Kapil’s team then found that you need the ability to query metadata through APIs. You must be able to query the metadata online, so other services can integrate.
• The team went back to the drawing board and re-architected the system from the ground up based on these learnings.
• The result was DataHub, the latest version of the company’s open source metadata management system, released last year.
• A major benefit of this metadata approach is the ability to drive a lot of other functions in the organization that depend on access to metadata.

Data Access APIs

• LinkedIn recently completely rebuilt its mobile experience, user tracking, and associated data models, essentially needing to “change engines in mid-flight.”
• The company needed a data API to meet this challenge. One did not exist, however, so they created a data access API named “Dali API” to provide an access layer to offline data.
• The company used Hive to build the infrastructure and get to market quickly. But in hindsight, using Hive added some dependencies.
• So LinkedIn built a new, more flexible library called Coral and made it available as open source software. This removed the Hive dependency, and the company will benefit from improvements made to Coral by the community.

If you’re interested in any of the topics discussed here, Sandeep and Kapil talked about even more in the full podcast. Keep an eye out for takeaways from the second part of this chat, and be sure to check out Part 2 of this blog post, the full episode and the rest of the Data+AI Battlescars podcast series!

The post Simplifying Data Management at LinkedIn – Metadata Management and APIs appeared first on Unravel.

Data+AI Battlescars Takeaways: Recruiting and Building the Data Science Team at Etsy

In this episode of Data+AI Battlescars (formerly CDO Battlescars), Sandeep Uttamchandani talks to Chu-Cheng, CDO at Etsy. This episode focuses on Chu-Cheng’s battlescars related to recruiting and building a data science team.

Chu-Cheng leads the global data organization at Etsy. He’s responsible for data science, AI innovation, machine learning and data infrastructure. Prior to Etsy, Chu-Cheng has led various data roles, including at Amazon, Intuit, Rakuten and eBay. Here are the key talking points from their chat.

Building a Data Science Team: The Early Stages

• At the early stages of building a data science team, it may be more useful to hire people who are generalists rather than, for example, specialized data scientists or machine learning engineers.
• In any successful team, you need a mix of different experience levels and skill sets.
• When building a data science team, Chu-Cheng actually looks for people who probably wouldn’t pass a traditional data science interview, but can still get the job done.
• For first hires, Chu-Cheng generally targets people who have at least a few years of work experience and have a lot of patience and willingness to learn.
• A good candidate is someone who can explain a difficult concept in a way that anyone can understand. You want to find someone who knows how to tell people what they are thinking.
• For example, you can follow them on LinkedIn and see their profile, how they write their own self description, and you get an idea even before the interview.
• In the past, Chu-Cheng often unconsciously looked for someone with whom he had a similar background.
• To counteract this bias, he learned to create a checklist of criteria prior to interviewing a candidate. He uses that checklist to evaluate the candidate’s qualifications, rather than just picking someone who has a similar background to him.

Building a Data Science Team: The Later Stages

• Eventually when you have a bigger team, you must start moving from hiring generalists to hiring specialists. If you only hire generalists, you’ll eventually run into a wall, because you have a bunch of people who are fungible.
• In interviews, Chu-Cheng tries to balance technical and soft skills, even when hiring scientists and engineers.
• If you are interviewing someone for a manager position, prioritize their mentoring, coaching, conflict resolution, and communication skills. The manager’s success is defined by the team’s success.
• As a manager, when coaching someone, instead of trying to give out or prescribe an answer, focus on how you can ask the right questions so that the person can come up with a solution on their own. Switch from telling to asking. Allow people to make mistakes so you can coach them to grow and learn a lesson from it.

Innovation at Etsy

• Chu-Cheng tries to teach his team how to write papers and patents. At Etsy, they encourage this innovation by sending a recognition gift or an innovation award.
• Papers and patents are not the only types of innovation, however. Innovation also involves the process of making something easier. Not everything can be patented or become a paper, but you can, for instance, write a blog sharing your learning. Innovation is a mindset.
• When considering a new technology, it is important to get a sense of the circumstances under which you should not use the technology, as well as when to use it. You must know when to use what.

If you’re interested in any of the topics discussed here, Sandeep and Chu-Cheng talked about even more in the full podcast. Be sure to check out the full episode and the rest of the Data+AI Battlescars podcast series!

The post Recruiting and Building the Data Science Team at Etsy appeared first on Unravel.

The following blog post appeared in its original form on Towards Data Science. It’s part of a series on DataOps for effective AI/ML. The author is CDO and VP Engineering here at Unravel Data. (GIF by giphy)

Let’s start with a real-world example from one of my past machine learning (ML) projects: We were building a customer churn model. “We urgently need an additional feature related to sentiment analysis of the customer support calls.” Creating the data pipeline to extract this dataset took about 4 months! Preparing, building, and scaling the Spark MLlib code took about 1.5-2 months! Later we realized that “an additional feature related to the time spent by the customer in accomplishing certain tasks in our app would further improve the model accuracy” — another 5 months gone in the data pipeline! Effectively, it took 2+ years to get the ML model deployed!

After driving dozens of ML initiatives (as well as advising multiple startups on this topic), I have reached the following conclusion: Given the iterative nature of AI/ML projects, having an agile process of building fast and reliable data pipelines (referred to as DataOps) has been the key differentiator in the ML projects that succeeded. (Unless there was a very exhaustive feature store available, which is typically never the case).

Behind every successful AI/ML product is a fast and reliable data pipeline developed using well-defined DataOps processes!

To level-set, what is DataOps? From Wikipedia: “DataOps incorporates the agile methodology to shorten the cycle time of analytics development in alignment with business goals.”

I define DataOps as a combination of process and technology to iteratively deliver reliable data pipelines with agility. Depending on the maturity of your data platform, you might be one of the following DataOps phases:

• Ad-hoc: No clear processes for DataOps
• Developing: Clear processes defined, but accomplished manually by the data team
• Optimal: Clear processes with self-service automation for data scientists, analysts, and users.

Similar to software development, DataOps can be visualized as an infinity loop

The DataOps lifecycle – shown as an infinity loop above – represents the journey in transforming raw data to insights. Before discussing the key processes in each lifecycle stage, the following is a list of top-of-mind battle scars I have encountered in each of the stages:

• Plan: “We cannot start a new project — we do not have the resources and need additional budget first”
• Create: “The query joins the tables in the data samples. I didn’t realize the actual data had a billion rows! ”
• Orchestrate: “Pipeline completes but the output table is empty — the scheduler triggered the ETL before the input table was populated”
• Test & Fix: “Tested in dev using a toy dataset — processing failed in production with OOM (out of memory) errors”
• Continuous Integration; “Poorly written data pipeline got promoted to production — the team is now firefighting”
• Deploy: “Did not anticipate the scale and resource contention with other pipelines”
• Operate & Monitor: “Not sure why the pipeline is running slowly today”
• Optimize & Feedback: “I tuned the query one time — didn’t realize the need to do it continuously to account for data skew, scale, etc.”

To avoid these battle scars and more, it is critical to mature DataOps from ad hoc, to developing, to self-service.

This blog series will help you go from ad hoc to well-defined DataOps processes, as well as share ideas on how to make them self-service, so that data scientists and users are not bottlenecked by data engineers.

DataOps at scale with Unravel

Create a free account

For each stage of the DataOps lifecycle stage, follow the links for the key processes to define and the experiences in making them self-service (some of the links below are being populated, so please bookmark this blog post and come back over time):

Plan Stage

• How to streamline finding datasets
• Formulating the scope and success criteria of the AI/ML problem
• How to select the right data processing technologies (batch, interactive, streaming) based on business needs

Create Stage

• How to streamline accessing metadata properties of the datasets
• How to streamline the data preparation process
• How to make behavioral data self-service

Orchestrate Stage

• Managing Data Governance
• Re-using ML Model Features
• Scheduling data pipelines

Test & Fix Stage

• Streamlining sandbox environment for testing
• Identify and remove data pipeline bottlenecks
• Verify data pipeline results for correctness, quality, performance, and efficiency

Continuous Integration & Deploy Stage

• Smoke test for data pipeline code integration
• Scheduling window selection for data pipelines
• Changes rollback

Operate Stage

• Detect anomalies to proactively avoid SLA violation
• Managing data incidents in production
• Alerting on rogue (resource hogging) jobs

Monitor Stage

• Building end-to-end observability of data pipelines
• Tracking lineage of data flows data
• Enforcing data quality with circuit breakers

Optimize & Feedback Stage

• Continuously optimize existing data pipelines
• Alerting on budgets

In summary, DataOps is the key to delivering fast and reliable AI/ML! It is a team sport. This blog series aims to demystify the required processes as well as build a common understanding across Data Scientists, Engineers, Operations, etc.

DataOps as a team sport

To learn more, check out the recent DataOps Unleashed Conference, as well as innovations in DataOps Observability at Unravel Data. Come back to get notified when the links above are populated.

The post AI/ML without DataOps is just a pipe dream! appeared first on Unravel.

The exponential adoption of IT technologies over the past several decades has had a profound impact on organizations of all sizes. Whether it is a small, medium, or large enterprise, the need to create web applications while managing an extensive set of data effectively is high on every CIO’s priority list.

As a result, there has been an ongoing effort to implement better approaches to software development, data analysis, and data management.

The efforts are so pervasive across industries that these approaches have been given names of their own. The approach to better manage software development and delivery is known as DevOps. The end-to-end approach to efficiently and effectively deliver data products – from responses to SQL queries, to data pipelines, to machine learning models and AI-powered insights – is known as DataOps.

DataOps and DevOps are similar in that they both aim to solve the need to scale delivery. The key difference is that DataOps focuses on the flow of data, and the use of data in analytics, rather than on the software development and delivery lifecycle.

There’s also a difference in impact. Strong DataOps practices are vital to the successful development and delivery of AI-powered applications, including machine learning models. AI and ML are powerful areas of innovation, perhaps the most important in decades. DataOps as a discipline is necessary for the successful development and deployment of AI.

To help you gain an understanding of DataOps vs DevOps, it’s helpful to provide an overview of both, discuss their respective goals, and then highlight the key differences between the two.

DataOps Overview

DataOps is sometimes seen as simply making data flows through an organization, and data transformations, work correctly. This misconception that DataOps is just DevOps applied to data analytics is common. Rather, DataOps is actually a holistic approach to solving business problems.

According to Gartner,

DataOps is a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization. The goal of DataOps is to deliver value faster by creating predictable delivery and change management of data, data models and related artifacts. DataOps uses technology to automate the design, deployment and management of data delivery with appropriate levels of governance, and it uses metadata to improve the usability and value of data in a dynamic environment.

As a concept, DataOps was first introduced by InformationWeek’s Lenny Liebmann in 2014. The term appeared in a blog post on the IBM Big Data & Analytics Hub, titled “3 reasons why DataOps is essential for big data success”.

A few years later, in 2017, DataOps began to get its own ecosystem, dedicated analyst coverage, increased keyword searches, inclusion in surveys, mention in books, and use in open source projects.

DataOps is Not Just Analytics

DataOps is sometimes seen as a set of best practices around analytics. Analytics can be considered to include most of AI and ML, so improving analytics functionality is not a trivial goal. But DataOps is much more than just DevOps applied to analyze data.

Data analytics happens at the end of a data pipeline, while DataOps encompasses nearly a dozen steps, including data ingestion and the entire data pipeline before analytics happens. DataOps also includes the delivery of analytics results and their ultimate business impact. And it serves as a framework for the development and delivery of useful capabilities from AI and ML.

As complex data environments are constantly changing, it is critical for an organization to possess and maintain a solid understanding of their data assets, and to add to their data assets when needed for business success. Understanding the origin of the data, analyzing data dependencies, and keeping documentation up to date are each resource-intensive, yet critical.

Having a high-performing DataOps team in place can help an organization accelerate the DataOps lifecycle – developing powerful, data-centric apps to deliver accurate insights to both internal and external customers.

DevOps Overview

Now that we’ve briefly described DataOps, let’s discuss DevOps. According to Atlassian, the DevOps movement started to come together between 2007 and 2008. At the time, software development and IT operations communities started raising concerns about an increase in what they felt was a near-fatal level of dysfunction in the industry.

The primary dysfunction these two groups saw was that in a traditional software development model, those who wrote the code were functionally and organizationally separate and apart from those who deployed and supported the code.

As such, software developers and IT Operations teams had competing objectives, different leadership, and different KPIs by which each group was measured and evaluated. As a result, teams were siloed and only concerned with what had a direct impact on them.

The result: poorly executed software releases, unhappy customers, and often unhappy development and IT Operations teams.

Over time, DevOps evolved to solve the pain caused by these siloed teams and poor lines of communication between software developers and IT Operations.

What is DevOps

DevOps describes an approach to software development that accelerates the build lifecycle using automation. By focusing on continuous integration and delivery of software, and by automating the integration, testing, and deployment of code, enterprises start to see many benefits. Specifically, this approach of merging DEVelopment and OPerationS, reduces time to deployment, increases time to market, keeps defects or bugs at a minimum, and generally shortens the time required to resolve any defects.

6 Primary Goals of DevOps

There are six key principles or goals the DevOps aims to deliver.

DevOps Goal 1: Speed

In order to quickly adapt to customer needs, changes in the market, or new business goals, the application development process and release capabilities need to be extremely fast. Practices such as continuous delivery and continuous integration help make it possible to maintain that speed throughout the development and operations phases.

DevOps Goal 2: Reliability

Continuous delivery and integration not only improve the time to market with new code, they also improve the overall stability and reliability of software. Integrating automated testing and exception handling helps software development teams identify problems immediately, minimizing the chances of errors being introduced and exposed to end users.

DevOps Goal 3: Rapid Delivery

DevOps aims to increase the pace and frequency of new software application releases, enabling development teams to improve an application as often as they’d like. Performing frequent, fast-paced releases ensures that the turnaround time for any given bug fix or new feature release is as short as possible.

DevOps Goal 4: Scalability

DevOps focuses on creating applications and infrastructure platforms that quickly and easily scale to address the constantly changing needs and demands of both business needs and end users. A practice that is gaining in popularity that helps scale applications is infrastructure as code, which is the process of managing and provisioning hardware data centers to immediately add resources and capacity for an application.

DevOps Goal 5: Security

DevOps encourages strong security practices by automating compliance policies. This simplifies the configuration process and introduces detailed security controls. This programmatic approach ensures that any resources that fall out of compliance are noticed immediately so they can be evaluated by the development team, in order to get them back into compliance immediately.

DevOps Goal 6: Collaboration

Just like other agile-based software development methodologies, DevOps strongly encourages collaboration throughout the entire software development life cycle. This leads to software development teams that are up to 25% more productive and 50% faster to market than non-agile teams.

Differences between DataOps and DevOps

As outlined above, DevOps is the transformation in the ability and capability of software development teams to continuously deliver and integrate their code.

DataOps focuses on the transformation of intelligence systems and analytic models by data analysts and data engineers.

DevOps brings software development teams and IT operations together with the primary goal to reduce the cost and time spent on the development and release cycle.

DataOps goes one step further, integrating data so that data teams can acquire the data, transform it, model it, and obtain insights that are highly actionable.

DataOps is not limited to making existing data pipelines work effectively, getting reports and Artificial Intelligence and Machine Learning outputs and inputs to appear as needed, and so on. DataOps actually includes all parts of the data management lifecycle.

DataOps, DevOps, and Competitive Advantage

DevOps, as a term and as a practice, grew rapidly in interest and activity throughout the last decade, but has plateaued recently. A decade ago, or even five years ago, aggressive adoption of DevOps as a practice could give an organization a significant competitive advantage. But DevOps is now “table stakes” for modern software development and delivery.

It’s now DataOps that’s in a phase of rapid growth. A big part of the reason for this is the need for strong DataOps practices in developing, and delivering value from, AI and ML. For IT practitioners and management, there are new skills to learn, new ways to deliver value, and in a sense, whole new worlds to conquer, all based on the development, growth, and institutionalization of new practices around data.

At the organizational level, DataOps gives companies the opportunity to innovate, to better serve customers, and to gain competitive advantage by rapid, effective adoption and innovation around DataOps as a practice.

Many of today’s largest and fastest-growing companies are DataOps innovators. Facebook, Apple, Alphabet, Netflix, and Uber are just the best-known among companies which have grown to a previously unheard-of degree, largely based on their innovative (and, often, controversial) practices around the use of data.

Adobe is an example of a company that has grown rapidly – increasing its market value by 4x in the last few years – by adding a data-driven business, the marketing-centered Experience Cloud, to their previously existing portfolio.

Algorithms are hard to protect, competitively, while a company’s data is its own. And, while AI algorithms and machine learning models that depend on this data can be shared, they don’t mean much without the flow of data that powers them. So all this means that innovation based on a company’s data, accelerated by the adoption and implementation of DataOps, is more able to yield lasting and protectable competitive advantage, and contribute to a company’s growth.