A Data Platform Leader’s Guide to Choosing the Right Data Observability Solution

Modern data platforms demand more than just basic monitoring. As Databricks adoption grows across the enterprise, platform owners are under pressure to optimize performance, control cloud costs, and deliver reliable data at scale. But with a fragmented vendor landscape and no one-size-fits-all solution, knowing where to start can be a challenge.

This guide simplifies the complex world of data observability and provides a clear, actionable framework for selecting and deploying the right solution for your Databricks environment.

Discover:

• The five core data observability domains every enterprise needs to cover (based on Gartner’s 2024 framework)
• How different solution types—DIY, FinOps, DevOps, native tools, and AI-native platforms—compare
• How the emerging discipline of DataFinOps is more than cost governance
• Which approach best aligns with your goals: cost control, data quality, performance tuning, and scalability
• A phased deployment roadmap for rolling out your selected solution with confidence

If you’re evaluating your data observability options or looking to optimize your Databricks cost and performance, this guide will help you make the best choice for your needs.

The post Databricks Data Observability Buyer’s Guide appeared first on Unravel.

According to Forrester, accurately forecasting cloud costs remains a significant challenge for 80% of data management professionals. This struggle often stems from a lack of granular visibility, control over usage, and ability to optimize code and infrastructure for cost and performance. Organizations utilizing modern data platforms like Snowflake, BigQuery, and Databricks often face unexpected budget overruns, missed performance SLAs, and inefficient resource allocation.

Transitioning from reactive spending to proactive optimization is crucial for effective cost management in modern data stack environments.

This shift requires a comprehensive approach that encompasses several key strategies:

1. Granular Visibility
Gain comprehensive insights into expenses by unifying fragmented data and breaking down silos, enabling precise financial planning and resource allocation for effective cost control. This unified approach allows teams to identify hidden cost drivers and inefficiencies across the entire data ecosystem.

By consolidating data from various sources, organizations can create a holistic view of their spending patterns, facilitating more accurate budget forecasting and informed decision-making. Additionally, this level of visibility empowers teams to pinpoint opportunities for optimization, such as underutilized resources or redundant processes, leading to significant cost savings over time.

2. ETL Pipeline Optimization
Design cost-effective pipelines from the outset, implementing resource utilization best practices and ongoing performance monitoring to identify and address inefficiencies. This approach involves carefully architecting ETL processes to minimize resource usage while maintaining optimal performance.

By employing advanced performance tuning techniques, such as optimizing query execution plans and leveraging built-in optimizations, organizations can significantly reduce processing time and associated costs. Continuous monitoring of pipeline performance allows for the early detection of bottlenecks or resource-intensive operations, enabling timely adjustments and ensuring sustained efficiency over time.

3. Intelligent Resource Management
Implement intelligent autoscaling to dynamically adjust resources based on workload demands, optimizing costs in real-time while maintaining performance. Efficiently manage data lake and compute resources to minimize unnecessary expenses during scaling. This approach allows organizations to provision automatically and de-provision resources as needed, ensuring optimal utilization and cost-efficiency.

By setting appropriate scaling policies and thresholds, you can avoid over-provisioning during periods of low demand and ensure sufficient capacity during peak usage times. Additionally, separating storage and compute resources enables more granular control over costs, allowing you to scale each component independently based on specific requirements.

4. FinOps Culture
Foster collaboration between data and finance teams, implementing cost allocation strategies like tagging and chargeback mechanisms to attribute expenses to specific projects or teams accurately. This approach creates a shared responsibility for cloud costs and promotes organizational transparency.

By establishing clear communication channels and regular meetings between technical and financial stakeholders, teams can align their efforts to optimize resource utilization and spending. A robust tagging system also allows for detailed cost breakdowns, enabling more informed decision-making and budget allocation based on actual usage patterns.

5. Advanced Forecasting
Develop sophisticated forecasting techniques and flexible budgeting strategies using historical data and AI-driven analytics to accurately predict future costs and create adaptive budgets that accommodate changing business needs. Organizations can identify trends and seasonal variations that impact costs by analyzing past usage patterns and performance metrics.

This data-driven approach enables more precise resource allocation and helps teams anticipate potential cost spikes, allowing for proactive adjustments to prevent budget overruns. Additionally, implementing AI-powered forecasting models can provide real-time insights and recommendations, enabling continuous optimization of environments as workloads and business requirements evolve.

Mastering these strategies can help you transform your approach to cost management from reactive to proactive, ensuring you maximize the value of your cloud investments while maintaining financial control.

To learn more about implementing these cost management strategies in your modern data environment, join our upcoming webinar series, “Controlling Cloud Costs.” This ten-part series will explore each aspect of effective cost management, providing actionable insights and best practices to gain control over your data platform costs.

Register for Controlling Databricks Cloud Cost webinars.

Register for Controlling Snowflake Cloud Cost webinars.

The post Mastering Cost Management: From Reactive Spending to Proactive Optimization appeared first on Unravel.

Mastering Databricks Cost Management and FinOps: A Comprehensive Checklist

In the era of big data and cloud computing, organizations increasingly turn to platforms like Databricks to handle their data processing and analytics needs. However, with great power comes great responsibility – and, in this case, the responsibility of managing costs effectively.

This checklist dives deep into cost management and FinOps for Databricks, exploring how to inform, govern, and optimize your usage while taking a holistic approach that considers code, configurations, datasets, and infrastructure.

While this checklist is comprehensive and very impactful when implemented fully, it can also be overwhelming to implement with limited staffing and resources. AI-driven insights and automation can solve this problem and are also explored at the bottom of this guide.

Understanding Databricks Cost Management

Before we delve into strategies for optimization, it’s crucial to understand that Databricks cost management isn’t just about reducing expenses. It’s about gaining visibility into where your spend is going, ensuring resources are being used efficiently, and aligning costs with business value. This comprehensive approach is often referred to as FinOps (Financial Operations).

The Holistic Approach: Key Areas to Consider

1. Code Optimization

Is code optimized? Efficient code is the foundation of cost-effective Databricks usage. Consider the following:

Query Optimization: Ensure your Spark SQL queries are optimized for performance. Use explain plans to understand query execution and identify bottlenecks.
Proper Data Partitioning: Implement effective partitioning strategies to minimize data scans and improve query performance.
Caching Strategies: Utilize Databricks’ caching mechanisms judiciously to reduce redundant computations.

2. Configuration Management

Are configurations managed appropriately? Proper configuration can significantly impact costs:

Cluster Sizing: Right-size your clusters based on workload requirements. Avoid over-provisioning resources.
Autoscaling: Implement autoscaling to adjust cluster size based on demand dynamically.
Instance Selection: Choose the appropriate instance types for your workloads, considering both performance and cost.

3. Dataset Management

Are datasets managed correctly? Efficient data management is crucial for controlling costs:

Data Lifecycle Management: Implement policies for data retention and archiving to avoid unnecessary storage costs.
Data Format Optimization: Use efficient file formats like Parquet or ORC to reduce storage and improve query performance.
Data Skew Handling: Address data skew issues to prevent performance bottlenecks and unnecessary resource consumption.

4. Infrastructure Optimization

Is infrastructure optimized? Optimize your underlying infrastructure for cost-efficiency:

Storage Tiering: Utilize appropriate storage tiers (e.g., DBFS, S3, Azure Blob Storage) based on data access patterns.
Spot Instances: Leverage spot instances for non-critical workloads to reduce costs.
Reserved Instances: Consider purchasing reserved instances for predictable, long-running workloads.

Implementing FinOps Practices

To truly master Databricks cost management, implement these FinOps practices:

1. Visibility and Reporting

Implement comprehensive cost allocation and tagging strategies.
Create dashboards to visualize spend across different dimensions (teams, projects, environments).
Set up alerts for unusual spending patterns or budget overruns.

2. Optimization

Regularly review and optimize resource usage based on actual consumption patterns.
Implement automated policies for shutting down idle clusters.
Encourage a culture of cost awareness among data engineers and analysts.

3. Governance

Establish clear policies and guidelines for resource provisioning and usage.
Implement role-based access control (RBAC) to ensure appropriate resource access.
Create approval workflows for high-cost operations or resource requests.

Setting Up Guardrails

Guardrails are essential for preventing cost overruns and ensuring responsible usage:

Budget Thresholds: Set up budget alerts at various thresholds (e.g., 50%, 75%, 90% of budget).
Usage Quotas: Implement quotas for compute hours, storage, or other resources at the user or team level.
Automated Policies: Use Databricks’ Policy Engine to enforce cost-saving measures automatically.
Cost Centers: Implement chargeback or showback models to make teams accountable for their spend.

The Need for Automated Observability and FinOps Solutions

While manual oversight is important, the scale and complexity of modern data operations often necessitate automated solutions. Tools like Unravel can provide:

Real-time cost visibility across your entire Databricks environment.
Automated recommendations for cost optimization.
Anomaly detection to identify unusual spending patterns quickly.
Predictive analytics to forecast future costs and resource needs.

These solutions can significantly enhance your ability to manage costs effectively, providing insights that would be difficult or impossible to obtain manually.

Conclusion

Effective cost management and FinOps for Databricks require a holistic approach considering all aspects of your data operations. By optimizing code, configurations, datasets, and infrastructure, and implementing robust FinOps practices, you can ensure that your Databricks investment delivers maximum value to your organization. Remember, the goal isn’t just to reduce costs, but to optimize spend and align it with business objectives. With the right strategies and tools in place, you can turn cost management from a challenge into a competitive advantage.

To learn more about how Unravel can help with Databricks cost management, request a health check report, view a self-guided product tour, or request a demo.

The post Databricks Cost Management appeared first on Unravel.

The Complexities of Code Optimization in Databricks: Problems, Challenges and Solutions

Databricks, with its unified analytics platform, offers powerful capabilities for big data processing and machine learning. However, with great power comes great responsibility – and in this case, the responsibility of efficient code optimization.

This blog post explores the complexities of code optimization in Databricks across SQL, Python, and Scala, the difficulties in diagnosing and resolving issues, and how automated solutions can simplify this process.

The Databricks Code Optimization Puzzle

1. Spark SQL Optimization Challenges

Problem: Inefficient Spark SQL queries can lead to excessive shuffling, out-of-memory errors, and slow execution times.

Diagnosis Challenge: Identifying the root cause of a slow Spark SQL query is complex. Is it due to poor join conditions, suboptimal partitioning, or inefficient use of Spark’s catalyst optimizer? The Spark UI provides a wealth of information, but parsing through stages, tasks, and shuffles to pinpoint the exact issue requires deep expertise and time.

Resolution Difficulty: Optimizing Spark SQL often involves a delicate balance. Techniques like broadcast joins might work well for small-to-medium datasets but fail spectacularly for large ones. Each optimization technique needs to be carefully tested across various data scales, which is time-consuming and can lead to performance regressions if not done meticulously.

2. Python UDF Performance Issues

Problem: While Python UDFs (User-Defined Functions) offer flexibility, they can be a major performance bottleneck due to serialization overhead and lack of Spark’s optimizations.

Diagnosis Challenge: The impact of Python UDFs isn’t always immediately apparent. A UDF that works well on a small dataset might become a significant bottleneck as data volumes grow. Identifying which UDFs are causing issues and why requires careful profiling and analysis of Spark jobs.

Resolution Difficulty: Optimizing Python UDFs often involves rewriting them in Scala or using Pandas UDFs, which requires a different skill set. Balancing between code readability, maintainability, and performance becomes a significant challenge, especially in teams with varying levels of Spark expertise.

3. Scala Code Complexity

Problem: While Scala can offer performance benefits, complex Scala code can lead to issues like object serialization problems, garbage collection pauses, and difficulty in code maintenance.

Diagnosis Challenge: Scala’s powerful features, like lazy evaluation and implicits, can make it difficult to trace the execution flow and identify performance bottlenecks. Issues like serialization problems might only appear in production environments, making them particularly challenging to diagnose.

Resolution Difficulty: Optimizing Scala code often requires a deep understanding of both Scala and the internals of Spark. Solutions might involve changing fundamental aspects of the code structure, which can be risky and time-consuming. Balancing between idiomatic Scala and Spark-friendly code is an ongoing challenge.

4. Memory Management Across Languages

Problem: Inefficient memory management, particularly in long-running Spark applications, can lead to out-of-memory errors or degraded performance over time.

Diagnosis Challenge: Memory issues in Databricks can be particularly elusive. Is the problem in the JVM heap, off-heap memory, or perhaps in Python’s memory management? Understanding the interplay between Spark’s memory management and the specifics of SQL, Python, and Scala requires expertise in all these areas.

Resolution Difficulty: Resolving memory issues often involves a combination of code optimization, configuration tuning, and sometimes fundamental architectural changes. This process can be lengthy and may require multiple iterations of testing in production-like environments.

The Manual Optimization Struggle

Traditionally, addressing these challenges involves a cycle of:

1. Manually analyzing Spark UI, logs, and metrics
2. Conducting time-consuming performance tests across various data scales
3. Carefully refactoring code and tuning configurations
4. Monitoring the impact of changes across different workloads and data sizes
5. Rinse and repeat

This process is not only time-consuming but also requires a rare combination of skills across SQL, Python, Scala, and Spark internals. Even for experts, keeping up with the latest best practices and Databricks features is an ongoing challenge.

Leveraging Automation for Databricks Optimization

Given the complexities and ongoing nature of these challenges, many organizations are turning to automated solutions to streamline their Databricks optimization efforts. Tools like Unravel can help by:

1. Cross-Language Performance Monitoring: Automatically tracking performance metrics across SQL, Python, and Scala code in a unified manner.

2. Intelligent Bottleneck Detection: Using machine learning to identify performance bottlenecks, whether they’re in SQL queries, Python UDFs, or Scala code.

3. Root Cause Analysis: Quickly pinpointing the source of performance issues, whether they’re related to code structure, data skew, or resource allocation.

4. Code-Aware Optimization Recommendations: Providing language-specific suggestions for code improvements, such as replacing Python UDFs with Pandas UDFs or optimizing Scala serialization.

5. Predictive Performance Modeling: Estimating the impact of code changes across different data scales before deployment.

6. Automated Tuning: In some cases, automatically adjusting Spark configurations based on workload patterns and performance goals.

By leveraging such automated solutions, data teams can focus their expertise on building innovative data products while ensuring their Databricks environment remains optimized and cost-effective. Instead of spending hours digging through Spark UIs and log files, teams can quickly identify and resolve issues, or even prevent them from occurring in the first place.

Conclusion

Code optimization in Databricks is a multifaceted challenge that spans across SQL, Python, and Scala. While the problems are complex and the manual diagnosis and resolution process can be daunting, automated solutions offer a path to simplify and streamline these efforts. By leveraging such tools, organizations can more effectively manage their Databricks performance, improve job reliability, and allow their data teams to focus on delivering value rather than constantly battling optimization challenges.

Remember, whether you’re using manual methods or automated tools, optimization in Databricks is an ongoing process. As your data volumes grow and processing requirements evolve, staying on top of performance management will ensure that your Databricks implementation continues to deliver the insights and data products your business needs, efficiently and reliably.

To learn more about how Unravel can help with Databricks code optimization, request a health check report, view a self-guided product tour, or request a demo.

The post Databricks Code Optimization appeared first on Unravel.

Navigating the Maze of Configuration Management in Modern Data Platforms: Problems, Challenges and Solutions

In the world of big data, configuration management is often the unsung hero of platform performance and cost-efficiency. Whether you’re working with Snowflake, Databricks, BigQuery, or any other modern data platform, effective configuration management can mean the difference between a sluggish, expensive system and a finely-tuned, cost-effective one.

This blog post explores the complexities of configuration management in data platforms, the challenges in optimizing these settings, and how automated solutions can simplify this critical task.

The Configuration Conundrum

1. Cluster and Warehouse Sizing

Problem: Improper sizing of compute resources (like Databricks clusters or Snowflake warehouses) can lead to either performance bottlenecks or unnecessary costs.

Diagnosis Challenge: Determining the right size for your compute resources is not straightforward. It depends on workload patterns, data volumes, and query complexity, all of which can vary over time. Identifying whether performance issues or high costs are due to improper sizing requires analyzing usage patterns across multiple dimensions.

Resolution Difficulty: Adjusting resource sizes often involves a trial-and-error process. Too small, and you risk poor performance; too large, and you’re wasting money. The impact of changes may not be immediately apparent and can affect different workloads in unexpected ways.

2. Caching and Performance Optimization Settings

Problem: Suboptimal caching strategies and performance settings can lead to repeated computations and slow query performance.

Diagnosis Challenge: The effectiveness of caching and other performance optimizations can be highly dependent on specific workload characteristics. Identifying whether poor performance is due to cache misses, inappropriate caching strategies, or other factors requires deep analysis of query patterns and platform-specific metrics.

Resolution Difficulty: Tuning caching and performance settings often requires a delicate balance. Aggressive caching might improve performance for some queries while causing staleness issues for others. Each adjustment needs to be carefully evaluated across various workload types.

3. Security and Access Control Configurations

Problem: Overly restrictive security settings can hinder legitimate work, while overly permissive ones can create security vulnerabilities.

Diagnosis Challenge: Identifying the root cause of access issues can be complex, especially in platforms with multi-layered security models. Is a performance problem due to a query issue, or is it because of an overly restrictive security policy?

Resolution Difficulty: Adjusting security configurations requires careful consideration of both security requirements and operational needs. Changes need to be thoroughly tested to ensure they don’t inadvertently create security holes or disrupt critical workflows.

4. Cost Control and Resource Governance

Problem: Without proper cost control measures, data platform expenses can quickly spiral out of control.

Diagnosis Challenge: Understanding the cost implications of various platform features and usage patterns is complex. Is a spike in costs due to inefficient queries, improper resource allocation, or simply increased usage?

Resolution Difficulty: Implementing effective cost control measures often involves setting up complex policies and monitoring systems. It requires balancing cost optimization with the need for performance and flexibility, which can be a challenging trade-off to manage.

The Manual Configuration Management Struggle

Traditionally, managing these configurations involves:

1. Continuously monitoring platform usage, performance metrics, and costs
2. Manually adjusting configurations based on observed patterns
3. Conducting extensive testing to ensure changes don’t negatively impact performance or security
4. Constantly staying updated with platform-specific best practices and new features
5. Repeating this process as workloads and requirements evolve

This approach is not only time-consuming but also reactive. By the time an issue is noticed and diagnosed, it may have already impacted performance or inflated costs. Moreover, the complexity of modern data platforms means that the impact of configuration changes can be difficult to predict, leading to a constant cycle of tweaking and re-adjusting.

Embracing Automation in Configuration Management

Given these challenges, many organizations are turning to automated solutions to manage and optimize their data platform configurations. Platforms like Unravel can help by:

Continuous Monitoring: Automatically tracking resource utilization, performance metrics, and costs across all aspects of the data platform.

Intelligent Analysis: Using machine learning to identify patterns and anomalies in platform usage and performance that might indicate configuration issues.

Predictive Optimization: Suggesting configuration changes based on observed usage patterns and predicting their impact before implementation.

Automated Adjustment: In some cases, automatically adjusting configurations within predefined parameters to optimize performance and cost.

Policy Enforcement: Helping to implement and enforce governance policies consistently across the platform.

Cross-Platform Optimization: For organizations using multiple data platforms, providing a unified view and consistent optimization approach across different environments.

By leveraging automated solutions, data teams can shift from a reactive to a proactive configuration management approach. Instead of constantly fighting fires, teams can focus on strategic initiatives while ensuring their data platforms remain optimized, secure, and cost-effective.

Conclusion

Configuration management in modern data platforms is a complex, ongoing challenge that requires continuous attention and expertise. While the problems are multifaceted and the manual management process can be overwhelming, automated solutions offer a path to simplify and streamline these efforts.

By embracing automation in configuration management, organizations can more effectively optimize their data platform performance, enhance security, control costs, and free up their data teams to focus on extracting value from data rather than endlessly tweaking platform settings.

Remember, whether using manual methods or automated tools, effective configuration management is an ongoing process. As your data volumes grow, workloads evolve, and platform features update, staying on top of your configurations will ensure that your data platform continues to meet your business needs efficiently and cost-effectively.

To learn more about how Unravel can help manage and optimize your data platform configurations with Databricks, Snowflake, and BigQuery: request a health check report, view a self-guided product tour, or request a demo.

The post Configuration Management in Modern Data Platforms appeared first on Unravel.

Databricks Health: Sample Data Estate Report

Download a sample report that includes insights into the health of a Databricks data estate:

• Performance insights: See the speedup possible with improved jobs and workflows execution.
• Productivity boost: Uncover top l improvements automatically without ever looking at logs and metrics again.
• Savings projection: View projected annualized savings for clusters and pipelines.
• SLA attainment: Measure potential improvements to data pipeline times.
• Job health: See which jobs are failing most frequently and solve these to improve your Databricks data estate.

The post Discover Your Databricks Health: Sample Data Estate Report appeared first on Unravel.

Purpose-built AI provides real-time cost and performance insights and efficiency recommendations for Databricks users

Palo Alto, CA — December 5, 2023 — Unravel Data, the first AI-enabled data observability and FinOps platform built to address the speed and scale of modern data platforms, today announced that it has joined the Databricks Partner Program to deliver AI-powered data observability into Databricks for granular visibility, performance optimizations, and cost governance of data pipelines and applications. With this new partnership, Unravel and Databricks will collaborate on Go-To-Market (GTM) efforts to enable Databricks customers to leverage Unravel’s purpose-built AI for the Lakehouse for real-time, continuous insights and recommendations to speed time to value of data and AI products and ensure optimal ROI.   

With organizations increasingly under pressure to deliver data and AI innovation at lightning speed, data teams are on the front line of delivering production-ready data pipelines at an exponential rate while optimizing performance and efficiency to deliver faster time to value. Unravel’s purpose-built AI for Databricks integrates with Lakehouse Monitoring and Lakehouse Observability to deliver performance and efficiency needed to achieve speed and scale for data analytics and AI products. Unravel’s integration with Unity Catalog enables Databricks users to speed up lakehouse transformation by providing real-time, AI-powered cost insights, code-level optimizations, accurate spending predictions, and performance recommendations to accelerate data pipelines and applications for greater returns on cloud data platform investments. AutoActions and alerts help automate governance with proactive guardrails.

“Most organizations today are receiving unprecedented amounts of data from a staggering number of sources, and they’re struggling to manage it all, which can quickly lead to unpredictable cloud data spend. This combination of rapid lakehouse adoption and the hyperfocus companies have on leveraging AI/ML models for additional revenue and competitive advantage, brings the importance of data observability to the forefront,” said Kunal Agarwal, CEO and co-founder, Unravel Data. “Lakehouse customers who use Unravel can now achieve the agility required for AI/ML innovation while having the predictability and cost governance guardrails needed to ensure a strong ROI.”

Unravel’s purpose-built AI for Databricks delivers insights based on Unravel’s deep observability at the job, user, and code level to supply AI-driven cost efficiency recommendations, including compute provisioning, query performance, autoscaling efficiencies, and more. 

Unravel for Databricks enables organizations to:

• Speed cloud transformation initiatives by having real-time cost visibility, predictive spend forecasting, and performance insights for their workloads 
• Enhance time to market of new AI initiatives by mitigating potential pipeline bottlenecks and associated costs before they occur
• Better manage and optimize the ROI of data projects with customized dashboards and alerts that offer insights on spend, performance, and unit economics

Unravel’s integration with popular DevOps tools like GitHub and Azure DevOps provides actionability in CI/CD workflows by enabling early issue detection during the code-merge phase and providing developers real-time insights into potential financial impacts of their code changes. This results in fewer production issues and improved cost efficiency.

Learn how Unravel and Databricks can help enterprises optimize their cloud data spend and increase ROI here.   

About Unravel Data

Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Unravel’s market-leading data observability and FinOps platform with purpose-built AI for each data platform, provides actionable recommendations needed for cost and performance data and AI pipeline efficiencies. A recent winner of the Best Data Tool & Platform of 2023 as part of the annual SIIA CODiE Awards, some of the world’s most recognized brands like Adobe, Maersk, Mastercard, Equifax, and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

The post Unravel Data Partners with Databricks for Lakehouse Observability and FinOps  appeared first on Unravel.

5 key differences between Overwatch and Unravel’s free observability for Databricks.

Before investing significant time and effort to configure Overwatch, compare how Unravel’s free data observability solution gives you end-to-end, real-time granular visibility into your Databricks Lakehouse platform out of the box.

The post Open Source Overwatch VS Unravel Free Comparison Infographic appeared first on Unravel.

“Someone’s sitting in the shade today because someone planted a tree a long time ago.” —Warren Buffet

CI/CD, a software development strategy, combines the methodologies of Continuous Integration and Continuous Delivery/Continuous Deployment to safely and reliably deliver new versions of code in iterative short cycles. This practice bridges the gap between developers and operations team by streamlining the building, testing, and deployment of the code by automating the series of steps involved in this otherwise complex process. Traditionally used to speed up the software development life cycle, today CI/CD is gaining popularity among data scientists and data engineers since it enables cross-team collaboration and rapid, secure integration and deployment of libraries, scripts, notebooks, and other ML workflow assets.

One recent report found that 80% of organizations have adopted agile practices, but for nearly two-thirds of developers it takes at least one week to get committed code successfully running in production. Implementing CI/CD can streamline data pipeline development and deployment, accelerating release times and frequency, while improving code quality. 

The evolving need for CI/CD for data teams

AI’s rapid adoption is driving the demand for fresh and reliable data for training, validation, verification, and drift analysis. Implementing CI/CD enhances your Databricks development process, streamlines pipeline deployment, and accelerates time-to-market. CI/CD revolutionizes how you build, test, and deploy code within your Databricks environment, helping you automate tasks, ensure a smooth transition from development to production, and enable lakehouse data engineering and data science teams to work more efficiently. And when it comes to cloud data platforms like Databricks, performance equals cost. The more optimized your pipelines are, the more optimized your Databricks spend will be.

Why incorporate Unravel into your existing DevOps workflow?

Unravel Data is the AI-powered data observability and FinOps platform for Databricks. By using Unravel’s CI/CD integration for Databricks, developers can catch performance problems early in development and deployment life cycles and proactively take actions to mitigate issues. This has shown to significantly reduce the time taken by data teams to act on critical timely insights. Unravel’s AI-powered efficiency recommendations, now embedded right into the DevOps environments, help foster a cost-conscious culture that compels developers to follow performance and cost-driven coding best practices. It also raises awareness of resource usage, configuration changes, and data layout issues that could impact service level agreements (SLAs) when the code is deployed in production. Accepting or ignoring insights suggested by Unravel helps promote accountability for developers’ actions and creates transparency for the DevOps and FinOps practitioners to attribute cost-saving wins and losses. 

With the advent of Generative Pre-Trained Transformer (GPT) AI models, data teams today have started using coding co-pilots to generate accurate and efficient code. With Unravel, this experience is a notch better with real-time visibility into code inefficiencies that can translate into production performance problems like bottlenecks, performance anomalies, missed SLAs, cost overruns, etc. Other code-assist tools like GitHub Copilot are limited in their scope of assistance to static code analysis based code rewrite suggestions, Unravel’s AI-driven Insights Engine built for Databricks considers the performance and cost impact of code and configuration changes and provides recommendations to make optimal suggestions. This helps you streamline your development process, identify bottlenecks, and ensure optimal performance throughout the life cycle of your data pipelines.

Unravel’s AI-powered analysis automatically provides deep, actionable insights.

Next, let’s look into what key benefits are provided by the Unravel integration into your DevOps workflows. 

Achieve operational excellence 

Unravel’s CI/CD integration for Databricks enhances data team and developer efficiency by seamlessly providing real-time, AI-powered insights to help optimize performance and troubleshoot issues in your data pipelines.  

Unravel integrates with your favorite CI/CD tools such as Azure DevOps and GitHub. When developers make changes to code and submit via a pull request, Unravel automatically conducts AI-powered checks to ensure the code is performant and efficient. This helps developers:

• Maximize resource utilization by gaining valuable insights into pipeline efficiency
• Achieve performance and cost goals by analyzing critical metrics during development
• Leverage specific, actionable recommendations to improve code for cost and performance optimization
• Identify and resolve bottlenecks promptly, reducing development time

Leverage developer pull request (PR) reviews

Developers play a crucial role in achieving cost efficiency through PR reviews. Encourage them to adopt best practices and follow established guidelines when submitting code for review. This ensures that all tests are run and results are thoroughly evaluated before merging into the main project branch.

By actively involving developers in the review process, you tap into their knowledge and experience to identify potential areas for cost savings within your pipelines. Their insights can help streamline workflows, improve resource allocation, and eliminate inefficiencies. Involving developers in PR reviews fosters collaboration among team members and encourages feedback, creating a culture of continuous improvement. 

Here are several ways developer PR reviews can enhance the reliability of data pipelines:

• Ensure code quality: Developer PR reviews serve as an effective mechanism to maintain high code-quality standards. Through these reviews, developers can catch coding errors, identify potential bugs, and suggest improvements before the code is merged into the production repository.
• Detect issues early: By involving developers in PR reviews, you ensure that potential issues are identified early in the development process. This allows for prompt resolution and prevents problems from propagating further down the pipeline.
• Mitigate risks: Faulty or inefficient code changes can have significant impacts on your pipelines and overall system stability. With developer PR reviews, you involve experts who understand the intricacies of the pipeline and can help mitigate risks by providing valuable insights and suggestions.
• Foster a collaborative environment: Developer PR reviews create a collaborative environment where team members actively engage with one another’s work. Feedback provided during these reviews promotes knowledge sharing, improves individual skills, and enhances overall team performance.

Real-world examples of CI/CD integration for Databricks

Companies in finance, healthcare, e-commerce, and more have successfully implemented CI/CD practices with Databricks. Enterprise organization across industries leverage Unravel to ensure that code is performant and efficient before it goes into production.

• Financial services: A Fortune Global 500 bank provides Unravel to their developers as a way to evaluate their pipelines before they do a code release.
• Healthcare: One of the largest health insurance providers in the United States uses Unravel to ensure that its business-critical data applications are optimized for performance, reliability, and cost in its development environment—before they go live in production.
• Logistics: One of the world’s largest logistics companies leverages Unravel to upskill their data teams at scale. They put Unravel in their CI/CD process to ensure that all code and queries are reviewed to ensure they meet the desired quality and efficiency bar before they go into production.

Unravel CI/CD integration for Databricks use cases

Incorporating Unravel’s real-time, AI insights into PR reviews helps developers ensure the reliability, performance, and cost efficiency of data pipelines before they go into production. This practice ensures that any code changes are thoroughly reviewed before being merged into the main project branch. By catching potential issues early on, you can prevent pipeline breaks, bottlenecks, and wasted compute tasks from running in production. 

Ensure pipeline reliability

Unravel’s purpose-built AI helps augment your PR reviews to ensure code quality and reliability in your release pipelines. Unravel integration into your Databricks CI/CD process helps developers identify potential issues early on and mitigate risks associated with faulty or inefficient code changes. Catching breaking changes in development and test environments helps developers improve productivity and helps ensure that you achieve your SLAs.

1-minute tour: Unravel’s AI-powered Speed, Cost, Reliability Optimizer

Achieve cost efficiency

Unravel provides immediate feedback and recommendations to improve cost efficiency. This enables you to catch inefficient code, and developers can make any necessary adjustments for optimal resource utilization before it impacts production environments. Using Unravel as part of PR reviews helps your organization optimize resource allocation and reduce cloud waste.

1-minute tour: Unravel’s AI-powered Databricks Cost Optimization

Boost pipeline performance

Collaborative code reviews provide an opportunity to identify bottlenecks, optimize code, and enhance data processing efficiency. By including Unravel’s AI recommendations in the review process, developers benefit from AI-powered insights to ensure code changes achieve performance objectives. 

1-minute tour: Unravel’s AI-powered Pipeline Bottleneck Analysis

Get started with Unravel CI/CD integration for Databricks

Supercharge your CI/CD process for Databricks using Unravel’s AI. By leveraging this powerful combination, you can significantly improve developer productivity, ensure pipeline reliability, achieve cost efficiency, and boost overall pipeline performance. Whether you choose to automate PR reviews with Azure DevOps or GitHub, Unravel’s CI/CD integration for Databricks has got you covered.

Now it’s time to take action and unleash the full potential of your Databricks environment. Integrate Unravel’s CI/CD solution into your workflow and experience the benefits firsthand. Don’t miss out on the opportunity to streamline your development process, save costs, and deliver high-quality code faster than ever before.

Next steps to learn more

Read Unravel’s CI/CD integration documentation

Watch this video

Book a live demo

The post Unravel CI/CD Integration for Databricks appeared first on Unravel.

In this fast-paced era of artificial intelligence (AI), the need for data is multiplying. The demand for faster data life cycles has skyrocketed, thanks to AI’s insatiable appetite for knowledge. According to a recent McKinsey survey, 75% expect generative AI (GenAI) to “cause significant or disruptive change in the nature of their industry’s competition in the next three years.”

Next-gen AI craves unstructured, streaming, industry-specific data. Although the pace of innovation is relentless, “when it comes to generative AI, data really is your moat.”

But here’s the twist: efficiency is now the new cool kid in town. Data product profitability hinges on optimizing every step of the data life cycle—from ingestion and transformation, to processing, curating, and refining. It’s no longer just about gathering mountains of information; it’s about collecting the right data efficiently.

As new, industry-specific GenAI use cases emerge, there is an urgent need for large data sets for training, validation, verification, and drift analysis. GenAI requires flexible, scalable, and efficient data architecture, infrastructure, code, and operating models to achieve success.

Leverage a Scalable Operating Model to Accelerate Your Data Life Cycle Velocity

To optimize your data life cycle, it’s crucial to leverage a scalable operating model that can accelerate the velocity of your data processes. By following a systematic approach and implementing efficient strategies, you can effectively manage your data from start to finish.

Databricks recently introduced a scalable operating model for data and AI to help customers achieve a positive Return on Data Assets (RODA).

Databricks’ iterative end-to-end operating pipeline

Define Use Cases and Business Requirements

Before diving into the data life cycle, it’s essential to clearly define your use cases and business requirements. This involves understanding what specific problems or goals you plan to address with your data. By identifying these use cases and related business requirements, you can determine the necessary steps and actions needed throughout the entire process.

Build, Test, and Iterate the Solution

Once you have defined your use cases and business requirements, it’s time to build, test, and iterate the solution. This involves developing the necessary infrastructure, tools, and processes required for managing your data effectively. It’s important to continuously test and iterate on your solution to ensure that it meets your desired outcomes.

During this phase, consider using agile methodologies that allow for quick iterations and feedback loops. This will enable you to make adjustments as needed based on real-world usage and feedback from stakeholders.

Scale Efficiently

As your data needs grow over time, it’s crucial to scale efficiently. This means ensuring that your architecture can handle increased volumes of data without sacrificing performance or reliability.

Consider leveraging cloud-based technologies that offer scalability on-demand. Cloud platforms provide flexible resources that can be easily scaled up or down based on your needs. Employing automation techniques such as machine learning algorithms or artificial intelligence can help streamline processes and improve efficiency.

By scaling efficiently, you can accommodate growing datasets while maintaining high-quality standards throughout the entire data life cycle.

Elements of the Business Use Cases and Requirements Phase

In the data life cycle, the business requirements phase plays a crucial role in setting the foundation for successful data management. This phase involves several key elements that contribute to defining a solution and ensuring measurable outcomes. Let’s take a closer look at these elements:

• Leverage design thinking to define a solution for each problem statement: Design thinking is an approach that focuses on understanding user needs, challenging assumptions, and exploring innovative solutions. In this phase, it is essential to apply design thinking principles to identify and define a single problem statement that aligns with business objectives.
• Validate the business case and define measurable outcomes: Before proceeding further, it is crucial to validate the business case for the proposed solution. This involves assessing its feasibility, potential benefits, and alignment with strategic goals. Defining clear and measurable outcomes helps in evaluating project success.
• Map out the MVP end user experiences: To ensure user satisfaction and engagement, mapping out Minimum Viable Product (MVP) end-user experiences is essential. This involves identifying key touchpoints and interactions throughout the data life cycle stages. By considering user perspectives early on, organizations can create intuitive and effective solutions.
• Understand the data requirements: A thorough understanding of data requirements is vital for successful implementation. It includes identifying what types of data are needed, their sources, formats, quality standards, security considerations, and any specific regulations or compliance requirements.
• Gather required capabilities with platform architects: Collaborating with platform architects helps gather insights into available capabilities within existing infrastructure or technology platforms. This step ensures compatibility between business requirements and technical capabilities while minimizing redundancies or unnecessary investments.
• Establish data management roles, responsibilities, and procedures: Defining clear roles and responsibilities within the organization’s data management team is critical for effective execution. Establishing procedures for data observability, stewardship practices, access controls, privacy policies ensures consistency in managing data throughout its life cycle.

By following these elements in the business requirements phase, organizations can lay a solid foundation for successful data management and optimize the overall data life cycle. It sets the stage for subsequent phases, including data acquisition, storage, processing, analysis, and utilization.

Build, Test, and Iterate the Solution

To successfully implement a data life cycle, it is crucial to focus on building, testing, and iterating the solution. This phase involves several key steps that ensure the development and deployment of a robust and efficient system.

• Plan development and deployment: The first step in this phase is to carefully plan the development and deployment process. This includes identifying the goals and objectives of the project, defining timelines and milestones, and allocating resources effectively. By having a clear plan in place, the data team can streamline their efforts towards achieving desired outcomes.
• Gather end-user feedback at every stage: Throughout the development process, it is essential to gather feedback from end users at every stage. This allows for iterative improvements based on real-world usage scenarios. By actively involving end users in providing feedback, the data team can identify areas for enhancement or potential issues that need to be addressed.
• Define CI/CD pipelines for fast testing and iteration: Implementing Continuous Integration (CI) and Continuous Deployment (CD) pipelines enables fast testing and iteration of the solution. These pipelines automate various stages of software development such as code integration, testing, deployment, and monitoring. By automating these processes, any changes or updates can be quickly tested and deployed without manual intervention.
• Data preparation, cleaning, and processing: Before training machine learning models or conducting experiments with datasets, it is crucial to prepare, clean, and process the data appropriately. This involves tasks such as removing outliers or missing values from datasets to ensure accurate results during model training.
• Feature engineering: Feature engineering plays a vital role in enhancing model performance by selecting relevant features from raw data or creating new features based on domain knowledge. It involves transforming raw data into meaningful representations that capture essential patterns or characteristics.
• Training and ML experiments: In this stage of the data life cycle, machine learning models are trained using appropriate algorithms on prepared datasets. Multiple experiments may be conducted, testing different algorithms or hyperparameters to find the best-performing model.
• Model deployment: Once a satisfactory model is obtained, it needs to be deployed in a production environment. This involves integrating the model into existing systems or creating new APIs for real-time predictions.
• Model monitoring and scoring: After deployment, continuous monitoring of the model’s performance is essential. Tracking key metrics and scoring the model’s outputs against ground truth data helps identify any degradation in performance or potential issues that require attention.

By following these steps and iterating on the solution based on user feedback, data teams can ensure an efficient and effective data life cycle from development to deployment and beyond.

Efficiently Scale and Drive Adoption with Your Operating Model

To efficiently scale your data life cycle and drive adoption, you need to focus on several key areas. Let’s dive into each of them:

• Deploy into production: Once you have built and tested your solution, it’s time to deploy it into production. This step involves moving your solution from a development environment to a live environment where end users can access and utilize it.
• Evaluate production results: After deploying your solution, it is crucial to evaluate its performance in the production environment. Monitor key metrics and gather feedback from users to identify any issues or areas for improvement.
• Socialize data observability and FinOps best practices: To ensure the success of your operating model, it is essential to socialize data observability and FinOps best practices among your team. This involves promoting transparency, accountability, and efficiency in managing data operations.
• Acknowledge engineers who “shift left” performance and efficiency: Recognize and reward engineers who prioritize performance and efficiency early in the development process. Encourage a culture of proactive optimization by acknowledging those who contribute to improving the overall effectiveness of the data life cycle.
• Manage access, incidents, support, and feature requests: Efficiently scaling your operating model requires effective management of access permissions, incident handling processes, support systems, and feature requests. Streamline these processes to ensure smooth operations while accommodating user needs.
• Track progress towards business outcomes by measuring and sharing KPIs: Measuring key performance indicators (KPIs) is vital for tracking progress towards business outcomes. Regularly measure relevant metrics related to adoption rates, user satisfaction levels, cost savings achieved through efficiency improvements, etc., then share this information across teams for increased visibility.

By implementing these strategies within your operating model, you can efficiently scale your data life cycle while driving adoption among users. Remember that continuous evaluation and improvement are critical for optimizing performance throughout the life cycle.

Drive for Performance with Purpose-Built AI

Unravel helps with many elements of the Databricks operating model:

• Quickly identify failed and inefficient Databricks jobs: One of the key challenges is identifying failed and inefficient Databricks jobs. However, with AI purpose-built for Databricks, this task becomes much easier. By leveraging advanced analytics and monitoring capabilities, you can quickly pinpoint any issues in your job executions.
• Creating ML models vs deploying them into production: Creating machine learning models is undoubtedly challenging, but deploying them into production is even harder. It requires careful consideration of factors like scalability, performance, and reliability. With purpose-built AI tools, you can streamline the deployment process by automating various tasks such as model versioning, containerization, and orchestration.
• Leverage Unravel’s Analysis tab for insights: To gain deeper insights into your application’s performance during job execution, leverage the analysis tab provided by purpose-built AI solutions. This feature allows you to examine critical details like memory usage errors or other bottlenecks that may be impacting job efficiency.
  
  

  Unravel’s AI-powered analysis automatically provides deep, actionable insights.

• Share links for collaboration: Collaboration plays a crucial role in data management and infrastructure optimization. Unravel enables you to share links with data scientists, developers, and data engineers to provide detailed information about specific test runs or failed Databricks jobs. This promotes collaboration and facilitates a better understanding of why certain jobs may have failed.
• Cloud data cost management made easy: Cloud cost management, also known as FinOps, is another essential aspect of data life cycle management. Purpose-built AI solutions simplify this process by providing comprehensive insights into cost drivers within your Databricks environment. You can identify the biggest cost drivers such as users, clusters, jobs, and code segments that contribute significantly to cloud costs.
• AI recommendations for optimization: To optimize your data infrastructure further, purpose-built AI platforms offer valuable recommendations across various aspects, including infrastructure configuration, parallelism settings, handling data skewness issues, optimizing Python/SQL/Scala/Java code snippets, and more. These AI-driven recommendations help you make informed decisions to enhance performance and efficiency.

Learn More & Next Steps

Unravel hosted a virtual roundtable, Accelerate the Data Analytics Life Cycle, with a panel of Unravel and Databricks experts. Unravel VP Clinton Ford moderated the discussion with Sanjeev Mohan, principal at SanjMo and former VP at Gartner, Subramanian Iyer, Unravel training and enablement leader and Databricks SME, and Don Hilborn, Unravel Field CTO and former Databricks lead strategic solutions architect.

FAQs

How can I implement a scalable operating model for my data life cycle?

To implement a scalable operating model for your data life cycle, start by clearly defining roles and responsibilities within your organization. Establish efficient processes and workflows that enable seamless collaboration between different teams involved in managing the data life cycle. Leverage automation tools and technologies to streamline repetitive tasks and ensure consistency in data management practices.

What are some key considerations during the Business Requirements Phase?

During the Business Requirements Phase, it is crucial to engage stakeholders from various departments to gather comprehensive requirements. Clearly define project objectives, deliverables, timelines, and success criteria. Conduct thorough analysis of existing systems and processes to identify gaps or areas for improvement.

How can I drive adoption of my data life cycle operational model?

To drive adoption of your data management solution, focus on effective change management strategies. Communicate the benefits of the solution to all stakeholders involved and provide training programs or resources to help them understand its value. Encourage feedback from users throughout the implementation process and incorporate their suggestions to enhance usability and address any concerns.

What role does AI play in optimizing the data life cycle?

AI can play a significant role in optimizing the data life cycle by automating repetitive tasks, improving data quality through advanced analytics and machine learning algorithms, and providing valuable insights for decision-making. AI-powered tools can help identify patterns, trends, and anomalies in large datasets, enabling organizations to make data-driven decisions with greater accuracy and efficiency.

How do I ensure performance while implementing purpose-built AI?

To ensure performance while implementing purpose-built AI, it is essential to have a well-defined strategy. Start by clearly defining the problem you want to solve with AI and set measurable goals for success. Invest in high-quality training data to train your AI models effectively. Continuously monitor and evaluate the performance of your AI system, making necessary adjustments as needed.

The post Rev Up Your Lakehouse: Lap the Field with a Databricks Operating Model appeared first on Unravel.

Overview

In today’s digital landscape, data-driven decisions form the crux of successful business strategies. However, the path to harnessing data’s full potential is strewn with challenges. Let’s delve into the hurdles organizations face and how Unravel is the key to unlocking seamless data operations.

The Roadblocks in the Fast Lane of Data Operations

In today’s data-driven landscape, organizations grapple with erratic spending, cloud constraints, AI complexities, and prolonged MTTR, urgently seeking solutions to navigate these challenges efficiently. The four most common roadblocks are:

• Data Spend Forecasting: Imagine a roller coaster with unpredictable highs and lows. That’s how most organizations view their data spend forecasting. Such unpredictability wreaks havoc on financial planning, making operational consistency a challenge.
• Constraints in Adding Data Workloads: Imagine tying an anchor to a speedboat. That’s how the constraints feel when trying to adopt cloud data solutions, holding back progress and limiting agility.
• Surge in AI Model Complexity: AI’s evolutionary pace is exponential. As it grows, so do the intricacies surrounding data volume and pipelines, which strain budgetary limitations.
• The MTTR Bottleneck: The multifaceted nature of modern tech stacks means longer Mean Time to Repair (MTTR). This slows down processes, consumes valuable resources, and stalls innovation.

By acting as a comprehensive data observability and FinOps solution, Unravel Data empowers businesses to move past the challenges and frictions that typically hinder data operations, ensuring smoother, more efficient data-driven processes. Here’s how Unravel Data aids in navigating the roadblocks in the high-speed lane of data operations:

• Predictive Data Spend Forecasting: With its advanced analytics, Unravel Data can provide insights into data consumption patterns, helping businesses forecast their data spending more accurately. This eliminates the roller coaster of unpredictable costs.
• Simplifying Data Workloads: Unravel Data optimizes and automates workload management. Instead of being anchored down by the weight of complex data tasks, businesses can efficiently run and scale their data processes in the cloud.
• Managing AI Model Complexity: Unravel offers visibility and insights into AI data pipelines. Analyzing and optimizing these pipelines ensure that growing intricacies do not overwhelm resources or budgets.
• Reducing MTTR: By providing a clear view of the entire tech stack and pinpointing bottlenecks or issues, Unravel Data significantly reduces Mean Time to Repair. With its actionable insights, teams can address problems faster, reducing downtime and ensuring smooth data operations.
• Streamlining Data Pipelines: Unravel Data offers tools to diagnose and improve data pipeline performance. This ensures that even as data grows in volume and complexity, pipelines remain efficient and agile.
• Efficiency and ROI: With its clear insights into resource consumption and requirements, Unravel Data helps businesses run 50% more workloads in their existing Databricks environments, ensuring they only pay for what they need, reducing wastage and excess expenditure.

The Skyrocketing Growth of Cloud Data Management

As the digital realm expands, cloud data management usage is soaring, with data services accounting for a significant chunk. According to the IDC, the public cloud IaaS and PaaS market is projected to reach $400 billion by 2025, growing at a 28.8% CAGR from 2021 to 2025. Some highlights are:

• Data management and application development account for 39% and 20% of the market, respectively, and are the main workloads backed by PaaS solutions, capturing a major share of its revenue.
• In IaaS revenue, IT infrastructure leads with 25%, trailed by business applications (21%) and data management (20%).
• Unstructured data analytics and media streaming are predicted to be the top-growing segments with CAGRs of 41.9% and 41.2%, respectively.

Unravel provides a comprehensive solution to address the growth associated with cloud data management. Here’s how:

• Visibility and Transparency: Unravel offers in-depth insights into your cloud operations, allowing you to understand where and how costs are accruing, ensuring no hidden fees or unnoticed inefficiencies.
• Optimization Tools: Through its suite of analytics and AI-driven tools, Unravel pinpoints inefficiencies, recommends optimizations, and automates the scaling of resources to ensure you’re only using (and paying for) what you need.
• Forecasting: With predictive analytics, Unravel provides forecasts of data usage and associated costs, enabling proactive budgeting and financial planning.
• Workload Management: Unravel ensures that data workloads run efficiently and without wastage, reducing both computational costs and storage overhead.
• Performance Tuning: By optimizing query performance and data storage strategies, Unravel ensures faster results using fewer resources, translating to 50% more workloads.
• Monitoring and Alerts: Real-time monitoring paired with intelligent alerts ensures that any resource-intensive operations or anomalies are flagged immediately, allowing for quick intervention and rectification.

By employing these strategies and tools, Unravel acts as a financial safeguard for businesses, ensuring that the ever-growing cloud data bill remains predictable, manageable, and optimized for efficiency.

The Tightrope Walk of Efficiency Tuning and Talent

Modern enterprises hinge on data and AI, but shrinking budgets and talent gaps threaten them. Gartner pinpoints overprovisioning and skills shortages as major roadblocks, while Google and IDC underscore the high demand for data analytics skills and the untapped potential of unstructured data. Here are some of the problems modern organizations face:

• Production environments are statically overprovisioned and therefore underutilized. On-premises, 30% utilization is common, but it’s all capital expenditures (capex), and as long as it’s within budget, no one has traditionally cared about the waste. However, in the cloud, you pay for that excess resource monthly, forcing you to confront the ongoing cost of the waste. – Gartner
• The cloud skills gap has reached a crisis level in many organizations – Gartner
• Revenue creation through digital transformation requires talent engagement that is currently scarce and difficult to acquire and maintain. – Gartner
• Lack of skills remains the biggest barrier to infrastructure modernization initiatives, with many organizations finding they cannot hire outside talent to fill these skills gaps. IT organizations will not succeed unless they prioritize organic skills growth. – Gartner
• Data analytics skills are in demand across industries as businesses of all types around the world recognize that strong analytics improve business performance.- Google via Coursera

Unravel Data addresses the delicate balancing act of budget and talent in several strategic ways:

• Operational Efficiency: Purpose-built AI provides actionable insights into data operations across Databricks, Spark, EMR, BigQuery, Snowflake, etc. Unravel Data reduces the need for trial-and-error and time-consuming manual interventions. At the core of Unravel’s data observability platform is our AI-powered Insights Engine. This sophisticated Artificial Intelligence engine incorporates AI techniques, algorithms, and tools to process and analyze vast amounts of data, learn from patterns, and make predictions or decisions based on that learning. This not only improves operational efficiency but also ensures that talented personnel spend their time innovating rather than on routine tasks.
• Skills Gap Bridging: The platform’s intuitive interface and AI-driven insights mean that even those without deep expertise in specific data technologies can navigate, understand, and optimize complex data ecosystems. This eases the pressure to hire or train ultra-specialized talent.
• Predictive Analysis: With Unravel’s ability to predict potential bottlenecks or inefficiencies, teams can proactively address issues, leading to more efficient budget allocation and resource utilization.
• Cost Insights: Unravel provides detailed insights into the efficiency of various data operations, allowing organizations to make informed decisions on where to invest and where to cut back.
• Automated Optimization: By automating many of the tasks traditionally performed by data engineers, like performance tuning or troubleshooting, Unravel ensures teams can do more with less, optimizing both budget and talent.
• Talent Focus Shift: With mundane tasks automated and insights available at a glance, skilled personnel can focus on higher-value activities, like data innovation, analytics, and strategic projects.

By enhancing efficiency, providing clarity, and streamlining operations, Unravel Data ensures that organizations can get more from their existing budgets while maximizing the potential of their talent, turning the tightrope walk into a more stable journey.

The Intricacies of Data-Centric Organizations

Data-centric organizations grapple with the complexities of managing vast and fast-moving data in the digital age. Ensuring data accuracy, security, and compliance, while integrating varied sources, is challenging. They must balance data accessibility with protecting sensitive information, all while adapting to evolving technologies, addressing talent gaps, and extracting actionable insights from their data reservoirs. Here is some relevant research on the topic:

• “Data is foundational to AI” yet “unstructured data remains largely untapped.” – IDC
• Even as organizations rush to adopt data-centric operations, challenges persist. For instance, manufacturing data projects often hit roadblocks due to outdated legacy technology, as observed by the World Economic Forum.
• Generative AI is supported by large language models (LLMs), which require powerful and highly scalable computing capabilities to process data in real-time. – Gartner

Unravel Data provides a beacon for data-centric organizations amid complex challenges. Offering a holistic view of data operations, it simplifies management using AI-driven tools. It ensures data security, accessibility, and optimized performance. With its intuitive interface, Unravel bridges talent gaps and navigates the data maze, turning complexities into actionable insights.

Embarking on the Unravel Journey: Your Step-By-Step Guide

• Beginning your data journey with Unravel is as easy as 1-2-3. We guide you through the sign-up process, ensuring a smooth and hassle-free setup.
• Unravel for Databricks page
  

• Unravel for Databricks plans & pricing
  

• Unravel for Databricks free account
  

Level Up with Unravel Premium

Ready for an enhanced data experience? Unravel’s premium account offers a plethora of advanced features that the free version can’t match. Investing in this upgrade isn’t just about more tools; it’s about supercharging your data operations and ROI.

Wrap-Up

Although rising demands on the modern data landscape are challenging, they are not insurmountable. With tools like Unravel, organizations can navigate these complexities, ensuring that data remains a catalyst for growth, not a hurdle. Dive into the Unravel experience and redefine your data journey today.

Unravel is a business’s performance sentinel in the cloud realm, proactively ensuring that burgeoning cloud data expenses are not only predictable and manageable but also primed for significant cost savings. Unravel Data transforms the precarious balance of budget and talent into a streamlined, efficient journey for organizations. Unravel Data illuminates the path for data-centric organizations, streamlining operations with AI tools, ensuring data security, and optimizing performance. Its intuitive interface simplifies complex data landscapes, bridging talent gaps and converting challenges into actionable insights.

The post Overcoming Friction & Harnessing the Power of Unravel: Try It for Free appeared first on Unravel.

“Data is the new oil, and analytics is the combustion engine.” – Peter Sondergaard

Cloud data analytics is the key to maximizing value from your data. The lakehouse has emerged as a flexible and efficient architecture, and Databricks has emerged as a popular choice. However, data lakehouse processing volumes can fluctuate, leading to unpredictable surges in cloud data spending that impact budgeting and profitability. Executives want to make sure they are getting the most from their lakehouse investments and not overspending.

Implementing a proactive data observability and FinOps approach early in your lakehouse journey helps ensure you achieve your business objectives and bring predictability to your financial planning. Choosing the right lakehouse observability and FinOps tool sets your team up for success. Since the goal is efficiency, starting with free tools makes sense. Two free options stand out:

• Overwatch – the open source Databricks observability tool
• Unravel Standard – the free version of Unravel’s data observability and FinOps platform

Below are 5 reasons to choose Unravel free observability vs. Databricks free observability:

Reason #1: Complete observability

Many organizations take a do-it-yourself approach, building piecemeal observability solutions in-house by cobbling together a variety of data sources using open source tools. The problem is that it takes months or even years to get something usable up and running. Unravel’s data observability and FinOps platform helps you get results fast.

Unravel provides a 360° out-of-the-box solution

Unravel provides a holistic view of your Databricks estate, reducing the time to value. Gain deep insights into cluster performance, job execution, resource utilization, and cost drivers through comprehensive lakehouse observability. Unravel’s observability solution provides you with detailed visibility into the performance of your Databricks clusters. 

Insights overview dashboard in Unravel Standard

Ensure no blind spots in your analysis by leveraging Unravel’s end-to-end visibility across all aspects of your Databricks environment. View your lakehouse landscape at a glance with the Insights Overview dashboard. You can see the overall health of your Databricks estate, including the number of clusters that are over- or under-provisioned, the total number of inefficient and failed apps, and other summary statistics to guide your efforts to optimize your lakehouse towards better performance and cost efficiency.

Purpose-built correlation

Unravel’s purpose-built correlation models help you identify inefficient jobs at code, data layout/partitioning, and infrastructure levels. Databricks logs, metrics, events, traces, and source code are automatically evaluated to simplify root cause analysis and issue resolution. You can dive deep into the execution details of your Databricks jobs, track the progress of each job, and see resource usage details. This helps you identify long-running and resource-intensive jobs that might be impacting the overall performance and efficiency of your lakehouse estate.

End-to-end visibility

Visual summaries provide a way to look across all the jobs and clusters in your Databricks workspace. No need to click around your Databricks workspace looking for issues, run queries, or pull details into a spreadsheet to summarize results. Unravel helps you easily see all the details in one place. 

Reason #2: Real-time visibility

A single errant job or underutilized cluster can derail your efficiency goals and delay critical data pipelines. The ability to see job and cluster performance and efficiency in real time provides an early warning system.

Live updates for running jobs and clusters

React promptly to any anomalies or bottlenecks in your clusters and jobs to ensure efficiency. Unravel’s real-time insights allow you to catch long-running jobs before they impact pipeline performance or consume unnecessary resources.

Workspace Trends dashboard in Unravel Standard

See DBU usage and cluster session trends

By understanding the real-time performance of your Databricks workloads, you can identify areas where improvements can be made to improve efficiency without sacrificing performance. Leverage Unravel’s real-time insights to make data-driven decisions for better resource allocation and workload management. 

Drill down to see DBU usage and tasks for a specific day

Quickly find resource consumption outliers by day to understand how usage patterns are driving costs. Unravel helps you identify opportunities to reduce waste and increase cluster utilization. By having visibility into the real-time cost implications of your jobs and clusters, you can make faster decisions to boost performance and improve business results.

User-level reporting for showback/chargeback

Granular reporting to the user and job level helps you produce accurate and timely showback and chargeback reports. With Unravel’s real-time visibility into your Databricks workloads, you have the power to see which teams are consuming the most resources and proactively manage costs to achieve efficient operations. Reacting quickly to anomalies and leveraging real-time, user-level insights enables better decision-making for resource allocation and utilization. Unravel enables central data platform and operations teams to provide a reliable, single source of truth for showback and chargeback reporting.

Reason #3: Automated Cluster Discovery

You can’t fix problems you can’t see. It all begins with getting visibility across all your workspace clusters and jobs. Unravel automates this process to save you time and ensure you don’t miss anything.

Easily connect to all of your clusters in the workspace

Simplify the process of connecting to your Databricks clusters with Unravel’s automated cluster discovery. This streamlines the observability and management of your compute clusters, so you can focus on resource optimization to boost productivity. Unravel lets you easily see all of your clusters without adding dependencies.

Compute dashboard shows clusters in Unravel Standard

Quickly discover clusters with errors, delays, and failures

Unravel lets you see clusters grouped by event type (e.g., Contended Driver, High I/O, Data Skew, Node Downsizing). This helps you quickly identify patterns in compute clusters that are not being fully utilized. This eliminates the need for manual monitoring and analysis, saving you time and effort.

View cluster resource trends

Unravel’s intelligent automation continuously monitors cluster activity and resource utilization over time. This helps you spot changing workload requirements and helps ensure optimal performance while keeping costs in check by avoiding overprovisioning or underutilization to make the most of your cloud infrastructure investments.

Reason #4: Ease of Entry

Open source and DIY solutions typically have a steep learning curve to ensure everything is correctly configured and connected. Frequent changes and updates add to your team’s already full load. Unravel offers a simpler approach.

Unravel is quick to set up and get started with minimal learning curve

Integrating Unravel into your existing Databricks environment is a breeze. No complex setup or configuration required. With Unravel, you can seamlessly bring data observability and FinOps capabilities to your data lakehouse estate without breaking a sweat.

Unravel SaaS makes setup and configuration a breeze

But what exactly does this mean for you? It means that you can focus on what matters most—getting the most out of your Databricks platform while keeping costs in check. Unravel’s data observability and FinOps capabilities are provided as a fully managed service, giving you the power to optimize performance and resources, spot efficiency opportunities, and ensure smooth operation of your data pipelines and data applications.

No DIY coding or development required

Unravel is trusted by large enterprise customers across many industries for its ease of integration into their Databricks environments. Whether you’re a small team or an enterprise organization, Unravel’s data observability and FinOps platform is designed to meet your specific needs and use cases without the need to build anything from scratch.

Reason #5: Avoid lock-in

A lakehouse architecture gives you flexibility. As your data analytics and data processing needs grow and evolve, you may choose additional analytics tools to complement your cloud data estate. Your data observability and FinOps tool should support those tools as well.

Unravel is purpose-built for Databricks, Snowflake, BigQuery, and other modern data stacks

Each cloud data platform is different and requires a deep understanding of its inner workings in order to provide the visibility you need to run efficiently. Unravel is designed from the ground up to help you get the most out of each modern data platform, leveraging the most relevant and valuable metadata sources and correlating them all into a unified view of your data estate.

No need to deploy a separate tool as your observability needs grow

Unravel provides a consistent approach to data observability and FinOps to minimize time spent deploying and learning new tools. Data teams spend less time upskilling and more time getting valuable insights.

Independent reporting for FinOps

Data analytics is the fastest growing segment of cloud computing as organizations invest in new use cases such as business intelligence (BI), AI and machine learning. Organizations are adopting FinOps practices to ensure transparency in resource allocation, usage, and reporting. Unravel provides an independent perspective of lakehouse utilization and efficiency to ensure objective, data-driven decisions.

Compare Unravel and Databricks free observability

Get started today

Achieve predictable spend and gain valuable insights into your Databricks usage. Get started today with Unravel’s complete data observability and FinOps platform for Databricks that provides real-time visibility, automated cluster discovery, ease of entry, and independent analysis to help you take control of your costs while maximizing the value of your Databricks investments. Create your free Unravel account today.

Unravel for Databricks FAQ 

Can I use Unravel’s data observability platform with other cloud providers?

Yes. Unravel’s data observability platform is designed to work seamlessly across multiple cloud providers including AWS, Azure, and Google Cloud. So regardless of which cloud provider you choose for your data processing needs, Unravel can help you optimize costs and gain valuable insights.

How does automated cluster discovery help in managing Databricks costs?

Automated cluster discovery provided by solutions like Unravel enables you to easily identify underutilized or idle clusters within your Databricks environment. By identifying these clusters, you can make informed decisions about resource allocation and ensure that you are only paying for what you actually need.

Does Unravel offer real-time visibility into my Databricks usage?

Yes. With Unravel’s real-time visibility feature, you can monitor your Databricks usage in real time. This allows you to quickly identify any anomalies or issues that may impact cost efficiency and take proactive measures to address them.

Can Unravel help me optimize my Databricks costs for different workloads?

Yes. Unravel’s data observability platform provides comprehensive insights into the performance and cost of various Databricks workloads. By analyzing this data, you can identify areas for optimization and make informed decisions to ensure cost efficiency across different workloads.

How easy is it to get started with Unravel’s data observability platform?

Getting started with Unravel is quick and easy. Simply sign up for a free account on our website, connect your Databricks environment, and start gaining valuable insights into your usage and costs. Our intuitive interface and user-friendly features make it simple for anyone to get started without any hassle.

The post Databricks Cost Efficiency: 5 Reasons Why Unravel Free Observability vs. Databricks Free Observability appeared first on Unravel.

By Craig Wiley, Senior Director of Product Management, Databricks and Clinton Ford, Director of Product Marketing, Unravel Data

Introduction

Machine learning (ML) enables organizations to extract more value from their data than ever before. Companies who successfully deploy ML models into production are able to leverage that data value at a faster pace than ever before. But deploying ML models requires a number of key steps, each fraught with challenges:

• Data preparation, cleaning, and processing
• Feature engineering
• Training and ML experiments
• Model deployment
• Model monitoring and scoring

Figure 1. Phases of the ML lifecycle with Databricks Machine Learning and Unravel Data

Challenges at each phase

Data preparation and processing

Data preparation is a data scientist’s most time-consuming task. While there are many phases in the data science lifecycle, an ML model can only be as good as the data that feeds it. Reliable and consistent data is essential for training and machine learning (ML) experiments. Despite advances in data processing, a significant amount of effort is required to load and prepare data for training and ML experimentation. Unreliable data pipelines slow the process of developing new ML models.

Training and ML experiments

Once data is collected, cleansed, and refined, it is ready for feature engineering, model training, and ML experiments. The process is often tedious and error-prone, yet machine learning teams also need a way to reproduce and explain their results for debugging, regulatory reporting, or other purposes. Recording all of the necessary information about data lineage, source code revisions, and experiment results can be time-consuming and burdensome. Before a model can be deployed into production, it must have all of the detailed information for audits and reproducibility, including hyperparameters and performance metrics.

Model deployment and monitoring

While building ML models is hard, deploying them into production is even more difficult. For example, data quality must be continuously validated and model results must be scored for accuracy to detect model drift. What makes this challenge even more daunting is the breadth of ML frameworks and the required handoffs between teams throughout the ML model lifecycle– from data preparation and training to experimentation and production deployment. Model experiments are difficult to reproduce as the code, library dependencies, and source data change, evolve, and grow over time.

The solution

The ultimate hack to productionize ML is data observability combined with scalable, serverless, and automated ML model serving. Unravel’s AI-powered data observability for Databricks on AWS and Azure Databricks simplifies the challenges of data operations, improves performance, saves critical engineering time, and optimizes resource utilization.

Databricks Model Serving deploys machine learning models as a REST API, enabling you to build real-time ML applications like personalized recommendations, customer service chatbots, fraud detection, and more – all without the hassle of managing serving infrastructure.

Databricks + data observability

Whether you are building a lakehouse with Databricks for ML model serving, ETL, streaming data pipelines, BI dashboards, or data science, Unravel’s AI-powered data observability for Databricks on AWS and Azure Databricks simplifies operations, increases efficiency, and boosts productivity. Unravel provides AI insights to proactively pinpoint and resolve data pipeline performance issues, ensure data quality, and define automated guardrails for predictability.

Scalable training and ML experiments with Databricks

Databricks uses pre-installed, optimized libraries to build and train machine learning models. With Databricks, data science teams can build and train machine learning models. Databricks provides pre-installed, optimized libraries. Examples include scikit-learn, TensorFlow, PyTorch, and XGBoost. MLflow integration with Databricks on AWS and Azure Databricks makes it easy to track experiments and store models in repositories.

MLflow monitors machine learning model training and running. Information about the source code, data, configuration information, and results are stored in a single location for quick and easy reference. MLflow also stores models and loads them in production. Because MLflow is built on open frameworks, many different services, applications, frameworks, and tools can access and consume the models and related details.

Serverless ML model deployment and serving

Databricks Serverless Model Serving accelerates data science teams’ path to production by simplifying deployments and reducing mistakes through integrated tools. With the new model serving service, you can do the following:

• Deploy a model as an API with one click in a serverless environment.
• Serve models with high availability and low latency using endpoints that can automatically scale up and down based on incoming workload.
• Safely deploy the model using flexible deployment patterns such as progressive rollout or perform online experimentation using A/B testing.
• Seamlessly integrate model serving with online feature store (hosted on Azure Cosmos DB), MLflow Model Registry, and monitoring, allowing for faster and error-free deployment.

Conclusion

You can now train, deploy, monitor, and retrain machine learning models, all on the same platform with Databricks Model Serving. Integrating the feature store with model serving and monitoring helps ensure that production models are leveraging the latest data to produce accurate results. The end result is increased availability and simplified operations for greater AI velocity and positive business impact.

Ready to get started and try it out for yourself? Watch this Databricks event to see it in action. You can read more about Databricks Model Serving and how to use it in the Databricks on AWS documentation and the Azure Databricks documentation. Learn more about data observability in the Unravel documentation.

The post Solving key challenges in the ML lifecycle with Unravel and Databricks Model Serving appeared first on Unravel.

The benefits of moving your on-prem Spark Hadoop environment to Databricks are undeniable. A recent Forrester Total Economic Impact (TEI) study reveals that deploying Databricks can pay for itself in less than six months, with a 417% ROI from cost savings and increased revenue & productivity over three years. But without the right methodology and tools, such modernization/migration can be a daunting task.

Capgemini’s VP of Analytics Pratim Das recently moderated a webinar with Unravel’s VP of Solutions Engineering Chris Santiago, Databricks’ Migrations Lead (EMEA) Amine Benhamza, and Microsoft’s Analytics Global Black Belt (EMEA) Imre Ruskal to discuss how to reduce the risk of unexpected complexities, avoid roadblocks, and present cost overruns. 

The session Optimizing and Migrating Hadoop to Azure Databricks is available on demand, and this post briefly recaps that presentation.

Pratim from Capgemini opened by reviewing the four phases of a cloud migration—assess; plan; test, fix, verify; optimize, manage, scale—and polling the attendees about where they were on their journey and the top challenges they have encountered. 

How Unravel helps migrate to Databricks from Hadoop

Chris ran through the stages an enterprise goes through when doing a cloud migration from Hadoop to Databricks (really, any cloud platform), with the different challenges associated with each phase. 

Specifically, profiling exactly what you have running on Hadoop can be a highly manual, time-consuming exercise than can take 4-6 months, requires domain experts, can cost over $500K—and even then is still usually inaccurate and incomplete by 30%.

This leads to problematic planning. Because you don’t have complete data and have missed crucial dependencies, you wind up with inaccurate “guesstimates” that delay migrations by 9-18 months and underestimate TCO by 3-5X

Then once you’ve actually started deploying workloads in the cloud, too often users are frustrated that workloads are running slower than they did on-prem. Manual tuning each job takes about 20 hours in order to meet SLAs, increasing migration expenses by a few million dollars. 

Finally, migration is never a one-and-done deal. Managing and optimizing the workloads is a constant exercise, but fragmented tooling leads to cumbersome manual management and lack of governance results in ballooning cloud costs.

Chris Santiago shows over a dozen screenshots illustrating Unravel capabilities to assess and plan a Databricks migration. Click on image or here to jump to his session.

Chris illustrated how Unravel’s data-driven approach to migrating to Azure Databricks helps alleviate and solve these challenges. Specifically, Unravel answers questions you need to ask to get a complete picture of your Hadoop inventory:

• What jobs are running in your environment—by application, by user, by queue? 
• How are your resources actually being utilized over a lifespan of a particular environment?
• What’s the velocity—the number of jobs that are submitted in a particular environment—how much Spark vs. Hive, etc.?
• What pipelines are running (think Airflow, Oozie)?
• Which data tables are actually being used, and how often? 

Then once you have a full understanding of what you’re running in the Hadoop environment, you can start forecasting what this would look like in Databricks. Unravel gathers all the information about what resources are actually being used, how many, and when for each job. This allows you to “slice” the cluster to start scoping out what this would look like from an architectural perspective. Unravel takes in all those resource constraints and provides AI-driven recommendations on the appropriate architecture: when and where to use auto-scaling, where spot instances could be leveraged, etc.

Then when planning, Unravel gives you a full application catalog, both at a summary and drill-down level, of what’s running either as repeated jobs or ad hoc. You also get complexity analysis and data dependency reports so you know what you need to migrate and when in your wave plan. This automated report takes into account the complexity of your jobs, the data level and app level dependencies, and ultimately spits out a sprint plan that gives you the level of effort required. 

Click on image or here to see Unravel’s AI recommendations in action

But Unravel also helps with monitoring and optimizing your Databricks environment post-deployment to make sure that (a) everyone is using Databricks most effectively and (b) you’re getting the most out of your investment. With Unravel, you get full-stack observability metrics to understand exactly what’s going on with your jobs. But Unravel goes “beyond observability” to not just tell you what’s going and why, but also tell you what to do about it. 

By collecting and contextualizing data from a bunch of different sources—logs, Spark UI, Databricks console, APIs—Unravel’s AI engine automatically identifies where jobs could be tuned to run for higher performance or lower cost, with pinpoint recommendations on how to “fix things for greater efficiency. This allows you to tune thousands of jobs on the fly, control costs proactively, and track actual vs. budgeted spend in real time. 

Why Databricks?

Amine then presented a concise summary of why he’s seen customers migrate to Databricks from Hadoop, recounting the high costs associated with Hadoop on-prem, the administrative complexity of managing the “zoo of technologies,” the need to decouple compute and storage to reduce waste of unused resources, the need to develop modern AI/ML use cases, not to mention the Cloudera end-of-life issue. He went on to illustrate the advantages and benefits of the Databricks data lakehouse platform, Delta Lake, and how by bringing together the best of Databricks and Azure into a single unified solution, you get a fully modern analytics and AI architecture.

He then went on to show how the kind of data-driven approach that Capgemnini and Unravel take might look for different technologies migrating from Hadoop to Databricks.

Hadoop migration beyond Databricks

The Hadoop ecosystem over time has become extremely complicated and fragmented. If you are looking at all the components that might be in your Hortonworks or Cloudera legacy distribution today, and are trying to map them to the Azure model analytics reference architecture layer, things get pretty complex.

Some things are relatively straightforward to migrate over to Databricks—Spark, HDFS, Hive—others, not so much. This is where his team at Azure Data Services can help out. He went through the considerations and mapping for a range of different components, including:

• Oozie
• Kudi
• Nifi
• Flume
• Kafka
• Storm
• Flink
• Solr
• Pig
• HBase
• MapReduce
• and more

He showed how these various services were used to make sure customers are covered, to fill in the gaps and complement Databricks for an end-to- end solution.

Check out the full webinar Optimizing and Migrating Hadoop to Azure Databricks on demand.
No form to fill out!

The post Webinar Recap: Optimizing and Migrating Hadoop to Azure Databricks appeared first on Unravel.

Unravel Co-Founder and CTO Shivnath Babu recently hosted an informal roundtable discussion with representatives from Fortune 500 insurance firms, healthcare providers, and other enterprise companies. It was a chance for some peer-to-peer conversation about the challenges of migrating to the cloud and talk shop about Databricks. Here are some of the highlights.

Where are you on your cloud journey?

Everybody is at a different place in their migration to the cloud. So, the first thing we wanted to understand is exactly where each participant stood. While cloud migration challenges are pretty much universally the same for anyone, the specific top-of-mind challenges are a bit different at each stage of the game.

About one-third of the participants said that they are running their data applications on a mix of on-prem and cloud environments. This is not surprising, as most of the attendees work in sectors that have been leveraging big data for some time—specifically, insurance and healthcare—and so have plenty of legacy systems to contend with. As one contributor noted, “If a company has been around for more than 10 years, they are going to have multiple systems and they’re going to have legacy systems. Every company is in some type of digital transformation. They start small with what they can afford. And as they grow, they’re able to buy better products.”

Half indicated that they are in multi-cloud environments—which presents a different set of challenges. One participant who has been on the cloud migration path for 3-4 years said that along the way, “we’ve accumulated new tech debt because we have both AWS Lake Formation and Databricks. This generated some complexity, having two flavors of data catalog. So we’re trying to figure out how to deal with that and get things back on track so we can have better governance over data access.”

What are your biggest challenges in migrating to Databricks?

We polled the participants on what their primary goals are in moving to the cloud and what the top challenges they are experiencing. The responses showed a dead heat (43% each) between “better cost governance, chargeback & forecasting” and “improve performance of critical data pipelines with SLAs,” with “reduce the number of tickets and time to resolution” coming in third.

Again, not surprising results given that the majority of the audience were data engineers on the hook for ensuring data application performance and/or people who have been running data pipelines in the cloud for a while. It’s usually not until you start running more and more data jobs in the cloud that cost becomes a big headache.

How does Unravel help with your Databricks environment?

Within the context of the poll questions, Shivnath addressed—at a very high level—how Unravel helps solve the roundtable’s most pressing challenges.

If you think about an entire data pipeline converting data into insights, it can be a multi-system operation: data gets ingested by maybe Kafka, getting into a data lake or lakehouse, but then the actual processing may happen in Databricks. It’s not uncommon to see a dozen or more different technologies among the various components of a modern data pipeline.

What happens quickly is that you end up with an architecture that is very modern, very agile, very cloud-friendly, very elastic but where it’s very difficult to even understand who’s using what and how many resources it takes. An enterprise may have hundreds of data users, with more constantly being added.

Data gets to be like a drug, where the company wants more and more of it, driving more and more insights. And when you compare the number of engineers tasked with fixing problems with the number of data users, it’s clear that there just isn’t enough operational muscle (or enough people with the operational know-how and skill sets) to tackle the challenges. 

Unravel helps fill that gap, and it solves the problem with AI and ML. In a nutshell, what Unravel does is collect all the telemetry information from every system—legacy, cloud, SQL, data pipelines, access to data, the infrastructure—and stream it into the Unravel platform, where it is analyzed with AI/ML to convert that telemetry data into insights to answer questions like

• Where are all my costs going?
• Are they being efficiently used?
• And if not, how do I improve?

Shivnath pointed out that he sees a lot of customers running very large Databricks clusters when they don’t need to. Maybe they’re over-provisioning jobs because of an inefficiency in how the data is stored, or in how the SQL is actually written. Unravel helps with this kind of tuning optimization, reducing costs and building better governance policies from the get-go—so applications are already optimized when they are pushed into production. 

Unravel helps different members of data teams. It serves the data operations folks, the SREs, the architects who are responsible for designing the right architecture and setting governance policies, the data engineers who are creating the applications. 

And everything is based on applying AI/ML to telemetry data.

Want to see Unravel AI in action? Create a free account here.

The post Roundtable Recap: Sharing Challenges in Migrating to Databricks appeared first on Unravel.

UNRAVEL FOR DATABRICKS

Make the Most of Your Databricks Platform

Unravel Data simplifies the way data teams monitor, observe, manage, troubleshoot, and optimize the performance and spend utilization of large-scale modern data applications on Databricks. Unravel also accelerates data migration to Databricks by providing a cloud data migration assessment with actionable insights.

Unravel for Databricks addresses the following key use cases:

• Plan and Migrate to Databricks

  An in-depth assessment of your data workloads being migrated, Unravel’s cloud data migration assessment can minimize data migration barriers and empower you to have a smoother, faster migration.

• Manage and Optimize Databricks

  Unravel provides full stack visibility into Databricks workloads to ensure operational stability. It enables you to simplify data operations, improve resource performance, and optimize the spending and utilization of Databricks.

Next steps

Download the datasheet to learn more.
Start your cloud data migration journey today. Talk to our migration experts.
See what Unravel can do for you. Create your free account.

The post Unravel for Databricks Datasheet appeared first on Unravel.

The post Managing Costs for Spark on Databricks appeared first on Unravel.

The post Troubleshooting Databricks – Challenges, Fixes, and Solutions appeared first on Unravel.

The post Strategies To Accelerate Performance for Databricks appeared first on Unravel.

The post Going Beyond Observability for Spark Applications & Databricks Environments appeared first on Unravel.

The post Why Enhanced Visibility Matters for Your Databricks Environment appeared first on Unravel.

Thank you for your interest in the Unravel Data for Databricks for AWS Datasheet.

You can download it here.

The post Unravel Data For Databricks For AWS Datasheet appeared first on Unravel.

New Offering Accelerates Unravel’s Mission to Support Big Data Workloads Wherever they Reside, Including on-Premises, Public Cloud, Multiple Cloud and Hybrid Environments

PALO ALTO, CALIFORNIA – May 27, 2020 – Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced Unravel for AWS Databricks, a solution to deliver comprehensive monitoring, troubleshooting, and application performance management for AWS Databricks environments. Unravel for AWS Databricks leverages Unravel’s AI-powered data operations platform to accelerate performance of Spark on AWS while providing unprecedented visibility into runtime behavior, resource usage, and cloud costs.

“As business needs evolve, data workloads are moving to a growing variety of settings, stretching across on-prem environments, public clouds, multiple clouds and a hybrid mix of all of these. It’s important that organizations can get the same performance, reliability and value out of their data applications no matter where they are,” said Kunal Agarwal, CEO, Unravel Data. “Unravel for AWS is our latest effort to expand the platform to accommodate Big Data wherever it exists. With this addition, Unravel now supports Databricks in both AWS and Azure, and the Unravel platform is broadly available in every major public cloud as well as on-premises and in hybrid settings. We were always committed to being infrastructure-agnostic and this is another milestone in that mission.”

The announcement is the latest development in a long relationship between Unravel and AWS. Unravel already supports Amazon EMR, as well as Cloudera/Hortonworks on IaaS for AWS. This release provides further support for customers deploying modern data apps on AWS. In addition, Unravel is an existing member of the AWS Partner Network and member of AWS global startup program.

AWS Databricks is a unified data analytics platform for accelerating innovation across data science, data engineering, and business analytics, integrated with AWS infrastructure. Unravel for AWS Databricks helps operationalize Spark apps on the platform: AWS Databricks customers will shorten the cycle of getting Spark applications into production by relying on the visibility, operational intelligence, and data driven insights and recommendations that only Unravel can provide. Users will enjoy greater productivity by eliminating the time spent on tedious, low value tasks such as log data collection, root cause analysis and application tuning.

Key features of Unravel for AWS Databricks include:

• Application Performance Management for AWS Databricks – Unravel delivers visibility and understanding of Spark applications, clusters, workflows, and the underlying software stack
• Automated root cause analysis of Spark apps – Unravel can automatically identify, diagnose, and remediate Spark jobs and the full Spark stack, achieving simpler and faster resolution of issues for Spark applications on AWS Databricks clusters
• Comprehensive reporting, alerting, and dashboards – AWS Databricks users can now enjoy detailed insights, plain-language recommendations, and a host of new dashboards, alerts, and reporting on chargeback accounting, cluster resource usage, Spark runtime behavior and much more

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Data Expands Support for Modern Data Workloads in the Cloud with Introduction of Unravel for AWS Databricks appeared first on Unravel.

PALO ALTO, Calif. – February 25, 2020– Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, introduced new migration, cost analytics and architectural mapping capabilities for Unravel for Azure Databricks, which is now generally available from Unravel and in the Azure Marketplace. The move further solidifies Unravel’s mission to support modern data workloads wherever they exist, whether on-premises, in the public cloud or a hybrid setting.

“With more and more big data deployments moving to the public cloud, Unravel has spent the last several years helping to simplify the process of cloud migration as well as improving the management and optimization of modern data workloads once in the cloud. We have recently introduced platforms for all major public cloud platforms,” said Bala Venkatrao, Chief Product Officer, Unravel Data. “This release, highlighted by the industry’s only slice and dice migration capabilities, makes it easier than ever to move data workloads to Azure Databricks, while minimizing costs and increasing performance. The platform also allows enterprises to unify their data pipelines end-to-end, such as Azure Databricks and Azure HDInsight.”

Unravel for Azure Databricks delivers comprehensive monitoring, troubleshooting, and application performance management for Azure Databricks environments. The new additions to the platform include:

• Slice and dice migration support – Unravel now includes robust migration intelligence to help customers assess their migration planning to Azure Databricks in version 4.5.5.0. Slice and dice migration support provides impact analysis by applications and workloads. It also features recommended cloud cluster topology and cost estimates by service-level agreement (SLA), as well as auto-scaling impact trend analysis as a result of cloud migration.
• Cost analytics – Unravel will soon add new cost management capabilities to help optimize Azure Databricks workloads as they scale. These new features include cost assurance, cost planning and cost forecasting tools. Together, these tools provide granular detail of individual jobs in Azure Databricks, providing visibility at the workspace, job, and job-run level to track costs or DBUs over time.
• Detailed architectural recommendations: Unravel for Azure Databricks will soon include right-sizing, a feature that recommends virtual machine or workload types that will achieve the same performance on cheaper clusters.

Unravel for Azure Databricks helps operationalize Spark apps on the platform: Azure Databricks customers can shorten the cycle of getting Spark applications into production by relying on the visibility, operational intelligence, and data driven insights and recommendations that only Unravel can provide. Users enjoy greater productivity by eliminating the time spent on tedious, low value tasks such as log data collection, root cause analysis and application tuning.

In addition to being generally available directly from Unravel, Unravel for Azure Databricks is also available on the Azure Marketplace, where users can try a free trial of the platform and get $2000 in Azure credits. Cloud marketplaces are quickly becoming the preferred way for organizations to procure, deploy and manage enterprise software. Unravel for Azure Databricks on Azure Marketplace offers one-click deployment of Databricks performance monitoring and management in Azure.

About Unravel Data
Unravel Data radically transforms the way businesses understand and optimize the performance and cost of their modern data applications – and the complex data pipelines that power those applications. Providing a unified view across the entire data stack, Unravel’s market-leading data observability platform leverages AI, machine learning, and advanced analytics to provide modern data teams with the actionable recommendations they need to turn data into insights. Some of the world’s most recognized brands like Adobe, 84.51˚ (a Kroger company), and Deutsche Bank rely on Unravel Data to unlock data-driven insights and deliver new innovations to market. To learn more, visit https://www.unraveldata.com.

Media Contact
Blair Moreland
ZAG Communications for Unravel Data
unraveldata@zagcommunications.com

The post Unravel Introduces Workload Migration and Cost Analytics Solution for Azure Databricks appeared first on Unravel.

Monitor, Troubleshoot and Optimize Apache Spark Applications Using Microsoft Azure Databricks 

We are super excited to announce our support for Azure Databricks! We continue to build out the capabilities of the Unravel Data Operations platform and specifically support for the Microsoft Azure data and AI ecosystem teams.  The business and technical imperative to strategically and tactically architect the journey to cloud for your organization has never been stronger. Businesses are increasingly dependent on data for decision making and by extension the services and platforms such as Azure HDI and Azure Databricks that underpin these modern data applications.

The large scale industry adoption of Spark, and Cloud services from Azure and other platforms. represent the heart of the modern data operations program for the next decade. The combination of Microsoft and Databricks and resulting Azure Databricks offering is a natural response to deliver a deployment platform for AI, machine learning, and streaming data applications.

Spark has largely eclipsed Hadoop/MapReduce as the development paradigm of choice to develop a new generation of data applications that provide new insights and user experiences. Databricks has added a rich development and operations environment for running Apache Spark applications in the cloud, while Microsoft Azure has rapidly evolved into an enterprise favorite for migrating and running these new data applications in the cloud. 

It is against this backdrop that Unravel announces support for the Azure Databricks platform to provide our AI-powered data operations solution for Spark applications and data pipelines running on Azure Databricks. While Azure Databricks provides a state of the art platform for developing and running Spark apps and data pipelines, Unravel provides the relentless monitoring, interrogating, modeling, learning, and guided tuning and troubleshooting to create the optimal conditions for Spark to perform and operate at its peak potential.

Unravel is able to ask and answer questions about Azure Databricks that are essential to provide the levels of intelligence that are required to:

• Provide a unified view across all of your Azure Databricks instances and workspaces
• Understand Spark runtime behavior and how it interacts with Azure infrastructure, and adjacent technologies like Apache Kafka
• Detect and avoid costly human error in configuration, tuning, and root cause analysis 
• Accurately report cluster usage patterns and be able to adjust resource usage on the fly with Unravel insights
• Set and guarantee enterprise service levels, based on correlated operational metadata

The Unravel Platform is constantly learning and our training models adapting. The intelligence you glean from Unravel today continues to extend and adapt over time as application and user behaviors themselves change and adapt to new business demands. These in-built capabilities of the Unravel platform and our extensible APIs enable us to move fast to support customer demands to support an expanding range of Data and AI services such as Azure Databricks.  More importantly though it provides the insights, recommendations and automation to assure your journey to cloud is accelerated and your ongoing Cloud operations is fully optimized for cost and performance.

Read on to learn more about today’s news from Unravel.

Unravel Data Introduces AI-powered Data Operations Solution to Monitor, Troubleshoot and Optimize Apache Spark Applications Using Microsoft Azure Databricks

New Offering Enables Azure Databricks Customers to Quickly Operationalize Spark Data Engineering Workloads with Unprecedented Visibility and Radically Simpler Remediation of Failures and Slowdowns

PALO ALTO, Calif. – Sep. 4, 2019 —Unravel Data, the only data operations platform providing full-stack visibility and AI-powered recommendations to drive more reliable performance in modern data applications, today announced Unravel for Azure Databricks, a  solution to deliver comprehensive monitoring, troubleshooting, and application performance management for Azure Databricks environments. The new offering leverages AI to enable Azure Databricks customers to significantly improve performance of Spark jobs while providing unprecedented visibility into runtime behavior, resource usage, and cloud costs.

“Spark, Azure, and Azure Databricks have become foundational technologies in the modern data stack landscape, with more and more Fortune 1000 organizations using them to build their modern data pipelines,” said Kunal Agarwal, CEO, Unravel Data. “Unravel is uniquely positioned to empower Azure Databricks customers to maximize the performance, reliability and return on investment of their Spark workloads.”

Unravel for Azure Databricks helps operationalize Spark apps on the platform: Azure Databricks customers will shorten the cycle of getting Spark applications into production by relying on the visibility, operational intelligence, and data driven insights and recommendations that only Unravel can provide. Users will enjoy greater productivity by eliminating the time spent on tedious, low value tasks such as log data collection, root cause analysis and application tuning.

“Unravel’s full-stack DataOps platform has already helped Azure customers get the most out of their cloud-based big data deployments. We’re excited to extend that relationship to Azure Databricks,” said Yatharth Gupta, principal group manager, Azure Data at Microsoft. “Unravel adds tremendous value by delivering an AI-powered solution for Azure Databricks customers that are looking to troubleshoot challenging operational issues and optimize cost and performance of their Azure Databricks workloads.”

Key features of Unravel for Azure Databricks include:

• Application Performance Management for Azure Databricks – Unravel delivers visibility and understanding of Spark applications, clusters, workflows, and the underlying software stack
• Automated root cause analysis of Spark apps – Unravel can automatically identify, diagnose, and remediate Spark jobs and the full Spark stack, achieving simpler and faster resolution of issues for Spark applications on Azure Databricks clusters
• Comprehensive reporting, alerting, and dashboards – Azure Databricks users can now enjoy detailed insights, plain-language recommendations, and a host of new dashboards, alerts, and reporting on chargeback accounting, cluster resource usage,  Spark runtime behavior and much more.

Azure Databricks is a Spark-based analytics platform optimized for Microsoft Azure. Azure Databricks provides one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

An early access release of Unravel for Azure Databricks available now.

About Unravel Data

Unravel radically simplifies the way businesses understand and optimize the performance of their modern data applications – and the complex pipelines that power those applications. Providing a unified view across the entire stack, Unravel’s data operations platform leverages AI, machine learning, and advanced analytics to offer actionable recommendations and automation for tuning, troubleshooting, and improving performance – both today and tomorrow. By operationalizing how you do data, Unravel’s solutions support modern big data leaders, including Kaiser Permanente, Adobe, Deutsche Bank, Wayfair, and Neustar. The company is headquartered in Palo Alto, California, and is backed by Menlo Ventures, GGV Capital, M12, Point72 Ventures, Harmony Partners, Data Elite Ventures, and Jyoti Bansal. To learn more, visit unraveldata.com.

Copyright Statement

The name Unravel Data is a trademark of Unravel Data. Other trade names used in this document are the properties of their respective owners.

Contacts

Jordan Tewell, 10Fold
unravel@10fold.com
1-415-666-6066

The post Unravel and Azure Databricks: Monitor, Troubleshoot and Optimize in the Cloud appeared first on Unravel.

AI-DRIVEN DATA OBSERVABILITY + FINOPS FOR AZURE DATABRICKS

Performance. Reliability. Cost-effectiveness.

Unravel’s automated, AI-powered data observability + FinOps platform for Azure Databricks and other modern data stacks provides 360° visibility to allocate costs with granular precision, accurately predict spend, run 50% more workloads at the same budget, launch new apps 3X faster, and reliably hit greater than 99% of SLAs.

Unravel Data Observability + FinOps for Azure Databricks you can:

• Launch new apps 3X faster: End-to-end observability of data-native applications and pipelines. Automatic improvement of performance, cost efficiency, and reliability.
• Run 50% more workloads for same budget: Break down spend and forecast accurately. Optimize apps and platforms by eliminating inefficiencies. Set guardrails and automate governance. Unravel’s AI helps you implement observability and FinOps to ensure you achieve efficiency goals.
• Reduce firefighting time by 99% using AI-enabled troubleshooting: Detect anomalies, drift, skew, missing and incomplete data end-to-end. Integrate with multiple data quality solutions. All in one place.
• Forecast budget with ⨦ 10% accuracy: Accurately anticipate cloud data spending to for more predictable ROI. Unravel helps you accurately forecast spending with granular cost allocation. Purpose-built AI, at job, user and workgroup levels, enables real-time visibility of ongoing usage.

To see Unravel Data for Azure Databricks in action contact: Data experts  | 650 741-3442

The post Unravel Data For Azure Databricks Datasheet appeared first on Unravel.