According to Forrester, accurately forecasting cloud costs remains a significant challenge for 80% of data management professionals. This struggle often stems from a lack of granular visibility, control over usage, and ability to optimize code and infrastructure for cost and performance. Organizations utilizing modern data platforms like Snowflake, BigQuery, and Databricks often face unexpected budget overruns, missed performance SLAs, and inefficient resource allocation.

Transitioning from reactive spending to proactive optimization is crucial for effective cost management in modern data stack environments.

This shift requires a comprehensive approach that encompasses several key strategies:

1. Granular Visibility
Gain comprehensive insights into expenses by unifying fragmented data and breaking down silos, enabling precise financial planning and resource allocation for effective cost control. This unified approach allows teams to identify hidden cost drivers and inefficiencies across the entire data ecosystem.

By consolidating data from various sources, organizations can create a holistic view of their spending patterns, facilitating more accurate budget forecasting and informed decision-making. Additionally, this level of visibility empowers teams to pinpoint opportunities for optimization, such as underutilized resources or redundant processes, leading to significant cost savings over time.

2. ETL Pipeline Optimization
Design cost-effective pipelines from the outset, implementing resource utilization best practices and ongoing performance monitoring to identify and address inefficiencies. This approach involves carefully architecting ETL processes to minimize resource usage while maintaining optimal performance.

By employing advanced performance tuning techniques, such as optimizing query execution plans and leveraging built-in optimizations, organizations can significantly reduce processing time and associated costs. Continuous monitoring of pipeline performance allows for the early detection of bottlenecks or resource-intensive operations, enabling timely adjustments and ensuring sustained efficiency over time.

3. Intelligent Resource Management
Implement intelligent autoscaling to dynamically adjust resources based on workload demands, optimizing costs in real-time while maintaining performance. Efficiently manage data lake and compute resources to minimize unnecessary expenses during scaling. This approach allows organizations to provision automatically and de-provision resources as needed, ensuring optimal utilization and cost-efficiency.

By setting appropriate scaling policies and thresholds, you can avoid over-provisioning during periods of low demand and ensure sufficient capacity during peak usage times. Additionally, separating storage and compute resources enables more granular control over costs, allowing you to scale each component independently based on specific requirements.

4. FinOps Culture
Foster collaboration between data and finance teams, implementing cost allocation strategies like tagging and chargeback mechanisms to attribute expenses to specific projects or teams accurately. This approach creates a shared responsibility for cloud costs and promotes organizational transparency.

By establishing clear communication channels and regular meetings between technical and financial stakeholders, teams can align their efforts to optimize resource utilization and spending. A robust tagging system also allows for detailed cost breakdowns, enabling more informed decision-making and budget allocation based on actual usage patterns.

5. Advanced Forecasting
Develop sophisticated forecasting techniques and flexible budgeting strategies using historical data and AI-driven analytics to accurately predict future costs and create adaptive budgets that accommodate changing business needs. Organizations can identify trends and seasonal variations that impact costs by analyzing past usage patterns and performance metrics.

This data-driven approach enables more precise resource allocation and helps teams anticipate potential cost spikes, allowing for proactive adjustments to prevent budget overruns. Additionally, implementing AI-powered forecasting models can provide real-time insights and recommendations, enabling continuous optimization of environments as workloads and business requirements evolve.

Mastering these strategies can help you transform your approach to cost management from reactive to proactive, ensuring you maximize the value of your cloud investments while maintaining financial control.

To learn more about implementing these cost management strategies in your modern data environment, join our upcoming webinar series, “Controlling Cloud Costs.” This ten-part series will explore each aspect of effective cost management, providing actionable insights and best practices to gain control over your data platform costs.

Register for Controlling Databricks Cloud Cost webinars.

Register for Controlling Snowflake Cloud Cost webinars.

The post Mastering Cost Management: From Reactive Spending to Proactive Optimization appeared first on Unravel.

3 Key Steps to Build a FinOps Ethos in Your Data Engineering Team

In today’s data-driven enterprises, the intersection of fiscal responsibility and technical innovation has never been more critical. As data processing costs continue to scale with business growth, building a FinOps culture within your Data Engineering team isn’t just about cost control, it’s about creating a mindset that views cost optimization as an integral part of technical excellence.

Unravel’s ‘actionability for everyone’ approach has enabled executive customers to adopt three transformative steps to embed FinOps principles into their Data Engineering team’s DNA, ensuring that cost awareness becomes as natural as code quality or data accuracy. In this article, we walk you through how executives can get the cost rating of their workloads from Uncontrolled to Optimized with clear, guided actionable pathways.

Step 1. Democratize cost visibility

The foundation of any successful FinOps implementation begins with transparency. However, raw cost data alone isn’t enough-it needs to be contextualized and actionable.

Breaking Down the Cost Silos

• Unravel provides real-time cost attribution dashboards to map cloud spending to specific business units, teams, projects, and data pipelines.

• The custom views allow different stakeholders, from engineers to executives to discover top actions to control cost spends.
• The ability to track key metrics like cost/savings per job/query, time-to-value, and wasted costs due to idleness, wait times, etc., transforms cost data from a passive reporting tool to an active management instrument.

Provide tools for cost decision making

Modern data teams need to understand how their architectural and implementation choices affect the bottom line. Unravel provides visualizations and insights to guide these teams to implement:

• Pre-deployment cost impact assessments for new data pipelines (for example, what is the cost impact of migrating this workload from All-purpose compute to Job compute?).
• What-if analysis tools for infrastructure changes (for example, will changing from the current instance types to recommended instance types affect performance if I save on cost?).
• Historical trend analysis to identify cost patterns, budget overrun costs, cost wasted due to optimizations neglected by their teams, etc.

Step 2. Embed cost intelligence into the development and operations lifecycles

The next evolution in FinOps maturity comes from making cost optimization an integral part of the development process, not a post-deployment consideration. Executives should consider leveraging specialized AI agents across their technology stack that helps to boost productivity and free up their teams’ time to focus on innovation. Unravel provides a suite of AI-agents driven features that foster cost ethos in the organization and maximizes operational excellence with auto-fix capabilities.

Automated Optimization Lifecycle
Unravel helps you establish a systematic approach to cost optimization with automated AI-agentic workflows to help your teams operationalize recommendations while getting a huge productivity boost. Below are some ways to drive automation with Unravel agents:

• Implement automated fix suggestions for most code and configuration inefficiencies
•  Assign auto-fix actions to AI agents for effortless adoption or route them to a human reviewer for approval
• Configure automated rollback capabilities for changes if unintended performance or cost degradation is detected

Push Down Cost Consciousness To Developers 

• Automated code reviews that flag potential cost inefficiencies
• Smart cost savings recommendations based on historical usage patterns
• Allow developers to see the impact of their change on future runs

Define Measurable Success Metrics
Executives can track the effectiveness of FinOps awareness and culture using Unravel through:

• Cost efficiency improvements over time (WoW, MoM, YTD)
• Team engagement and rate of adoption with Accountability dashboards
• Time-to-resolution for code and configuration changes

Step 3. Create a self-sustaining FinOps culture

The final and most crucial step is transforming FinOps from an initiative into a cultural cornerstone of your data engineering practice.

Operationalize with AI agents

FinOps AI Agent

Implement timely alerting systems that help to drive value-oriented decisions for cost optimization and governance. Unravel provides email, Slack, Teams integrations for ensuring all necessary stakeholders get timely notifications and insights into opportunities and risks.

DataOps AI Agent

• Pipeline optimization suggestions for better resource utilization and mitigate SLA risks
• Job signature level cost and savings impact analysis to help with prioritization of recommendations
• Intelligent workload migration recommendations

Data Engineering AI Agent

• Storage tier optimization recommendations to avoid wasted costs due to cold tables
• Partition strategy optimization for cost-effective querying
• Avoid recurring failures and bottlenecks due to inefficiencies not acted upon for several weeks

Continuous Evolution

Finally, it is extremely important to foster and track the momentum of FinOps growth by:

• Regularly performing FinOps retrospectives with wider teams
• Revisiting which Business units and Cost Centers are contributing to wasted costs, neglected cost due to unadopted recommendations and budget overruns despite timely alerting

The path forward

Building a FinOps ethos in your Data Engineering team is a journey that requires commitment, tools, and cultural change. By following the above three key steps – democratizing cost visibility, embedding cost intelligence, and creating a self-sustaining culture – you can transform how your team thinks about and manages cloud costs.

The most successful implementations don’t treat FinOps as a separate discipline but rather as an integral part of technical excellence. When cost optimization becomes as natural as writing tests or documenting code, you have achieved true FinOps maturity. Unravel provides a comprehensive set of features and tools to aid your teams in accelerating FinOps best practices throughout the organization.

Remember, the goal isn’t just to reduce costs – it is to maximize the value derived from every dollar spent on your infrastructure. This mindset shift, combined with the right tools and processes, will position your data engineering team for sustainable growth and success in an increasingly cost-conscious technology landscape.

To learn more on how Unravel can help, contact us or request a demo.

The post Building a FinOps Ethos appeared first on Unravel.

Mastering BigQuery Cost Management and FinOps: A Comprehensive Checklist

Effective cost management becomes crucial as organizations increasingly rely on Google BigQuery for their data warehousing and analytics needs. This checklist delves into the intricacies of cost management and FinOps for BigQuery, exploring strategies to inform, govern, and optimize usage while taking a holistic approach that considers queries, datasets, infrastructure, and more.

While this checklist is comprehensive and very impactful when implemented fully, it can also be overwhelming to implement with limited staffing and resources. AI-driven insights and automation can solve this problem and are also explored at the bottom of this guide.

Understanding Cost Management for BigQuery

BigQuery’s pricing model is primarily based on data storage and query processing. While this model offers flexibility, it also requires careful management to ensure costs align with business value. Effective cost management for BigQuery is about more than reducing expenses—it’s also about optimizing spend, ensuring efficient resource utilization, and aligning costs with business outcomes. This comprehensive approach falls under the umbrella of FinOps (Financial Operations).

The Holistic Approach: Key Areas to Consider

1. Query Optimization

Are queries optimized? Efficient queries are fundamental to cost-effective BigQuery usage:

Query Structure: Write efficient SQL queries that minimize data scanned.
Partitioning and Clustering: Implement appropriate table partitioning and clustering strategies to reduce query costs.
Materialized Views: Use materialized views for frequently accessed or complex query results.
Query Caching: Leverage BigQuery’s query cache to avoid redundant processing.

2. Dataset Management

Are datasets managed correctly? Proper dataset management is crucial for controlling costs:

Data Lifecycle Management: Implement policies for data retention and expiration to manage storage costs.
Table Expiration: Set up automatic table expiration for temporary or test datasets.
Data Compression: Use appropriate compression methods to reduce storage costs.
Data Skew: Address data skew issues to prevent performance bottlenecks and unnecessary resource consumption.

3. Infrastructure Optimization

Is infrastructure optimized? While BigQuery is a managed service, there are still infrastructure considerations:

Slot Reservations: Evaluate and optimize slot reservations for predictable workloads.
Flat-Rate Pricing: Consider flat-rate pricing for high-volume, consistent usage patterns.
Multi-Region Setup: Balance data residency requirements with cost implications of multi-region setups.

4. Access and Governance

Are the right policies and governance in place? Proper access controls and governance are essential for cost management:

IAM Roles: Implement least privilege access using Google Cloud IAM roles.
Resource Hierarchies: Utilize resource hierarchies (organizations, folders, projects) for effective cost allocation.
VPC Service Controls: Implement VPC Service Controls to manage data access and potential egress costs.

Implementing FinOps Practices

To master cost management for BiqQuery, consider these FinOps practices:

1. Visibility and Reporting

Implement comprehensive labeling strategies for resources.
Create custom dashboards in Google Cloud Console or Data Studio for cost visualization.
Set up budget alerts and export detailed billing data for analysis.

2. Optimization

Regularly review and optimize queries based on BigQuery’s query explanation and job statistics.
Implement automated processes to identify and optimize high-cost queries.
Foster a culture of cost awareness among data analysts and engineers.

3. Governance

Establish clear policies for dataset creation, query execution, and resource provisioning.
Implement approval workflows for high-cost operations or large-scale data imports.
Create and enforce organizational policies to prevent costly misconfigurations.

Setting Up Guardrails

Implementing guardrails is crucial to prevent unexpected costs:

Query Limits: Set daily query limit quotas at the project or user level.
Cost Controls: Implement custom cost controls using Cloud Functions and the BigQuery API.
Data Access Controls: Use column-level and row-level security to restrict access to sensitive or high-volume data.
Budgets and Alerts: Set up project-level budgets and alerts in Google Cloud Console.

The Need for Automated Observability and FinOps Solutions

Given the scale and complexity of modern data operations, automated solutions can significantly enhance cost management efforts. Automated observability and FinOps solutions can provide the following:

Real-time cost visibility across your entire BigQuery environment.
Automated recommendations for query optimization and cost reduction.
Anomaly detection to quickly identify unusual spending patterns.
Predictive analytics to forecast future costs and resource needs.

These solutions can offer insights that would be difficult or impossible to obtain manually, helping you make data-driven decisions about your BigQuery usage and costs.

BigQuery-Specific Cost Optimization Techniques

Avoid SELECT: Instead, specify only the columns you need to reduce data processed.
Use Approximate Aggregation Functions: For large-scale aggregations where precision isn’t critical, use approximate functions like APPROX_COUNT_DISTINCT().
Optimize JOIN Operations: Ensure the larger table is on the left side of the JOIN to potentially reduce shuffle and processing time.
Leverage BigQuery ML: Use BigQuery ML for in-database machine learning to avoid data movement costs.
Use Scripting: Utilize BigQuery scripting to perform complex operations without multiple query executions.

Conclusion

Effective BigQuery cost management and FinOps require a holistic approach that considers all aspects of your data operations. By optimizing queries, managing datasets efficiently, leveraging appropriate pricing models, and implementing robust FinOps practices, you can ensure that your BigQuery investment delivers maximum value to your organization.

Remember, the goal isn’t just to reduce costs, but to optimize spend and align it with business objectives. With the right strategies and tools in place, you can transform cost management from a challenge into a competitive advantage, enabling your organization to make the most of BigQuery’s powerful capabilities while maintaining control over expenses.

To learn more about how Unravel can help with BigQuery cost management, request a health check report, view a self-guided product tour, or request a demo.

The post BigQuery Cost Management appeared first on Unravel.

Mastering Databricks Cost Management and FinOps: A Comprehensive Checklist

In the era of big data and cloud computing, organizations increasingly turn to platforms like Databricks to handle their data processing and analytics needs. However, with great power comes great responsibility – and, in this case, the responsibility of managing costs effectively.

This checklist dives deep into cost management and FinOps for Databricks, exploring how to inform, govern, and optimize your usage while taking a holistic approach that considers code, configurations, datasets, and infrastructure.

While this checklist is comprehensive and very impactful when implemented fully, it can also be overwhelming to implement with limited staffing and resources. AI-driven insights and automation can solve this problem and are also explored at the bottom of this guide.

Understanding Databricks Cost Management

Before we delve into strategies for optimization, it’s crucial to understand that Databricks cost management isn’t just about reducing expenses. It’s about gaining visibility into where your spend is going, ensuring resources are being used efficiently, and aligning costs with business value. This comprehensive approach is often referred to as FinOps (Financial Operations).

The Holistic Approach: Key Areas to Consider

1. Code Optimization

Is code optimized? Efficient code is the foundation of cost-effective Databricks usage. Consider the following:

Query Optimization: Ensure your Spark SQL queries are optimized for performance. Use explain plans to understand query execution and identify bottlenecks.
Proper Data Partitioning: Implement effective partitioning strategies to minimize data scans and improve query performance.
Caching Strategies: Utilize Databricks’ caching mechanisms judiciously to reduce redundant computations.

2. Configuration Management

Are configurations managed appropriately? Proper configuration can significantly impact costs:

Cluster Sizing: Right-size your clusters based on workload requirements. Avoid over-provisioning resources.
Autoscaling: Implement autoscaling to adjust cluster size based on demand dynamically.
Instance Selection: Choose the appropriate instance types for your workloads, considering both performance and cost.

3. Dataset Management

Are datasets managed correctly? Efficient data management is crucial for controlling costs:

Data Lifecycle Management: Implement policies for data retention and archiving to avoid unnecessary storage costs.
Data Format Optimization: Use efficient file formats like Parquet or ORC to reduce storage and improve query performance.
Data Skew Handling: Address data skew issues to prevent performance bottlenecks and unnecessary resource consumption.

4. Infrastructure Optimization

Is infrastructure optimized? Optimize your underlying infrastructure for cost-efficiency:

Storage Tiering: Utilize appropriate storage tiers (e.g., DBFS, S3, Azure Blob Storage) based on data access patterns.
Spot Instances: Leverage spot instances for non-critical workloads to reduce costs.
Reserved Instances: Consider purchasing reserved instances for predictable, long-running workloads.

Implementing FinOps Practices

To truly master Databricks cost management, implement these FinOps practices:

1. Visibility and Reporting

Implement comprehensive cost allocation and tagging strategies.
Create dashboards to visualize spend across different dimensions (teams, projects, environments).
Set up alerts for unusual spending patterns or budget overruns.

2. Optimization

Regularly review and optimize resource usage based on actual consumption patterns.
Implement automated policies for shutting down idle clusters.
Encourage a culture of cost awareness among data engineers and analysts.

3. Governance

Establish clear policies and guidelines for resource provisioning and usage.
Implement role-based access control (RBAC) to ensure appropriate resource access.
Create approval workflows for high-cost operations or resource requests.

Setting Up Guardrails

Guardrails are essential for preventing cost overruns and ensuring responsible usage:

Budget Thresholds: Set up budget alerts at various thresholds (e.g., 50%, 75%, 90% of budget).
Usage Quotas: Implement quotas for compute hours, storage, or other resources at the user or team level.
Automated Policies: Use Databricks’ Policy Engine to enforce cost-saving measures automatically.
Cost Centers: Implement chargeback or showback models to make teams accountable for their spend.

The Need for Automated Observability and FinOps Solutions

While manual oversight is important, the scale and complexity of modern data operations often necessitate automated solutions. Tools like Unravel can provide:

Real-time cost visibility across your entire Databricks environment.
Automated recommendations for cost optimization.
Anomaly detection to identify unusual spending patterns quickly.
Predictive analytics to forecast future costs and resource needs.

These solutions can significantly enhance your ability to manage costs effectively, providing insights that would be difficult or impossible to obtain manually.

Conclusion

Effective cost management and FinOps for Databricks require a holistic approach considering all aspects of your data operations. By optimizing code, configurations, datasets, and infrastructure, and implementing robust FinOps practices, you can ensure that your Databricks investment delivers maximum value to your organization. Remember, the goal isn’t just to reduce costs, but to optimize spend and align it with business objectives. With the right strategies and tools in place, you can turn cost management from a challenge into a competitive advantage.

To learn more about how Unravel can help with Databricks cost management, request a health check report, view a self-guided product tour, or request a demo.

The post Databricks Cost Management appeared first on Unravel.

The Complexities of Code Optimization in BigQuery: Problems, Challenges and Solutions

Google BigQuery offers a powerful, serverless data warehouse solution that can handle massive datasets with ease. However, this power comes with its own set of challenges, particularly when it comes to code optimization.

This blog post delves into the complexities of code optimization in BigQuery, the difficulties in diagnosing and resolving issues, and how automated solutions can simplify this process.

The BigQuery Code Optimization Puzzle

1. Query Performance and Cost Management

Problem: In BigQuery, query performance and cost are intimately linked. Inefficient queries can not only be slow but also extremely expensive, especially when dealing with large datasets.

Diagnosis Challenge: Identifying the root cause of a poorly performing and costly query is complex. Is it due to inefficient JOIN operations, suboptimal table structures, or simply the sheer volume of data being processed? BigQuery provides query explanations, but interpreting these for complex queries and understanding their cost implications requires significant expertise.

Resolution Difficulty: Optimizing BigQuery queries often involves a delicate balance between performance and cost. Techniques like denormalizing data might improve query speed but increase storage costs. Each optimization needs to be carefully evaluated for its impact on both performance and billing, which can be a time-consuming and error-prone process.

2. Partitioning and Clustering Challenges

Problem: Improper partitioning and clustering can lead to excessive data scanning, resulting in slow queries and unnecessary costs.

Diagnosis Challenge: The effects of suboptimal partitioning and clustering may not be immediately apparent and can vary depending on query patterns. Identifying whether poor performance is due to partitioning issues, clustering issues, or something else entirely requires analyzing query patterns over time and understanding the intricacies of BigQuery’s architecture.

Resolution Difficulty: Changing partitioning or clustering strategies is not a trivial operation, especially for large tables. It requires careful planning and can temporarily impact query performance during the restructuring process. Determining the optimal strategy often requires extensive A/B testing and monitoring across various query types and data sizes.

3. Nested and Repeated Fields Complexity

Problem: While BigQuery’s support for nested and repeated fields offers flexibility, it can lead to complex queries that are difficult to optimize and prone to performance issues.

Diagnosis Challenge: Understanding the performance characteristics of queries involving nested and repeated fields is like solving a multidimensional puzzle. The query explanation may not provide clear insights into how these fields are being processed, making it difficult to identify bottlenecks.

Resolution Difficulty: Optimizing queries with nested and repeated fields often requires restructuring the data model or rewriting queries in non-intuitive ways. This process can be time-consuming and may require significant changes to ETL processes and downstream analytics.

4. UDF and Stored Procedure Performance

Problem: User-Defined Functions (UDFs) and stored procedures in BigQuery can lead to unexpected performance issues if not implemented carefully.

Diagnosis Challenge: The impact of UDFs and stored procedures on query performance isn’t always clear from the query explanation. Identifying whether these are the source of performance issues requires careful analysis and benchmarking.

Resolution Difficulty: Optimizing UDFs and stored procedures often involves rewriting them from scratch or finding ways to eliminate them altogether. This can be a complex process, especially if these functions are widely used across your BigQuery projects.

Manual Optimization Struggle

Traditionally, addressing these challenges involves a cycle of:

1. Manually analyzing query explanations and job statistics
2. Conducting time-consuming A/B tests with different query structures and table designs
3. Carefully monitoring the impact of changes on both performance and cost
4. Continuously adjusting as data volumes and query patterns evolve

This process is not only time-consuming but also requires deep expertise in BigQuery’s architecture, SQL optimization techniques, and cost management strategies. Even then, optimizations that work today might become inefficient as your data and usage patterns change.

Harnessing Automation for BigQuery Optimization

Given the complexities and ongoing nature of these challenges, many organizations are turning to automated solutions to streamline their BigQuery optimization efforts. Tools like Unravel can help by:

Continuous Performance and Cost Monitoring: Automatically tracking query performance, resource utilization, and cost metrics across your entire BigQuery environment.

Intelligent Query Analysis: Using machine learning algorithms to identify patterns and anomalies in query performance and cost that might be missed by manual analysis.

Root Cause Identification: Quickly pinpointing the source of performance issues, whether they’re related to query structure, data distribution, or BigQuery-specific features like partitioning and clustering.

Optimization Recommendations: Providing actionable suggestions for query rewrites, partitioning and clustering strategies, and cost-saving measures.

Impact Prediction: Estimating the potential performance and cost impacts of suggested changes before you implement them.

Automated Policy Enforcement: Helping enforce best practices and cost controls automatically across your BigQuery projects.

By leveraging such automated solutions, data teams can focus their expertise on deriving insights from data while ensuring their BigQuery environment remains optimized and cost-effective. Instead of spending hours digging through query explanations and job statistics, teams can quickly identify and resolve issues, or even prevent them from occurring in the first place.

Conclusion

Code optimization in BigQuery is a complex, ongoing challenge that requires continuous attention and expertise. While the problems are multifaceted and the manual diagnosis and resolution process can be daunting, automated solutions offer a path to simplify and streamline these efforts. By leveraging such tools, organizations can more effectively manage their BigQuery performance and costs, improve query efficiency, and allow their data teams to focus on delivering value rather than constantly grappling with optimization challenges.

Remember, whether you’re using manual methods or automated tools, optimization in BigQuery is an ongoing process. As your data volumes grow and query patterns evolve, staying on top of performance and cost management will ensure that your BigQuery implementation continues to deliver the insights your business needs, efficiently and cost-effectively.

To learn more about how Unravel can help with BigQuery code optimization, request a health check report, view a self-guided product tour, or request a demo.

The post BigQuery Code Optimization appeared first on Unravel.

Mastering Snowflake Cost Management and FinOps: A Comprehensive Checklist

Effective cost management becomes paramount as organizations leverage Snowflake’s powerful cloud data platform for their analytics and data warehousing needs. This comprehensive checklist explores the intricacies of cost management and FinOps for Snowflake, delving into strategies to inform, govern, and optimize usage while taking a holistic approach that considers queries, storage, compute resources, and more.

While this checklist is comprehensive and very impactful when implemented fully, it can also be overwhelming to implement with limited staffing and resources. AI-driven insights and automation can solve this problem and are also explored at the bottom of this guide.

Understanding Cost Management for Snowflake

Snowflake’s unique architecture separates compute and storage, offering a flexible pay-as-you-go model. While this provides scalability and performance benefits, it also requires careful management to ensure costs align with business value.

Effective Snowflake cost management is about more than reducing expenses—it’s also about optimizing spend, ensuring efficient resource utilization, and aligning costs with business outcomes. This comprehensive approach falls under the umbrella of FinOps (Financial Operations).

The Holistic Approach: Key Areas to Consider

1. Compute Optimization

Are compute resources allocated efficiently?

Virtual Warehouse Sizing: Right-size your virtual warehouses based on workload requirements.
Auto-suspend and Auto-resume: Leverage Snowflake’s auto-suspend and auto-resume features to minimize idle time.
Query Optimization: Write efficient SQL queries to reduce compute time and costs.
Materialized Views: Use materialized views for frequently accessed or complex query results.
Result Caching: Utilize Snowflake’s result caching to avoid redundant computations.

2. Resource Monitoring and Governance

Are the right policies and governance in place? Proper monitoring and governance are essential for cost management:

Resource Monitors: Set up resource monitors to track and limit credit usage.
Account Usage and Information Schema Views: Utilize these views to gain insights into usage patterns and costs.
Role-Based Access Control (RBAC): Implement RBAC to ensure appropriate resource access and usage.

3. Storage Management

Is storage managed efficiently? While storage is typically a smaller portion of Snowflake costs, it’s still important to manage efficiently:

Data Lifecycle Management: Implement policies for data retention and archiving.
Time Travel and Fail-safe: Optimize usage of Time Travel and Fail-safe features based on your data recovery needs.
Zero-copy Cloning: Leverage zero-copy cloning for testing and development to avoid duplicating storage costs.
Data Compression: Use appropriate compression methods to reduce storage requirements.

4. Data Sharing and Marketplace

Are data sharing and marketplace usage optimized?

Secure Data Sharing: Leverage Snowflake’s secure data sharing to reduce data movement and associated costs.
Marketplace Considerations: Carefully evaluate the costs and benefits of data sets or applications from Snowflake Marketplace.

Implementing FinOps Practices

To master Snowflake cost management, consider these FinOps practices:

1. Visibility and Reporting

Implement comprehensive tagging strategies for resources.
Create custom dashboards using Snowsight or third-party BI tools for cost visualization.
Set up alerts for unusual spending patterns or budget overruns.

2. Optimization

Regularly review and optimize warehouse configurations and query performance.
Implement automated processes to identify and optimize high-cost queries or inefficient warehouses.
Foster a culture of cost awareness among data analysts, engineers, and scientists.

3. Governance

Establish clear policies for warehouse creation, data ingestion, and resource provisioning.
Implement approval workflows for high-cost operations or large-scale data imports.
Create and enforce organizational policies to prevent costly misconfigurations.

Setting Up Guardrails

Implementing guardrails is crucial to prevent unexpected costs:

Resource Monitors: Set up resource monitors with actions (suspend or notify) when thresholds are reached.
Warehouse Size Limits: Establish policies on maximum warehouse sizes for different user groups.
Query Timeouts: Configure appropriate query timeouts to prevent runaway queries.
Data Retention Policies: Implement automated data retention and archiving policies.

The Need for Automated Observability and FinOps Solutions

Given the complexity of modern data operations, automated solutions can significantly enhance cost management efforts. Automated observability and FinOps solutions can provide the following:

Real-time cost visibility across your entire Snowflake environment.
Automated recommendations for query optimization and warehouse right-sizing.
Anomaly detection to quickly identify unusual spending patterns.
Predictive analytics to forecast future costs and resource needs.

These solutions can offer insights that would be difficult or impossible to obtain manually, helping you make data-driven decisions about your Snowflake usage and costs.

Snowflake-Specific Cost Optimization Techniques

Cluster Keys: Properly define cluster keys to improve data clustering and query performance.
Search Optimization: Use search optimization service for tables with frequent point lookup queries.
Multi-cluster Warehouses: Leverage multi-cluster warehouses for concurrency without over-provisioning.
Resource Classes: Utilize resource classes to manage priorities and costs for different workloads.
Snowpipe: Consider Snowpipe for continuous, cost-effective data ingestion.

Conclusion

Effective Snowflake cost management and FinOps require a holistic approach considering all aspects of your data operations. By optimizing compute resources, managing storage efficiently, implementing robust governance, and leveraging Snowflake-specific features, you can ensure that your Snowflake investment delivers maximum value to your organization.

Remember, the goal isn’t just to reduce costs, but to optimize spend and align it with business objectives. With the right strategies and tools in place, you can transform cost management from a challenge into a competitive advantage, enabling your organization to make the most of Snowflake’s powerful capabilities while maintaining control over expenses.

By continuously monitoring, optimizing, and governing your Snowflake usage, you can achieve a balance between performance, flexibility, and cost-efficiency, ultimately driving better business outcomes through data-driven decision-making.

To learn more about how Unravel can help optimize your Snowflake cost, request a health check report, view a self-guided product tour, or request a personalized demo.

The post Snowflake Cost Management appeared first on Unravel.

The Complexities of Code Optimization in Databricks: Problems, Challenges and Solutions

Databricks, with its unified analytics platform, offers powerful capabilities for big data processing and machine learning. However, with great power comes great responsibility – and in this case, the responsibility of efficient code optimization.

This blog post explores the complexities of code optimization in Databricks across SQL, Python, and Scala, the difficulties in diagnosing and resolving issues, and how automated solutions can simplify this process.

The Databricks Code Optimization Puzzle

1. Spark SQL Optimization Challenges

Problem: Inefficient Spark SQL queries can lead to excessive shuffling, out-of-memory errors, and slow execution times.

Diagnosis Challenge: Identifying the root cause of a slow Spark SQL query is complex. Is it due to poor join conditions, suboptimal partitioning, or inefficient use of Spark’s catalyst optimizer? The Spark UI provides a wealth of information, but parsing through stages, tasks, and shuffles to pinpoint the exact issue requires deep expertise and time.

Resolution Difficulty: Optimizing Spark SQL often involves a delicate balance. Techniques like broadcast joins might work well for small-to-medium datasets but fail spectacularly for large ones. Each optimization technique needs to be carefully tested across various data scales, which is time-consuming and can lead to performance regressions if not done meticulously.

2. Python UDF Performance Issues

Problem: While Python UDFs (User-Defined Functions) offer flexibility, they can be a major performance bottleneck due to serialization overhead and lack of Spark’s optimizations.

Diagnosis Challenge: The impact of Python UDFs isn’t always immediately apparent. A UDF that works well on a small dataset might become a significant bottleneck as data volumes grow. Identifying which UDFs are causing issues and why requires careful profiling and analysis of Spark jobs.

Resolution Difficulty: Optimizing Python UDFs often involves rewriting them in Scala or using Pandas UDFs, which requires a different skill set. Balancing between code readability, maintainability, and performance becomes a significant challenge, especially in teams with varying levels of Spark expertise.

3. Scala Code Complexity

Problem: While Scala can offer performance benefits, complex Scala code can lead to issues like object serialization problems, garbage collection pauses, and difficulty in code maintenance.

Diagnosis Challenge: Scala’s powerful features, like lazy evaluation and implicits, can make it difficult to trace the execution flow and identify performance bottlenecks. Issues like serialization problems might only appear in production environments, making them particularly challenging to diagnose.

Resolution Difficulty: Optimizing Scala code often requires a deep understanding of both Scala and the internals of Spark. Solutions might involve changing fundamental aspects of the code structure, which can be risky and time-consuming. Balancing between idiomatic Scala and Spark-friendly code is an ongoing challenge.

4. Memory Management Across Languages

Problem: Inefficient memory management, particularly in long-running Spark applications, can lead to out-of-memory errors or degraded performance over time.

Diagnosis Challenge: Memory issues in Databricks can be particularly elusive. Is the problem in the JVM heap, off-heap memory, or perhaps in Python’s memory management? Understanding the interplay between Spark’s memory management and the specifics of SQL, Python, and Scala requires expertise in all these areas.

Resolution Difficulty: Resolving memory issues often involves a combination of code optimization, configuration tuning, and sometimes fundamental architectural changes. This process can be lengthy and may require multiple iterations of testing in production-like environments.

The Manual Optimization Struggle

Traditionally, addressing these challenges involves a cycle of:

1. Manually analyzing Spark UI, logs, and metrics
2. Conducting time-consuming performance tests across various data scales
3. Carefully refactoring code and tuning configurations
4. Monitoring the impact of changes across different workloads and data sizes
5. Rinse and repeat

This process is not only time-consuming but also requires a rare combination of skills across SQL, Python, Scala, and Spark internals. Even for experts, keeping up with the latest best practices and Databricks features is an ongoing challenge.

Leveraging Automation for Databricks Optimization

Given the complexities and ongoing nature of these challenges, many organizations are turning to automated solutions to streamline their Databricks optimization efforts. Tools like Unravel can help by:

1. Cross-Language Performance Monitoring: Automatically tracking performance metrics across SQL, Python, and Scala code in a unified manner.

2. Intelligent Bottleneck Detection: Using machine learning to identify performance bottlenecks, whether they’re in SQL queries, Python UDFs, or Scala code.

3. Root Cause Analysis: Quickly pinpointing the source of performance issues, whether they’re related to code structure, data skew, or resource allocation.

4. Code-Aware Optimization Recommendations: Providing language-specific suggestions for code improvements, such as replacing Python UDFs with Pandas UDFs or optimizing Scala serialization.

5. Predictive Performance Modeling: Estimating the impact of code changes across different data scales before deployment.

6. Automated Tuning: In some cases, automatically adjusting Spark configurations based on workload patterns and performance goals.

By leveraging such automated solutions, data teams can focus their expertise on building innovative data products while ensuring their Databricks environment remains optimized and cost-effective. Instead of spending hours digging through Spark UIs and log files, teams can quickly identify and resolve issues, or even prevent them from occurring in the first place.

Conclusion

Code optimization in Databricks is a multifaceted challenge that spans across SQL, Python, and Scala. While the problems are complex and the manual diagnosis and resolution process can be daunting, automated solutions offer a path to simplify and streamline these efforts. By leveraging such tools, organizations can more effectively manage their Databricks performance, improve job reliability, and allow their data teams to focus on delivering value rather than constantly battling optimization challenges.

Remember, whether you’re using manual methods or automated tools, optimization in Databricks is an ongoing process. As your data volumes grow and processing requirements evolve, staying on top of performance management will ensure that your Databricks implementation continues to deliver the insights and data products your business needs, efficiently and reliably.

To learn more about how Unravel can help with Databricks code optimization, request a health check report, view a self-guided product tour, or request a demo.

The post Databricks Code Optimization appeared first on Unravel.

The Complexities of Code Optimization in Snowflake: Problems, Challenges, and Solutions

In the world of Snowflake data warehousing, code optimization is crucial for managing costs and ensuring efficient resource utilization. However, this process is fraught with challenges that can leave even experienced data teams scratching their heads.

This blog post explores the complexities of code optimization in Snowflake, the difficulties in diagnosing and resolving issues, and how automated solutions can simplify this process.

The Snowflake Code Optimization Puzzle

1. Inefficient JOIN Operations

Problem: Large table joins often lead to excessive data shuffling and prolonged query times, significantly increasing credit consumption.

Diagnosis Challenge: Pinpointing the exact cause of a slow JOIN is like finding a needle in a haystack. Is it due to poor join conditions, lack of proper clustering, or simply the volume of data involved? The query plan might show a large data shuffle, but understanding why it’s happening and how to fix it requires deep expertise and time-consuming investigation.

Resolution Difficulty: Optimizing JOINs often involves a trial-and-error process. You might need to experiment with different join types, adjust clustering keys, or even consider restructuring your data model. Each change requires careful testing to ensure it doesn’t negatively impact other queries or downstream processes.

2. Suboptimal Data Clustering

Problem: Poor choices in clustering keys lead to inefficient data access patterns, increasing query times and, consequently, costs.

Diagnosis Challenge: The effects of suboptimal clustering are often subtle and vary depending on query patterns. A clustering key that works well for one set of queries might be terrible for another. Identifying the root cause requires analyzing a wide range of queries over time, a task that’s both time-consuming and complex.

Resolution Difficulty: Changing clustering keys is not a trivial operation. It requires careful planning, as it can temporarily increase storage costs and impact query performance during the re-clustering process. Determining the optimal clustering strategy often requires extensive A/B testing and monitoring.

3. Inefficient Use of UDFs

Problem: While powerful, User-Defined Functions (UDFs) can lead to unexpected performance issues and increased credit consumption if not used correctly.

Diagnosis Challenge: UDFs are often black boxes from a performance perspective. Traditional query profiling tools might show that a UDF is slow, but they can’t peer inside to identify why. This opacity makes it extremely difficult to pinpoint the root cause of UDF-related performance issues.

Resolution Difficulty: Optimizing UDFs often requires rewriting them from scratch, which can be time-consuming and risky. You might need to balance between UDF performance and maintainability, and in some cases, completely rethink your approach to the problem the UDF was solving.

4. Complex, Monolithic Queries

Problem: Large, complex queries can be incredibly difficult to optimize and may not leverage Snowflake’s MPP architecture effectively, leading to increased execution times and costs.

Diagnosis Challenge: Understanding the performance characteristics of a complex query is like solving a multidimensional puzzle. Each part of the query interacts with others in ways that can be hard to predict. Traditional query planners may struggle to provide useful insights for such queries.

Resolution Difficulty: Optimizing complex queries often requires breaking them down into smaller, more manageable parts. This process can be incredibly time-consuming and may require significant refactoring of not just the query, but also the surrounding ETL processes and downstream dependencies.

The Manual Optimization Struggle

Traditionally, addressing these challenges involves a cycle of:

1. Manually sifting through query histories and execution plans
2. Conducting time-consuming A/B tests
3. Carefully monitoring the impact of changes across various workloads
4. Rinse and repeat

This process is not only time-consuming but also prone to human error. It requires deep expertise in Snowflake’s architecture, SQL optimization techniques, and your specific data model. Even then, optimizations that work today might become inefficient as your data volumes and query patterns evolve.

The Power of Automation in Snowflake Optimization

Given the complexities and ongoing nature of these challenges, many organizations are turning to automated solutions to simplify and streamline their Snowflake optimization efforts. Tools like Unravel can help by:

Continuous Monitoring: Automatically tracking query performance, resource utilization, and cost metrics across your entire Snowflake environment.

Intelligent Analysis: Using machine learning algorithms to identify patterns and anomalies that might be missed by manual analysis.

Root Cause Identification: Quickly pinpointing the source of performance issues, whether they’re related to query structure, data distribution, or resource allocation.

Optimization Recommendations: Providing actionable suggestions for query rewrites, clustering key changes, and resource allocation adjustments.

Impact Prediction: Estimating the potential performance and cost impacts of suggested changes before you implement them.

Automated Tuning: In some cases, automatically applying optimizations based on predefined rules and thresholds.

By leveraging such automated solutions, data teams can focus their expertise on higher-value tasks while ensuring their Snowflake environment remains optimized and cost-effective. Instead of spending hours digging through query plans and execution logs, teams can quickly identify and resolve issues, or even prevent them from occurring in the first place.

Conclusion

Code optimization in Snowflake is a complex, ongoing challenge that requires continuous attention and expertise. While the problems are multifaceted and the manual diagnosis and resolution process can be daunting, automated solutions offer a path to simplify and streamline these efforts. By leveraging such tools, organizations can more effectively manage their Snowflake costs, improve query performance, and allow their data teams to focus on delivering value rather than constantly fighting optimization battles.

Remember, whether you’re using manual methods or automated tools, optimization is an ongoing process. As your data volumes grow and query patterns evolve, staying on top of performance and cost management will ensure that your Snowflake implementation continues to deliver the insights your business needs, efficiently and cost-effectively.

To learn more about how Unravel can help optimize your code in Snowflake, request a health check report, view a self-guided product tour, or request a demo.

The post Snowflake Code Optimization appeared first on Unravel.

Navigating the Maze of Configuration Management in Modern Data Platforms: Problems, Challenges and Solutions

In the world of big data, configuration management is often the unsung hero of platform performance and cost-efficiency. Whether you’re working with Snowflake, Databricks, BigQuery, or any other modern data platform, effective configuration management can mean the difference between a sluggish, expensive system and a finely-tuned, cost-effective one.

This blog post explores the complexities of configuration management in data platforms, the challenges in optimizing these settings, and how automated solutions can simplify this critical task.

The Configuration Conundrum

1. Cluster and Warehouse Sizing

Problem: Improper sizing of compute resources (like Databricks clusters or Snowflake warehouses) can lead to either performance bottlenecks or unnecessary costs.

Diagnosis Challenge: Determining the right size for your compute resources is not straightforward. It depends on workload patterns, data volumes, and query complexity, all of which can vary over time. Identifying whether performance issues or high costs are due to improper sizing requires analyzing usage patterns across multiple dimensions.

Resolution Difficulty: Adjusting resource sizes often involves a trial-and-error process. Too small, and you risk poor performance; too large, and you’re wasting money. The impact of changes may not be immediately apparent and can affect different workloads in unexpected ways.

2. Caching and Performance Optimization Settings

Problem: Suboptimal caching strategies and performance settings can lead to repeated computations and slow query performance.

Diagnosis Challenge: The effectiveness of caching and other performance optimizations can be highly dependent on specific workload characteristics. Identifying whether poor performance is due to cache misses, inappropriate caching strategies, or other factors requires deep analysis of query patterns and platform-specific metrics.

Resolution Difficulty: Tuning caching and performance settings often requires a delicate balance. Aggressive caching might improve performance for some queries while causing staleness issues for others. Each adjustment needs to be carefully evaluated across various workload types.

3. Security and Access Control Configurations

Problem: Overly restrictive security settings can hinder legitimate work, while overly permissive ones can create security vulnerabilities.

Diagnosis Challenge: Identifying the root cause of access issues can be complex, especially in platforms with multi-layered security models. Is a performance problem due to a query issue, or is it because of an overly restrictive security policy?

Resolution Difficulty: Adjusting security configurations requires careful consideration of both security requirements and operational needs. Changes need to be thoroughly tested to ensure they don’t inadvertently create security holes or disrupt critical workflows.

4. Cost Control and Resource Governance

Problem: Without proper cost control measures, data platform expenses can quickly spiral out of control.

Diagnosis Challenge: Understanding the cost implications of various platform features and usage patterns is complex. Is a spike in costs due to inefficient queries, improper resource allocation, or simply increased usage?

Resolution Difficulty: Implementing effective cost control measures often involves setting up complex policies and monitoring systems. It requires balancing cost optimization with the need for performance and flexibility, which can be a challenging trade-off to manage.

The Manual Configuration Management Struggle

Traditionally, managing these configurations involves:

1. Continuously monitoring platform usage, performance metrics, and costs
2. Manually adjusting configurations based on observed patterns
3. Conducting extensive testing to ensure changes don’t negatively impact performance or security
4. Constantly staying updated with platform-specific best practices and new features
5. Repeating this process as workloads and requirements evolve

This approach is not only time-consuming but also reactive. By the time an issue is noticed and diagnosed, it may have already impacted performance or inflated costs. Moreover, the complexity of modern data platforms means that the impact of configuration changes can be difficult to predict, leading to a constant cycle of tweaking and re-adjusting.

Embracing Automation in Configuration Management

Given these challenges, many organizations are turning to automated solutions to manage and optimize their data platform configurations. Platforms like Unravel can help by:

Continuous Monitoring: Automatically tracking resource utilization, performance metrics, and costs across all aspects of the data platform.

Intelligent Analysis: Using machine learning to identify patterns and anomalies in platform usage and performance that might indicate configuration issues.

Predictive Optimization: Suggesting configuration changes based on observed usage patterns and predicting their impact before implementation.

Automated Adjustment: In some cases, automatically adjusting configurations within predefined parameters to optimize performance and cost.

Policy Enforcement: Helping to implement and enforce governance policies consistently across the platform.

Cross-Platform Optimization: For organizations using multiple data platforms, providing a unified view and consistent optimization approach across different environments.

By leveraging automated solutions, data teams can shift from a reactive to a proactive configuration management approach. Instead of constantly fighting fires, teams can focus on strategic initiatives while ensuring their data platforms remain optimized, secure, and cost-effective.

Conclusion

Configuration management in modern data platforms is a complex, ongoing challenge that requires continuous attention and expertise. While the problems are multifaceted and the manual management process can be overwhelming, automated solutions offer a path to simplify and streamline these efforts.

By embracing automation in configuration management, organizations can more effectively optimize their data platform performance, enhance security, control costs, and free up their data teams to focus on extracting value from data rather than endlessly tweaking platform settings.

Remember, whether using manual methods or automated tools, effective configuration management is an ongoing process. As your data volumes grow, workloads evolve, and platform features update, staying on top of your configurations will ensure that your data platform continues to meet your business needs efficiently and cost-effectively.

To learn more about how Unravel can help manage and optimize your data platform configurations with Databricks, Snowflake, and BigQuery: request a health check report, view a self-guided product tour, or request a demo.

The post Configuration Management in Modern Data Platforms appeared first on Unravel.

AI Agents for Data Teams

Data is the driving force of the world’s modern economies, but data teams are struggling to meet demand to support generative AI (GenAI), including rapid data volume growth and the increasing complexity of data pipelines. More than 88% of software engineers, data scientists, and SQL analysts surveyed say they are turning to AI for more effective bug-fixing and troubleshooting. And 84% of engineers who use AI said it frees up their time to focus on high-value activities.

AI Agents represent the next wave of AI innovation and have arrived just in time to help data teams make more efficient use of their limited bandwidth to build, operate, and optimize data pipelines and GenAI applications on modern data platforms.

Data Teams Grapple with High Demand for GenAI

A surge in adoption of new technologies such as GenAI is putting tremendous pressure on data teams, leading to broken apps and burnout. In order to support new GenAI products, data teams must deliver more production data pipelines and data apps, faster. The result is that data teams have too much on their plates, the pipelines are too complex, there is not enough time, and not everyone has the deep tech skills required. No surprise that 70% of organizations have difficulty integrating data into AI models and only 48% of AI projects get deployed into production.

Understanding AI Agents

Defining AI Agents

AI agents are software-based systems that gather information, provide recommended actions, initiate and complete tasks in collaboration with or on behalf of humans to achieve a goal. AI agents can act independently, utilizing components like perception and reasoning, provide step-by-step guidance to augment human abilities, or can provide supporting information to support complex human-led tasks. AI agents play a crucial role in automating tasks and simplifying data-driven decision-making, and achieving greater productivity and efficiency.

How AI Agents Work

AI agents operate by leveraging a wide range of data sources and signals, using algorithms and data processing to identify anomalies and actions, then interact with their environment and users to effectively achieve specific goals. AI agents can achieve >90% accuracy, primarily driven by the reliability, volume, and variety of input data and telemetry to which they have access.

Types of Intelligent Agents

• Reactive and proactive agents are two primary categories of intelligent agents.
• Some agents perform work for you, while others help complete tasks with you or provide information to support your work.
• Each type of intelligent agent has distinct characteristics and applications tailored to specific functions, enhancing productivity and efficiency.

AI for Data Driven Organizations

Enhancing Decision Making

AI agents empower teams by improving data support decision-making processes for you, with you, or by you. Examples of how AI agents act on your behalf include reducing toil and handling routine decisions based on AI insights. In various industries, AI agents optimize decision-making and provide recommendations to support your decisions. For complex tasks, AI agents provide supporting information needed to build data pipelines, write SQL queries, and partition data.

Benefits of broader telemetry sources for AI agents

Integrating telemetry from various platforms and systems enhances AI agents’ ability to provide accurate recommendations. Incorporating AI agents into root cause analysis (RCA) systems offers significant benefits. Meta’s AI-based root cause analysis system shows how AI agents enhance tools and applications.

Overcoming Challenges

Enterprises running modern data stacks face common challenges like high costs, slow performance, and impaired productivity. Leveraging AI agents can automate tasks for you, with you, and by you. Unravel customers such as Equifax, Maersk, and Novartis have successfully overcome these challenges using AI.

The Value of AI Agents for Data Teams

Reducing Costs

When implementing AI agents, businesses benefit from optimized data stacks, reducing operational costs significantly. These agents continuously analyze telemetry data, adapting to new information dynamically. Unravel customers have successfully leveraged AI to achieve operational efficiency and cost savings.

Accelerating Performance

Performance is crucial in data analytics, and AI agents play a vital role in enhancing it. By utilizing these agents, enterprise organizations can make well-informed decisions promptly. Unravel customers have experienced accelerated data analytics performance through the implementation of AI technologies.

Improving Productivity

AI agents are instrumental in streamlining processes within businesses, leading to increased productivity levels. By integrating these agents into workflows, companies witness substantial productivity gains. Automation of repetitive tasks by AI agents simplify troubleshooting to boost overall productivity and efficiency.

Future Trends in AI Agents for FinOps, DataOps, and Data Engineering

Faster Innovation with AI Agents

By 2026 conversational AI will reduce agent labor costs by $80 billion. AI agents are advancing, providing accurate recommendations to address more issues automatically. This allows your team to focus on innovation. For example, companies like Meta use AI agents to simplify root cause analysis (RCA) for complex applications.

Accelerated Data Pipelines with AI Agents

Data processing is shifting towards real-time analytics, enabling faster revenue growth. However, this places higher demands on data teams. Equifax leverages AI to serve over 12 million daily requests in near real time.

Improved Data Analytics Efficiency with AI Agents

Data management is the fastest-growing segment of cloud spending. In the cloud, time is money; faster data processing reduces costs. One of the word’s largest logistics companies improved efficiency by up to 70% in just 6 months using Unravel’s AI recommendations.

Empower Your Team with AI Agents

Harnessing the power of AI agents can revolutionize your business operations, enhancing efficiency, decision-making, and customer experiences. Embrace this technology to stay ahead in the competitive landscape and unlock new opportunities for growth and innovation.

Learn more about our FinOps Agent, DataOps Agent, and Data Engineering Agent.

The post AI Agents: Empower Data Teams With Actionability^TM for Transformative Results appeared first on Unravel.

A Recap of the Unravel Data Customer Session at Gartner Data & Analytics Summit 2024

The Gartner Data & Analytics Summit 2024 (“D&A”) in London is a pivotal event for Chief Data Analytics Officers (CDAOs) and data and analytics leaders, drawing together a global community eager to dive deep into the insights, strategies, and frameworks that are shaping the future of data and analytics. With an unprecedented assembly of over 3,000 attendees, spanning 150+ knowledge-packed sessions and joined by 90+ innovative partners, the D&A Summit was designed to catalyze data transformation within organizations, offering attendees unique insights to think big and drive real change.

The Unravel Data customer session, titled “How to Stop Burning Money (or At Least Slow the Burn)”, emerged as a highlight of the D&A Summit, drawing attention to the pressing need for cost-efficient data processing in today’s rapid digital evolution. The session—presented by one of the largest logistics giants globally, including over 100,000 employees and a fleet of 700+ container vessels operating across 130 countries—captivated CDAOs & data and analytics leaders from over 150+ attendees. The audience was from 140+ companies across 30+ industries such as banking, retail, and pharma, spanning 110+ cities across 20+ countries. This compelling turnout underscored the universal relevance and urgency of cost-optimization strategies in data engineering. Watch the highlight reel here.

The session was presented by Peter Rees, Director of GenAI, AI, Data and Integration at Maersk, who garnered unprecedented accolades, including a 170% higher-than-average attendance. Peter Rees’ session was the third highest-rated of all 40 partner sessions. These results reflect the session’s relevance and the invaluable insights shared on revolutionizing the efficiency of data processing pipelines, cost allocation, and optimization techniques.
The Gartner Data & Analytics Summit 2024, and particularly the Unravel Data customer session, brought together organizations striving to align their data engineering costs with the value of their data and analytics. Unravel Data’s innovative approach, showcased through the success of a world-leading logistics company, provides a blueprint for organizations across industries looking to dramatically enhance the speed, productivity, and efficiency of their data processing and AI investments.

We invite you to explore how your organization can benefit from Unravel Data’s groundbreaking platform. Take the first step towards transforming your data processing strategy by scheduling an Unravel Data Health Check. Embark on your journey towards optimal efficiency and cost savings today.

The post How to Stop Burning Money (or at Least Slow the Burn) appeared first on Unravel.

In a recently published report, “Beyond FinOps: Optimizing Your Public Cloud Costs”, Gartner shares a graphic which is adapted from Unravel.

Unravel’s Perspective

How FinOps Helps

Unravel helps organizations adopt a FinOps approach to improve cloud data spending efficiency. FinOps helps organizations address overspend, including under- and over-provisioned cloud services, suboptimal architectures, and inefficient pricing strategies. Infrastructure and operations (I&O) leaders and practitioners can use FinOps principles to optimize cloud service design, configuration, and spending commitments to reduce costs. Data and analytics (D&A) leaders and teams are using FinOps to achieve predictable spend for their cloud data platforms.

Introducing Cloud Financial Management and Governance

Cloud Financial Management (CFM) and Governance empowers organizations to quickly adapt in today’s dynamic landscape, ensuring agility and competitiveness. CFM principles help organizations take advantage of the cloud’s variable consumption model through purpose-built tools tailored for your modern cloud finance needs. A well-defined cloud cost governance model lets cloud users monitor and manage their cloud costs and balance cloud spending vs. performance and end-user experience.

Three Keys to Optimizing Your Modern Data Stack

1. Cloud data platform usage analysis plays a crucial role in cloud financial management by providing insights into usage patterns, cost allocation, and optimization opportunities. By automatically gathering, analyzing, and correlating data from various sources, such as traces, logs, metrics, source code, and configuration files, organizations can identify areas for cost savings and efficiency improvements. Unravel’s purpose-built AI reduces the toil required to manually examine metrics, such as resource utilization, spending trends, and performance benchmarks for modern data stacks such as Databricks, Snowflake, and BigQuery.
2. Cost allocation, showback, and chargeback are essential for effective cloud cost optimization. Organizations need to accurately assign costs to different departments or projects based on their actual resource consumption. This ensures accountability and helps in identifying areas of overspending or underutilization. Automated tools and platforms like Unravel can streamline the cost allocation process for cloud data platforms such as Databricks, Snowflake, and BigQuery, making it easier to track expenses and optimize resource usage.
3. Budget forecasting and management is another critical aspect of cloud financial management. By analyzing historical data and usage trends, organizations can predict future expenses more accurately. This enables them to plan budgets effectively, avoid unexpected costs, and allocate resources efficiently. Implementing robust budget forecasting processes can help organizations achieve greater financial control and optimize their cloud spending.

Next Steps

You now grasp the essence of cloud financial management and governance to optimize cloud spending and your cloud data platform. Start your journey and embrace these concepts to enhance your cloud strategy and drive success. Take charge of your Databricks, Snowflake, and BigQuery optimization today with a free health check.

Gartner, Beyond FinOps: Optimizing Your Public Cloud Costs, By Marco Meinardi, 21 March 2024

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The post Unravel Data was Mentioned in a Graphic source in the 2024 Gartner® Report appeared first on Unravel.

Palo Alto, CA – Apr. 24, 2024 – Unravel Data is proud to announce A.P. Moller – Maersk, a global leader in container logistics, is participating at the upcoming Gartner Data & Analytics Summit. Scheduled from May 13-15, 2024, at ExCeL London, this summit is renowned for gathering visionaries and innovators in the realm of data and analytics.

Peter Rees, Lead Architect at Maersk, together with Mark Sear, Director of Insight Analytics, Data, and Integration at Maersk, will be leading a must-attend session titled “Unravel Data: How to stop burning money (or at least slow the burn).” This session is meticulously designed to address the escalating concern of unpredictable growth in data analytics costs which can significantly hinder the progression of data-driven innovation.

Session Overview:

Data analytics costs are spiraling, and businesses are searching for methodologies to efficiently manage and optimize these expenses without compromising on innovation or operational agility. Maersk, leveraging Unravel Data’s cutting-edge solutions, has pioneered a cost-optimization framework that not only streamlines development and data delivery processes but also aligns with the enterprise’s mission to deliver a more connected, agile, and sustainable future for global logistics.

During the session, attendees will gain exclusive insights into how Maersk has successfully harnessed the power of Unravel Data within its infrastructure to ensure the business remains at the forefront of cost efficiency while bolstering its data-driven decision-making capabilities.

Speaker Highlights:

• Peter Rees, as Maersk’s Lead Architect specializing in Enterprise Data & AI/ML, brings a wealth of knowledge in data mesh and event-driven architectures. His extensive track record in AI strategies and analytics, complemented by an innovative mindset, positions him as a cornerstone in the conversation on bridging data technology with business value.
• Mark Sear elevates the discourse with his profound expertise in digital transformation and business intelligence as Maersk’s Director of Insight Analytics, Data, and Integration. Mark’s academic and professional achievements underscore his commitment to leveraging data for actionable insights, thus fostering strategic business growth and operational efficiencies.

Event Details:

• What: Gartner Data & Analytics Summit
• When: May 13-15, 2024
• Where: ExCeL London, UK
• Session Title: Unravel Data: How to stop burning money (or at least slow the burn)

Unravel Data invites all attendees who are looking to navigate the challenges of data analytics cost growth to join this session. It promises to be an enlightening exploration of practical solutions, real-world applications, and visionary strategies for any organization aiming to optimize their data-driven initiatives and investments.

About A.P. Moller – Maersk:
A.P. Moller – Maersk is an integrated container logistics company working to connect and simplify its customers’ supply chains. As a global leader in shipping services, the company operates in 130 countries and employs over 80,000 people. For further information, visit www.maersk.com.

About Unravel Data:
Unravel’s automated, AI-powered data observability + FinOps provides 360° visibility to allocate costs with granular precision, accurately predict spend, run 50% more workloads at the same budget, launch new apps 3X faster, and reliably hit greater than 99% of SLAs. For further information, visit www.unraveldata.com.

Media Contact:
Keith Alsheimer
CMO, Unravel Data
hello@unraveldata.com

The post Maersk data leaders speaking at Gartner Data & Analytics Summit London appeared first on Unravel.

This article explains the two pricing models—on-demand and capacity—for the compute component of the BigQuery pricing, the challenges of calculating chargeback on compute, and what Unravel can do.

1. On-demand compute pricing: You are charged based on the billed bytes used by your queries. So if the price is $X/TiB and a query uses Y-TiB of billed bytes, you will be billed $(X*Y) for that query.
2. Capacity compute pricing: You buy slots and you are charged based on the number of slots and the time for which slots were made available. 

The following section describes compute pricing in more detail.

Capacity Pricing

To use capacity pricing, you start by creating reservations. You can create one or more reservations inside an admin project. All the costs related to the slots for the reservations will be attributed to the admin project in the bill.

Reservations

While creating a reservation, you need to specify “baseline slots” and “max slots,” in multiples of 100. You will be charged for the baseline slots at the minimum for the duration of the reservation. When all the baseline slots have been utilized by queries running in that reservation, more slots can be made available via autoscaling. Autoscaling happens in multiple of 100 slots. The number of slots available for autoscaling is (max slots – baseline slots). 

You can modify the baseline and max slots after you create the reservation. You can increase the values at any point in time, but you cannot decrease the values within 1 hour of creating a reservation or updating the reservation.

Assignments

After you have created a reservation, to enable the queries in projects to use slots from that reservation, you have to create assignments. You can assign projects, folders ,or organizations to a reservation. Whenever a query is started, it will first try to use the slots from this reservation. You can create or delete assignments at any point in time.

Pricing

BigQuery provides 3 editions —Standard, Enterprise, and Enterprise Plus. These editions have different capabilities and different pricing rates. The rates are defined in terms of slot hours. For example, the rate is $0.04 per slot hour for Standard edition in the US and $0.06 per slot hour for Enterprise edition in the same region.

In capacity pricing, you are charged for the number of slots made available and the time for which slots are made available. Suppose you have a reservation with 100 baseline slots and 500 max slots in the Standard edition. Consider the following usage:

• In the first hour, no queries are running, so the slot requirement is 0.
• In the second hour, there are queries running, but the slot requirement is less than 100.
• In the third hour, more queries are running and the slot requirement is 150.

In the first 2 hours, even though the slot requirement is less than 100, you will still be charged for 100 slots—the baseline slots—for each of the first 2 hours.

In the third hour, we need 50 more slots than the baseline, so autoscaling kicks in to provide more slots. Since autoscaling only scales up or down in multiples of 100, 100 more slots are added. Hence, a total of 200 slots (100 baseline + 100 from autoscaling) are made available in this hour. 

The number of slot-hours from this 3-hour period is 100 + 100 + 200 = 400. With a rate of $0.04 per slot-hour for Standard edition, you will be charged 0.04*400 = $16 for this usage.

Pay-as-you-go

Recall that you can create/delete/update reservations and baseline/max slots whenever you want. Also, you will be charged for just the number of slots made available to you and for the time the slots are made available. This model is called pay-as-you-go, as you are paying for the usage you are using. 

Capacity Commitment

If you expect to use a certain number of slots over a long period of time, you can make a commitment of X slots over a 1-year or 3-year period, for a better rate. You will be charged for those slots for the entire period regardless of whether you use them or not. This model is called capacity commitment.

Consider the following example of capacity commitment. Let’s say you have:

• 1-year commitment of 1600 slots in the Enterprise edition. 
• Created 1 reservation with max size of 1500 slots and baseline of 1000 slots. 
• Hence your autoscaling slots are 1500-1000 = 500.
• Pay-as-you-go price for enterprise edition is $0.06 per slot-hour.
• 1-year commitment price for the enterprise edition is $0.048 per slot hour.

Consider this scenario:

• In the first hour, the requirement is less than 1000 slots.
• In the second hour, the requirement is 1200 slots.
• In the third hour, the requirement is 1800 slots.

In the first hour, the baseline slots of 1000 are made available for the reservation; these slots are available from the commitment slots. Since we have a commitment of 1600 slots, all the 1600 slots are actually available. The 1000 slots are available for the reservation as baseline. The remaining 600 are called idle slots and are also charged. So for the first hour, we are charged for 1600 slots as per commitment price, with a cost of $(1600 * 0.048).

In the second hour, since the requirement is 1200 slots, there is an additional requirement of 200 slots beyond the baseline of 1000 slots. Since 600 idle slots are available from the committed capacity, the additional requirement of 200 slots will come from these idle slots, while the remaining 400 slots will remain idle. Notice that autoscaling was not needed in this case. Before going for autoscaling, BigQuery will try to use idle slots (unless ignore_idle_slots config is set to True for that reservation). So how much are we charged for the second hour? The answer is 1600 slots, since that is what is committed. These 1600 slots are charged as per the commitment price, so the cost for the second hour is $(1600 * 0.048).

In the third hour, the requirement is 1800 slots: the first 1600 slots will come from commitment slots, and the other 200 will now come from autoscaling slots. The 1600 slots will be charged as per 1-year commit pricing, and the 200 slots coming from autoscale slots will be charged as per pay-as-you-go pricing at $0.06/slot-hour in this case. Therefore, the cost for the third hour is $((1600 * 0.048) + (200 * 0.06)).

Notes

Some points to note regarding capacity pricing:

1. The slots are charged with a maximum granularity of 1 second, and the charge is for a minimum of 1 minute.
2. Autoscaling always happens in increments/decrements of 100 slots.
3. Queries running in a reservation automatically use idle slots from other reservations within the same admin project. Unless ignore_idle_slots is set to True for the reservation.
4. The capacity commitment is specific to a region, organization, and edition. The idle slots can’t be shared across regions or editions.
5. Idle slot capacity is not shared between reservations in different admin projects.

The Cost Chargeback Problem

In an organization, typically there are one or more capacity commitments, one or more admin projects, and multiple reservations in these admin projects. GCP provides billing data that gives you the hourly reservation costs for a given admin project, edition, and location combination. However, there are multiple possible ways to map a team to a reservation: a team can be assigned to one reservation, multiple teams can be assigned to the same reservation, or multiple teams can be assigned to multiple reservations.

In any case, it is a tricky task to find out which team or user is contributing how much to your cost. How do you map the different teams and users to the different projects, editions, and locations? And how do you then track the cost incurred by these teams and users? Common chargeback approaches—such as chargeback by accounts, projects, and reservations—simply cannot provide clarity at the user or team level. There is also no direct data source that gives this information from GCP.

Unravel provides cost chargeback at the user and team levels by combining data from different sources such as billing, apps, and tags. The crux of our proprietary approach is providing an accurate cost estimate at the query level. We then associate the query costs with users and teams (via tags) to derive the user-level or team-level chargeback. 

Computing query-level cost estimates involves addressing a number of challenges. Some of the challenges include:

• A query may have different stages where chargeback could be different.
• A query may use slots with commitment pricing or pay-as-you-go pricing.
• Capacity is billed at one minute minimum.
• Autoscaling increments by multiples of 100 slots.
• Chargeback policy for idle resource.

Let’s understand these challenges further with a few scenarios. In all the examples below, we assume that we use the Enterprise edition in the US region, with rates of $0.06/slot-hour for pay-as-you-go, and $0.048/slot-hour for a 1-year commitment.

Scenario 1: Slot-hours billed differently at different stages of a query 

Looking at the total slot-hours for chargeback could be misleading because the slot-hours at different stages of a query may be billed differently.

Consider the following scenario:

• A reservation with a baseline of 0 slot and max 200 slots.
• No capacity commitment.
• Query Q1 runs from 5am to 6am with 200 slots for the whole hour.
• Query Q2 runs from 6am to 8am in 2 stages:
  • In the first stage, from 6am to 7am, it uses 150 slots for the whole hour.
  • In the second stage, from 7am to 8am, it uses 50 slots for the whole hour.

Both queries use the exact same slot-hours total, i.e., 200 slot-hours, and use the same reservation and edition. Hence we may think the chargeback to both queries should be the same.

But if you look closely, the two queries do not incur the same amount of cost.

Q1 uses 200 slots for 1 hour. Given the reservation with a baseline of 0 slot and a max of 200 slots, 200 slots are made available in this hour, and the cost of the query is $(200*0.06) = $12.

In contrast, Q2’s usage is split into 150 slots for the first hour and 50 slots for the second hour. Since slots are autoscaled in increments of 100, to run Q2, 200 slots are made available in the first hour and 100 slots are made available in the second hour. The total slot-hours for Q2 is therefore 300, and the cost is $(300*0.06) = $18.

Summary: The cost chargeback to a query needs to account for how many slots are used in different stages of the query and not just the total slot-hours (or slot-ms) used.

Scenario 2: Capacity commitment pricing vs. pay-as-you-go pricing from autoscaling

At different stages, a query may use slots from capacity commitment or from autoscaling that are charged at the pay-as-you-go price. 

Consider the following scenario:

• 1 reservation with a baseline of 100 slots and max slots as 300 slots.
• 1-year capacity commitment of 100 slots.
• Query Q1 runs from 5am to 6am and uses 300 slots for the whole 1 hour.
• Query Q2 runs from 6am to 8am in 2 stages. It uses 100 slots from 6am to 7am, and uses 200 slots from 7am to 8am.

Once again, the total slot-hours for both queries are the same, i.e., 300 slot hours, and we might chargeback the same cost to both queries.

But if you look closely, the queries do not incur the same amount of cost.

For Q1, 100 slots come from committed capacity and are charged at the 1-year commit price ($0.048/slot-hour), whereas 200 are autoscale slots that are charged at the pay-as-you-go price ($0.06/slot-hour). So the cost of Q1 is  $((100*0.048) + (200*0.06)) = $16.80.

For Q2, from 6-7am, 100 slots come from committed capacity and are charged at the 1-year commit price ($0.048/slot-hour), so the cost for 6-7am is  = $(100*0.048) = $4.80.

From 7-8am, 100 slots from committed capacity are charged at 1-year commit price ($0.048/slot-hour), and the other 100 are autoscale slots charged at pay-as-you-go price ($0.06/slot-hour). So the cost from 7-8am is  = $(100*0.048) + (100*0.06) = $10.80.

Hence the cost between 6-8am (duration when Query-2 is running) is = $4.80 + $10.80 = $15.60.

Summary: The cost chargeback to a query needs to account for whether the slots come from committed capacity or from autoscaling charged at pay-as-you-go price. A query may use both at different stages.

Scenario 3: Minimum slot capacity and increment in autoscaling

A query may be billed for more than the resource it actually needs because of the minimum slot capacity and the minimum increment in autoscaling.

• 1 reservation with a baseline of 0 slots and max slots as 300 slots.
• No capacity commitment.
• Query Q1 uses 50 slots for 10 seconds between 05:00:00 to 05:00:10.
• There is no other query running between 04:59:00 to 05:02:00.

If you were to chargeback by slot-ms, you would say that the query uses 50 slots for 10 seconds, or  500,000 slot-ms.

However, this assumption is flawed because of these two conditions:

1. Slot capacity is billed for a minimum of 1 minute before being billed per second.
2. Autoscaling happens in increments of 100 slots.

For Q1, 100 slots (not 50) are actually made available, for 1 minute (60,000 ms) and hence you will be charged for 6,000,000 slot-ms in your bill. 

Summary: The cost chargeback needs to account for minimum slot capacity and autoscaling increments. 

Scenario 4: Chargeback policy for idle resource

In the previous scenario, we see that a query that actually uses 500,000 slot-ms is billed for 6,000,000 slot-ms. Here we make the assumption that whatever resource is made available but not used is also included in the chargeback of the queries running at the same time. What happens if there are multiple queries running concurrently, with unused resources? Continuing with the example in Scenario 3, if there is another query, Q2, that uses 50 slots for 30s, from 05:00:10 to 05:00:40, then: 

• Q1 still uses 500,000 slot-ms like before.
• Q2 uses 1,500,000 slot-ms.
• The total bill remains 6,000,000 slot-ms as before, because slot capacity is billed for a minimum of 1 min and autoscaling increments by 100 slots.

There are several ways to consider the chargeback to Q1 and Q2:

1. Charge each query by its actual slot-ms, and have a separate “idle” category. In this case, Q1 is billed for 500,000 slot-ms, Q2 is billed for 1,500,000 slot-ms, and the remaining 4,000,000 slot-ms is attributed to the “idle” category.

1. Divide idle resources equally among the queries. In this case, Q1 is billed 2,500,000 slot-ms, and Q2 is billed 3,500,000 slot-ms.

1. Divide idle resources proportionally among the queries based on the queries’ usage. In this case, Q1 uses 1,833,333 slot-ms, while Q2 uses 4,166,667 slot-ms.

Summary: Chargeback policy needs to consider how to handle idle resources. Without a clear policy, there could be mismatches between users’ assumptions and the implementation, even leading to inconsistencies, such as the sum of the queries costs deviating from the bill. Moreover, different organizations may prefer different chargeback policies, and there’s no one-size-fits-all approach.

Conclusion

To conclude, providing accurate and useful chargeback for an organization’s usage of BigQuery presents a number of challenges. The common approaches of chargeback by accounts, reservations, and projects are often insufficient for most organizations, as they need user-level and team-level chargeback. However, chargeback by users and teams require us to be able to provide query-level cost estimates, and then aggregate by users and teams (via tags). Computing the query-level cost estimate is another tricky puzzle where simply considering the total slot usage of a query will not work. Instead, we need to consider various factors such as different billing for different stages of the same query, commitment pricing cs. pay-as-you-go pricing from autoscaling, minimum slot capacity and minimum autoscaling increments, and idle policy.

Fortunately, Unravel has information for all the pieces of the puzzle. Its proprietary algorithm intelligently combines these pieces of information and considers the scenarios discussed. Unravel recognizes that chargeback often doesn’t have a one-size-fits-all approach, and can work with customers to adapt its algorithm to specific requirements and use cases.

The post Understanding BigQuery Cost appeared first on Unravel.

5 key differences between Overwatch and Unravel’s free observability for Databricks.

Before investing significant time and effort to configure Overwatch, compare how Unravel’s free data observability solution gives you end-to-end, real-time granular visibility into your Databricks Lakehouse platform out of the box.

The post Open Source Overwatch VS Unravel Free Comparison Infographic appeared first on Unravel.

Snowflake’s data cloud has expanded to become a top choice among organizations looking to leverage data and AI—including large language models (LLMs) and other types of generative AI—to deliver innovative new products to end users and customers. However, the democratization of AI often leads to inefficient usage that results in a cost explosion and decreases the business value of Snowflake. The inefficient usage of Snowflake can occur at various levels. Below are just some examples.

• Warehouses: A warehouse that is too large or has too many clusters for a given workload will be underutilized and incur a higher cost than necessary, whereas the opposite (the warehouse being too small or having too few clusters) will not do the work fast enough. 
• Workload: The democratization of AI results in a rapidly increasing number of SQL users, many of whom focus on getting value out of the data and do not think about the performance and the cost aspects of running SQL on Snowflake. This often leads to costly practices such as:
  • SELECT * just to check out the schema
  • Running a long, timed-out query repeatedly without checking why it timed out
  • Expensive joins such as cartesian products
• Data: No or poorly chosen cluster or partition keys lead to many table scans. Unused tables accumulate over time and by the time users notice the data explosion, they have a hard time knowing which tables may be deleted safely.

Snowflake, like other leading cloud data platforms, provides various compute options and automated features to help with the control of resource usage and spending. However, you need to understand the characteristics of your workload and other KPIs, and have domain expertise, to pick the right options and settings for these features—not to mention there’s no native support to identify bad practices and anti-patterns in SQL. Lastly, optimization is not a one-time exercise. As business needs evolve, so do workloads and data; optimizing for cost and performance becomes part of a continued governance of data and AI operations.

Introducing Unravel for Snowflake

Unravel for Snowflake is a purpose-built AI-driven data observability and FinOps solution that enables organizations to get maximum business value from Snowflake by achieving high cost efficiency and performance. It does so by providing deep, correlated visibility into cost, workload, and performance, with AI-powered insights and automated guardrails for continued optimization and governance. Expanding the existing portfolio of purpose-built AI solutions for Databricks, BigQuery, Amazon EMR, and Cloudera, Unravel for Snowflake is the latest data observability and FinOps product from Unravel Data.

The new Unravel for Snowflake features align with the FinOps phases of inform, optimize, operate:

INFORM

• A “Health Check” that provides a top-down summary view of cost, usage, insights to improve inefficiencies, and projected annualized savings from applying these insights
• A Cost 360 view that captures complete cost across compute, storage and data transfer, and shows chargeback and trends of cost and usage across warehouses, users, tags, and queries
• Top K most expensive warehouses, users, queries
• Detailed warehouse and query views with extensive KPIs
• Side-by-side comparison of queries

OPTIMIZE

• Warehouse insights 
• SQL insights
• Data and storage insights

OPERATE

• OpenSearch-based alerts on query duration and credits
• Alert customization: ability to create custom alerts

Let us first take a look at the Health Check feature that kick-starts the journey of cost and performance optimization.

Health Check for Cost and Performance Inefficiencies

Dashboard-level summary of usage & costs, AI insights into inefficiencies, and projected savings.

The Health Check feature automatically analyzes the workload and cost over the past 15 days. It generates a top-down summary of the cost and usage during this period and, more important, shows insights to improve the inefficiencies for cost and performance—along with the projected annualized savings from applying these insights. 

See at a glance your most expensive “query signatures,” with AI-driven insights on reducing costs.

Users can easily spot the most impactful insights at the warehouse and query levels, and drill down to find out the details. They can also view the top 10 most expensive “query signatures,” or groups of similar queries. Lastly, it recommends alerting policies specific to your workload. 

Users can use the Health Check feature regularly to find new insights and their impact in savings. As the workloads evolve with new business use cases, new inefficiencies may arise and require continued monitoring and governance.

Digging Deep into Understanding Spending

Cost chargeback breakdown and trends by users & queries

Unravel also enables you to easily visualize and understand where money is spent and whether there are anomalies that you should investigate. The Cost 360 view provides cost breakdown and trends across warehouses, users, queries, and tags. It also shows top offenders by listing the most expensive warehouses, users, and query signatures, so that users can address them first.

Cost chargeback breakdown and trends by warehouses & tags.

Debugging and Optimizing Failed, Costly, and Slow Queries

Drill-down AI-driven insights and recommendations into query cost & performance.

Unravel captures extensive data and metadata about cost, workload, and data, and automatically applies AI to generate insights and recommendations for each query and warehouse. Users can filter queries based on status, insights, duration, etc., to find interesting queries, and drill down to look at query details, including the insights for cost and performance optimization. They can also see similar queries to a given query and do side-by-side comparison to spot the difference between two runs.

Get Started with Unravel for Snowflake

To conclude, Unravel supports a variety of use cases in FinOps, from understanding cost and usage, to optimizing inefficiencies and providing alerts for governance. Learn more about Unravel for Snowflake by reading the Unravel for Snowflake docs and request a personalized Snowflake Health Check report. 

The post Announcing Unravel for Snowflake: Faster Time to Business Value in the Data Cloud appeared first on Unravel.

Organizations are transforming their industries through the power of data analytics and AI. A recent McKinsey survey finds that 75% expect generative AI (GenAI) to “cause significant or disruptive change in the nature of their industry’s competition in the next three years.” AI enables businesses to launch innovative new products, gain insights into their business, and boost profitability through technologies that help them outperform competitors. Organizations that don’t leverage data and AI risk falling behind.

Despite all the opportunities with data and AI, many find ROI with advanced technologies like IoT, AI, and predictive analytics elusive. For example, companies find it difficult to get accurate and granular reporting on compute and storage for cloud data and analytics workloads. In speaking with enterprise customers, we hear several recurring barriers they face to achieve their desired ROI on the data cloud.

Cloud data spend is challenging to forecast

About 80% of 157 data management professionals express difficulty predicting data-related cloud costs. Data cloud spend can be difficult to reliably predict. Sudden spikes in data volumes, new analytics use cases, and new data products require additional cloud resources. In addition, cloud service providers can unexpectedly increase prices. Soaring prices and usage fluctuations can disrupt financial operations. Organizations frequently lack visibility into cloud data spending to effectively manage their data analytics and AI budgets.

• Workload fluctuations: Snowflake data processing and storage costs are driven by the amount of compute and storage resources used. As data cloud usage increases for new applications, dashboards, and uses, it becomes challenging to accurately estimate the required data processing and storage costs. This unpredictability can result in budget overruns that affect 60% of infrastructure and operations (I&O) leaders.
• Unanticipated expenses: Spikes in streaming data volumes, large amounts of unstructured and semi-structured data, and shared warehouse consumption can quickly exceed cloud data budgets. These unforeseen usage peaks can catch organizations off guard, leading to unexpected data cloud costs.
• Limited visibility: Accurately allocating costs across the company requires detailed visibility into the data cloud bill. Without query-level or user-level reporting, it becomes impossible to accurately attribute costs to various teams and departments. The result is confusion, friction and finger-pointing between groups as leaders blame high chargeback costs on reporting discrepancies.

Organizations can establish spending guardrails and implement controls by implementing a FinOps approach and leveraging granular data to implement smart and effective controls over their data cloud spend, set up budgets, and utilize alerts to avoid data cloud cost overruns.

Data cloud workloads constrained by budget and staff limits

In 2024, IT organizations expect to shift their focus towards controlling costs, improving efficiency, and increasing automation. Cloud service provider price increases and growing usage add to existing economic pressures, while talent remains scarce and expensive. These cost and bandwidth factors are limiting the number of new data cloud workloads that can be launched.

“Data analytics, engineering & storage” are among the top 3 biggest skill gaps and 54% of data teams say the talent shortage and time required to upskill employees are the biggest challenges to adoption of their AI strategy.

Global demand for AI and machine learning professionals is expected to increase by 40% over the next five years. Approximately one million new jobs will be created as companies look to leverage data and AI for a wide variety of use cases—from automation and risk analysis, to security and supply chain forecasting.

AI adoption and data volume demand

Since ChatGPT broke usage records, generative AI is driving increased data cloud demand and usage. Data teams are struggling to maintain productivity as AI projects scale “due to increasing complexity, inefficient collaboration, and lack of standardized processes and tools” (McKinsey).

Data is foundational for AI and much of it is unstructured, yet IDC found most unstructured data is not leveraged by organizations. A lack of production-ready data pipelines for diverse data sources was the second-most-cited reason (31%) for AI project failure.

Data pipeline failures slow innovation

Data pipelines are becoming more complex, increasing the time required for root cause analysis (RCA) for breaks and delays. Data teams struggle most with data processing speed. Time is a critical factor that pulls skilled and valuable talent into unproductive firefighting. The more time they spend dealing with pipeline issues or failures, the greater the impact on productivity and delivery of new innovation.

Automated data pipeline monitoring and testing is essential for data cloud applications, since teams rapidly iterate and adapt to changing end-user needs and product requirements. Failed queries and data pipelines create data issues for downstream users and workloads such as analytics, BI dashboards, and AI/ML model training. These delays and failures can have a ripple effect that impacts end user decision-making and AI models that rely on accurate, timely content.

Conclusion

Unravel for Snowflake combines the power of AI and automation to help you overcome these challenges. With Unravel, Snowflake users get improved visibility to allocate costs for showback/chargeback, AI-driven recommendations to boost query efficiency, and real-time spend reporting and alerts to accurately predict costs. Unravel for Snowflake helps you optimize your workloads and get more value from your data cloud investments.

Take the next step and check out a self-guided tour or request a free Snowflake Health Check report.

The post Top 4 Challenges to Scaling Snowflake for AI appeared first on Unravel.

This article was originally published by the Forbes Technology Council, September 13, 2023.

When gold was discovered in California in 1848, it triggered one of the largest migrations in U.S. history, accelerated a transportation revolution and helped revitalize the U.S. economy. There’s another kind of Gold Rush happening today: a mad dash to invest in artificial intelligence (AI) and machine learning (ML).

The speed at which AI-related technologies have been embraced by businesses means that companies can’t afford to sit on the sidelines. Companies also can’t afford to invest in models that fail to live up to their promises.

But AI comes with a cost. McKinsey estimates that developing a single generative AI model costs up to $200 million, up to $10 million to customize an existing model with internal data and up to $2 million to deploy an off-the-shelf solution.

The volume of generative AI/ML workloads—and data pipelines that power them—has also skyrocketed at an exponential rate as various departments run differing use cases with this transformational technology. Bloomberg Intelligence reports that the generative AI market is poised to explode, growing to $1.3 trillion over the next 10 years from a market size of just $40 billion in 2022. And every job, every workload, and every data pipeline costs money.

Because of the cost factor, winning the AI race isn’t just about getting there first; it’s about making the best use of resources to achieve maximum business goals.

The Snowball Effect

There was a time when IT teams were the only ones utilizing AI/ML models. Now, teams across the enterprise—from marketing to risk to finance to product and supply chain—are all utilizing AI in some capacity, many of whom lack the training and expertise to run these models efficiently.

AI/ML models process exponentially more data, requiring massive amounts of cloud compute and storage resources. That makes them expensive: A single training run for GPT-3 costs $12 million.

Enterprises today may have upwards of tens—even hundreds—of thousands of pipelines running at any given time. Running sub-optimized pipelines in the cloud often causes costs to quickly spin out of control.

The most obvious culprit is oversized infrastructure, where users are simply guessing how much compute resources they need rather than basing it on actual usage requirements. Same thing with storage costs, where teams may be using more expensive options than necessary for huge amounts of data that they rarely use.

But data quality and inefficient code often cause costs to soar even higher: data schema, data skew and load imbalances, idle time, and a rash of other code-related issues that make data pipelines take longer to run than necessary—or even fail outright.

Like a snowball gathering size as it rolls down a mountain, the more data pipelines you have running, the more problems, headaches, and, ultimately, costs you’re likely to have.

And it’s not just cloud costs that need to be considered. Modern data pipelines and AI workloads are complex. It takes a tremendous amount of troubleshooting expertise just to keep models working and meeting business SLAs—and that doesn’t factor in the costs of downtime or brand damage. For example, if a bank’s fraud detection app goes down for even a few minutes, how much would that cost the company?

Optimized Data Workloads on the Cloud

Optimizing cloud data costs is a business strategy that ensures a company’s resources are being allocated appropriately and in the most cost-efficient manner. It’s fundamental to the success of an AI-driven company as it ensures that cloud data budgets are being used effectively and providing maximum ROI.

But business and IT leaders need to first understand exactly where resources are being used efficiently and where waste is occurring. To do so, keep in mind the following when developing a cloud data cost optimization strategy.

• Reuse building blocks. Everything costs money on the cloud. Every file you store, every record you access, every piece of code you run incur a cost. Data processing can usually be broken down into a series of steps, and a smart data team should be able to reuse those steps for other processing. For example, code written to move data about a company’s sales records could be reused by the pricing and product teams rather than both teams building their own code separately and incurring twice the cost.

• Truly leverage cloud capabilities. The cloud allows you to quickly adjust the resources needed to process data workloads. Unfortunately, too many companies operate under “just in case” scenarios that lead to allocating more resources than actually needed. By understanding usage patterns and leveraging cloud’s auto-scaling capabilities, it’s possible for companies to dynamically control how they scale up and, more importantly, create guardrails to manage the maximum.

• Analyze compute and storage spend by job and by user. The ability to really dig down to the granular details of who is spending what on which project will likely yield a few surprises. You might find that the most expensive jobs are not the ones that are making your company millions. You may find that you’re paying way more for exploration than for data models that will be put to good use. Or, you may find that the same group of users are responsible for the jobs with the biggest spend and the lowest ROI (in which case, it might be time to tighten up on some processes).

Given the data demands that generative AI models and use cases place on a company, business and IT leaders need to have a deep understanding of what’s going on under the proverbial hood. As generative AI evolves, business leaders will need to address new challenges. Keeping cloud costs under control shouldn’t be one of them.

The post Why Optimizing Cost Is Crucial to AI/ML Success appeared first on Unravel.

“Someone’s sitting in the shade today because someone planted a tree a long time ago.” —Warren Buffet

CI/CD, a software development strategy, combines the methodologies of Continuous Integration and Continuous Delivery/Continuous Deployment to safely and reliably deliver new versions of code in iterative short cycles. This practice bridges the gap between developers and operations team by streamlining the building, testing, and deployment of the code by automating the series of steps involved in this otherwise complex process. Traditionally used to speed up the software development life cycle, today CI/CD is gaining popularity among data scientists and data engineers since it enables cross-team collaboration and rapid, secure integration and deployment of libraries, scripts, notebooks, and other ML workflow assets.

One recent report found that 80% of organizations have adopted agile practices, but for nearly two-thirds of developers it takes at least one week to get committed code successfully running in production. Implementing CI/CD can streamline data pipeline development and deployment, accelerating release times and frequency, while improving code quality. 

The evolving need for CI/CD for data teams

AI’s rapid adoption is driving the demand for fresh and reliable data for training, validation, verification, and drift analysis. Implementing CI/CD enhances your Databricks development process, streamlines pipeline deployment, and accelerates time-to-market. CI/CD revolutionizes how you build, test, and deploy code within your Databricks environment, helping you automate tasks, ensure a smooth transition from development to production, and enable lakehouse data engineering and data science teams to work more efficiently. And when it comes to cloud data platforms like Databricks, performance equals cost. The more optimized your pipelines are, the more optimized your Databricks spend will be.

Why incorporate Unravel into your existing DevOps workflow?

Unravel Data is the AI-powered data observability and FinOps platform for Databricks. By using Unravel’s CI/CD integration for Databricks, developers can catch performance problems early in development and deployment life cycles and proactively take actions to mitigate issues. This has shown to significantly reduce the time taken by data teams to act on critical timely insights. Unravel’s AI-powered efficiency recommendations, now embedded right into the DevOps environments, help foster a cost-conscious culture that compels developers to follow performance and cost-driven coding best practices. It also raises awareness of resource usage, configuration changes, and data layout issues that could impact service level agreements (SLAs) when the code is deployed in production. Accepting or ignoring insights suggested by Unravel helps promote accountability for developers’ actions and creates transparency for the DevOps and FinOps practitioners to attribute cost-saving wins and losses. 

With the advent of Generative Pre-Trained Transformer (GPT) AI models, data teams today have started using coding co-pilots to generate accurate and efficient code. With Unravel, this experience is a notch better with real-time visibility into code inefficiencies that can translate into production performance problems like bottlenecks, performance anomalies, missed SLAs, cost overruns, etc. Other code-assist tools like GitHub Copilot are limited in their scope of assistance to static code analysis based code rewrite suggestions, Unravel’s AI-driven Insights Engine built for Databricks considers the performance and cost impact of code and configuration changes and provides recommendations to make optimal suggestions. This helps you streamline your development process, identify bottlenecks, and ensure optimal performance throughout the life cycle of your data pipelines.

Unravel’s AI-powered analysis automatically provides deep, actionable insights.

Next, let’s look into what key benefits are provided by the Unravel integration into your DevOps workflows. 

Achieve operational excellence 

Unravel’s CI/CD integration for Databricks enhances data team and developer efficiency by seamlessly providing real-time, AI-powered insights to help optimize performance and troubleshoot issues in your data pipelines.  

Unravel integrates with your favorite CI/CD tools such as Azure DevOps and GitHub. When developers make changes to code and submit via a pull request, Unravel automatically conducts AI-powered checks to ensure the code is performant and efficient. This helps developers:

• Maximize resource utilization by gaining valuable insights into pipeline efficiency
• Achieve performance and cost goals by analyzing critical metrics during development
• Leverage specific, actionable recommendations to improve code for cost and performance optimization
• Identify and resolve bottlenecks promptly, reducing development time

Leverage developer pull request (PR) reviews

Developers play a crucial role in achieving cost efficiency through PR reviews. Encourage them to adopt best practices and follow established guidelines when submitting code for review. This ensures that all tests are run and results are thoroughly evaluated before merging into the main project branch.

By actively involving developers in the review process, you tap into their knowledge and experience to identify potential areas for cost savings within your pipelines. Their insights can help streamline workflows, improve resource allocation, and eliminate inefficiencies. Involving developers in PR reviews fosters collaboration among team members and encourages feedback, creating a culture of continuous improvement. 

Here are several ways developer PR reviews can enhance the reliability of data pipelines:

• Ensure code quality: Developer PR reviews serve as an effective mechanism to maintain high code-quality standards. Through these reviews, developers can catch coding errors, identify potential bugs, and suggest improvements before the code is merged into the production repository.
• Detect issues early: By involving developers in PR reviews, you ensure that potential issues are identified early in the development process. This allows for prompt resolution and prevents problems from propagating further down the pipeline.
• Mitigate risks: Faulty or inefficient code changes can have significant impacts on your pipelines and overall system stability. With developer PR reviews, you involve experts who understand the intricacies of the pipeline and can help mitigate risks by providing valuable insights and suggestions.
• Foster a collaborative environment: Developer PR reviews create a collaborative environment where team members actively engage with one another’s work. Feedback provided during these reviews promotes knowledge sharing, improves individual skills, and enhances overall team performance.

Real-world examples of CI/CD integration for Databricks

Companies in finance, healthcare, e-commerce, and more have successfully implemented CI/CD practices with Databricks. Enterprise organization across industries leverage Unravel to ensure that code is performant and efficient before it goes into production.

• Financial services: A Fortune Global 500 bank provides Unravel to their developers as a way to evaluate their pipelines before they do a code release.
• Healthcare: One of the largest health insurance providers in the United States uses Unravel to ensure that its business-critical data applications are optimized for performance, reliability, and cost in its development environment—before they go live in production.
• Logistics: One of the world’s largest logistics companies leverages Unravel to upskill their data teams at scale. They put Unravel in their CI/CD process to ensure that all code and queries are reviewed to ensure they meet the desired quality and efficiency bar before they go into production.

Unravel CI/CD integration for Databricks use cases

Incorporating Unravel’s real-time, AI insights into PR reviews helps developers ensure the reliability, performance, and cost efficiency of data pipelines before they go into production. This practice ensures that any code changes are thoroughly reviewed before being merged into the main project branch. By catching potential issues early on, you can prevent pipeline breaks, bottlenecks, and wasted compute tasks from running in production. 

Ensure pipeline reliability

Unravel’s purpose-built AI helps augment your PR reviews to ensure code quality and reliability in your release pipelines. Unravel integration into your Databricks CI/CD process helps developers identify potential issues early on and mitigate risks associated with faulty or inefficient code changes. Catching breaking changes in development and test environments helps developers improve productivity and helps ensure that you achieve your SLAs.

1-minute tour: Unravel’s AI-powered Speed, Cost, Reliability Optimizer

Achieve cost efficiency

Unravel provides immediate feedback and recommendations to improve cost efficiency. This enables you to catch inefficient code, and developers can make any necessary adjustments for optimal resource utilization before it impacts production environments. Using Unravel as part of PR reviews helps your organization optimize resource allocation and reduce cloud waste.

1-minute tour: Unravel’s AI-powered Databricks Cost Optimization

Boost pipeline performance

Collaborative code reviews provide an opportunity to identify bottlenecks, optimize code, and enhance data processing efficiency. By including Unravel’s AI recommendations in the review process, developers benefit from AI-powered insights to ensure code changes achieve performance objectives. 

1-minute tour: Unravel’s AI-powered Pipeline Bottleneck Analysis

Get started with Unravel CI/CD integration for Databricks

Supercharge your CI/CD process for Databricks using Unravel’s AI. By leveraging this powerful combination, you can significantly improve developer productivity, ensure pipeline reliability, achieve cost efficiency, and boost overall pipeline performance. Whether you choose to automate PR reviews with Azure DevOps or GitHub, Unravel’s CI/CD integration for Databricks has got you covered.

Now it’s time to take action and unleash the full potential of your Databricks environment. Integrate Unravel’s CI/CD solution into your workflow and experience the benefits firsthand. Don’t miss out on the opportunity to streamline your development process, save costs, and deliver high-quality code faster than ever before.

Next steps to learn more

Read Unravel’s CI/CD integration documentation

Watch this video

Book a live demo

The post Unravel CI/CD Integration for Databricks appeared first on Unravel.

In today’s data-driven world, there is a huge amount of data flowing into the business. Engineers spend a large part of their time in building pipelines—to collect the data from different sources, process it, and transform it to useful datasets that can be sent to business intelligence applications or machine learning models. Tools like Airflow are used to orchestrate complex data pipelines by programmatically authoring, scheduling, and monitoring the workflow pipelines. From a 10,000-foot view, Airflow may look like just a powerful cron, but it has additional capabilities like monitoring, generating logs, retrying jobs, adding dependencies between tasks, enabling a huge number of operators to run different types of tasks, etc. 

Pipelines become increasingly complex because of the interdependence of several tools needed to complete a workflow. Common tasks done in day-to-day pipeline activities call for a number of different technologies to process data for different purposes: running a Spark job, executing a Snowflake query, executing a data transfer across different platforms (like from a GCS bucket to an AWS S3 bucket), and pushing data into a queue system like SQS. 

Airflow orchestrates pipelines running several third-party tools and supports about 80 different providers.  But you need to monitor what’s going on underneath pipeline operations across all those tools and get deeper insights about pipeline execution. Unravel supports this by bringing in all the information monitored from Airflow via API, StatsD, and notifications. Unravel also collects metrics from Spark, BigQuery, Snowflake to connect the dots to find out how the pipeline execution in Airflow impacts other systems.

Let’s examine two common use cases to illustrate how Unravel stitches together all pipeline details to solve problems that you can’t address with Airflow alone.

View DAG Run Information

Let’s say a financial services company is using Airflow to run Spark jobs on their instances and notices that a few of the DAG runs are taking longer to complete. Airflow UI shows information about the duration of each task run and the status of the task run, but that’s about it. For deeper insight into why the runs are taking longer, you’d need to gather detailed information about the Spark ID that is created via this task run. 

By tightly integrating with the Spark ecosystem, Unravel fills that gap and gives you deeper insights about the Spark job run, its duration, and the resource utilized. These details, along with information collected from Airflow, can help you see the bigger picture of what’s happening underneath, inside your data pipelines. 

This is an example test_spark DAG that runs the Spark job via the SparkSubmitOperator in the company’s environment. The SparkSubmitOperator task is taking longer to execute, and the screen shot below shows the long duration of the Spark submit task.This information flows into Unravel and is visualized in the Pipeline Observer UI.  A summary list of failed/degraded pipeline runs helps to filter DAG runs with problems to debug. Clicking on a specific pipeline run ID drills down into more information about the pipeline, e.g., detailed information about the task, duration of each task, total duration, baseline compared to previous runs, and stack traces about the failure. Unravel also shows a list of all pipeline runs.Clicking on one of the Spark DAG runs with a long duration reveals more detailed information about the DAG and allows comparison with previous DAG runs. You can also find information about the Spark ID that is run for this particular DAG.

Clicking on the Spark app ID takes you to the Unravel UI with more information about the Spark job that is collected from Spark: the resources used to run the Spark job, any error logs, configurations made to run that Spark job, duration of the Spark jobs, and more.

Here we have more detail about the Spark job and the Spark ID created during the DAG run that helps verify the list of Spark jobs, understand which DAG run created the specific Spark job, see the resources consumed by that Spark job, or find out the configuration of the Spark job. 

Tracking down a specific Spark job from a DAG run is difficult since although Airflow initiates the task run, Spark takes care of running the task. Unravel can monitor and collect details from Spark, identify the Spark ID, and correlate the Spark job to the Airflow DAG run that initiated the job.

Delay in Scheduling Pipeline Run 

Unravel helps pinpoint the delay in scheduling DAG runs.

The same financial services company runs multiple DAGs as part of their pipeline system. While the DAG runs are working fine, the team stumbles upon a different problem: the DAG that is expected to run at the scheduled time is running at a delayed time. This delay affects running subsequent DAG runs, resulting in tasks not getting completed on time.

Airflow has the capability to alert if a Task/DAG that’s running misses its SLA. But there are cases when the DAG run is not initiated at all in the first place—e.g., no slot is available or the number of DAGs that can run in parallel exceeds the configured maximum number.  

Unravel helps bring the problem to the surface by rendering underlying details about the DAG run delay right in the UI. For a DAG run ID, Unravel shows the useful information collected from StatsD like the total parse time for processing the DAG, DAG run schedule delay, etc. This information provides pipeline insights that help to identify why there is delay in scheduling the DAG run around the DAG execution time.

The Pipeline Details page presents DAG run ID information, with pipeline insights about the schedule delay in the right-hand pane. It clearly shows that the DAG run was delayed well beyond the acceptable threshold. 

Unravel helps keep the pipeline running in check and automatically delivers insights about what’s happening inside Airflow. Unravel monitors systems like Spark, BigQuery, and Airflow and has the granular data about each of the systems, connecting that information and rendering the right insights to make it a powerful tool for monitoring cloud data systems.

The post How Unravel Enhances Airflow appeared first on Unravel.

In this fast-paced era of artificial intelligence (AI), the need for data is multiplying. The demand for faster data life cycles has skyrocketed, thanks to AI’s insatiable appetite for knowledge. According to a recent McKinsey survey, 75% expect generative AI (GenAI) to “cause significant or disruptive change in the nature of their industry’s competition in the next three years.”

Next-gen AI craves unstructured, streaming, industry-specific data. Although the pace of innovation is relentless, “when it comes to generative AI, data really is your moat.”

But here’s the twist: efficiency is now the new cool kid in town. Data product profitability hinges on optimizing every step of the data life cycle—from ingestion and transformation, to processing, curating, and refining. It’s no longer just about gathering mountains of information; it’s about collecting the right data efficiently.

As new, industry-specific GenAI use cases emerge, there is an urgent need for large data sets for training, validation, verification, and drift analysis. GenAI requires flexible, scalable, and efficient data architecture, infrastructure, code, and operating models to achieve success.

Leverage a Scalable Operating Model to Accelerate Your Data Life Cycle Velocity

To optimize your data life cycle, it’s crucial to leverage a scalable operating model that can accelerate the velocity of your data processes. By following a systematic approach and implementing efficient strategies, you can effectively manage your data from start to finish.

Databricks recently introduced a scalable operating model for data and AI to help customers achieve a positive Return on Data Assets (RODA).

Databricks’ iterative end-to-end operating pipeline

Define Use Cases and Business Requirements

Before diving into the data life cycle, it’s essential to clearly define your use cases and business requirements. This involves understanding what specific problems or goals you plan to address with your data. By identifying these use cases and related business requirements, you can determine the necessary steps and actions needed throughout the entire process.

Build, Test, and Iterate the Solution

Once you have defined your use cases and business requirements, it’s time to build, test, and iterate the solution. This involves developing the necessary infrastructure, tools, and processes required for managing your data effectively. It’s important to continuously test and iterate on your solution to ensure that it meets your desired outcomes.

During this phase, consider using agile methodologies that allow for quick iterations and feedback loops. This will enable you to make adjustments as needed based on real-world usage and feedback from stakeholders.

Scale Efficiently

As your data needs grow over time, it’s crucial to scale efficiently. This means ensuring that your architecture can handle increased volumes of data without sacrificing performance or reliability.

Consider leveraging cloud-based technologies that offer scalability on-demand. Cloud platforms provide flexible resources that can be easily scaled up or down based on your needs. Employing automation techniques such as machine learning algorithms or artificial intelligence can help streamline processes and improve efficiency.

By scaling efficiently, you can accommodate growing datasets while maintaining high-quality standards throughout the entire data life cycle.

Elements of the Business Use Cases and Requirements Phase

In the data life cycle, the business requirements phase plays a crucial role in setting the foundation for successful data management. This phase involves several key elements that contribute to defining a solution and ensuring measurable outcomes. Let’s take a closer look at these elements:

• Leverage design thinking to define a solution for each problem statement: Design thinking is an approach that focuses on understanding user needs, challenging assumptions, and exploring innovative solutions. In this phase, it is essential to apply design thinking principles to identify and define a single problem statement that aligns with business objectives.
• Validate the business case and define measurable outcomes: Before proceeding further, it is crucial to validate the business case for the proposed solution. This involves assessing its feasibility, potential benefits, and alignment with strategic goals. Defining clear and measurable outcomes helps in evaluating project success.
• Map out the MVP end user experiences: To ensure user satisfaction and engagement, mapping out Minimum Viable Product (MVP) end-user experiences is essential. This involves identifying key touchpoints and interactions throughout the data life cycle stages. By considering user perspectives early on, organizations can create intuitive and effective solutions.
• Understand the data requirements: A thorough understanding of data requirements is vital for successful implementation. It includes identifying what types of data are needed, their sources, formats, quality standards, security considerations, and any specific regulations or compliance requirements.
• Gather required capabilities with platform architects: Collaborating with platform architects helps gather insights into available capabilities within existing infrastructure or technology platforms. This step ensures compatibility between business requirements and technical capabilities while minimizing redundancies or unnecessary investments.
• Establish data management roles, responsibilities, and procedures: Defining clear roles and responsibilities within the organization’s data management team is critical for effective execution. Establishing procedures for data observability, stewardship practices, access controls, privacy policies ensures consistency in managing data throughout its life cycle.

By following these elements in the business requirements phase, organizations can lay a solid foundation for successful data management and optimize the overall data life cycle. It sets the stage for subsequent phases, including data acquisition, storage, processing, analysis, and utilization.

Build, Test, and Iterate the Solution

To successfully implement a data life cycle, it is crucial to focus on building, testing, and iterating the solution. This phase involves several key steps that ensure the development and deployment of a robust and efficient system.

• Plan development and deployment: The first step in this phase is to carefully plan the development and deployment process. This includes identifying the goals and objectives of the project, defining timelines and milestones, and allocating resources effectively. By having a clear plan in place, the data team can streamline their efforts towards achieving desired outcomes.
• Gather end-user feedback at every stage: Throughout the development process, it is essential to gather feedback from end users at every stage. This allows for iterative improvements based on real-world usage scenarios. By actively involving end users in providing feedback, the data team can identify areas for enhancement or potential issues that need to be addressed.
• Define CI/CD pipelines for fast testing and iteration: Implementing Continuous Integration (CI) and Continuous Deployment (CD) pipelines enables fast testing and iteration of the solution. These pipelines automate various stages of software development such as code integration, testing, deployment, and monitoring. By automating these processes, any changes or updates can be quickly tested and deployed without manual intervention.
• Data preparation, cleaning, and processing: Before training machine learning models or conducting experiments with datasets, it is crucial to prepare, clean, and process the data appropriately. This involves tasks such as removing outliers or missing values from datasets to ensure accurate results during model training.
• Feature engineering: Feature engineering plays a vital role in enhancing model performance by selecting relevant features from raw data or creating new features based on domain knowledge. It involves transforming raw data into meaningful representations that capture essential patterns or characteristics.
• Training and ML experiments: In this stage of the data life cycle, machine learning models are trained using appropriate algorithms on prepared datasets. Multiple experiments may be conducted, testing different algorithms or hyperparameters to find the best-performing model.
• Model deployment: Once a satisfactory model is obtained, it needs to be deployed in a production environment. This involves integrating the model into existing systems or creating new APIs for real-time predictions.
• Model monitoring and scoring: After deployment, continuous monitoring of the model’s performance is essential. Tracking key metrics and scoring the model’s outputs against ground truth data helps identify any degradation in performance or potential issues that require attention.

By following these steps and iterating on the solution based on user feedback, data teams can ensure an efficient and effective data life cycle from development to deployment and beyond.

Efficiently Scale and Drive Adoption with Your Operating Model

To efficiently scale your data life cycle and drive adoption, you need to focus on several key areas. Let’s dive into each of them:

• Deploy into production: Once you have built and tested your solution, it’s time to deploy it into production. This step involves moving your solution from a development environment to a live environment where end users can access and utilize it.
• Evaluate production results: After deploying your solution, it is crucial to evaluate its performance in the production environment. Monitor key metrics and gather feedback from users to identify any issues or areas for improvement.
• Socialize data observability and FinOps best practices: To ensure the success of your operating model, it is essential to socialize data observability and FinOps best practices among your team. This involves promoting transparency, accountability, and efficiency in managing data operations.
• Acknowledge engineers who “shift left” performance and efficiency: Recognize and reward engineers who prioritize performance and efficiency early in the development process. Encourage a culture of proactive optimization by acknowledging those who contribute to improving the overall effectiveness of the data life cycle.
• Manage access, incidents, support, and feature requests: Efficiently scaling your operating model requires effective management of access permissions, incident handling processes, support systems, and feature requests. Streamline these processes to ensure smooth operations while accommodating user needs.
• Track progress towards business outcomes by measuring and sharing KPIs: Measuring key performance indicators (KPIs) is vital for tracking progress towards business outcomes. Regularly measure relevant metrics related to adoption rates, user satisfaction levels, cost savings achieved through efficiency improvements, etc., then share this information across teams for increased visibility.

By implementing these strategies within your operating model, you can efficiently scale your data life cycle while driving adoption among users. Remember that continuous evaluation and improvement are critical for optimizing performance throughout the life cycle.

Drive for Performance with Purpose-Built AI

Unravel helps with many elements of the Databricks operating model:

• Quickly identify failed and inefficient Databricks jobs: One of the key challenges is identifying failed and inefficient Databricks jobs. However, with AI purpose-built for Databricks, this task becomes much easier. By leveraging advanced analytics and monitoring capabilities, you can quickly pinpoint any issues in your job executions.
• Creating ML models vs deploying them into production: Creating machine learning models is undoubtedly challenging, but deploying them into production is even harder. It requires careful consideration of factors like scalability, performance, and reliability. With purpose-built AI tools, you can streamline the deployment process by automating various tasks such as model versioning, containerization, and orchestration.
• Leverage Unravel’s Analysis tab for insights: To gain deeper insights into your application’s performance during job execution, leverage the analysis tab provided by purpose-built AI solutions. This feature allows you to examine critical details like memory usage errors or other bottlenecks that may be impacting job efficiency.
  
  

  Unravel’s AI-powered analysis automatically provides deep, actionable insights.

• Share links for collaboration: Collaboration plays a crucial role in data management and infrastructure optimization. Unravel enables you to share links with data scientists, developers, and data engineers to provide detailed information about specific test runs or failed Databricks jobs. This promotes collaboration and facilitates a better understanding of why certain jobs may have failed.
• Cloud data cost management made easy: Cloud cost management, also known as FinOps, is another essential aspect of data life cycle management. Purpose-built AI solutions simplify this process by providing comprehensive insights into cost drivers within your Databricks environment. You can identify the biggest cost drivers such as users, clusters, jobs, and code segments that contribute significantly to cloud costs.
• AI recommendations for optimization: To optimize your data infrastructure further, purpose-built AI platforms offer valuable recommendations across various aspects, including infrastructure configuration, parallelism settings, handling data skewness issues, optimizing Python/SQL/Scala/Java code snippets, and more. These AI-driven recommendations help you make informed decisions to enhance performance and efficiency.

Learn More & Next Steps

Unravel hosted a virtual roundtable, Accelerate the Data Analytics Life Cycle, with a panel of Unravel and Databricks experts. Unravel VP Clinton Ford moderated the discussion with Sanjeev Mohan, principal at SanjMo and former VP at Gartner, Subramanian Iyer, Unravel training and enablement leader and Databricks SME, and Don Hilborn, Unravel Field CTO and former Databricks lead strategic solutions architect.

FAQs

How can I implement a scalable operating model for my data life cycle?

To implement a scalable operating model for your data life cycle, start by clearly defining roles and responsibilities within your organization. Establish efficient processes and workflows that enable seamless collaboration between different teams involved in managing the data life cycle. Leverage automation tools and technologies to streamline repetitive tasks and ensure consistency in data management practices.

What are some key considerations during the Business Requirements Phase?

During the Business Requirements Phase, it is crucial to engage stakeholders from various departments to gather comprehensive requirements. Clearly define project objectives, deliverables, timelines, and success criteria. Conduct thorough analysis of existing systems and processes to identify gaps or areas for improvement.

How can I drive adoption of my data life cycle operational model?

To drive adoption of your data management solution, focus on effective change management strategies. Communicate the benefits of the solution to all stakeholders involved and provide training programs or resources to help them understand its value. Encourage feedback from users throughout the implementation process and incorporate their suggestions to enhance usability and address any concerns.

What role does AI play in optimizing the data life cycle?

AI can play a significant role in optimizing the data life cycle by automating repetitive tasks, improving data quality through advanced analytics and machine learning algorithms, and providing valuable insights for decision-making. AI-powered tools can help identify patterns, trends, and anomalies in large datasets, enabling organizations to make data-driven decisions with greater accuracy and efficiency.

How do I ensure performance while implementing purpose-built AI?

To ensure performance while implementing purpose-built AI, it is essential to have a well-defined strategy. Start by clearly defining the problem you want to solve with AI and set measurable goals for success. Invest in high-quality training data to train your AI models effectively. Continuously monitor and evaluate the performance of your AI system, making necessary adjustments as needed.

The post Rev Up Your Lakehouse: Lap the Field with a Databricks Operating Model appeared first on Unravel.

Overview

In today’s digital landscape, data-driven decisions form the crux of successful business strategies. However, the path to harnessing data’s full potential is strewn with challenges. Let’s delve into the hurdles organizations face and how Unravel is the key to unlocking seamless data operations.

The Roadblocks in the Fast Lane of Data Operations

In today’s data-driven landscape, organizations grapple with erratic spending, cloud constraints, AI complexities, and prolonged MTTR, urgently seeking solutions to navigate these challenges efficiently. The four most common roadblocks are:

• Data Spend Forecasting: Imagine a roller coaster with unpredictable highs and lows. That’s how most organizations view their data spend forecasting. Such unpredictability wreaks havoc on financial planning, making operational consistency a challenge.
• Constraints in Adding Data Workloads: Imagine tying an anchor to a speedboat. That’s how the constraints feel when trying to adopt cloud data solutions, holding back progress and limiting agility.
• Surge in AI Model Complexity: AI’s evolutionary pace is exponential. As it grows, so do the intricacies surrounding data volume and pipelines, which strain budgetary limitations.
• The MTTR Bottleneck: The multifaceted nature of modern tech stacks means longer Mean Time to Repair (MTTR). This slows down processes, consumes valuable resources, and stalls innovation.

By acting as a comprehensive data observability and FinOps solution, Unravel Data empowers businesses to move past the challenges and frictions that typically hinder data operations, ensuring smoother, more efficient data-driven processes. Here’s how Unravel Data aids in navigating the roadblocks in the high-speed lane of data operations:

• Predictive Data Spend Forecasting: With its advanced analytics, Unravel Data can provide insights into data consumption patterns, helping businesses forecast their data spending more accurately. This eliminates the roller coaster of unpredictable costs.
• Simplifying Data Workloads: Unravel Data optimizes and automates workload management. Instead of being anchored down by the weight of complex data tasks, businesses can efficiently run and scale their data processes in the cloud.
• Managing AI Model Complexity: Unravel offers visibility and insights into AI data pipelines. Analyzing and optimizing these pipelines ensure that growing intricacies do not overwhelm resources or budgets.
• Reducing MTTR: By providing a clear view of the entire tech stack and pinpointing bottlenecks or issues, Unravel Data significantly reduces Mean Time to Repair. With its actionable insights, teams can address problems faster, reducing downtime and ensuring smooth data operations.
• Streamlining Data Pipelines: Unravel Data offers tools to diagnose and improve data pipeline performance. This ensures that even as data grows in volume and complexity, pipelines remain efficient and agile.
• Efficiency and ROI: With its clear insights into resource consumption and requirements, Unravel Data helps businesses run 50% more workloads in their existing Databricks environments, ensuring they only pay for what they need, reducing wastage and excess expenditure.

The Skyrocketing Growth of Cloud Data Management

As the digital realm expands, cloud data management usage is soaring, with data services accounting for a significant chunk. According to the IDC, the public cloud IaaS and PaaS market is projected to reach $400 billion by 2025, growing at a 28.8% CAGR from 2021 to 2025. Some highlights are:

• Data management and application development account for 39% and 20% of the market, respectively, and are the main workloads backed by PaaS solutions, capturing a major share of its revenue.
• In IaaS revenue, IT infrastructure leads with 25%, trailed by business applications (21%) and data management (20%).
• Unstructured data analytics and media streaming are predicted to be the top-growing segments with CAGRs of 41.9% and 41.2%, respectively.

Unravel provides a comprehensive solution to address the growth associated with cloud data management. Here’s how:

• Visibility and Transparency: Unravel offers in-depth insights into your cloud operations, allowing you to understand where and how costs are accruing, ensuring no hidden fees or unnoticed inefficiencies.
• Optimization Tools: Through its suite of analytics and AI-driven tools, Unravel pinpoints inefficiencies, recommends optimizations, and automates the scaling of resources to ensure you’re only using (and paying for) what you need.
• Forecasting: With predictive analytics, Unravel provides forecasts of data usage and associated costs, enabling proactive budgeting and financial planning.
• Workload Management: Unravel ensures that data workloads run efficiently and without wastage, reducing both computational costs and storage overhead.
• Performance Tuning: By optimizing query performance and data storage strategies, Unravel ensures faster results using fewer resources, translating to 50% more workloads.
• Monitoring and Alerts: Real-time monitoring paired with intelligent alerts ensures that any resource-intensive operations or anomalies are flagged immediately, allowing for quick intervention and rectification.

By employing these strategies and tools, Unravel acts as a financial safeguard for businesses, ensuring that the ever-growing cloud data bill remains predictable, manageable, and optimized for efficiency.

The Tightrope Walk of Efficiency Tuning and Talent

Modern enterprises hinge on data and AI, but shrinking budgets and talent gaps threaten them. Gartner pinpoints overprovisioning and skills shortages as major roadblocks, while Google and IDC underscore the high demand for data analytics skills and the untapped potential of unstructured data. Here are some of the problems modern organizations face:

• Production environments are statically overprovisioned and therefore underutilized. On-premises, 30% utilization is common, but it’s all capital expenditures (capex), and as long as it’s within budget, no one has traditionally cared about the waste. However, in the cloud, you pay for that excess resource monthly, forcing you to confront the ongoing cost of the waste. – Gartner
• The cloud skills gap has reached a crisis level in many organizations – Gartner
• Revenue creation through digital transformation requires talent engagement that is currently scarce and difficult to acquire and maintain. – Gartner
• Lack of skills remains the biggest barrier to infrastructure modernization initiatives, with many organizations finding they cannot hire outside talent to fill these skills gaps. IT organizations will not succeed unless they prioritize organic skills growth. – Gartner
• Data analytics skills are in demand across industries as businesses of all types around the world recognize that strong analytics improve business performance.- Google via Coursera

Unravel Data addresses the delicate balancing act of budget and talent in several strategic ways:

• Operational Efficiency: Purpose-built AI provides actionable insights into data operations across Databricks, Spark, EMR, BigQuery, Snowflake, etc. Unravel Data reduces the need for trial-and-error and time-consuming manual interventions. At the core of Unravel’s data observability platform is our AI-powered Insights Engine. This sophisticated Artificial Intelligence engine incorporates AI techniques, algorithms, and tools to process and analyze vast amounts of data, learn from patterns, and make predictions or decisions based on that learning. This not only improves operational efficiency but also ensures that talented personnel spend their time innovating rather than on routine tasks.
• Skills Gap Bridging: The platform’s intuitive interface and AI-driven insights mean that even those without deep expertise in specific data technologies can navigate, understand, and optimize complex data ecosystems. This eases the pressure to hire or train ultra-specialized talent.
• Predictive Analysis: With Unravel’s ability to predict potential bottlenecks or inefficiencies, teams can proactively address issues, leading to more efficient budget allocation and resource utilization.
• Cost Insights: Unravel provides detailed insights into the efficiency of various data operations, allowing organizations to make informed decisions on where to invest and where to cut back.
• Automated Optimization: By automating many of the tasks traditionally performed by data engineers, like performance tuning or troubleshooting, Unravel ensures teams can do more with less, optimizing both budget and talent.
• Talent Focus Shift: With mundane tasks automated and insights available at a glance, skilled personnel can focus on higher-value activities, like data innovation, analytics, and strategic projects.

By enhancing efficiency, providing clarity, and streamlining operations, Unravel Data ensures that organizations can get more from their existing budgets while maximizing the potential of their talent, turning the tightrope walk into a more stable journey.

The Intricacies of Data-Centric Organizations

Data-centric organizations grapple with the complexities of managing vast and fast-moving data in the digital age. Ensuring data accuracy, security, and compliance, while integrating varied sources, is challenging. They must balance data accessibility with protecting sensitive information, all while adapting to evolving technologies, addressing talent gaps, and extracting actionable insights from their data reservoirs. Here is some relevant research on the topic:

• “Data is foundational to AI” yet “unstructured data remains largely untapped.” – IDC
• Even as organizations rush to adopt data-centric operations, challenges persist. For instance, manufacturing data projects often hit roadblocks due to outdated legacy technology, as observed by the World Economic Forum.
• Generative AI is supported by large language models (LLMs), which require powerful and highly scalable computing capabilities to process data in real-time. – Gartner

Unravel Data provides a beacon for data-centric organizations amid complex challenges. Offering a holistic view of data operations, it simplifies management using AI-driven tools. It ensures data security, accessibility, and optimized performance. With its intuitive interface, Unravel bridges talent gaps and navigates the data maze, turning complexities into actionable insights.

Embarking on the Unravel Journey: Your Step-By-Step Guide

• Beginning your data journey with Unravel is as easy as 1-2-3. We guide you through the sign-up process, ensuring a smooth and hassle-free setup.
• Unravel for Databricks page
  

• Unravel for Databricks plans & pricing
  

• Unravel for Databricks free account
  

Level Up with Unravel Premium

Ready for an enhanced data experience? Unravel’s premium account offers a plethora of advanced features that the free version can’t match. Investing in this upgrade isn’t just about more tools; it’s about supercharging your data operations and ROI.

Wrap-Up

Although rising demands on the modern data landscape are challenging, they are not insurmountable. With tools like Unravel, organizations can navigate these complexities, ensuring that data remains a catalyst for growth, not a hurdle. Dive into the Unravel experience and redefine your data journey today.

Unravel is a business’s performance sentinel in the cloud realm, proactively ensuring that burgeoning cloud data expenses are not only predictable and manageable but also primed for significant cost savings. Unravel Data transforms the precarious balance of budget and talent into a streamlined, efficient journey for organizations. Unravel Data illuminates the path for data-centric organizations, streamlining operations with AI tools, ensuring data security, and optimizing performance. Its intuitive interface simplifies complex data landscapes, bridging talent gaps and converting challenges into actionable insights.

The post Overcoming Friction & Harnessing the Power of Unravel: Try It for Free appeared first on Unravel.

“Data is the new oil, and analytics is the combustion engine.” – Peter Sondergaard

Cloud data analytics is the key to maximizing value from your data. The lakehouse has emerged as a flexible and efficient architecture, and Databricks has emerged as a popular choice. However, data lakehouse processing volumes can fluctuate, leading to unpredictable surges in cloud data spending that impact budgeting and profitability. Executives want to make sure they are getting the most from their lakehouse investments and not overspending.

Implementing a proactive data observability and FinOps approach early in your lakehouse journey helps ensure you achieve your business objectives and bring predictability to your financial planning. Choosing the right lakehouse observability and FinOps tool sets your team up for success. Since the goal is efficiency, starting with free tools makes sense. Two free options stand out:

• Overwatch – the open source Databricks observability tool
• Unravel Standard – the free version of Unravel’s data observability and FinOps platform

Below are 5 reasons to choose Unravel free observability vs. Databricks free observability:

Reason #1: Complete observability

Many organizations take a do-it-yourself approach, building piecemeal observability solutions in-house by cobbling together a variety of data sources using open source tools. The problem is that it takes months or even years to get something usable up and running. Unravel’s data observability and FinOps platform helps you get results fast.

Unravel provides a 360° out-of-the-box solution

Unravel provides a holistic view of your Databricks estate, reducing the time to value. Gain deep insights into cluster performance, job execution, resource utilization, and cost drivers through comprehensive lakehouse observability. Unravel’s observability solution provides you with detailed visibility into the performance of your Databricks clusters. 

Insights overview dashboard in Unravel Standard

Ensure no blind spots in your analysis by leveraging Unravel’s end-to-end visibility across all aspects of your Databricks environment. View your lakehouse landscape at a glance with the Insights Overview dashboard. You can see the overall health of your Databricks estate, including the number of clusters that are over- or under-provisioned, the total number of inefficient and failed apps, and other summary statistics to guide your efforts to optimize your lakehouse towards better performance and cost efficiency.

Purpose-built correlation

Unravel’s purpose-built correlation models help you identify inefficient jobs at code, data layout/partitioning, and infrastructure levels. Databricks logs, metrics, events, traces, and source code are automatically evaluated to simplify root cause analysis and issue resolution. You can dive deep into the execution details of your Databricks jobs, track the progress of each job, and see resource usage details. This helps you identify long-running and resource-intensive jobs that might be impacting the overall performance and efficiency of your lakehouse estate.

End-to-end visibility

Visual summaries provide a way to look across all the jobs and clusters in your Databricks workspace. No need to click around your Databricks workspace looking for issues, run queries, or pull details into a spreadsheet to summarize results. Unravel helps you easily see all the details in one place. 

Reason #2: Real-time visibility

A single errant job or underutilized cluster can derail your efficiency goals and delay critical data pipelines. The ability to see job and cluster performance and efficiency in real time provides an early warning system.

Live updates for running jobs and clusters

React promptly to any anomalies or bottlenecks in your clusters and jobs to ensure efficiency. Unravel’s real-time insights allow you to catch long-running jobs before they impact pipeline performance or consume unnecessary resources.

Workspace Trends dashboard in Unravel Standard

See DBU usage and cluster session trends

By understanding the real-time performance of your Databricks workloads, you can identify areas where improvements can be made to improve efficiency without sacrificing performance. Leverage Unravel’s real-time insights to make data-driven decisions for better resource allocation and workload management. 

Drill down to see DBU usage and tasks for a specific day

Quickly find resource consumption outliers by day to understand how usage patterns are driving costs. Unravel helps you identify opportunities to reduce waste and increase cluster utilization. By having visibility into the real-time cost implications of your jobs and clusters, you can make faster decisions to boost performance and improve business results.

User-level reporting for showback/chargeback

Granular reporting to the user and job level helps you produce accurate and timely showback and chargeback reports. With Unravel’s real-time visibility into your Databricks workloads, you have the power to see which teams are consuming the most resources and proactively manage costs to achieve efficient operations. Reacting quickly to anomalies and leveraging real-time, user-level insights enables better decision-making for resource allocation and utilization. Unravel enables central data platform and operations teams to provide a reliable, single source of truth for showback and chargeback reporting.

Reason #3: Automated Cluster Discovery

You can’t fix problems you can’t see. It all begins with getting visibility across all your workspace clusters and jobs. Unravel automates this process to save you time and ensure you don’t miss anything.

Easily connect to all of your clusters in the workspace

Simplify the process of connecting to your Databricks clusters with Unravel’s automated cluster discovery. This streamlines the observability and management of your compute clusters, so you can focus on resource optimization to boost productivity. Unravel lets you easily see all of your clusters without adding dependencies.

Compute dashboard shows clusters in Unravel Standard

Quickly discover clusters with errors, delays, and failures

Unravel lets you see clusters grouped by event type (e.g., Contended Driver, High I/O, Data Skew, Node Downsizing). This helps you quickly identify patterns in compute clusters that are not being fully utilized. This eliminates the need for manual monitoring and analysis, saving you time and effort.

View cluster resource trends

Unravel’s intelligent automation continuously monitors cluster activity and resource utilization over time. This helps you spot changing workload requirements and helps ensure optimal performance while keeping costs in check by avoiding overprovisioning or underutilization to make the most of your cloud infrastructure investments.

Reason #4: Ease of Entry

Open source and DIY solutions typically have a steep learning curve to ensure everything is correctly configured and connected. Frequent changes and updates add to your team’s already full load. Unravel offers a simpler approach.

Unravel is quick to set up and get started with minimal learning curve

Integrating Unravel into your existing Databricks environment is a breeze. No complex setup or configuration required. With Unravel, you can seamlessly bring data observability and FinOps capabilities to your data lakehouse estate without breaking a sweat.

Unravel SaaS makes setup and configuration a breeze

But what exactly does this mean for you? It means that you can focus on what matters most—getting the most out of your Databricks platform while keeping costs in check. Unravel’s data observability and FinOps capabilities are provided as a fully managed service, giving you the power to optimize performance and resources, spot efficiency opportunities, and ensure smooth operation of your data pipelines and data applications.

No DIY coding or development required

Unravel is trusted by large enterprise customers across many industries for its ease of integration into their Databricks environments. Whether you’re a small team or an enterprise organization, Unravel’s data observability and FinOps platform is designed to meet your specific needs and use cases without the need to build anything from scratch.

Reason #5: Avoid lock-in

A lakehouse architecture gives you flexibility. As your data analytics and data processing needs grow and evolve, you may choose additional analytics tools to complement your cloud data estate. Your data observability and FinOps tool should support those tools as well.

Unravel is purpose-built for Databricks, Snowflake, BigQuery, and other modern data stacks

Each cloud data platform is different and requires a deep understanding of its inner workings in order to provide the visibility you need to run efficiently. Unravel is designed from the ground up to help you get the most out of each modern data platform, leveraging the most relevant and valuable metadata sources and correlating them all into a unified view of your data estate.

No need to deploy a separate tool as your observability needs grow

Unravel provides a consistent approach to data observability and FinOps to minimize time spent deploying and learning new tools. Data teams spend less time upskilling and more time getting valuable insights.

Independent reporting for FinOps

Data analytics is the fastest growing segment of cloud computing as organizations invest in new use cases such as business intelligence (BI), AI and machine learning. Organizations are adopting FinOps practices to ensure transparency in resource allocation, usage, and reporting. Unravel provides an independent perspective of lakehouse utilization and efficiency to ensure objective, data-driven decisions.

Compare Unravel and Databricks free observability

Get started today

Achieve predictable spend and gain valuable insights into your Databricks usage. Get started today with Unravel’s complete data observability and FinOps platform for Databricks that provides real-time visibility, automated cluster discovery, ease of entry, and independent analysis to help you take control of your costs while maximizing the value of your Databricks investments. Create your free Unravel account today.

Unravel for Databricks FAQ 

Can I use Unravel’s data observability platform with other cloud providers?

Yes. Unravel’s data observability platform is designed to work seamlessly across multiple cloud providers including AWS, Azure, and Google Cloud. So regardless of which cloud provider you choose for your data processing needs, Unravel can help you optimize costs and gain valuable insights.

How does automated cluster discovery help in managing Databricks costs?

Automated cluster discovery provided by solutions like Unravel enables you to easily identify underutilized or idle clusters within your Databricks environment. By identifying these clusters, you can make informed decisions about resource allocation and ensure that you are only paying for what you actually need.

Does Unravel offer real-time visibility into my Databricks usage?

Yes. With Unravel’s real-time visibility feature, you can monitor your Databricks usage in real time. This allows you to quickly identify any anomalies or issues that may impact cost efficiency and take proactive measures to address them.

Can Unravel help me optimize my Databricks costs for different workloads?

Yes. Unravel’s data observability platform provides comprehensive insights into the performance and cost of various Databricks workloads. By analyzing this data, you can identify areas for optimization and make informed decisions to ensure cost efficiency across different workloads.

How easy is it to get started with Unravel’s data observability platform?

Getting started with Unravel is quick and easy. Simply sign up for a free account on our website, connect your Databricks environment, and start gaining valuable insights into your usage and costs. Our intuitive interface and user-friendly features make it simple for anyone to get started without any hassle.

The post Databricks Cost Efficiency: 5 Reasons Why Unravel Free Observability vs. Databricks Free Observability appeared first on Unravel.

Hey there! Have you ever found yourself scratching your head over unpredictable cloud data costs? It’s no secret that accurately forecasting cloud data spend can be a real headache. Fluctuating costs make it challenging to plan and allocate resources effectively, leaving businesses vulnerable to budget overruns and financial challenges. But don’t worry, we’ve got you covered!

Uncontrolled fluctuations in cloud data spend can hinder growth and profitability, disrupting the smooth sailing of your business operations. That’s why it’s crucial to gain control over your cloud data workload forecasts. By understanding the changes in your cloud data spend and having a clear picture of your billing data, you can make informed decisions that align with your company’s goals.

We’ll discuss practical strategies to improve forecast accuracy, identify data pipelines and analytics workloads that are above or below budget, and enhance accountability across different business units.

So let’s dive right in and discover how you can steer your business towards cost-effective cloud management!

Unanticipated cloud data spend

Last year, over $16 billion was wasted in cloud spend. Data management is the largest and fastest-growing category of cloud spending, representing 39% of the typical cloud bill. Gartner noted that in 2022, 98% of the overall database management system (DBMS) market growth came from cloud-based database platforms. Cloud data costs are often the most difficult to predict due to fluctuating workloads. 82% of 157 data management professionals surveyed by Forrester cited difficulty predicting data-related cloud costs. On top of the fluctuations that are inherent with data workloads, a lack of visibility into cloud data spend makes it challenging to manage budgets effectively.

• Fluctuating workloads: Data processing and storage costs are driven by the amount of data stored and analyzed. With varying workloads, it becomes challenging to accurately estimate the required data processing and storage costs. This unpredictability can result in budget overruns that affect 60% of infrastructure and operations (I&O) leaders.
• Unexpected expenses: Streaming data, large amounts of unstructured and semi-structured data, and shared slot pool consumption can quickly drive up cloud data costs. These factors contribute to unforeseen spikes in usage that may catch organizations off guard, leading to unexpected expenses on their cloud bills.
  
• Lack of visibility: Without granular visibility into cloud data analytics billing information, businesses have no way to accurately allocate costs down to the job or user level. This makes it difficult for them to track usage patterns and identify areas where budgets will be over- or under-spent, or where performance and cost optimization are needed.
  

By implementing a FinOps approach, businesses can gain better control over their cloud data spend, optimize their budgets effectively, and avoid unpleasant surprises when it comes time to pay the bill.

Why cloud data costs are so unpredictable

Cloud data costs can be a source of frustration for many businesses due to their unpredictability. Cloud and data platform providers often have complex pricing models that make it challenging to accurately forecast expenses. Here are some key reasons why cloud data analytics costs can be so difficult to predict:

• Variety of factors affect analytics costs: Cloud and data platform providers offer a range of services and pricing options, making it hard to determine the exact cost of using specific resources. Factors such as compute instance and cluster sizes, regional pricing, and additional features all contribute to the final cloud and data platform bill.

• Usage patterns impact cost: Fluctuations in usage patterns can significantly affect cloud data costs. Peak demand periods or sudden increases in data volume can result in unexpected expenses. Without proper planning, businesses may find themselves facing higher bills during these periods.

• Lack of visibility into resource utilization: Inefficient workload management and a lack of visibility into resource utilization can lead to higher expenses. Without the ability to monitor and optimize resource allocation, businesses may end up paying for unused or underutilized resources.

• Inability to allocate historical spend: A lack of granular visibility into data costs at the job, project, and user level makes it nearly impossible to accurately allocate historical spend or forecast future investments. This makes budgeting and financial planning challenging for businesses that rely on cloud data platforms.

• Changes in technology or service offerings: Cloud and data platform providers frequently introduce new technologies or adjust their service offerings, which can impact cost structures. Businesses must stay updated on these changes as they may influence their overall cloud expenditure.

Navigating the complexities of cloud data forecasting requires careful analysis and proactive management or resource consumption fluctuations and cost unpredictability. By understanding usage patterns, optimizing capacity utilization, and staying informed about changes from cloud and data platform providers, businesses can gain better control over their cloud data costs.

5 key ingredients of an accurate cloud data cost forecast

To ensure an accurate cloud data cost forecast, several key ingredients must be considered. These include:

1. Comprehensive understanding of historical usage patterns and trends: Analyzing past usage data provides valuable insights into resource consumption and enables more accurate predictions of future spending.
2. Granular visibility into data resource usage: It is essential to have detailed visibility into the utilization of resources down to the job and user level. This level of granularity enables a more precise estimation of costs associated with specific tasks or individuals.
3. Analysis of current platform configurations and workload requirements: Evaluating the existing data platform settings, data access patterns, and workload demands help predict growth rates and changes in cloud data spend.
4. Consideration of external factors: External factors such as market conditions or regulatory changes can significantly impact cloud data processing costs. Incorporating these variables into the forecasting model ensures a more realistic projection.
5. Utilization of advanced forecasting techniques and algorithms: Leveraging advanced techniques and algorithms enhances the accuracy of predictions by accounting for various factors simultaneously, resulting in more reliable forecasts.

By incorporating these key ingredients into your cloud data forecasting strategy, you can gain better control over your forecast and achieve higher accuracy in predicting future expenses. With a comprehensive understanding of historical patterns, granular visibility into resource usage, analysis of configurations and workload requirements, consideration of external influences, and advanced forecasting techniques, you can make informed decisions to increase the accuracy of your cloud data spend forecasts.

Remember that accurate cloud data cost forecasting is crucial for effective financial planning within your organization’s cloud environment.

Explore different methods and tools for accurate cloud data forecasting

Statistical modeling techniques like time series analysis can be used to predict future trends based on patterns in historical data. These predictive models help improve forecast accuracy by identifying recurring patterns and extrapolating them into the future.

Machine learning algorithms offer another powerful tool for cloud data forecasting. By analyzing vast amounts of information, these algorithms can generate accurate forecasts by identifying complex relationships and trends within the data. This enables organizations to make informed estimates of their cloud data processing patterns to help anticipate cyclical usage and growth.

Cloud management platforms provide built-in forecasting capabilities. However, it is important to note that forecasts are only as good as the underlying data. Without granular visibility into the jobs, projects, and users utilizing the cloud resources, forecast models cannot take key drivers into account. To ensure accurate predictions, it is crucial to distinguish one-time or end-of-period data processing from ongoing processing related to growing customer and end-user activity.

How purpose-built AI improves cloud data cost forecasting

Purpose-built AI is a game-changer. By leveraging advanced algorithms and machine learning capabilities, businesses can unlock the full potential of their cloud data cost management. 

Here’s how purpose-built AI improves cloud data cost forecasting:

• Identifying hidden cost drivers: Purpose-built AI has the ability to analyze vast amounts of data and identify subtle factors that contribute to increased costs. It goes beyond surface-level analysis and uncovers underlying patterns, enabling businesses to accurately anticipate cloud analytics resource needs.

• Continuous learning for improved accuracy: Machine learning models are continuously trained on past performance, enabling them to learn from historical data and improve accuracy over time. This means that as more data becomes available, the forecasts become increasingly reliable.

• Proactive decision-making with predictive analytics: Predictive analytics powered by purpose-built AI enable businesses to make proactive decisions regarding their cloud expenditure. By analyzing trends and patterns, organizations can anticipate future costs and take necessary steps to avoid unnecessary expenses or mitigate risks associated with fluctuating costs.

• AI-driven recommendations for enhanced cost efficiency: Purpose-built AI provides valuable recommendations based on its analysis of cloud data cost patterns. These recommendations help businesses improve their overall cost efficiency, ensuring that resources are allocated optimally to accelerate cloud data platform usage.

Using Unravel to Improve Cloud Data Cost Forecast Accuracy to within ±10%

These different approaches help improve the accuracy of your cloud data cost forecasts. But how can you take it a step further? That’s where Unravel comes in.

Unravel is a purpose-built AI platform that can revolutionize the accuracy of your cloud data cost forecasts. Unravel provides real-time insights into your data usage patterns, identifies budget trends, and predicts future costs with remarkable accuracy. With its intuitive interface and easy-to-use features, Unravel empowers you to make informed decisions about resource allocation, budget planning, and overall cost management in the cloud.

Ready to take control of your cloud data costs? Start using Unravel today and unlock the full potential of accurate cloud data cost forecasting.

FAQs

Q: How does Unravel improve cloud data cost forecasting?

A: Unravel leverages advanced machine learning algorithms to analyze historical usage patterns and identify trends in your cloud data costs. By understanding these patterns, it can accurately predict future costs and provide actionable insights for optimizing resource allocation.

Q: Can I integrate Unravel with my existing cloud data platform?

A: Yes! Unravel seamlessly integrates with popular cloud platforms such as AWS, Azure, and Google Cloud Platform. It supports both on-premises and hybrid environments as well.

Q: Is Unravel suitable for businesses of all sizes?

A: Absolutely! Whether you’re a small startup or a large enterprise, Unravel caters to businesses of all sizes with large data estates looking to leverage data to maximize business value. Unravel’s scalable architecture ensures that it can handle the needs of organizations with 10,000s of data platform jobs, such as DBS Bank.

Q: How long does it take to see results with Unravel?

A: You’ll start seeing immediate benefits once you integrate Unravel into your infrastructure. Its real-time insights and actionable recommendations allow you to make informed decisions right from the get-go.

Q: Can Unravel help with other aspects of cloud data management?

A: Yes, Unravel offers a comprehensive suite of features for end-to-end cloud data performance management, FinOps for your data cloud, intelligent data quality, and forecast accuracy. From performance optimization to cost governance, Unravel provides a holistic solution for your cloud analytics needs.

The post 5 Key Ingredients to Accurate Cloud Data Budget Forecasting appeared first on Unravel.

I was an Enterprise Data Architect at Boeing NASA Systems the day of the Columbia Shuttle disaster. The tragedy has had a profound impact in my career on how I look at data. Recently I have been struck by how, had it been available back then, purpose-built AI observability might have altered the course of events. 

The Day of the Disaster

With complete reverence and respect for the Columbia disaster, I can remember that day with incredible amplification. I was standing in front of my TV watching the Shuttle Columbia disintegrate on reentry. As the Enterprise Data Architect at Boeing NASA Systems, I received a call from my boss: “We have to pull and preserve all the Orbitor data. I will meet you at the office.” Internally, Boeing referred to the Shuttles as Oribitors. It was only months earlier I had the opportunity to share punch and cookies with some of the crew. To say I was being flooded with emotions would be a tremendous understatement.

When I arrived at Boeing NASA Systems, near Johnson Space Center, the mood in the hallowed aeronautical halls was somber. I sat intently, with eyes scanning lines of data from decades of space missions. The world had just witnessed a tragedy — the Columbia Shuttle disintegrated on reentry, taking the lives of seven astronauts. The world looked on in shock and sought answers. As a major contractor for NASA, Boeing was at the forefront of this investigation.

As the Enterprise Data Architect, I was one of an army helping dissect the colossal troves of data associated with the shuttle, looking for anomalies, deviations — anything that could give a clue as to what had gone wrong. Days turned into nights, and nights turned into weeks and months as we tirelessly pieced together Columbia’s final moments. But as we delved deeper, a haunting reality began to emerge.

Every tiny detail of the shuttle was monitored, from the heat patterns of its engines to the radio signals it emitted. But there was a blind spot, an oversight that no one had foreseen. In the myriad of data sets, there was nothing that indicated the effects of a shuttle’s insulation tiles colliding with a piece of Styrofoam, especially at speeds exceeding 500 miles per hour.

The actual incident was seemingly insignificant — a piece of foam insulation had broken off and struck the shuttle’s left wing. But in the vast expanse of space and the brutal conditions of reentry, even minor damage could prove catastrophic.

Video footage confirmed: the foam had struck the shuttle. But without concrete data on what such an impact would do, the team was left to speculate and reconstruct potential scenarios. The lack of this specific data had turned into a gaping void in the investigation.

As a seasoned Enterprise Data Architect, I always believed in the power of information. I absolutely believed that in the numbers, in the bytes and bits, we find the stories that the universe whispers to us. But this time, the universe had whispered a story that we didn’t have the data to understand fully.

Key Findings

After the accident, NASA formed the Columbia Accident Investigation Board (CAIB) to investigate the disaster. The board consisted of experts from various fields outside of NASA to ensure impartiality.

1. Physical Cause: The CAIB identified the direct cause of the accident as a piece of foam insulation from the shuttle’s external fuel tank that broke off during launch. This foam struck the leading edge of the shuttle’s left wing. Although foam shedding was a known issue, it had been wrongly perceived as non-threatening because of prior flights where the foam was lost but didn’t lead to catastrophe.

2. Organizational Causes: Beyond the immediate physical cause, the CAIB highlighted deeper institutional issues within NASA. They found that there were organizational barriers preventing effective communication of safety concerns. Moreover, safety concerns had been normalized over time due to prior incidents that did not result in visible failure. Essentially, since nothing bad had happened in prior incidents where the foam was shed, the practice had been erroneously deemed “safe.”

3. Decision Making: The CAIB pointed out issues in decision-making processes that relied heavily on past success as a predictor of future success, rather than rigorous testing and validation.

4. Response to Concerns: There were engineers who were concerned about the foam strike shortly after Columbia’s launch, but their requests for satellite imagery to assess potential damage were denied. The reasons were multifaceted, ranging from beliefs that nothing could be done even if damage was found, to a misunderstanding of the severity of the situation.

The CAIB made a number of recommendations to NASA to improve safety for future shuttle flights. These included:

1. Physical Changes: Improving the way the shuttle’s external tank was manufactured to prevent foam shedding and enhancing on-orbit inspection and repair techniques to address potential damage.

2. Organizational Changes: Addressing the cultural issues and communication barriers within NASA that led to the accident, and ensuring that safety concerns were more rigorously addressed.

3. Continuous Evaluation: Establishing an independent Technical Engineering Authority responsible for technical requirements and all waivers to them, and building an independent safety program that oversees all areas of shuttle safety.

Could Purpose-built AI Observability Have Helped?

In the aftermath, NASA grounded the shuttle fleet for more than two years after the disaster. They then implemented the CAIB’s recommendations before resuming shuttle flights. Columbia’s disaster, along with the Challenger explosion in 1986, are stark reminders of the risks of space travel and the importance of a diligent and transparent safety culture. The lessons from Columbia shaped many of the safety practices NASA follows in its current human spaceflight programs.

The Columbia disaster led to profound changes in how space missions were approached, with a renewed emphasis on data collection and eliminating informational blind spots. But for me, it became a deeply personal mission. I realized that sometimes, the absence of data could speak louder than the most glaring of numbers. It was a lesson I would carry throughout my career, ensuring that no stone was left unturned, and no data point overlooked.

The Columbia disaster, at its core, was a result of both a physical failure (foam insulation striking the shuttle’s wing) and organizational oversights (inadequate recognition and response to potential risks). Purpose-built AI Observability, which involves leveraging artificial intelligence to gain insights into complex systems and predict failures, might have helped in several key ways:

1. Real-time Anomaly Detection: Modern AI systems can analyze vast amounts of data in real time to identify anomalies. If an AI-driven observability platform had been monitoring the shuttle’s various sensors and systems, it might have detected unexpected changes or abnormalities in the shuttle’s behavior after the foam strike, potentially even subtle ones that humans might overlook.

2. Historical Analysis: An AI system with access to all previous shuttle launch and flight data might have detected patterns or risks associated with foam-shedding incidents, even if they hadn’t previously resulted in a catastrophe. The system could then raise these as potential long-term risks.

3. Predictive Maintenance: AI-driven tools can predict when components of a system are likely to fail based on current and historical data. If applied to the shuttle program, such a system might have provided early warnings about potential vulnerabilities in the shuttle’s design or wear-and-tear.

4. Decision Support: AI systems could have aided human decision-makers in evaluating the potential risks of continuing the mission after the foam strike, providing simulations, probabilities of failure, or other key metrics to help guide decisions.

5. Enhanced Imaging and Diagnosis: If equipped with sophisticated imaging capabilities, AI could analyze images of the shuttle (from external cameras or satellites) to detect potential damage, even if it’s minor, and then assess the risks associated with such damage.

6. Overcoming Organizational Blind Spots: One of the major challenges in the Columbia disaster was the normalization of deviance, where foam shedding became an “accepted” risk because it hadn’t previously caused a disaster. An AI system, being objective, doesn’t suffer from these biases. It would consistently evaluate risks based on data, not on historical outcomes.

7. Alerts and Escalations: An AI system can be programmed to escalate potential risks to higher levels of authority, ensuring that crucial decisions don’t get caught in bureaucratic processes.

While AI Observability could have provided invaluable insights and might have changed the course of events leading to the Columbia disaster, it’s essential to note that the integration of such AI systems also requires organizational openness to technological solutions and a proactive attitude toward safety. The technology is only as effective as the organization’s willingness to act on its findings.

The tragedy served as a grim reminder for organizations worldwide: It’s not just about collecting data; it’s about understanding the significance of what isn’t there. Because in those blind spots, destiny can take a drastic turn.

In Memory and In Action

The Columbia crew and their families deserve our utmost respect and admiration for their unwavering commitment to space exploration and the betterment of humanity. 

• Rick D. Husband: As the Commander of the mission, Rick led with dedication, confidence, and unparalleled skill. His devotion to space exploration was evident in every decision he made. We remember him not just for his expertise, but for his warmth and his ability to inspire those around him. His family’s strength and grace, in the face of the deepest pain, serve as a testament to the love and support they provided him throughout his journey.
• William C. McCool: As the pilot of Columbia, William’s adeptness and unwavering focus were essential to the mission. His enthusiasm and dedication were contagious, elevating the spirits of everyone around him. His family’s resilience and pride in his achievements are a reflection of the man he was — passionate, driven, and deeply caring.
• Michael P. Anderson: As the payload commander, Michael’s role was vital, overseeing the myriad of experiments and research aboard the shuttle. His intellect was matched by his kindness, making him a cherished member of the team. His family’s courage and enduring love encapsulate the essence of Michael’s spirit — bright, optimistic, and ever-curious.
• Ilan Ramon: As the first Israeli astronaut, Ilan represented hope, unity, and the bridging of frontiers. His enthusiasm for life was infectious, and he inspired millions with his historic journey. His family’s grace in the face of the unthinkable tragedy is a testament to their shared dream and the values that Ilan stood for.
• Kalpana Chawla: Known affectionately as ‘KC’, Kalpana’s journey from a small town in India to becoming a space shuttle mission specialist stands as an inspiration to countless dreamers worldwide. Her determination, intellect, and humility made her a beacon of hope for many. Her family’s dignity and strength, holding onto her legacy, reminds us all of the power of dreams and the sacrifices made to realize them.
• David M. Brown: As a mission specialist, David brought with him a zest for life, a passion for learning, and an innate curiosity that epitomized the spirit of exploration. He ventured where few dared and achieved what many only dreamt of. His family’s enduring love and their commitment to preserving his memory exemplify the close bond they shared and the mutual respect they held for each other.
• Laurel B. Clark: As a mission specialist, Laurel’s dedication to scientific exploration and discovery was evident in every task she undertook. Her warmth, dedication, and infectious enthusiasm made her a beloved figure within her team and beyond. Her family’s enduring spirit, cherishing her memories and celebrating her achievements, is a tribute to the love and support that were foundational to her success.

To each of these remarkable individuals and their families, we extend our deepest respect and gratitude. Their sacrifices and contributions will forever remain etched in the annals of space exploration, reminding us of the human spirit’s resilience and indomitable will.

For those of us close to the Columbia disaster, it was more than a failure; it was a personal loss. Yet, in memory of those brave souls, we are compelled to look ahead. In the stories whispered to us by data, and in the painful lessons from their absence, we seek to ensure that such tragedies remain averted in the future.

While no technology can turn back time, the promise of AI Observability beckons a future where every anomaly is caught, every blind spot illuminated, and every astronaut returns home safely.

The above narrative seeks to respect the gravity of the Columbia disaster while emphasizing the potential of AI Observability. It underlines the importance of data, both in understanding tragedies and in preventing future ones.

The post Blind Spots in Your System: The Grave Risks of Overlooking Observability appeared first on Unravel.

The data landscape is constantly evolving, and with it come new challenges and opportunities for data teams. While generative AI and large language models (LLMs) seem to be all everyone is talking about, they are just the latest manifestation of a trend that has been evolving over the past several years: organizations tapping into petabyte-scale data volumes and running increasingly massive data pipelines to deliver ever more data analytics projects and AI/ML models. 

The scale and pace of data consumption is rising exponentially.

Data is now core to every business. Untapping its potential to uncover business insights, drive operational efficiencies, and develop new data products is a key competitive differentiator that separates winners from losers. But running the enormous data workloads that fuel such innovation is expensive. Businesses are already struggling to keep their cloud data costs within budget. So, how can companies increase business-critical data output without sending cloud data costs through the roof?

Effective cost management becomes paramount. That’s where FinOps principles come into play, helping organizations to optimize their cloud resources and align them with business goals. 

A recent virtual fireside chat with Sanjeev Mohan, former Gartner Research VP for Big Data & Advanced Analytics and founder/principal of SanjMo, Unravel VP of Solutions Engineering Chris Santiago, and DataOps Champion and Certified FinOps Practitioner Clinton Ford discussed five tips for getting started with FinOps for data workloads. 

Why FinOps for Data Makes Sense

The virtual event kicked off discussing Gartner analyst Lydia Leong’s argument that building a dedicated FinOps team is a waste of time. Our panelists broke down why FinOps for data teams is actually crucial for companies running big data workloads in the cloud. Then they talked about how Spotify fixed a $1 million query, using that as an example of FinOps principles in practice. 

The experts emphasized that having a clear strategy and plan in place is critical for ensuring that resources are allocated effectively and in line with business objectives.

Five Best Practices for DataFinOps

To set data teams on the path to FinOps success, Sanjeev Mohan and Chris Santiago shared five practical tips during the presentation:

1. Discover: Begin by tracking and visualizing your costs at three different layers: the query level, user level, and workload level. You must first know where the money is going, to ensure that resources are properly aligned.
2. Understand Cost Drivers: Dig into the top cost drivers—what’s actually incurring the most cost—at a granular level. Not only is this vital for accurately forecasting budgets, but you can prioritize workloads based on their strategic value, ensuring that you’re focusing on tasks that contribute meaningfully to your bottom line.
3. Collaborate: FinOps is a team sport among Finance, Engineering, and the business. For many organizations, this is a cultural shift as much as anything. A good start is to leverage chargeback/showback models to hold teams accountable and encourage better management of resources.
4. Build Early Warning Systems: Implement guardrails to nip cost overruns in the bud. It’s best to catch performance/cost problems early on in development, whether as part of your CI/CD process or even as simple as triggering an alert when some cost/performance threshold is violated.
5. Automate and Optimize: Continuously monitor, automate, and optimize key processes to minimize waste, save time, and achieve better results.

Audience Questions

In addition to discussing FinOps best practices, the panelists fielded several questions from the audience. They addressed topics such as calculating unit costs, selecting impactful visualization tools, and employing cost reduction strategies tailored for their organizations. Throughout the session, the experts emphasized collaboration and partnership, showcasing Unravel’s commitment to empowering data teams to reach their full potential.

The Unravel Advantage

With its AI-powered Insights Engine built specifically for modern data platforms like Databricks, Snowflake, Google Cloud BigQuery, and Amazon EMR, Unravel provides data teams with the performance/cost optimization insights and automation they need to thrive in the competitive data landscape. Just as the recent case study of a leading health insurance provider demonstrated, Unravel’s capabilities are instrumental in helping organizations optimize code and infrastructure to help run more data workloads without increasing their budget. The five FinOps best practices shared by Sanjeev and Chris offer actionable insights for data teams looking to optimize costs, drive efficiency, and achieve their goals in an ever-changing data landscape.

With Unravel as your trusted partner, you can approach FinOps with confidence, knowing that you have access to the expertise, tools, and support required to succeed.

Next Steps

• Schedule a demo with Unravel to discover how it can revolutionize your cost optimization efforts.
  
• Download Eckerson Group’s Governing Cost with FinOps for Cloud Analytics.

The post Unlocking Success with FinOps: Top Insights from Expert Virtual Event appeared first on Unravel.

Data is a powerful force that can generate business value with immense potential for businesses and organizations across industries. Leveraging data and analytics has become a critical factor for successful digital transformation that can accelerate revenue growth and AI innovation. Data and AI leaders enable business insights, product and service innovation, and game-changing technology that helps them outperform their peers in terms of operational efficiency, revenue, and customer retention, among other key business metrics. Organizations that fail to harness the power of data are at risk of falling behind their competitors.

Despite all the benefits of data and AI, businesses face common challenges.

Unanticipated cloud data spend

Last year, over $16 billion was wasted in cloud spend. Data management is the largest and fastest-growing category of cloud spending, representing 39% of the typical cloud bill. Gartner noted that in 2022, 98% of the overall database management system (DBMS) market growth came from cloud-based database platforms. Cloud data costs are often the most difficult to predict due to fluctuating workloads. 82% of 157 data management professionals surveyed by Forrester cited difficulty predicting data-related cloud costs. On top of the fluctuations that are inherent with data workloads, a lack of visibility into cloud data spend makes it challenging to manage budgets effectively.

• Fluctuating workloads: Google Cloud BigQuery data processing and storage costs are driven by the amount of data stored and analyzed. With varying workloads, it becomes challenging to accurately estimate the required data processing and storage costs. This unpredictability can result in budget overruns that affect 60% of infrastructure and operations (I&O) leaders.
• Unexpected expenses: Streaming data, large amounts of unstructured and semi-structured data, and shared slot pool consumption can quickly drive up cloud data costs. These factors contribute to unforeseen spikes in usage that may catch organizations off guard, leading to unexpected expenses on their cloud bills.
• Lack of visibility: Without granular visibility into cloud data analytics billing information, businesses have no way to accurately allocate costs down to the job or user level. This makes it difficult for them to track usage patterns and identify areas where budgets will be over- or under-spent, or where performance and cost optimization are needed.

By implementing a FinOps approach, businesses can gain better control over their cloud data spend, optimize their budgets effectively, and avoid unpleasant surprises when it comes time to pay the bill.

Budget and staff constraints limit new data workloads

In 2023, CIOs are expecting an average increase of only 5.1% in their IT budgets, which is lower than the projected global inflation rate of 6.5%. Economic pressures, scarcity and high cost of talent, and ongoing supply challenges are creating urgency to achieve more value in less time.

Limited budget and staffing resources can hinder the implementation of new data workloads. For example, “lack of resources/knowledge to scale” is the leading reason preventing IoT data deployments. Budget and staffing resources constraints pose real risks to launching profitable data and AI projects.

Exponential data volume growth for AI

The rapid growth of disruptive technologies such as generative AI, has led to an exponential increase in cloud computing data volumes. However, managing and analyzing massive amounts of data poses significant challenges for organizations.

Data is foundational for AI and much of it is unstructured, yet IDC found most unstructured data is not leveraged by organizations. A lack of production-ready data pipelines for diverse data sources was the second most cited reason (31%) for AI project failure.

Data pipeline failures slow innovation

Data pipelines are becoming increasingly complex, increasing the Mean Time To Repair (MTTR) breaks and delays. Time is a critical factor that pulls skilled and valuable talent into unproductive firefighting. The more time they spend dealing with pipeline issues or failures, the greater the impact on productivity and new innovation.

Manually testing and running release process checklists are heavy burdens for new and growing data engineering teams. With all of the manual toil, it is no surprise that over 70% of data projects in manufacturing stall at Proof of Concept (PoC) stage and do not see sustainable value realization.

Downtime resulting from pipeline disruptions can have a significant negative impact on the service level agreements (SLAs). It not only affects the efficiency of data processing, but also impacts downstream tasks like analysis and reporting. These slowdowns directly affect the ability of team members and business leaders to make timely decisions based on data insights.

Conclusion

Unravel 4.8.1 for BigQuery provides improved visibility to accelerate performance, boost query efficiency, allocate costs, and accurately predict spend. This launch aligns with the recent BigQuery pricing model change. With Unravel for BigQuery, customers can easily choose the best pricing plan to match their usage. Unravel helps you optimize your workloads and get more value from your cloud data investments.

The post Harnessing Google Cloud BigQuery for Speed and Scale: Data Observability, FinOps, and Beyond appeared first on Unravel.

Google recently introduced significant changes to its existing BigQuery pricing models, affecting both compute and storage. They announced the end of sale for flat-rate and flex slots for all BigQuery customers not currently in a contract. Google announced an increase to the price of on-demand analysis by 25% across all regions, starting on July 5, 2023.

Main Components of BigQuery Pricing

Understanding the pricing structure of BigQuery is crucial to effectively manage expenses. There are two big components to BigQuery pricing:

• Compute (analysis) pricing is the cost to process queries, including SQL queries, user-defined functions, scripts, and certain data manipulation language (DML) and data definition language (DDL) statements
• Storage pricing is the cost to store data that you load into BigQuery. Storage options are logical (the default) or physical. If data storage is converted from logical to physical, customers cannot go back to logical storage.

Selecting the appropriate edition and accurately forecasting data processing needs is essential to cloud data budget planning and maximizing the value derived from Google Cloud BigQuery.

Introducing Unravel 4.8.1 for BigQuery

Unravel 4.8.1 for BigQuery includes AI-driven FinOps and performance optimization features and enhancements, empowering Google Cloud BigQuery customers to see and better manage their cloud data costs. Unravel helps users understand specific cost drivers, allocation insights, and performance and cost optimization of SQL queries. New Unravel features are align with the FinOps phases:

Inform

• Compute and storage costs
• Unit costs and trends for projects, users, and jobs

Optimize

• Reservation insights
• SQL insights
• Data and storage insights
• Scheduling insights

Operate

• OpenSearch-based alerts on job duration and slot-ms
• Alert customization: ability to create custom alerts

Improving visibility, optimizing data performance, and automating spending guardrails can help organizations overcome resource limitations to get more out of their existing data environments.

Visibility into BigQuery compute and storage spend

Getting insights into your cloud data spending starts with understanding your cloud bill. With Unravel, BigQuery users can see their overall spend as well as spending trends for their selected time window, such as the past 30 days.

The cost dashboard shows details and trends, including compute, storage, and services by pricing tier, project, job, and user.

Unravel provides cost analysis, including the average cost of both compute and storage per project, job, and user over time. Compute spending can be further split between on-demand and reserved capacity pricing.

Armed with this detail, BigQuery customers can better understand both infrastructure and pricing tier usage as well as efficiencies by query, user, department, and project. This granular visibility enables accurate, precise cost allocation, trend visualization, and forecasting.

This dashboard provides BigQuery project chargeback details and trends, including a breakdown by compute and storage tier.

Unravel’s AI-driven cloud cost optimization for BigQuery delivers insights based on Unravel’s deep observability of the job, user, and code level to supply AI-driven cost optimization recommendations for slots and SQL queries, including slot provisioning, query duration, autoscaling efficiencies, and more.

With Unravel, BigQuery users can speed cloud transformation initiatives by having real-time cost visibility, predictive spend forecasting, and performance insights for their workloads.

AI-driven cloud cost optimization for BigQuery

At the core of Unravel Data’s data observability and FinOps platform is the AI-powered Insights Engine. It is purpose-built for data platforms—including BigQuery—to understand all the unique aspects and capabilities of each modern data stack and the underlying infrastructure to optimize efficiency and performance.

Unravel’s AI-powered Insights Engine continuously ingests and interprets millions of metadata inputs to provide real-time insights into application and system performance, along with recommendations to improve performance and efficiency for faster results and greater positive business impact for your existing cloud data spend.

Unravel provides insights and recommendations to optimize BigQuery reservations.

Using Unravel’s cost and performance optimization intelligence based on its deep observability at the job, user, and code level, users get recommendations such as:

• Reservation sizing that achieves optimal cost efficiency and performance
• SQL insights and anti-patterns to avoid
• Scheduling insights for recurring jobs
• Quota insights with respect to workload patterns
• and more

With Unravel, BigQuery customers can speed cloud transformation initiatives by having predictive cost and performance insights of existing workloads prior to moving them to the cloud.

Visualization dashboards and unit costs

Visualizing unit costs not only simplifies cost management but also enhances decision-making processes within your organization. With clear insights into spending patterns and resource utilization, you can make informed choices regarding optimization strategies or budget allocation.

With Unravel, BigQuery customers can customize dashboards and alerts with easy-to-use widgets to enable at-a-glance and drill down dashboards on:

• Spend
• Performance
• Unit economics

Unravel displays user count and cost trends by compute pricing tier.

From a unit economics perspective, BigQuery customers can build dashboards to show unit costs in terms of average cost per user, per project, and per job.

Take advantage of visualization dashboards in Unravel for BigQuery to effortlessly gain valuable insights into unit costs.

Additional features included in this release

Unravel 4.8.1 includes additional features, such as showback/chargeback reports, SQL insights and anti-patterns. You can compare two jobs side-by-side, enabling you to point out any metrics that are different between two runs, even if the queries are different.

With this release, you also get:

• Top-K projects, users, and jobs
• Showback by compute and storage types, services, pricing plans, etc.
• Chargeback by projects and users
• Out-of-the-box and custom alerts and dashboards
• Project/Job views of insights and details
• Side-by-side job comparisons
• Data KPIs, metrics, and insights such as size and number of tables and partitions, access by jobs, hot/warm/cold tables

Use case scenarios

Unravel for BigQuery provides a single source of truth to improve collaboration across functional teams and accelerates workflows for common use cases. Below are just a few examples of how Unravel helps BigQuery users for specific situations:

Get Started with Unravel for BigQuery

Learn more about Unravel for BigQuery by reviewing the docs and creating your own free account. 

The post Announcing Unravel 4.8.1: Maximize business value with Google Cloud BigQuery Editions pricing appeared first on Unravel.

With the dramatic increase in the volume, velocity, and variety of data analytics projects, understanding costs and optimizing expenditure is crucial for success. Data teams often face challenges in effectively managing costs, accurately attributing them, and finding ways to enhance cost efficiency. Fortunately, Unravel Data, with its comprehensive platform, addresses these pain points and empowers data teams to unlock their full cost optimization potential—enabling them to run more cloud data workloads without increasing their budget. In this blog, we will delve into the key takeaways from the recent FinOps Camp Episode #1, focusing on the importance of understanding costs, attributing costs, and achieving cost efficiency with Unravel.

Understanding Costs: The Foundation of Cost Optimization

One of the main challenges faced by data teams is gaining deep insights into cost drivers. Traditional observability platforms like Cloudability, or vendor-specific tools like AWS Cost Explorer and Azure Cost Dashboard, often provide limited visibility, focusing solely on infrastructure costs. This lack of granular insights hinders data teams from making informed decisions about resource allocation and identifying areas for cost optimization. Unravel, however, offers a comprehensive dashboard that provides a holistic view of cost breakdowns. This enables data teams to understand exactly where costs are being incurred, facilitating smarter decision-making around resource allocation and optimization efforts.

Accurate Cost Attribution: The Key to Fairness and Transparency

Accurately attributing costs is another hurdle for data teams. Shared services, such as MySQL or Kafka, require meticulous cost allocation to ensure fairness and transparency within an organization. Unravel understands this challenge and provides a solution that simplifies cost allocation. By seamlessly attributing costs to individual users, jobs, teams/departments/lines of business, projects, applications and pipelines, clusters, etc., Unravel enables data teams to break down the proportional costs of shared services. This not only enhances cost tracking but also promotes accountability and fairness within the organization.

Unlocking Cost Efficiency: Recommendations for Optimization

Cost efficiency is the goal of every data team striving for excellence. In the virtual event, Unravel highlighted its powerful feature of identifying inefficiencies within data jobs and providing actionable recommendations for optimization. By analyzing tasks with its AI-powered Recommendation Engine designed specifically for specific platforms like Databricks, Snowflake, BigQuery, and Amazon EMR, Unravel can pinpoint resources that have been oversized and lines of code that contribute to performance bottlenecks. Armed with these insights, data teams can collaborate with developers to address these inefficiencies effectively, resulting in improved resource utilization, reduced costs, and accelerated application performance. Unravel’s proactive optimization recommendations enable data teams to achieve peak cost efficiency and deliver exceptional results.

Operationalizing Unravel: Going From Reactive to Proactive

Unravel’s platform lets data teams go beyond responding to cost and performance inefficiencies after the fact to getting ahead of cost issues beforehand. Unravel empowers ongoing cost governance by enabling team leaders to set up automated guardrails that trigger an alert (or even autonomous “circuit breaker” actions) whenever a predefined threshold is violated—be it projected budget overrun, jobs exceeding a certain size, runtime, cost, etc. Essentially, these automated guardrails take the granular cost allocation information and the AI-powered recommendations and apply them to context-specific workloads to track real-time spending against budgets, prevent runaway jobs and rogue users, identify optimizations to meet SLAs, and nip cost overruns in the bud.

Conclusion: Unleash Your Cost Optimization Potential with Unravel

Understanding costs, attributing costs accurately, and achieving cost efficiency are critical components of any successful data analytics strategy. In the FinOps Camp Episode #1, Unravel showcased its ability to address these concerns and empower data teams to optimize costs effectively. By providing in-depth insights, seamless cost attribution, and proactive optimization recommendations, Unravel enables data teams to understand at a deep level exactly where their cloud data spend is going, predict spending with data-driven accuracy, and optimize data applications/pipelines so organizations can run more high-value workloads within existing budgets. Unravel unlocks your cost optimization potential and maximizes the value of your data analytics efforts. Together, we can transform cost optimization from a challenge into a competitive advantage.

Next Steps:

•  Schedule a demo with Unravel to discover how it can revolutionize your cost optimization efforts..
• Download Eckerson Group’s Governing Cost with FinOps for Cloud Analytics

The post Unlocking Cost Optimization: Insights from FinOps Camp Episode #1 appeared first on Unravel.

Introduction

In the era of Big Data, the importance of data quality cannot be overstated. The vast volumes of information generated every second hold immense potential for organizations across industries. However, this potential can only be realized when the underlying data is accurate, reliable, and consistent. Data quality serves as the bedrock upon which crucial business decisions are made, insights are derived, and strategies are formulated. It empowers organizations to gain a comprehensive understanding of their operations, customers, and market trends. High-quality data ensures that analytics, machine learning, and artificial intelligence algorithms produce meaningful and actionable outcomes. From detecting patterns and predicting trends to identifying opportunities and mitigating risks, data quality is the driving force behind data-driven success. It instills confidence in decision-makers, fosters innovation, and unlocks the full potential of Big Data, enabling organizations to thrive in today’s data-driven world.

Overview

In this seven-part blog we will explore using ML/AI for data quality. Machine learning and artificial intelligence can be instrumental in improving the quality of data. Machine learning models like logistic regression, decision trees, random forests, gradient boosting machines, and neural networks predict categories of data based on past examples, correcting misclassifications. Linear regression, polynomial regression, support vector regression, and neural networks predict numeric values, filling in missing entries. Clustering techniques like K-means, hierarchical clustering, and DBSCAN identify duplicates or near-duplicates. Models such as Isolation Forest, Local Outlier Factor, and Auto-encoders detect outliers and anomalies. To handle missing data, k-Nearest Neighbors and Expectation-Maximization predict and fill in the gaps. NLP models like BERT, GPT, and RoBERTa process and analyze text data, ensuring quality through tasks like entity recognition and sentiment analysis. CNNs fix errors in image data, while RNNs and Transformer models handle sequence data. The key to ensuring data quality with these models is a well-labeled and accurate training set. Without good training data, the models may learn to reproduce the errors present in the data. We will focus on using the following models to apply critical data quality to our lakehouse.

Machine learning and artificial intelligence can be instrumental in improving the quality of data. Here are a few models and techniques that can be used for various use cases:

• Classification Algorithms: Models such as logistic regression, decision trees, random forests, gradient boosting machines, or neural networks can be used to predict categories of data based on past examples. This can be especially useful in cases where data entries have been misclassified or improperly labeled.
• Regression Algorithms: Similarly, algorithms like linear regression, polynomial regression, or more complex techniques like support vector regression or neural networks can predict numeric values in the data set. This can be beneficial for filling in missing numeric values in a data set.
• Clustering Algorithms: Techniques like K-means clustering, hierarchical clustering, or DBSCAN can be used to identify similar entries in a data set. This can help identify duplicates or near-duplicates in the data.
• Anomaly Detection Algorithms: Models like Isolation Forest, Local Outlier Factor (LOF), or Auto-encoders can be used to detect outliers or anomalies in the data. This can be beneficial in identifying and handling outliers or errors in the data set.
• Data Imputation Techniques: Missing data is a common problem in many data sets. Machine learning techniques, such as k-Nearest Neighbors (KNN) or Expectation-Maximization (EM), can be used to predict and fill in missing data.
• Natural Language Processing (NLP): NLP models like BERT, GPT, or RoBERTa can be used to process and analyze text data. These models can handle tasks such as entity recognition, sentiment analysis, text classification, which can be helpful in ensuring the quality of text data.
• Deep Learning Techniques: Convolutional Neural Networks (CNNs) can be used for image data to identify and correct errors, while Recurrent Neural Networks (RNNs) or Transformer models can be useful for sequence data.

Remember that the key to ensuring data quality with these models is a well-labeled and accurate training set. Without good training data, the models may learn to reproduce the errors presented in the data.

In this first blog of seven we will focus on Classification Algorithms. The code examples provided below can be found in this GitHub location. Below is a simple example of using a classification algorithm in a Databricks Notebook to address a data quality issue using a classification algorithm.

Data Engineers Leveraging AI/ML for Data Quality

Machine learning (ML) and artificial intelligence (AI) play a crucial role in the field of data engineering. Data engineers leverage ML and AI techniques to process, analyze, and extract valuable insights from large and complex datasets. Overall, ML and AI provide data engineers with powerful tools and techniques to extract insights, improve data quality, automate processes, and enable data-driven decision-making. They enhance the efficiency and effectiveness of data engineering workflows, enabling organizations to unlock the full potential of their data assets.

AI/ML can help with numerous data quality use cases. Models such as logistic regression, decision trees, random forests, gradient boosting machines, or neural networks can be used to predict categories of data based on past examples. This can be especially useful in cases where data entries have been misclassified or improperly labeled. Similarly, algorithms like linear regression, polynomial regression, or more complex techniques like support vector regression or neural networks can predict numeric values in the data set. This can be beneficial for filling in missing numeric values in a data set. Techniques like K-means clustering, hierarchical clustering, or DBSCAN can be used to identify similar entries in a data set. This can help identify duplicates or near-duplicates in the data. Models like Isolation Forest, Local Outlier Factor (LOF), or Auto-encoders can be used to detect outliers or anomalies in the data. This can be beneficial in identifying and handling outliers or errors in the data set.

Missing data is a common problem in many data sets. Machine learning techniques, such as k-Nearest Neighbors (KNN) or Expectation-Maximization (EM), can be used to predict and fill in missing data. NLP models like BERT, GPT, or RoBERTa can be used to process and analyze text data. These models can handle tasks such as entity recognition, sentiment analysis, text classification, which can be helpful in ensuring the quality of text data. Convolutional Neural Networks (CNNs) can be used for image data to identify and correct errors, while Recurrent Neural Networks (RNNs) or Transformer models can be useful for sequence data.

Suppose we have a dataset with some missing categorical values. We can use logistic regression to fill in the missing values based on the other features. For the purpose of this example, let’s assume we have a dataset with ‘age’, ‘income’, and ‘job_type’ features. Suppose ‘job_type’ is a categorical variable with some missing entries.

Classification Algorithms — Using Logistic Regression to Fix the Data

Logistic regression is primarily used for binary classification problems, where the goal is to predict a binary outcome variable based on one or more input variables. However, it is not typically used directly for assessing data quality. Data quality refers to the accuracy, completeness, consistency, and reliability of data. 

That being said, logistic regression can be indirectly used as a tool for identifying potential data quality issues. Here are some examples of how it can be used in data quality. Logistic regression can be used to define the specific data quality issue to be addressed. For example, you may be interested in identifying data records with missing values or outliers. Logistic regression can be used for feature engineering. This helps identify relevant features (variables) that may indicate the presence of the data quality issue. Data preparation is one of the most common uses of logistic regression. Here the ML helps prepare the dataset by cleaning, transforming, and normalizing the data as necessary. This step involves handling missing values, outliers, and any other data preprocessing tasks.

It’s important to note that logistic regression alone generally cannot fix data quality problems, but it can help identify potential issues by predicting their presence based on the available features. Addressing the identified data quality issues usually requires domain knowledge, data cleansing techniques, and appropriate data management processes. In the simplified example below we see a problem and then use logistic regression to predict what the missing values should be.

Step 1

Create a table with the columns “age”, “income”, and “job_type” in a SQL database, you can use the following SQL statement:

Step1: Create Table

Step 2

Load data to table. Notice that three records are missing job_type. This will be the column that we will use ML to predict. We load a very small set of data for this example. This same technique can be used against billions or trillions of rows. More data will almost always yield better results.

Step 2: Load Data to Table

Step 3

Load the data into a data frame. If you need to create a unique index for your data frame, please refer to this article.

Step 3: Load Data to Data Frame

Step 4

In the context of PySpark DataFrame operations, filter() is a transformation function used to filter the rows in a DataFrame based on a condition.

Step 4: Filter the Rows in a Data Frame Based on a Condition

df is the original DataFrame. We’re creating two new DataFrames, df_known and df_unknown, from this original DataFrame.

• df_known = df.filter(df.job_type.isNotNull()) is creating a new DataFrame that only contains rows where the job_type is not null (i.e., rows where the job_type is known).
• df_unknown = df.filter(df.job_type.isNull()) is creating a new DataFrame that only contains rows where the job_type is null (i.e., rows where the job_type is unknown).

By separating the known and unknown job_type rows into two separate DataFrames, we can perform different operations on each. For instance, we use df_known to train the machine learning model, and then use that model to predict the job_type for the df_unknown DataFrame.

Step 5

In this step we will vectorize the features. Vectorizing the features is a crucial pre-processing step in machine learning and AI. In the context of machine learning, vectorization refers to the process of converting raw data into a format that can be understood and processed by a machine learning algorithm. A vector is essentially an ordered list of values, which in machine learning represent the ‘features’ or attributes of an observation.

Step 5: Vectorize Features

Step 6

In this step we will convert categorical labels to indices. Converting categorical labels to indices is a common preprocessing step in ML and AI when dealing with categorical data. Categorical data represents information that is divided into distinct categories or classes, such as “red,” “blue,” and “green” for colors or “dog,” “cat,” and “bird” for animal types. Machine learning algorithms typically require numerical input, so converting categorical labels to indices allows these algorithms to process the data effectively.

Converting categorical labels to indices is important for ML and AI algorithms because it allows them to interpret and process the categorical data as numerical inputs. This conversion enables the algorithms to perform computations, calculate distances, and make predictions based on the numerical representations of the categories. It is worth noting that label encoding does not imply any inherent order or numerical relationship between the categories; it simply provides a numerical representation that algorithms can work with.

It’s also worth mentioning that in some cases, label encoding may not be sufficient, especially when the categorical data has a large number of unique categories or when there is no inherent ordinal relationship between the categories. In such cases, additional techniques like one-hot encoding or feature hashing may be used to represent categorical data effectively for ML and AI algorithms.

Step 6: Converting Categorical Labels to Indices

Step 7

In this step we will train the model. Training a logistic regression model involves the process of estimating the parameters of the model based on a given dataset. The goal is to find the best-fitting line or decision boundary that separates the different classes in the data.

The process of training a logistic regression model aims to find the optimal parameters that minimize the cost function and provide the best separation between classes in the given dataset. With the trained model, it becomes possible to make predictions on new data points and classify them into the appropriate class based on their features.

Step 7: Train the Model

Step 8

Predict the missing value in this case job_type. Logistic regression, despite its name, is a classification algorithm rather than a regression algorithm. It is used to predict the probability of an instance belonging to a particular class or category.

Logistic regression is widely used in various applications such as sentiment analysis, fraud detection, spam filtering, and medical diagnosis. It provides a probabilistic interpretation and flexibility in handling both numerical and categorical independent variables, making it a popular choice for classification tasks.

Step 8: Predict the Missing Value in This Case job_type

Step 9

Convert predicted indices back to original labels. Converting predicted indices back to original labels in AI and ML involves the reverse process of encoding categorical labels into numerical indices. When working with classification tasks, machine learning models often output predicted class indices instead of the original categorical labels.

It’s important to note that this reverse mapping process assumes a one-to-one mapping between the indices and the original labels. In cases where the original labels are not unique or there is a more complex relationship between the indices and the labels, additional handling may be required to ensure accurate conversion.

Step 9: Convert Predicted Indices Back to Original Labels

Recap

Machine learning (ML) and artificial intelligence (AI) have a significant impact on data engineering, enabling the processing, analysis, and extraction of insights from complex datasets. ML and AI empower data engineers with tools to improve data quality, automate processes, and facilitate data-driven decision-making. Logistic regression, decision trees, random forests, gradient boosting machines, and neural networks can predict categories based on past examples, aiding in correcting misclassified or improperly labeled data. Algorithms like linear regression, polynomial regression, support vector regression, or neural networks predict numeric values, addressing missing numeric entries. Clustering techniques like K-means, hierarchical clustering, or DBSCAN identify duplicates or near-duplicates. Models like Isolation Forest, Local Outlier Factor (LOF), or Auto-encoders detect outliers or anomalies, handling errors in the data. Machine learning techniques such as k-Nearest Neighbors (KNN) or Expectation-Maximization (EM) predict and fill in missing data. NLP models like BERT, GPT, or RoBERTa process text data for tasks like entity recognition and sentiment analysis. CNNs correct errors in image data, while RNNs or Transformer models handle sequence data.

Data engineers can use Databricks Notebook, for example, to address data quality issues by training a logistic regression model to predict missing categorical values. This involves loading and preparing the data, separating known and unknown instances, vectorizing features, converting categorical labels to indices, training the model, predicting missing values, and converting predicted indices back to original labels.

Ready to unlock the full potential of your data? Embrace the power of machine learning and artificial intelligence in data engineering. Improve data quality, automate processes, and make data-driven decisions with confidence. From logistic regression to neural networks, leverage powerful algorithms to predict categories, address missing values, detect anomalies, and more. Utilize clustering techniques to identify duplicates and near-duplicates. Process text, correct image errors, and handle sequential data effortlessly. Try out tools like Databricks Notebook to train models and resolve data quality issues. Empower your data engineering journey and transform your organization’s data into valuable insights. Take the leap into the world of ML and AI in data engineering today! You can start with this Notebook.

The post Unleashing the Power of Data: How Data Engineers Can Harness AI/ML to Achieve Essential Data Quality appeared first on Unravel.

DBS Bank leverages Unravel to identify inefficiencies across 10,000s of data applications/pipelines and guide individual developers and engineers on how, where, and what to improve—no matter what technology or platform they’re using.  

DBS Bank is one of the largest financial services institutions in Asia—the biggest in Southeast Asia—with a presence in 19 markets. Headquartered in Singapore, DBS is recognized globally for its technological innovation and leadership, having been named World’s Best Bank by both Global Finance and Euromoney, Global Bank of the Year by The Banker, World’s Best Digital Bank by Euromoney (three times), and one of the world’s top 10 business transformations of the decade by Harvard Business Review. In addition, DBS has been given Global Finance’s Safest Bank in Asia award for 14 consecutive years. 

DBS is, in its words, “redefining the future of banking as a technology company.” Says the bank’s Executive Director of Automation, Infrastructure for Big Data, AI, and Analytics Luis Carlos Cruz Huertas, “Technology is in our DNA, from the top down. Six years ago, when we started our data journey, the philosophy was that we’re going to be a data-driven organization, 100%. Everything we decide is through data.” 

Realizing innovation and efficiency at scale and speed

DBS backs up its commitment to being a leading data-forward company. Almost 40% of the bank’s employees—some 23,000—are native developers. The volume, variety, and velocity of DBS’ data products, initiatives, and analytics fuel an incredibly complex environment. The bank has developed more than 200 data-use cases to deliver business value: detecting fraud, providing value to customers via their DBS banking app, and providing hyper personalized “nudges” that guide customers in making more informed banking and investment decisions.

As with all financial services enterprises, the DBS data estate is a multi-everything mélange: on-prem, hybrid, and several cloud providers; all the principal platforms; open source, commercial, and proprietary solutions; and a wide variety of technologies, old and new. Adding to this complexity are the various country-specific requirements and restrictions throughout the bank’s markets. As Luis says, “There are many ways to do the cloud, but there’s also the banking way to do cloud. Finance is ring-fenced by regulation all over the world. To do cloud in Asia is quite complex. We don’t [always] get to choose the cloud, sometimes the cloud chooses us.” (For example, India now requires all data to run in India. Google was the only cloud available in Indonesia at the time. Data cannot leave China; there are no hyperscalers there other than Chinese.)

The pace of today’s innovation at a data-forward bank like DBS makes ensuring efficient, reliable performance an ever-growing challenge.

Today DBS runs more than 40,000 data jobs, with tens of thousands of ML models. “And that actually keeps growing,” says Luis. “We have exponentially grown—almost 100X what we used to run five years ago. There’s a lot of innovation we’re bringing, month to month.”

As Luis points out, “Sometimes innovation is a disruptor of stability.” With thousands  of developers all using whatever technologies are best-suited for their particular workload, DBS has to contend with the triple-headed issues of exponentially increasing data volumes (think: OpenAI); increasingly more technologies, platforms, and cloud providers; and the ever-present challenge of having enough skilled people with enough time to ensure that everything is running reliably and efficiently on time, every time—at population scale.  

DBS empowers self-service optimization

One of DBS’ lessons learned in its modern data journey is that, as Luis says, “a lot of the time the efficiencies relied on the engineering team to fix the problems. We didn’t really have a viable mechanism to put into the [business] users’ hands for them to analyze and understand their code, their deficiencies, and how to improve.”

That’s where Unravel comes in. The Unravel platform harnesses deep full-stack visibility, contextual awareness, AI-powered intelligence, and automation to not only quickly show what’s going on and why, but provide crisp, prescriptive recommendations on exactly how to make things better and then keep them that way proactively.

“In Unravel, [we] have a source to identify areas of improvement for our current operational jobs. By leveraging Unravel, we’ve been able to put an analytical tool in the hands of business users to evaluate themselves before they actually do a code release. [They now get] a lot of additional insights they can use to learn, and it puts a lot more responsibility into how they’re doing their job.”

To learn more about how Unravel puts self-service performance and cost optimization capabilities into the hands of individual engineers, see the Unravel Platform Overview.

Marrying data engineering and data science

DBS is perpetually innovating with data—40% of the bank’s workforce are data science developers—yet the scale and speed of DBS’ data operations bring the data engineering side of things to the forefront. As Luis points out, there are thousands of developers committing code but only a handful of data engineers to make sure everything runs reliably and efficiently.

While few organizations have such a high percentage of developers, virtually everyone has the same lopsided developer : engineer ratio. Tackling this reality is at the heart of DataOps and means new roles, where data science and data engineering meet. Unravel has helped facilitate this new intersection of roles and responsibilities by making it easier to pinpoint code inefficiencies and provide insights into what, where, and how to optimize code — without having to “throw it over the fence” to already overburdened engineers.

Luis discusses how DBS addresses the issue: “In hardware, you have something called system performance engineers. They’re dedicated to optimizing and tuning how a processor operates, until it’s pristine. I said, ‘Why don’t we apply that concept and bring it over to data?’ What we need is a very good data scientist who wants to learn data engineering. And a data engineer who wants to become a data scientist. Because I need to connect and marry both worlds. It’s the only way a person can actually say, ‘Oh, this is why pandas doesn’t work in Spark.’ For a data engineer, it’s very clear. For a data scientist, it’s not. Because the first thing they learned in Python is pandas. And they love it. And Spark hates it. It’s a perfect divorce. So we need to teach data scientists how to break their pandas into Spark. Our system performance engineers became masters at doing this. They use Unravel to highlight all the data points that we need to attack in the code, so let’s just go see it. 

“We call data jobs ‘mutants,’ because they mutate from the hands of one data scientist to another. You can literally see the differences on how they write the code. Some of it might be perfect, with Markdown files, explainability, and then there’s an entire chunk of code that doesn’t have that. So we tune and optimize the code. Unravel helps us facilitate the journey on deriving what to do first, what to attack first, from the optimization perspective.”

DBS usually builds their own products—why did they buy Unravel?

DBS has made the bold decision to develop its own proprietary technology stack as wrappers for its governance, control plane, data plane, etc. “We ring-fence all compute, storage, services—every resource the cloud provides. The reason we create our own products is that we’ve been let down way too many times. We don’t really control what happens in the market. It’s the world we live in. We accept that. Our data protection mechanism used to be BlueTalon, which was then acquired by Microsoft. And Microsoft decided to dispose of [BlueTalon] and use their own product. Our entire security framework depended on BlueTalon. . . . We decided to just build our own [framework].

“In a way DBS protects itself from being forced to just do what the providers want us to do. And there’s a resistance from us—a big resistance—to oblige that.” Luis uses a cooking analogy to describe the DBS approach. At one extreme is home cooking, where you go buy all the ingredients and make meals yourself. At the other end of the spectrum is going out to restaurants, where you choose items from the menu and you get a meal the way they prepare it. The DBS data platform is more like a cooking studio—Luis  provides the kitchen, developers pick the ingredients they want, and then cook the recipe with their own particular flavor. “That’s what we provide to our lines of business,” says Luis. “The good thing is, we can plug in anything new at any given point in time. The downside is that you need very, very good technical people. How do we sustain the pace of rebuilding new products, and keep up with the open source side, which moves astronomically fast—we’re very connected to the open source mission, close to 50% of our developers contribute to the open source world—and at the same time keep our [internal and external] customers happy?”

Luis explains the build vs. buy decision boils down to whether the product “drives the core of what we do, whether it’s the ‘spinal cord’ of what we do. We’ve had Unravel for a long, long time. It’s one of those products where we said, ‘Should we do it [i.e., build it ourselves]? Or should we find something on the market?’ With Unravel, it was an easy choice to say, ‘If they can do what they claim they can do, bring ’em in.’ We don’t need to develop our own [data observability solution]. Unravel has demonstrated value in [eliminating] hours of toil—nondeterministic tasks performed by engineers because of repetitive incidents. So, long story short: that [Unravel] is still with us demonstrates that there is value from the lines of business.”

Luis emphasizes his data engineering team are not the users of Unravel. “The users are the business units that are creating their jobs. We just drive adoption to make sure everyone in the bank uses it. Because ultimately our data engineers cannot ‘overrun’ a troop of 3,000 data scientists. So we gave [Unravel] to them, they can use it, they can optimize, and control their own fate.”

DBS realizes high ROI from Unravel

Unravel’s granular data observability, automation, and AI enable DBS to control its cloud data usage at the speed and scale demanded by today’s FinServ environment. “Unravel has saved potentially several $100,000s in [unnecessary] compute capacity,” says Luis. “This is a very big transformational year for us. We’re moving more and more into some sort of data mesh and to a more containerized world. Unravel is part of that journey and we expect that we’ll get the same level of return we’ve gotten so far—or even better.” 

The post DBS Bank Uplevels Individual Engineers at Scale appeared first on Unravel.

Overview: The Right Instance Type

This is the first blog in a five-blog series. For an overview of this blog series please review my post All Data Ecosystems Are Real-Time, It Is Just a Matter of Time.

Choosing the right instance type in the cloud is an important decision that can have a significant impact on the performance, cost, and scalability of your applications. Here are some steps to help you choose the right instance types:

1. Determine your application requirements. Start by identifying the resource requirements of your application, such as CPU, memory, storage, and network bandwidth. You should also consider any special hardware or software requirements, such as GPUs for machine learning workloads.
2. Evaluate the available instance types. Each cloud provider offers a range of instance types with varying amounts of resources, performance characteristics, and pricing. Evaluate the available instance types and their specifications to determine which ones best match your application requirements.
3. Consider the cost. The cost of different instance types can vary significantly, so it’s important to consider the cost implications of your choices. You should consider not only the hourly rate but also the overall cost over time, taking into account any discounts or usage commitments.
4. Optimize for scalability. As your application grows, you may need to scale up or out by adding more instances. Choose instance types that can easily scale horizontally (i.e., adding more instances) or vertically (i.e., upgrading to a larger instance type).
5. Test and optimize. Once you have chosen your instance types, test your application on them to ensure that it meets your performance requirements. Monitor the performance of your application and optimize your instance types as necessary to achieve the best balance of performance and cost.

Choosing the right instance types in the cloud requires careful consideration of your application requirements, available instance types, cost, scalability, and performance. By following these steps, you can make informed decisions and optimize your cloud infrastructure to meet your business needs.

Determine Application Requirements

Determining your application’s CPU, memory, storage, and network requirements is a crucial step in choosing the right instance types in the cloud. Here are some steps to help you determine these requirements:

1. CPU requirements Start by identifying the CPU-intensive tasks in your application, such as video encoding, machine learning, or complex calculations. Determine the number of cores and clock speed required for these tasks, as well as any requirements for CPU affinity or hyperthreading.
2. Memory requirements Identify the memory-intensive tasks in your application, such as caching, database operations, or in-memory processing. Determine the amount of RAM required for these tasks, as well as any requirements for memory bandwidth or latency.
3. Storage requirements Determine the amount of storage required for your application data, as well as any requirements for storage performance (e.g., IOPS, throughput) or durability (e.g., replication, backup).
4. Network requirements Identify the network-intensive tasks in your application, such as data transfer, web traffic, or distributed computing. Determine the required network bandwidth, latency, and throughput, as well as any requirements for network security (e.g., VPN, encryption).

To determine these requirements, you can use various monitoring and profiling tools to analyze your application’s resource usage, such as CPU and memory utilization, network traffic, and storage I/O. You can also use benchmarks and performance tests to simulate different workloads and measure the performance of different instance types.

By understanding your application’s resource requirements, you can choose the right instance types in the cloud that provide the necessary CPU, memory, storage, and network resources to meet your application’s performance and scalability needs.

Evaluate the Available Instance Types

Evaluating the available instance types in a cloud provider requires careful consideration of several factors, such as the workload requirements, the performance characteristics of the instances, and the cost. Here are some steps you can take to evaluate the available instance types in a cloud provider:

1. Identify your workload requirements. Before evaluating instance types, you should have a clear understanding of your workload requirements. For example, you should know the amount of CPU, memory, and storage your application needs, as well as any specific networking or GPU requirements.
2. Review the instance types available. Cloud providers offer a range of instance types with varying configurations and performance characteristics. You should review the available instance types and select the ones that are suitable for your workload requirements.
3. Evaluate performance characteristics. Each instance type has its own performance characteristics, such as CPU speed, memory bandwidth, and network throughput. You should evaluate the performance characteristics of each instance type to determine if they meet your workload requirements.
4. Consider cost. The cost of each instance type varies based on the configuration and performance characteristics. You should evaluate the cost of each instance type and select the ones that are within your budget.
5. Conduct benchmarks. Once you have selected a few instance types that meet your workload requirements and budget, you should conduct benchmarks to determine which instance type provides the best performance for your workload.
6. Consider other factors. Apart from performance and cost, you should also consider other factors such as availability, reliability, and support when evaluating instance types.

The best way to evaluate the available instance types in a cloud provider is to carefully consider your workload requirements, performance characteristics, and cost, and conduct benchmarks to determine the best instance type for your specific use case.

Consider the Cost

When evaluating cloud instance types, it is important to consider both the hourly rate and the overall cost over time, as these factors can vary significantly depending on the provider and the specific instance type. Here are some steps you can take to determine the hourly rate and overall cost over time for cloud instance types:

1. Identify the instance types you are interested in. Before you can determine the hourly rate and overall cost over time, you need to identify the specific instance types you are interested in.
2. Check the hourly rate. Most cloud providers offer a pricing calculator that allows you to check the hourly rate for a specific instance type. You can use this calculator to compare the hourly rate for different instance types and providers.
3. Consider the length of time you will use the instance. While the hourly rate is an important consideration, it is also important to consider the length of time you will use the instance. If you plan to use the instance for a long period of time, it may be more cost-effective to choose an instance type with a higher hourly rate but lower overall cost over time.
4. Look for cost-saving options. Many cloud providers offer cost-saving options such as reserved instances or spot instances. These options can help reduce the overall cost over time, but may require a longer commitment or be subject to availability limitations.
5. Factor in any additional costs. In addition to the hourly rate, there may be additional costs such as data transfer fees or storage costs that can significantly impact the overall cost over time.
6. Consider the potential for scaling. If you anticipate the need for scaling in the future, you should also consider the potential cost implications of adding additional instances over time.

By carefully considering the hourly rate, overall cost over time, and any additional costs or cost-saving options, you can make an informed decision about the most cost-effective instance type for your specific use case.

Optimize for Scalability

To optimize resources for scalability in the cloud, you can follow these best practices:

1. Design for scalability. When designing your architecture, consider the needs of your application and design it to scale horizontally. This means adding more resources, such as servers or containers, to handle an increase in demand.
2. Use auto-scaling. Auto-scaling allows you to automatically increase or decrease the number of resources based on the demand for your application. This helps ensure that you are using only the necessary resources at any given time, and can also save costs by reducing resources during low demand periods.
3. Use load balancing. Load balancing distributes incoming traffic across multiple resources, which helps prevent any one resource from being overloaded. This also helps with failover and disaster recovery.
4. Use caching. Caching can help reduce the load on your servers by storing frequently accessed data in a cache. This reduces the number of requests that need to be processed by your servers, which can improve performance and reduce costs.
5. Use cloud monitoring. Cloud monitoring tools can help you identify potential performance issues and bottlenecks before they become problems. This can help you optimize your resources more effectively and improve the overall performance of your application.
6. Use serverless architecture. With serverless architecture, you don’t need to manage servers or resources. Instead, you pay only for the resources that your application uses, which can help you optimize your resources and reduce costs.

By following the above best practices, you can optimize your resources for scalability in the cloud and ensure that your application can handle an increase in demand.

Test & Optimize

Testing and optimizing your cloud environment is a critical aspect of ensuring that your applications and services are performing optimally and that you’re not overspending on cloud resources. Here are some steps you can take to test and optimize your cloud environment:

1. Set up monitoring. Before you can start testing and optimizing, you need to have visibility into your cloud environment. Set up monitoring tools that can give you insights into key metrics such as CPU utilization, network traffic, and storage usage.
2. Conduct load testing. Conduct load testing to determine how your applications and services perform under different levels of traffic. This can help you identify bottlenecks and performance issues, and make optimizations to improve performance.
3. Optimize resource allocation. Make sure that you’re not overspending on cloud resources by optimizing resource allocation. This includes things like resizing virtual machines, choosing the right storage options, and using auto-scaling to automatically adjust resource allocation based on demand.
4. Implement security measures. Make sure that your cloud environment is secure by implementing appropriate security measures such as firewalls, access controls, and encryption.
5. Use automation. Automating routine tasks can help you save time and reduce the risk of errors. This includes things like automating deployments, backups, and resource provisioning.
6. Review cost optimization options. Consider reviewing your cloud provider’s cost optimization options, such as reserved instances or spot instances. These can help you save money on your cloud bill while still maintaining performance.
7. Continuously monitor and optimize. Continuous monitoring and optimization is key to ensuring that your cloud environment is performing optimally. Set up regular reviews to identify opportunities for optimization and ensure that your cloud environment is meeting your business needs.

By following these steps, you can test and optimize your cloud environment to ensure that it’s secure, cost-effective, and performing optimally.

Recap

Following the above steps will help to make informed decisions and optimize your cloud infrastructure to meet your business needs. Carefully consider your workload requirements. Evaluating performance characteristics, cost, and conducting benchmarks requires understanding your application’s resource requirements. With resource requirements you can choose the right instance types in the cloud that provide the necessary CPU, memory, storage, and network resources to meet your application’s performance and scalability needs.

The post The Modern Data Ecosystem: Choose the Right Instance appeared first on Unravel.

Monitor Cloud Resources

When monitoring cloud resources, there are several factors to consider:

1. Performance It is essential to monitor the performance of your cloud resources, including their availability, latency, and throughput. You can use metrics such as CPU usage, network traffic, and memory usage to measure the performance of your resources.
2. Scalability You should monitor the scalability of your cloud resources to ensure that they can handle changes in demand. You can use tools such as auto-scaling to automatically adjust the resources based on demand.
3. Security You must monitor the security of your cloud resources to ensure that they are protected from unauthorized access or attacks. You can use tools such as intrusion detection systems and firewalls to monitor and protect your resources.
4. Cost It is important to monitor the cost of your cloud resources to ensure that you are not overspending on resources that are not being used. You can use tools such as cost optimization and billing alerts to manage your costs.
5. Compliance If you are subject to regulatory compliance requirements, you should monitor your cloud resources to ensure that you are meeting those requirements. You can use tools such as audit logs and compliance reports to monitor and maintain compliance.
6. Availability It is important to monitor the availability of your cloud resources to ensure that they are up and running when needed. You can use tools such as load balancing and failover to ensure high availability.
7. User Experience You should also monitor the user experience of your cloud resources to ensure that they are meeting the needs of your users. You can use tools such as user monitoring and feedback to measure user satisfaction and identify areas for improvement.

Performance Monitoring

Here are some best practices for performance monitoring in the cloud:

1. Establish baselines. Establish baseline performance metrics for your applications and services. This will allow you to identify and troubleshoot performance issues more quickly.
2. Monitor resource utilization. Monitor resource utilization such as CPU usage, memory usage, network bandwidth, and disk I/O. This will help you identify resource bottlenecks and optimize resource allocation.
3. Use automated monitoring tools. Use automated monitoring tools such as CloudWatch, DataDog, and New Relic to collect performance metrics and analyze them in real time. This will allow you to identify and address performance issues as they arise.
4. Set alerts. Set up alerts for critical performance metrics such as CPU utilization, memory utilization, and network bandwidth. This will allow you to proactively address performance issues before they impact end users.
5. Use load testing. Use load testing tools to simulate heavy loads on your applications and services. This will help you identify performance bottlenecks and optimize resource allocation before going live.
6. Monitor end-user experience. Monitor end-user experience using tools such as synthetic monitoring and real user monitoring (RUM) . This will allow you to identify and address performance issues that impact end users.
7. Analyze logs. Analyze logs to identify potential performance issues. This will help you identify the root cause of performance issues and optimize resource allocation.
8. Continuously optimize. Continuously optimize your resources based on performance metrics and end-user experience. This will help you ensure that your applications and services perform at their best at all times.

Scalability Monitoring

Here are some best practices for scalability monitoring in the cloud:

1. Establish baselines. Establish baseline performance metrics for your applications and services. This will allow you to identify and troubleshoot scalability issues more quickly.
2. Monitor auto-scaling. Monitor auto-scaling metrics to ensure that your resources are scaling up or down according to demand. This will help you ensure that you have the right amount of resources available to meet demand.
3. Use load testing. Use load testing tools to simulate heavy loads on your applications and services. This will help you identify scalability bottlenecks and optimize resource allocation before going live.
4. Set alerts. Set up alerts for critical scalability metrics such as CPU utilization, memory utilization, and network bandwidth. This will allow you to proactively address scalability issues before they impact end users.
5. Use horizontal scaling. Use horizontal scaling to add more instances of your application or service to handle increased traffic. This will allow you to scale quickly and efficiently.
6. Use vertical scaling. Use vertical scaling to increase the size of your resources to handle increased traffic. This will allow you to scale quickly and efficiently.
7. Analyze logs. Analyze logs to identify potential scalability issues. This will help you identify the root cause of scalability issues and optimize resource allocation.
8. Continuously optimize. Continuously optimize your resources based on scalability metrics and end-user experience. This will help you ensure that your applications and services can handle any level of demand.

Security Monitoring

Here are some best practices for handling security monitoring in the cloud:

1. Use security services. Use cloud-based security services such as AWS Security Hub, Azure Security Center, and Google Cloud Security Command Center to centralize and automate security monitoring across your cloud environment.
2. Monitor user activity. Monitor user activity across your cloud environment, including login attempts, resource access, and changes to security policies. This will help you identify potential security threats and ensure that access is granted only to authorized users.
3. Use encryption. Use encryption to protect data at rest and in transit. This will help you protect sensitive data from unauthorized access.
4. Set up alerts. Set up alerts for critical security events such as failed login attempts, unauthorized access, and changes to security policies. This will allow you to respond quickly to security threats.
5. Use multi-factor authentication. Use multi-factor authentication to add an extra layer of security to user accounts. This will help prevent unauthorized access even if a user’s password is compromised.
6. Use firewalls. Use firewalls to control network traffic to and from your cloud resources. This will help you prevent unauthorized access and ensure that only authorized traffic is allowed.
7. Implement access controls. Implement access controls to ensure that only authorized users have access to your cloud resources. This will help you prevent unauthorized access and ensure that access is granted only to those who need it.
8. Perform regular security audits. Perform regular security audits to identify potential security threats and ensure that your cloud environment is secure. This will help you identify and address security issues before they become major problems.

Cost Monitoring

Here are some best practices for monitoring cost in the cloud:

1. Use cost management tools. Use cloud-based cost management tools such as AWS Cost Explorer, Azure Cost Management, and Google Cloud Billing to monitor and optimize your cloud costs.
2. Set budgets. Set budgets for your cloud spending to help you stay within your financial limits. This will help you avoid unexpected charges and ensure that you are using your cloud resources efficiently.
3. Monitor usage. Monitor your cloud resource usage to identify any unnecessary or underutilized resources. This will help you identify opportunities for optimization and cost savings.
4. Analyze cost data. Analyze your cost data to identify trends and areas of high spending. This will help you identify opportunities for optimization and cost savings.
5. Use cost allocation. Use cost allocation to assign costs to individual users, teams, or projects. This will help you identify which resources are being used most and which users or teams are driving up costs.
6. Use reserved instances. Use reserved instances to save money on long-term cloud usage. This will help you save money on your cloud costs over time.
7. Optimize resource allocation. Optimize your resource allocation to ensure that you are using the right amount of resources for your needs. This will help you avoid over-provisioning and under-provisioning.
8. Implement cost optimization strategies. Implement cost optimization strategies such as using spot instances, turning off non-critical resources when not in use, and using serverless architectures. This will help you save money on your cloud costs without sacrificing performance or reliability.

Compliance Monitoring

Here are some best practices for monitoring compliance in the cloud:

1. Understand compliance requirements. Understand the compliance requirements that apply to your organization and your cloud environment, such as HIPAA, PCI-DSS, or GDPR.
2. Use compliance services. Use cloud-based compliance services such as AWS Artifact, Azure Compliance Manager, and Google Cloud Compliance to streamline compliance management and ensure that you are meeting your regulatory requirements.
3. Conduct regular audits. Conduct regular audits to ensure that your cloud environment is in compliance with regulatory requirements. This will help you identify and address compliance issues before they become major problems.
4. Implement security controls. Implement security controls such as access controls, encryption, and multi-factor authentication to protect sensitive data and ensure compliance with regulatory requirements.
5. Monitor activity logs. Monitor activity logs across your cloud environment to identify potential compliance issues, such as unauthorized access or data breaches. This will help you ensure that you are meeting your regulatory requirements and protect sensitive data.
6. Use automation. Use automation tools to help you enforce compliance policies and ensure that your cloud environment is compliant with regulatory requirements.
7. Establish incident response plans. Establish incident response plans to help you respond quickly to compliance issues or data breaches. This will help you minimize the impact of any incidents and ensure that you are meeting your regulatory requirements.
8. Train your employees. Train your employees on compliance policies and procedures to ensure that they understand their roles and responsibilities in maintaining compliance with regulatory requirements. This will help you ensure that everyone in your organization is working together to maintain compliance in the cloud.

Monitor Availability

Here are some best practices for monitoring resource availability in the cloud:

1. Use monitoring services. Use cloud-based monitoring services such as AWS CloudWatch, Azure Monitor, and Google Cloud Monitoring to monitor the availability of your cloud resources.
2. Set up alerts. Set up alerts to notify you when there are issues with resource availability, such as when a server goes down or a service becomes unresponsive. This will help you respond quickly to issues and minimize downtime.
3. Monitor performance metrics. Monitor performance metrics such as CPU usage, memory usage, and network latency to identify potential issues before they become major problems. This will help you ensure that your resources are performing optimally and prevent performance issues from affecting availability.
4. Conduct regular load testing. Conduct regular load testing to ensure that your resources can handle the expected levels of traffic and usage. This will help you identify potential performance bottlenecks and ensure that your resources are available when you need them.
5. Use high availability architectures. Use high availability architectures such as load balancing, auto-scaling, and multi-region deployments to ensure that your resources are available even in the event of a failure. This will help you minimize downtime and ensure that your resources are always available.
6. Monitor service-level agreements (SLAs). Monitor SLAs to ensure that your cloud providers are meeting their service-level commitments. This will help you hold your providers accountable and ensure that your resources are available as expected.
7. Conduct disaster recovery drills. Conduct disaster recovery drills to ensure that you can recover from major outages or disasters. This will help you minimize downtime and ensure that your resources are available even in the event of a major failure.
8. Implement redundancy. Implement redundancy for critical resources to ensure that they are always available. This can include redundant servers, databases, or storage systems. This will help you ensure that your critical resources are always available and minimize downtime.

Monitor User Experience

Here are some best practices for monitoring user experience in the cloud:

1. Define user experience metrics. Define user experience metrics that are important to your business, such as page load times, error rates, and response times. This will help you track user experience and identify areas for improvement.
2. Use synthetic transactions. Use synthetic transactions to simulate user interactions with your applications and services. This will help you identify performance issues and ensure that your applications and services are delivering a good user experience.
3. Monitor real user traffic. Monitor real user traffic to identify issues that may not be apparent in synthetic transactions. This will help you understand how your users are actually using your applications and services and identify any performance issues that may be impacting the user experience.
4. Monitor third-party services. Monitor third-party services that your applications and services rely on, such as payment gateways and content delivery networks. This will help you identify any issues that may be impacting the user experience and ensure that your users have a seamless experience.
5. Use application performance management (APM) tools. Use APM tools to monitor application performance and identify potential issues that may be impacting the user experience. This will help you quickly identify and resolve issues that may be impacting your users.
6. Monitor network latency. Monitor network latency to ensure that your applications and services are delivering a good user experience. This will help you identify any network-related issues that may be impacting the user experience.
7. Set up alerts. Set up alerts to notify you when user experience metrics fall below acceptable levels. This will help you respond quickly to issues and ensure that your users have a good experience.
8. Continuously test and optimize. Continuously test and optimize your applications and services to ensure that they are delivering a good user experience. This will help you identify and fix issues before they impact your users and ensure that your applications and services are always performing optimally.

Recap

When monitoring cloud resources, there are several factors to consider. First, performance. It is essential to monitor the performance of your cloud resources, including their availability, latency, and throughput. This will allow you to identify and address performance issues that impact end users. You can use tools such as cost optimization and billing alerts to manage your costs. This will help you avoid unexpected charges and ensure that you are using your cloud resources efficiently. Conduct regular load testing to ensure that your resources can handle the expected levels of traffic and usage. Define user experience metrics that are important to your business, such as page load times, error rates, and response times.

The post The Modern Data Ecosystem: Monitor Cloud Resources appeared first on Unravel.

Use Managed Services

Using managed services in the cloud can help you reduce your operational burden, increase efficiency, and improve scalability. However, to fully realize the benefits of managed services, you need to follow some best practices. Here are some best practices to consider when using managed services in the cloud:

1. Understand your service-level agreements (SLAs). Before using any managed service, you should understand the SLAs offered by your cloud provider. This will help you understand the availability and reliability of the service, as well as any limitations or restrictions that may impact your use of the service.
2. Choose the right service. You should choose the right managed service for your needs. This means selecting a service that meets your requirements and offers the features and functionality you need. You should also consider the cost of the service and how it will impact your budget.
3. Plan for scalability. Managed services in the cloud are designed to be highly scalable, so you should plan for scalability when using them. This means understanding how the service will scale as your needs change and ensuring that you can easily scale the service up or down as required.
4. Monitor service performance. You should monitor the performance of your managed services to ensure that they are meeting your expectations. This may involve setting up monitoring tools to track service usage, performance, and availability. You should also define appropriate thresholds and alerts to notify you when issues arise.
5. Secure your services. Security is critical when using managed services in the cloud. You should ensure that your services are secured according to best practices, such as using strong passwords, encrypting data in transit and at rest, and implementing access controls. You should also regularly audit your services to ensure that they remain secure.
6. Stay up-to-date. Managed services in the cloud are continually evolving, so you should stay up-to-date with the latest features, updates, and releases. This will help you take advantage of new features and ensure that your services are up-to-date and secure.

By following these best practices, you can ensure that your managed services in the cloud are efficient, reliable, and secure.

Understand Your Service-Level Agreements (SLAs)

Understanding cloud service-level agreements (SLAs) is crucial when you use cloud services. SLAs define the level of service you can expect from a cloud provider and outline the terms and conditions of the service. Here are some ways to help you understand cloud SLAs:

1. Read the SLA. The best way to understand cloud SLAs is to read the SLA itself. The SLA provides details on what services are offered, how they are delivered, and what level of availability you can expect. It also outlines the terms and conditions of the service and what you can expect in the event of an outage or other issues.
2. Understand the metrics. Cloud SLAs typically include metrics that define the level of service you can expect. These metrics may include availability, performance, and response time. You should understand how these metrics are measured and what level of service is guaranteed for each metric.
3. Know the guarantees. The SLA also specifies the guarantees that the cloud provider offers for each metric. You should understand what happens if the provider fails to meet these guarantees, such as compensation or penalties.
4. Identify exclusions. The SLA may also include exclusions or limitations to the service, such as scheduled maintenance, force majeure events, or issues caused by your own actions. You should understand what these exclusions are and how they may impact your use of the service.
5. Ask questions. If you are unsure about any aspect of the SLA, you should ask questions. The cloud provider should be able to provide clarification on any issues and help you understand the SLA better.
6. Get expert help. If you are still unsure about the SLA or need help negotiating SLAs with multiple providers, consider getting expert help. Cloud consultants or legal advisors can help you understand the SLA better and ensure that you get the best possible terms for your needs.

By following these steps, you can better understand cloud SLAs and make informed decisions about the cloud services you use.

Choose the Right Service

Choosing the right cloud service is a critical decision that can have a significant impact on your organization. Here are some factors to consider when choosing a cloud service:

1. Business needs The first step in choosing a cloud service is to understand your business needs. What are your specific requirements? What do you need the cloud service to do? Consider factors such as scalability, security, compliance, and cost when evaluating your options.
2. Reliability and availability Reliability and availability are critical when choosing a cloud service. Look for a provider with a strong track record of uptime and availability. Also, ensure that the provider has a robust disaster recovery plan in place in case of service disruptions or outages.
3. Security Security is a top priority when using cloud services. Choose a provider that has robust security measures in place, such as encryption, access controls, and multi-factor authentication. Also, consider whether the provider meets any relevant compliance requirements, such as HIPAA or GDPR.
4. Cost Cost is another critical factor to consider when choosing a cloud service. Look for a provider that offers transparent pricing and provides a clear breakdown of costs. Also, consider any hidden fees or charges, such as data transfer costs or support fees.
5. Support Choose a cloud service provider that offers robust support options, such as 24/7 customer support or online resources. Ensure that the provider has a reputation for providing excellent support and responding quickly to issues.
6. Integration Ensure that the cloud service provider integrates with any existing systems or applications that your organization uses. This can help reduce the complexity of your IT environment and improve productivity.
7. Scalability Choose a cloud service provider that can scale as your needs change. Ensure that the provider can handle your growth and provides flexibility in terms of scaling up or down.

By considering these factors, you can choose the right cloud service that meets your business needs, is secure, reliable, and scalable, and provides good value.

Plan for Scalability

Scalability is a key advantage of cloud computing, allowing organizations to quickly and easily increase or decrease resources as needed. Here are some best practices for planning for scalability in the cloud:

1. Start with a solid architecture. A solid architecture is essential for building scalable cloud applications. Ensure that your architecture is designed to support scalability from the beginning, by leveraging horizontal scaling, load balancing, and other cloud-native capabilities.
2. Automate everything. Automation is critical for scaling in the cloud. Automate deployment, configuration, and management tasks to reduce manual effort and increase efficiency. Use tools like cloud orchestration or infrastructure-as-code (IAC) to automate the provisioning and configuration of resources.
3. Use elasticity. Elasticity is the ability to automatically adjust resource capacity to meet changes in demand. Use auto-scaling to automatically increase or decrease resources based on utilization or other metrics. This can help ensure that you always have the right amount of resources to handle traffic spikes or fluctuations.
4. Monitor and optimize. Monitoring is critical for maintaining scalability. Use monitoring tools to track application performance and identify potential bottlenecks or areas for optimization. Optimize your applications, infrastructure, and processes to ensure that you can scale as needed without encountering issues.
5. Plan for failure. Scalability also means being prepared for failure. Ensure that your application is designed to handle failures and that you have a plan in place for dealing with outages or other issues. Use fault tolerance and high availability to ensure that your application can continue running even if a component fails.

By following these best practices, you can plan for scalability in the cloud and ensure that your applications can handle changes in demand without encountering issues.

Monitor Service Performance

Monitoring service performance is essential to ensure that your cloud applications are running smoothly and meeting service-level agreements (SLAs). Here are some best practices for monitoring service performance in the cloud:

1. Define key performance indicators (KPIs). Define KPIs that are relevant to your business needs, such as response time, throughput, and error rates. These KPIs will help you track how well your applications are performing and whether they are meeting your SLAs.
2. Use monitoring tools. Use monitoring tools to collect and analyze data on your application’s performance. These tools can help you identify issues before they become critical and track how well your application is meeting your KPIs.
3. Set alerts. Set up alerts based on your KPIs to notify you when something goes wrong. This can help you quickly identify and resolve issues before they impact your application’s performance.
4. Monitor end-to-end performance. Monitor end-to-end performance, including network latency, database performance, and third-party services, to identify any potential bottlenecks or issues that could impact your application’s performance.
5. Analyze and optimize. Analyze performance data to identify patterns and trends. Use this information to optimize your application’s performance and identify areas for improvement. Optimize your application code, database queries, and network configurations to improve performance.
6. Use machine learning. Leverage machine learning to analyze performance data and identify anomalies or issues automatically. This can help you identify issues before they become critical and take proactive steps to resolve them.

By following these best practices, you can monitor service performance in the cloud effectively and ensure that your applications are meeting your SLAs and delivering the best possible user experience.

Secure Your Services

Securing your services in the cloud is critical to protect your data and applications from cyber threats. Here are some best practices for securing your services in the cloud:

1. Implement strong access control. Implement strong access control mechanisms to restrict access to your cloud resources. Use least privilege principles to ensure that users only have access to the resources they need. Implement multi-factor authentication (MFA) and use strong passwords to protect against unauthorized access.
2. Encrypt your data. Encrypt your data both at rest and in transit to protect against data breaches. Use SSL/TLS protocols for data in transit and encryption mechanisms like AES or RSA for data at rest. Additionally, consider encrypting data before it is stored in the cloud to provide an additional layer of protection.
3. Implement network security. Implement network security measures to protect against network-based attacks, such as DDoS attacks, by using firewalls, intrusion detection/prevention systems (IDS/IPS), and VPNs. Segregate your network into logical segments to reduce the risk of lateral movement by attackers.
4. Use security groups and network ACLs. Use security groups and network ACLs to control inbound and outbound traffic to your resources. Implement granular rules to restrict traffic to only what is necessary, and consider using security groups and network ACLs together to provide layered security.
5. Implement logging and monitoring. Implement logging and monitoring to detect and respond to security incidents. Use cloud-native tools like CloudTrail or CloudWatch to monitor activity in your environment and alert you to any unusual behavior.
6. Perform regular security audits. Perform regular security audits to identify potential vulnerabilities and ensure that your security controls are effective. Conduct regular penetration testing and vulnerability assessments to identify and remediate any weaknesses in your environment.

By following these best practices, you can secure your services in the cloud and protect your applications and data from cyber threats.

Stay Up-to-Date on Managed Services

Staying up to date on managed services in the cloud is essential to ensure that you are using the latest features and capabilities and making the most out of your cloud investment. Here are some ways to stay up-to-date on managed services in the cloud:

1. Subscribe to cloud service providers’ blogs. Cloud service providers like AWS, Google Cloud, and Microsoft Azure regularly publish blog posts announcing new features and services. By subscribing to these blogs, you can stay up-to-date on the latest developments and updates.
2. Attend cloud conferences. Attending cloud conferences like AWS re:Invent, Google Cloud Next, and Microsoft Ignite is an excellent way to learn about new and upcoming managed services in the cloud. These events feature keynote speeches, technical sessions, and hands-on workshops that can help you stay up-to-date with the latest trends and technologies.
3. Join cloud user groups. Joining cloud user groups like AWS User Group, Google Cloud User Group, and Azure User Group is a great way to connect with other cloud professionals and learn about new managed services in the cloud. These groups often hold meetings and events where members can share their experiences and discuss the latest developments.
4. Participate in online communities. Participating in online communities like Reddit, Stack Overflow, and LinkedIn Groups is an excellent way to stay up-to-date on managed services in the cloud. These communities often have active discussions about new features and services, and members can share their experiences and insights.
5. Follow industry experts. Following industry experts on social media platforms like Twitter, LinkedIn, and Medium is an excellent way to stay up-to-date on managed services in the cloud. These experts often share their thoughts and insights on the latest developments and can provide valuable guidance and advice.

By following these methods, you can stay up-to-date on managed services in the cloud and ensure that you are using the latest features and capabilities to achieve your business goals.

Recap

Understand your cloud SLAs and make informed decisions about the cloud services you use. By following these best practices, you can ensure that your managed services in the cloud are efficient, reliable, and secure. Leveraging scalability in the cloud ensures that your applications can handle changes in demand without encountering issues. Ensuring that your applications are meeting your SLAs and delivering the best possible user experience requires constant vigilance. Make sure you can secure your services in the cloud and protect your applications and data from cyber threats. By considering the right factors, you can choose the right cloud service that meets your business needs, is secure, reliable, and scalable, and provides good value. Ultimately this helps achieve key business objectives for every customer using the cloud.

The post The Modern Data Ecosystem: Use Managed Services appeared first on Unravel.

Optimize Storage

There are several ways to optimize cloud storage, depending on your specific needs and circumstances. Here are some general tips that can help:

1. Understand your data. Before you start optimizing, it’s important to understand what data you have and how it’s being used. This can help you identify which files or folders are taking up the most space, and which ones are being accessed the most frequently.
2. Use storage compression. Compression can reduce the size of your files, which can save you storage space and reduce the amount of data you need to transfer over the network. However, keep in mind that compressed files may take longer to access and may not be suitable for all types of data.
3. Use deduplication. Deduplication can identify and eliminate duplicate data, which can save you storage space and reduce the amount of data you need to transfer over the network. However, keep in mind that deduplication may increase the amount of CPU and memory resources required to manage your data.
4. Choose the right storage class. Most cloud storage providers offer different storage classes that vary in performance, availability, and cost. Choose the storage class that best meets your needs and budget.
5. Set up retention policies. Retention policies can help you automatically delete old or outdated data, which can free up storage space and reduce your storage costs. However, be careful not to delete data that you may need later.
6. Monitor your usage. Regularly monitor your cloud storage usage to ensure that you’re not exceeding your storage limits or paying for more storage than you need. You can use cloud storage monitoring tools or third-party services to help you with this.
7. Consider a multi-cloud strategy. If you have very large amounts of data, you may want to consider using multiple cloud storage providers to spread your data across multiple locations. This can help you optimize performance, availability, and cost, while also reducing the risk of data loss.

Overall, optimizing cloud storage requires careful planning, monitoring, and management. By following these tips, you can reduce your storage costs, improve your data management, and get the most out of your cloud storage investment.

Understand Your Data

Analyzing data in the cloud can be a powerful way to gain insights and extract value from large datasets. Here are some best practices for analyzing data in the cloud:

1. Choose the right cloud platform. There are several cloud platforms available, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform. Choose the one that suits your needs and budget.
2. Store data in a scalable, secure, and cost-effective way. You can store data in cloud-based databases, data lakes, or data warehouses. Make sure that you choose a storage solution that is scalable, secure, and cost-effective.
3. Choose the right data analysis tool. There are several cloud-based data analysis tools available, such as Amazon SageMaker, Microsoft Azure Machine Learning, and Google Cloud AI Platform. Choose the one that suits your needs and budget.
4. Prepare data for analysis. Data preparation involves cleaning, transforming, and structuring the data for analysis. This step is crucial for accurate analysis results.
5. Choose the right analysis technique. Depending on the nature of the data and the business problem you are trying to solve, you may choose from various analysis techniques such as descriptive, diagnostic, predictive, or prescriptive.
6. Visualize data. Visualization helps to communicate insights effectively. Choose a visualization tool that suits your needs and budget.
7. Monitor and optimize performance. Monitor the performance of your data analysis system and optimize it as necessary. This step helps to ensure that you get accurate and timely insights from your data.

Overall, analyzing data in the cloud can be a powerful way to gain insights and extract value from large datasets. By following these best practices, you can ensure that you get the most out of your cloud-based data analysis system.

Use Storage Compression

Storage compression is a useful technique for reducing storage costs and improving performance in the cloud. Here are some best practices for using storage compression in the cloud:

1. Choose the right compression algorithm. There are several compression algorithms available, such as gzip, bzip2, and LZ4. Choose the algorithm that suits your needs and budget. Consider factors such as compression ratio, speed, and memory usage.
2. Compress data at the right time. Compress data when it is written to storage or when it is not frequently accessed. Avoid compressing data that is frequently accessed, as this can slow down performance.
3. Monitor compression performance. Monitor the performance of your compression system to ensure that it is not slowing down performance. Use tools such as monitoring dashboards to track the performance of your system.
4. Test compression performance. Test the performance of your compression system with different types of data to ensure that it is effective. Consider testing with data that has varying levels of redundancy, such as log files or images.
5. Use compression in conjunction with other techniques. Consider using compression in conjunction with other storage optimization techniques such as deduplication, tiering, and archiving. This can further reduce storage costs and improve performance.
6. Consider the cost of decompression. Decompressing data can be a resource-intensive process. Consider the cost of decompression when choosing a compression algorithm and when designing your storage architecture.

Overall, using storage compression in the cloud can be an effective way to reduce storage costs and improve performance. By following these best practices, you can ensure that you get the most out of your storage compression system.

Data Deduplication 

Deduplication is a technique used to reduce the amount of data stored in the cloud by identifying and removing duplicate data. Here are some best practices for deduplicating cloud data:

1. Choose the right deduplication algorithm. There are several deduplication algorithms available, such as content-defined chunking and fixed-size chunking. Choose the algorithm that suits your needs and budget. Consider factors such as data type, deduplication ratio, and resource usage.
2. Deduplicate data at the right time. Deduplicate data when it is written to storage or when it is not frequently accessed. Avoid deduplicating data that is frequently accessed, as this can slow down performance.
3. Monitor deduplication performance. Monitor the performance of your deduplication system to ensure that it is not slowing down performance. Use tools such as monitoring dashboards to track the performance of your system.
4. Test deduplication performance. Test the performance of your deduplication system with different types of data to ensure that it is effective. Consider testing with data that has varying levels of redundancy, such as log files or images.
5. Consider the tradeoff between storage cost and compute cost. Deduplicating data can be a resource-intensive process. Consider the tradeoff between storage cost and compute cost when choosing a deduplication algorithm and when designing your storage architecture.
6. Use deduplication in conjunction with other techniques. Consider using deduplication in conjunction with other storage optimization techniques such as compression, tiering, and archiving. This can further reduce storage costs and improve performance.

Overall, deduplicating cloud data can be an effective way to reduce storage costs and improve performance. By following these best practices, you can ensure that you get the most out of your deduplication system.

Use the Right Storage Class

Choosing the right storage class for data in the cloud involves considering factors such as access frequency, durability, availability, and cost. Here are some steps to follow when choosing the right storage class for your data in the cloud:

1. Determine your access needs. Consider how frequently you need to access your data. If you need to access your data frequently, you should choose a storage class that provides low latency and high throughput. If you don’t need to access your data frequently, you can choose a storage class that provides lower performance and lower cost.
2. Consider your durability needs. Durability refers to the probability of losing data due to hardware failure. If your data is critical and needs high durability, you should choose a storage class that provides high durability, such as Amazon S3 Standard or Google Cloud Storage Nearline.
3. Evaluate your availability needs. Availability refers to the ability to access your data when you need it. If your data is critical and needs high availability, you should choose a storage class that provides high availability, such as Amazon S3 Standard or Google Cloud Storage Nearline.
4. Determine your cost needs. Cost is also an important factor when choosing a storage class. If you have a limited budget, you should choose a storage class that provides lower cost, such as Amazon S3 Infrequent Access or Google Cloud Storage Coldline.
5. Consider any compliance requirements. Some industries have compliance requirements that dictate how data must be stored. If you have compliance requirements, you should choose a storage class that meets those requirements.
6. Consider data lifecycle management. Depending on the type of data, you may need to store it for a certain period of time before deleting it. Some storage classes may provide lifecycle management features to help you manage your data more efficiently.

By considering these factors, you can choose the right storage class for your data in the cloud that meets your needs and helps you save costs.

Set Data Retention Policies

Setting up retention policies for your cloud data is an important step in managing your data and ensuring that you are in compliance with regulatory requirements. Here are some steps you can follow to set up retention policies for your cloud data:

1. Identify the types of data you need to retain. The first step in setting up retention policies is to identify the types of data that you need to retain. This could include data related to financial transactions, employee records, customer information, and other types of data that are important for your business.
2. Determine the retention periods. Next, you will need to determine how long each type of data needs to be retained. This will depend on the regulatory requirements for your industry as well as your own internal policies. 
3. Decide on the retention strategy. There are several different retention strategies you can use for your cloud data. For example, you could choose to retain all data for a certain period of time, or you could choose to delete data after a certain period of time has elapsed. You could also choose to retain data based on certain triggers, such as when a legal or regulatory inquiry is initiated.
4. Implement the retention policies. Once you have determined the types of data you need to retain, the retention periods, and the retention strategy, you can implement your retention policies in your cloud storage provider. Most cloud storage providers have built-in tools for setting up retention policies.
5. Monitor the retention policies. It’s important to regularly monitor your retention policies to ensure that they are working as intended. You should periodically review the types of data being retained, the retention periods, and the retention strategy to ensure that they are still appropriate. You should also regularly audit your retention policies to ensure that they are in compliance with any changes in regulatory requirements.

By following these steps, you can set up retention policies for your cloud data that will help you manage your data effectively, ensure compliance with regulatory requirements, and reduce your risk of data breaches or loss.

Monitor Usage 

Monitoring your cloud usage is essential for managing your costs, optimizing your resources, and ensuring the security of your data. Here are some of the best ways to monitor your cloud usage:

1. Cloud provider monitoring tools Most cloud providers offer built-in monitoring tools that allow you to track your usage, monitor your costs, and receive alerts when you approach your resource limits. These tools typically provide real-time insights into your cloud usage and can help you identify areas where you can optimize your resources.
2. Third-party monitoring tools There are many third-party monitoring tools available that can help you monitor your cloud usage across multiple cloud providers. These tools offer more advanced features and can help you identify usage patterns, forecast future usage, and detect anomalies that may indicate security threats or performance issues.
3. Cost optimization tools Cost optimization tools can help you identify areas where you can reduce your costs, such as by using more efficient resource configurations or by identifying idle resources that can be decommissioned. These tools typically integrate with your cloud provider’s monitoring tools to provide a comprehensive view of your usage and costs.
4. Security and compliance tools Security and compliance tools can help you monitor your cloud usage for security threats and compliance violations. These tools typically monitor your cloud resources for suspicious activity, such as unauthorized access attempts, and can help you stay in compliance with regulatory requirements.
5. Regular audits Regular audits of your cloud usage can help you identify areas where you can optimize your resources, reduce costs, and improve security. You should periodically review your cloud usage and costs, and adjust your resources and policies as necessary to ensure that you are getting the most value from your cloud investment.

By using these monitoring tools and strategies, you can gain better visibility into your cloud usage, optimize your resources, reduce costs, and ensure the security and compliance of your cloud resources.

Pursue a Multi-Cloud Strategy

Pursuing a multi-cloud strategy can offer several benefits, such as increased resilience, reduced vendor lock-in, and improved performance. However, there are several considerations you should keep in mind before pursuing a multi-cloud strategy. Here are some of the key considerations:

1. Business objectives The first consideration is your business objectives. You need to determine why you want to pursue a multi-cloud strategy and what you hope to achieve. For example, you may be looking to improve the resilience of your applications or reduce vendor lock-in.
2. Compatibility The next consideration is the compatibility of your applications and workloads across different cloud providers. You need to ensure that your applications and workloads are compatible with the different cloud providers you plan to use. You may need to modify your applications and workloads to ensure they can run on multiple cloud platforms.
3. Data management Another important consideration is data management. You need to ensure that your data is managed securely and efficiently across all the cloud providers you use. This may involve implementing data management policies and tools to ensure that your data is always available and protected.
4. Cost management Managing costs is also a critical consideration. You need to ensure that you can manage costs effectively across all the cloud providers you use. This may involve using cost management tools and monitoring usage and costs to identify areas where you can optimize spending.
5. Security Security is always a key consideration, but it becomes even more important when using multiple cloud providers. You need to ensure that your applications and data are secure across all the cloud providers you use. This may involve implementing security policies and using security tools to detect and respond to security threats.
6. Skills and resources Finally, you need to consider the skills and resources required to manage a multi-cloud environment. This may involve hiring additional staff or up-skilling existing staff to ensure that they have the necessary expertise to manage a multi-cloud environment.

By considering these key factors, you can develop a successful multi-cloud strategy that meets your business objectives and helps you achieve your goals.

Recap

Optimizing cloud storage requires careful planning, monitoring, and management. Analyzing data in the cloud can be a powerful way to gain insights and extract value. Using storage compression is an effective way to reduce storage costs and improve performance. Using monitoring tools and strategies, you can gain better visibility into your cloud usage, optimize your resources, reduce costs, and ensure the security and compliance of your cloud resources. By considering these key factors, you can develop a successful multi-cloud strategy that meets your business objectives and helps you achieve your goals. By following these tips, you can reduce your storage costs, improve your data management, and get the most out of your cloud storage investment.

The post The Modern Data Ecosystem: Optimize Your Storage appeared first on Unravel.

Auto-Scaling Overview

This is the second blog in a five-blog series. For an overview of this blog series, please review my post All Data Ecosystems Are Real-Time, It Is Just a Matter of Time. The series should be read in order.

Auto-scaling is a powerful feature of cloud computing that allows you to automatically adjust the resources allocated to your applications based on changes in demand. Here are some best practices for using auto-scaling in the cloud:

1. Set up appropriate triggers. Set up triggers based on metrics such as CPU utilization, network traffic, or memory usage to ensure that your application scales up or down when needed.
2. Use multiple availability zones. Deploy your application across multiple availability zones to ensure high availability and reliability. This will also help you to avoid any potential single points of failure.
3. Start with conservative settings. Start with conservative settings for scaling policies to avoid over-provisioning or under-provisioning resources. You can gradually increase the thresholds as you gain more experience with your application.
4. Monitor your auto-scaling. Regularly monitor the performance of your auto-scaling policies to ensure that they are working as expected. You can use monitoring tools such as CloudWatch to track metrics and troubleshoot any issues.
5. Use automated configuration management. Use tools such as Chef, Puppet, or Ansible to automate the configuration management of your application. This will make it easier to deploy and scale your application across multiple instances.
6. Test your auto-scaling policies. Test your auto-scaling policies under different load scenarios to ensure that they can handle sudden spikes in traffic. You can use load testing tools such as JMeter or Gatling to simulate realistic load scenarios.

By following these best practices, you can use auto-scaling in the cloud to improve the availability, reliability, and scalability of your applications.

Set Up Appropriate Triggers

Setting up appropriate triggers is an essential step when using auto-scaling in the cloud. Here are some best practices for setting up triggers:

1. Identify the right metrics. Start by identifying the metrics that are most relevant to your application. Common metrics include CPU utilization, network traffic, and memory usage. You can also use custom metrics based on your application’s specific requirements.
2. Determine threshold values. Once you have identified the relevant metrics, determine the threshold values that will trigger scaling. For example, you might set a threshold of 70% CPU utilization to trigger scaling up, and 30% CPU utilization to trigger scaling down.
3. Set up alarms. Set up CloudWatch alarms to monitor the relevant metrics and trigger scaling based on the threshold values you have set. For example, you might set up an alarm to trigger scaling up when CPU utilization exceeds 70% for a sustained period of time.
4. Use hysteresis. To avoid triggering scaling up and down repeatedly in response to minor fluctuations in metrics, use hysteresis. Hysteresis introduces a delay before triggering scaling in either direction, helping to ensure that scaling events are only triggered when they are really needed.
5. Consider cooldown periods. Cooldown periods introduce a delay between scaling events, helping to prevent over-provisioning or under-provisioning of resources. When a scaling event is triggered, a cooldown period is started during which no further scaling events will be triggered. This helps to ensure that the system stabilizes before further scaling events are triggered.

By following these best practices, you can set up appropriate triggers for scaling in the cloud, ensuring that your application can scale automatically in response to changes in demand.

Use Multiple Availability Zones

Using multiple availability zones is a best practice in the cloud to improve the availability and reliability of your application. Here are some best practices for using multiple availability zones:

1. Choose an appropriate region. Start by choosing a region that is geographically close to your users to minimize latency. Consider the regulatory requirements, cost, and availability of resources when choosing a region.
2. Deploy across multiple availability zones. Deploy your application across multiple availability zones within the same region to ensure high availability and fault tolerance. Availability zones are isolated data centers within a region that are designed to be independent of each other.
3. Use load balancers. Use load balancers to distribute traffic across multiple instances in different availability zones. This helps to ensure that if one availability zone goes down, traffic can be automatically redirected to other availability zones.
4. Use cross-zone load balancing. Enable cross-zone load balancing to distribute traffic evenly across all available instances, regardless of which availability zone they are in. This helps to ensure that instances in all availability zones are being utilized evenly.
5. Monitor availability zones. Regularly monitor the availability and performance of instances in different availability zones. You can use CloudWatch to monitor metrics such as latency, network traffic, and error rates, and to set up alarms to alert you to any issues.
6. Use automatic failover. Configure automatic failover for your database and other critical services to ensure that if one availability zone goes down, traffic can be automatically redirected to a standby instance in another availability zone.

By following these best practices, you can use multiple availability zones in the cloud to improve the availability and reliability of your application, and to minimize the impact of any potential disruptions.

Start with Conservative Settings

Over-provisioning or under-provisioning resources can lead to wasted resources or poor application performance, respectively. Here are some best practices to avoid these issues:

1. Monitor resource usage. Regularly monitor the resource usage of your application, including CPU, memory, storage, and network usage. Use monitoring tools such as CloudWatch to collect and analyze metrics, and set up alarms to alert you to any resource constraints.
2. Set appropriate thresholds. Set appropriate thresholds for scaling based on your application’s resource usage. Start with conservative thresholds, and adjust them as needed based on your monitoring data.
3. Use automation. Use automation tools such as AWS Auto Scaling to automatically adjust resource provisioning based on demand. This can help to ensure that resources are provisioned efficiently and that you are not over-provisioning or under-provisioning.
4. Use load testing. Use load testing tools such as JMeter or Gatling to simulate realistic traffic loads and test your application’s performance. This can help you to identify any performance issues before they occur in production.
5. Optimize application architecture. Optimize your application architecture to reduce resource usage, such as by using caching, minimizing database queries, and using efficient algorithms.
6. Use multiple availability zones. Deploy your application across multiple availability zones to ensure high availability and fault tolerance, and to minimize the impact of any potential disruptions.

By following these best practices, you can ensure that you are not over-provisioning or under-provisioning resources in your cloud infrastructure, and that your application is running efficiently and reliably.

Monitor and Auto-Scale Your Cloud

The best way to monitor and auto-scale your cloud applications is by using a combination of monitoring tools, scaling policies, and automation tools. Here are some best practices for monitoring and auto-scaling your cloud apps:

1. Monitor application performance. Use monitoring tools such as AWS CloudWatch to monitor the performance of your application. Collect metrics such as CPU utilization, memory usage, and network traffic, and set up alarms to notify you of any performance issues.
2. Define scaling policies. Define scaling policies for each resource type based on the performance metrics you are monitoring. This can include policies for scaling based on CPU utilization, network traffic, or other metrics.
3. Set scaling thresholds. Set conservative thresholds for scaling based on your initial analysis of resource usage, and adjust them as needed based on your monitoring data.
4. Use automation tools. Use automation tools to automatically adjust resource provisioning based on demand. This can help to ensure that resources are provisioned efficiently and that you are not over-provisioning or under-provisioning.
5. Use load testing. Use load testing tools such as JMeter or Gatling to simulate realistic traffic loads and test your application’s performance. This can help you to identify any performance issues before they occur in production.
6. Use multiple availability zones. Deploy your application across multiple availability zones to ensure high availability and fault tolerance, and to minimize the impact of any potential disruptions.
7. Monitor and optimize. Regularly monitor the performance of your application and optimize your scaling policies based on the data you collect. This will help you to ensure that your application is running efficiently and reliably.

By following these best practices, you can ensure that your cloud applications are monitored and auto-scaled effectively, helping you to optimize performance and minimize the risk of downtime.

Use Automated Configuration Management

Automated configuration management in the cloud can help you manage and provision your infrastructure efficiently and consistently. Here are some best practices for using automated configuration management in the cloud:

1. Use infrastructure as code. Use infrastructure as code tools such as AWS CloudFormation or Terraform to define your infrastructure as code. This can help to ensure that your infrastructure is consistent across different environments and can be easily reproduced.
2. Use configuration management tools. Use configuration management tools such as Chef, Puppet, or Ansible to automate the configuration of your servers and applications. These tools can help you ensure that your infrastructure is configured consistently and can be easily scaled.
3. Use version control. Use version control tools such as Git to manage your infrastructure code and configuration files. This can help you to track changes to your infrastructure and roll back changes if necessary.
4. Use testing and validation. Use testing and validation tools to ensure that your infrastructure code and configuration files are valid and that your infrastructure is properly configured. This can help you to avoid configuration errors and reduce downtime.
5. Use monitoring and logging. Use monitoring and logging tools to track changes to your infrastructure and to troubleshoot any issues that arise. This can help you to identify problems quickly and resolve them before they impact your users.
6. Use automation. Use automation tools such as AWS OpsWorks or AWS CodeDeploy to automate the deployment and configuration of your infrastructure. This can help you to deploy changes quickly and efficiently.

By following these best practices, you can use automated configuration management in the cloud to manage your infrastructure efficiently and consistently, reducing the risk of configuration errors and downtime, and enabling you to scale your infrastructure easily as your needs change.

Testing Auto-Scaling Policies

Testing your auto-scaling policies is an important step in ensuring that your cloud infrastructure can handle changes in demand effectively. Here are some best practices for testing your auto-scaling policies:

1. Use realistic test scenarios. Use realistic test scenarios to simulate the traffic patterns and demand that your application may experience in production. This can help you to identify any potential issues and ensure that your auto-scaling policies can handle changes in demand effectively.
2. Test different scenarios. Test your auto-scaling policies under different scenarios, such as high traffic loads or sudden spikes in demand. This can help you to ensure that your policies are effective in a variety of situations.
3. Monitor performance. Monitor the performance of your application during testing to identify any performance issues or bottlenecks. This can help you to optimize your infrastructure and ensure that your application is running efficiently.
4. Validate results. Validate the results of your testing to ensure that your auto-scaling policies are working as expected. This can help you to identify any issues or areas for improvement.
5. Use automation tools. Use automation tools such as AWS CloudFormation or Terraform to automate the testing process and ensure that your tests are consistent and reproducible.
6. Use load testing tools. Use load testing tools such as JMeter or Gatling to simulate realistic traffic loads and test your auto-scaling policies under different scenarios.

By following these best practices, you can ensure that your auto-scaling policies are effective and can handle changes in demand effectively, reducing the risk of downtime and ensuring that your application is running efficiently.

Recap

Auto-scaling can be leveraged to improve the availability, reliability, and scalability of your applications. Multiple availability zones in the cloud improve the availability and reliability of cloud applications, and minimizes the impact of any potential disruptions. Make changes conservatively. Increase resources incrementally. This makes sure you don’t oversize. We must manage our infrastructure efficiently and consistently to reduce the risk of configuration errors and downtime. Conservative scaling enables us to scale our infrastructure easily as our needs change. Handle changes in demand effectively and reduce the risk of downtime and ensure our application is running efficiently.

The post The Modern Data Ecosystem: Use Auto-Scaling appeared first on Unravel.

Overview: Six-Part Blog

In this six-part blog I will demonstrate why what I call Services Oriented Data Architecture (SΘDΔ^®) is the right data architecture for now and the foreseeable future. I will drill into specific examples of how to build the most optimal cloud data architecture regardless of your cloud provider. This will lay the foundation for SΘDΔ^®. We will also define the Data Asset Management System(DΔḾṢ)^®. DΔḾṢ is the modern data management system approach for advanced data ecosystems. The modern data ecosystem must focus on interchangeable interoperable services and let the system focus on optimally storing, retrieving, and processing data. DΔḾṢ takes care of this for the modern data ecosystem.

We will drill into the exercises necessary to optimize the full stack of your cloud data ecosystem. These exercises will work regardless of the cloud provider. We will look at the best ways to store data regardless of type. Then we will drill into how to optimize your compute in the cloud. The compute is generally the most expensive of all cloud assets. We will also drill into how to optimize memory use. Finally, we will wrap up with examples of SΘDΔ^®.

Modern data architecture is a framework for designing, building, and managing data systems that can effectively support modern data-driven business needs. It is focused on achieving scalability, flexibility, reliability, and cost-effectiveness, while also addressing modern data requirements such as real-time data processing, machine learning, and analytics.

Some of the key components of modern data architecture include:

1. Data ingestion and integration This involves collecting and integrating data from various sources, including structured and unstructured data, and ingesting it into the data system.
2. Data storage and processing This involves storing and processing data in a scalable, distributed, and fault-tolerant manner using technologies such as cloud storage, data lakes, and data warehouses.
3. Data management and governance This involves ensuring that data is properly managed, secured, and governed, including policies around data access, privacy, and compliance.
4. Data analysis and visualization This involves leveraging advanced analytics tools and techniques to extract insights from data and present them in a way that is understandable and actionable.
5. Machine learning and artificial intelligence This involves leveraging machine learning and AI technologies to build predictive models, automate decision-making, and enable advanced analytics.
6. Data streaming and real-time processing This involves processing and analyzing data in real time, allowing organizations to respond quickly to changing business needs.

Overall, modern data architecture is designed to help organizations leverage data as a strategic asset and gain a competitive advantage by making better data-driven decisions.

Cloud Optimization Best Practices

Running efficiently on the large cloud providers requires careful consideration of various factors, including your application’s requirements, the size and type of instances needed, and the selected services to leverage.

Here are some general tips to help you run efficiently on the large cloud providers’ cloud:

1. Choose the right instance types. The large cloud providers offer a wide range of instance types optimized for different workloads. Choose the instance type that best fits your application’s requirements to avoid over-provisioning or under-provisioning.
2. Use auto-scaling. Auto-scaling allows you to scale your infrastructure up or down based on demand. This ensures that you have enough capacity to handle traffic spikes while minimizing costs during periods of low usage.
3. Optimize your storage. The large cloud providers offer various storage options, each with its own performance characteristics and costs. Select the storage type that best fits your application’s needs.
4. Use managed services. The large cloud providers provide various managed services, These services allow you to focus on your application’s business logic while the large cloud providers take care of the underlying infrastructure. SaaS vendors manage the software, and PaaS vendors manage the platform.
5. Monitor your resources. The major cloud providers provide various monitoring and logging tools that allow you to track your application’s performance and troubleshoot issues quickly. Use these tools to identify bottlenecks and optimize your infrastructure.
6. Use a content delivery network (CDN). If your application serves static content, consider using a CDN to cache content closer to your users, reducing latency and improving performance.

By following these best practices, you can ensure that your application runs efficiently on the large cloud providers, providing a great user experience while minimizing costs.

The Optimized Way to Store Data in the Cloud

The best structure for storing data for reporting depends on various factors, including the type and volume of data, the reporting requirements, and the performance considerations. Here are some general guidelines for choosing a suitable structure for storing data for reporting:

1. Use a dimensional modeling approach. Dimensional modeling is a database design technique that organizes data into dimensions and facts. It is optimized for reporting and analysis and can help simplify complex queries and improve performance. The star schema and snowflake schema are popular dimensional modeling approaches.
2. Choose a suitable database type. Depending on the size and type of data, you can choose a suitable database type for storing data for reporting. Relational databases are the most common type of database used for reporting, but NoSQL databases can also be used for certain reporting scenarios.
3. Normalize data appropriately. Normalization is the process of organizing data in a database to minimize data redundancy and improve data integrity. However, over-normalization can make querying complex and slow down reporting. Therefore, it is important to normalize data appropriately based on the reporting requirements.
4. Use indexes to improve query performance. Indexes can help improve query performance by allowing the database to quickly find the data required for a report. Choose appropriate indexes based on the reporting requirements and the size of the data.
5. Consider partitioning. Partitioning involves splitting large tables into smaller, more manageable pieces. It can improve query performance by allowing the database to quickly access the required data.
6. Consider data compression. Data compression can help reduce the storage requirements of data and improve query performance by reducing the amount of data that needs to be read from disk.

Overall, the best structure for storing data for reporting depends on various factors, and it is important to carefully consider the reporting requirements and performance considerations when choosing a suitable structure.

Optimal Processing of Data in the Cloud

The best way to process data in the cloud depends on various factors, including the type and volume of data, the processing requirements, and the performance considerations. Here are some general guidelines for processing data in the cloud:

1. Use cloud-native data processing services. Cloud providers offer a wide range of data processing services, such as AWS Lambda, GCP Cloud Functions, and Azure Functions, which allow you to process data without managing the underlying infrastructure. These services are highly scalable and can be cost-effective for small- to medium-sized workloads.
2. Use serverless computing. Serverless computing is a cloud computing model in which the cloud provider manages the infrastructure and automatically scales the resources based on the workload. Serverless computing can be a cost-effective and scalable solution for processing data, especially for sporadic or bursty workloads.
3. Use containerization. Containerization allows you to package your data processing code and dependencies into a container image and deploy it to a container orchestration platform, such as Kubernetes or Docker Swarm. This approach can help you achieve faster deployment, better resource utilization, and improved scalability.
4. Use distributed computing frameworks. Distributed computing frameworks, such as Apache Hadoop, Spark, and Flink, allow you to process large volumes of data in a distributed manner across multiple nodes. These frameworks can be used for batch processing, real-time processing, and machine learning workloads.
5. Use data streaming platforms. Data streaming platforms, such as Apache Kafka and GCP Pub/Sub, allow you to process data in real time and respond quickly to changing business needs. These platforms can be used for real-time processing, data ingestion, and event-driven architectures.
6. Use machine learning and AI services. Cloud providers offer a wide range of machine learning and AI services, such as AWS SageMaker, GCP AI Platform, and Azure Machine Learning, which allow you to build, train, and deploy machine learning models in the cloud. These services can be used for predictive analytics, natural language processing, computer vision, and other machine learning workloads.

Overall, the best way to process data in the cloud depends on various factors, and it is important to carefully consider the processing requirements and performance considerations when choosing a suitable approach.

Optimize Memory

The best memory size for processing 1 terabyte of data depends on the specific processing requirements and the type of processing being performed. In general, the memory size required for processing 1 terabyte of data can vary widely depending on the data format, processing algorithms, and performance requirements. For example, if you are processing structured data in a relational database, the memory size required will depend on the specific SQL query being executed and the size of the result set. In this case, the memory size required may range from a few gigabytes to several hundred gigabytes or more, depending on the complexity of the query and the number of concurrent queries being executed.

On the other hand, if you are processing unstructured data, such as images or videos, the memory size required will depend on the specific processing algorithm being used and the size of the data being processed. In this case, the memory size required may range from a few gigabytes to several terabytes or more, depending on the complexity of the algorithm and the size of the input data.

Therefore, it is not possible to give a specific memory size recommendation for processing 1 terabyte of data without knowing more about the specific processing requirements and the type of data being processed. It is important to carefully consider the memory requirements when designing the processing system and to allocate sufficient memory resources to ensure optimal performance.

Service Oriented Data Architecture Is the Future for Data Ecosystems 

A Services Oriented Data Architecture (SΘDΔ^®) is an architectural approach used in cloud computing that focuses on creating and deploying software systems as a set of interconnected services. Each service performs a specific business function, and communication between services occurs over a network, typically using web-based protocols such as RESTful APIs.

In the cloud, SΘDΔ can be implemented using a variety of cloud computing technologies, including infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). In an SΘDΔ-based cloud architecture, services are hosted on cloud infrastructure, such as virtual machines or containers, and can be dynamically scaled up or down based on demand.

One of the key benefits of SΘDΔ in the cloud is its ability to enable greater agility and flexibility in software development and deployment. By breaking down a complex software system into smaller, more manageable services, SΘDΔ makes it easier to build, test, and deploy new features and updates. It also allows for more granular control over resource allocation, making it easier to optimize performance and cost.

Overall, service-based architecture is a powerful tool for building scalable, flexible, and resilient software systems in the cloud, especially data ecosystems.

Recap

In this blog we began a conversation about the modern data ecosystem. By following best practices, we can ensure that our cloud applications run efficiently, on the large cloud providers, providing a great user experience while minimizing costs. We covered the following:

1. The modern data architecture is designed to help organizations leverage data as a strategic asset and gain a competitive advantage by making better data-driven decisions.
2. The best way to process data in the cloud depends on various factors, and it is important to carefully consider the processing requirements and performance considerations when choosing a suitable approach.
3. Overall, service-based architecture is a powerful tool for building scalable, flexible, and resilient software systems in the cloud, especially data ecosystems.

The post All Data Ecosystems Are Real Time, It Is Just a Matter of Time appeared first on Unravel.

Leverage Content Delivery Network

Here are some best practices when using a Content Delivery Network (CDN):

1. Choose the right CDN provider. Choose a CDN provider that is reliable, scalable, and has a global network of servers. Look for providers that offer features such as caching, load balancing, and DDoS protection.
2. Configure your CDN properly. Configure your CDN properly to ensure that it is delivering content efficiently and securely. This may include setting up caching rules, configuring SSL/TLS encryption, and configuring firewall rules.
3. Monitor your CDN. Monitor your CDN to ensure that it is performing optimally and delivering content efficiently. This may include monitoring CDN usage, network latency, and caching effectiveness.
4. Optimize your CDN. Optimize your CDN to ensure that it is delivering content as efficiently as possible. This may include using compression, optimizing images, and minifying JavaScript and CSS files.
5. Use multiple CDNs. Consider using multiple CDNs to ensure that your content is always available and is delivered quickly. This may include using multiple providers or using multiple CDNs from the same provider.
6. Test your CDN. Test your CDN to ensure that it is delivering content as expected. This may include conducting load testing to ensure that your CDN can handle expected levels of traffic and testing performance from different geographical locations.
7. Use CDN analytics. Use CDN analytics to track CDN usage and monitor performance. This will help you identify any issues and optimize your CDN for better performance.
8. Implement security measures. Implement security measures to protect your content and your users. This may include configuring SSL/TLS encryption, setting up firewalls, and using DDoS protection.

By following these best practices, you can ensure that your CDN is delivering content efficiently and securely, and is providing a positive user experience.

Choosing the Right CDN Provider

Here are some best practices when choosing the right Content Delivery Provider (CDP) in the cloud:

1. Understand your requirements. Before selecting a CDP, determine your requirements for content delivery. This includes identifying the types of content you want to deliver, the expected traffic volume, and the geographical locations of your users.
2. Research multiple providers. Research multiple CDP providers to determine which one best meets your requirements. Look for providers with a global network of servers, high reliability, and good performance.
3. Evaluate performance. Evaluate the performance of each provider by conducting tests from different geographical locations. This will help you determine which provider delivers content quickly and efficiently to your users.
4. Consider cost. Consider the cost of each provider, including the cost of data transfer, storage, and other associated fees. Look for providers that offer flexible pricing models and transparent pricing structures.
5. Evaluate security features. Evaluate the security features of each provider, including DDoS protection, SSL/TLS encryption, and firewalls. Look for providers that offer comprehensive security features to protect your content and your users.
6. Check for integration. Check if the provider integrates well with your existing infrastructure and tools, such as your content management system, analytics tools, and monitoring tools.
7. Look for analytics. Look for providers that offer analytics and reporting tools to help you track content delivery performance and optimize your content delivery.
8. Check for support. Check the level of support offered by each provider, including support for troubleshooting, upgrades, and maintenance. Look for providers with a responsive support team that can help you resolve issues quickly.

By following these best practices, you can select the right CDP that meets your requirements and delivers content efficiently and securely to your users.

Configure Your CDN Properly

Here are some best practices when configuring your Content Delivery Network (CDN) properly in the cloud:

1. Configure caching. Caching is a key feature of CDNs that enables content to be delivered quickly and efficiently. Configure caching rules to ensure that frequently accessed content is cached and delivered quickly.
2. Configure content compression. Configure content compression to reduce the size of your files and improve performance. Gzip compression is a popular option that can be configured at the CDN level.
3. Configure SSL/TLS encryption. Configure SSL/TLS encryption to ensure that content is delivered securely. Look for CDNs that offer free SSL/TLS certificates or have the option to use your own certificate.
4. Configure firewall rules. Configure firewall rules to protect your content and your users. This includes setting up rules to block traffic from malicious sources and to restrict access to your content.
5. Use multiple CDNs. Consider using multiple CDNs to improve performance and ensure availability. Use a multi-CDN strategy to distribute traffic across multiple CDNs, which can reduce latency and increase reliability.
6. Configure DNS settings. Configure DNS settings to ensure that traffic is directed to your CDN. This includes configuring CNAME records or using a DNS provider that integrates with your CDN.
7. Test your CDN configuration. Test your CDN configuration to ensure that it is working properly. This includes testing performance, testing from different geographical locations, and testing content delivery from different devices.
8. Monitor your CDN configuration. Monitor your CDN configuration to ensure that it is delivering content efficiently and securely. This includes monitoring CDN usage, network latency, caching effectiveness, and security events.

By following these best practices, you can configure your CDN properly to ensure that content is delivered quickly, efficiently, and securely to your users.

Monitor Your CDN

Here are some best practices when monitoring your Content Delivery Network (CDN) in the cloud:

1. Monitor CDN usage. Monitor CDN usage to track how much content is being delivered and where it is being delivered. This can help you identify potential issues and optimize content delivery.
2. Monitor network latency. Monitor network latency to ensure that content is being delivered quickly and efficiently. Use tools like Pingdom or KeyCDN’s real-time monitoring to identify network latency issues.
3. Monitor caching effectiveness. Monitor caching effectiveness to ensure that frequently accessed content is being cached and delivered quickly. Use CDN analytics and monitoring tools to track caching effectiveness.
4. Monitor security events. Monitor security events to ensure that your content and your users are protected from security threats. This includes monitoring for DDoS attacks, intrusion attempts, and other security events.
5. Monitor CDN performance. Monitor CDN performance to ensure that content is being delivered efficiently and that the CDN is performing optimally. This includes monitoring server response times, cache hit rates, and other performance metrics.
6. Monitor user experience. Monitor user experience to ensure that users are able to access content quickly and efficiently. Use tools like Google Analytics or Pingdom to monitor user experience and identify issues.
7. Monitor CDN costs. Monitor CDN costs to ensure that you are staying within your budget. Use CDN cost calculators to estimate costs and monitor usage to identify potential cost savings.
8. Set up alerts. Set up alerts to notify you when issues arise, such as network latency spikes, security events, or server downtime. Use CDN monitoring tools to set up alerts and notifications.

By following these best practices, you can monitor your CDN effectively to ensure that content is being delivered quickly, efficiently, and securely to your users, while staying within your budget.

Optimize Your Content Delivery Network

Optimizing your Content Delivery Network (CDN) in the cloud is crucial to ensure fast and reliable content delivery to your users. Here are some best practices to follow:

1. Choose the right CDN provider. Research different CDN providers and choose the one that meets your needs in terms of cost, performance, and geographical coverage.
2. Use a multi-CDN approach. Consider using multiple CDN providers to improve your content delivery performance and reliability.
3. Optimize your CDN configuration. Configure your CDN to serve static assets, such as images, videos, and files, directly from the CDN cache, while serving dynamic content, such as HTML pages, from your origin server.
4. Use caching effectively. Set appropriate cache control headers for your content to ensure that it is cached by the CDN and served quickly to your users.
5. Monitor your CDN performance. Monitor your CDN performance regularly and identify any issues or bottlenecks that may be affecting your content delivery performance.
6. Use compression. Use compression techniques, such as gzip compression, to reduce the size of your content and improve its delivery speed.
7. Optimize DNS resolution. Use a global DNS service to optimize DNS resolution and reduce the time it takes for your users to access your content.
8. Implement HTTPS. Implement HTTPS to ensure secure content delivery and improve your search engine ranking.
9. Consider using edge computing. Consider using edge computing to offload some of the processing and caching tasks to the CDN edge servers, which can help reduce the load on your origin servers and improve content delivery performance.

By following these best practices, you can optimize your CDN in the cloud and ensure fast, reliable, and secure content delivery to your users.

Use Multiple Content Delivery Networks

Using multiple Content Delivery Networks (CDNs) in the cloud can improve the performance and reliability of your content delivery, but it also requires careful management to ensure optimal results. Here are some best practices to follow when using multiple CDNs:

1. Use a multi-CDN management platform. Consider using a multi-CDN management platform to manage your CDNs and monitor their performance. These platforms allow you to configure your CDNs and dynamically route traffic to the optimal CDN based on real-time performance metrics.
2. Define a clear CDN selection strategy. Develop a clear strategy for selecting the CDN to use for each request. Consider factors such as geographical proximity, latency, and availability when selecting a CDN.
3. Test your CDNs. Regularly test the performance of your CDNs using real-world scenarios. This will help you identify any issues and ensure that your CDNs are performing optimally.
4. Implement consistent caching policies. Ensure that your caching policies are consistent across all your CDNs. This will help avoid cache misses and improve content delivery performance.
5. Implement a failover strategy. Define a failover strategy to ensure that your content is always available even if one of your CDNs experiences an outage. This may involve dynamically routing traffic to a backup CDN or using your origin server as a fallback.
6. Monitor and optimize costs. Monitor your CDN costs and optimize your usage to ensure that you are getting the best value for your investment.
7. Ensure security. Implement appropriate security measures, such as SSL encryption and DDoS protection, to ensure that your content is delivered securely and your CDNs are protected from attacks.

By following these best practices, you can successfully use multiple CDNs in the cloud to improve your content delivery performance and reliability.

Test Your Content Delivery Network

Testing your Content Delivery Network (CDN) in the cloud is crucial to ensure that it is performing optimally and delivering content quickly and reliably to your users. Here are some best practices to follow when testing your CDN:

1. Define clear testing objectives. Define clear objectives for your CDN testing, such as identifying bottlenecks or measuring performance metrics. This will help you design your test scenarios and evaluate the results effectively.
2. Use real-world testing scenarios. Use real-world scenarios to test your CDN, such as simulating traffic from different geographic locations, devices, and network conditions. This will help you identify any issues that may affect your users’ experience.
3. Measure key performance metrics. Measure key performance metrics such as response time, latency, throughput, and cache hit rate. This will help you identify any areas of improvement and optimize your CDN performance.
4. Test caching behavior. Test the caching behavior of your CDN by measuring cache hit and miss rates for different types of content. This will help you optimize your caching policies and reduce your origin server load.
5. Test CDN failover and disaster recovery. Test your CDN failover and disaster recovery mechanisms to ensure that your content is always available even if one or more CDNs experience an outage.
6. Use automated testing tools. Use automated testing tools to simulate traffic and measure performance metrics. This will help you perform tests more efficiently and accurately.
7. Test regularly. Regularly test your CDN to ensure that it is performing optimally and identify any issues as soon as possible.

By following these best practices, you can effectively test your CDN in the cloud and ensure fast and reliable content delivery to your users.

Use CDN Analytics

Analytics play a critical role in optimizing your Content Delivery Network (CDN) in the cloud, as they provide insights into your CDN performance and user behavior. Here are some best practices to follow when using CDN analytics in the cloud:

1. Define clear goals. Define clear goals for your CDN analytics, such as identifying areas of improvement or measuring user engagement. This will help you select the appropriate analytics tools and collect the right data.
2. Use a multi-CDN analytics platform. If you are using multiple CDNs, consider using a multi-CDN analytics platform to consolidate data from multiple CDNs and provide a unified view of your content delivery performance.
3. Monitor key performance metrics. Monitor key performance metrics such as page load time, bounce rate, and conversion rate. This will help you identify any performance issues and optimize your content delivery.
4. Use real-time analytics. Use real-time analytics to monitor your CDN performance and user behavior in real time. This will help you identify any issues as soon as they occur and take immediate action.
5. Use A/B testing. Use A/B testing to test different CDN configurations or content variations and measure the impact on user engagement and performance.
6. Use data visualization tools. Use data visualization tools to help you visualize and analyze your CDN performance data effectively. This will help you identify trends and patterns that may be difficult to detect using raw data.
7. Optimize CDN configuration based on analytics insights. Use insights from your CDN analytics to optimize your CDN configuration, such as adjusting caching policies, optimizing content delivery routes, or reducing page load time.

By following these best practices, you can use CDN analytics in the cloud to optimize your content delivery performance, improve user engagement, and achieve your business goals.

Implement Security Measures

Implementing security measures in the cloud is essential to protect your applications, data, and infrastructure from cyber threats. Here are some best practices to follow when implementing security measures in the cloud:

1. Use strong authentication and access controls. Implement strong authentication mechanisms such as multi-factor authentication and use access controls to restrict access to your cloud resources based on the principle of least privilege.
2. Implement network security controls. Use network security controls such as firewalls, intrusion detection and prevention systems (IDPS), and virtual private networks (VPNs) to protect your cloud infrastructure from network-based attacks.
3. Implement encryption. Use encryption to protect sensitive data, both in transit and at rest. Implement encryption for data stored in the cloud, and use secure protocols such as HTTPS and TLS for data in transit.
4. Regularly apply security patches and updates. Regularly apply security patches and updates to your cloud infrastructure and applications to ensure that you are protected against known vulnerabilities.
5. Implement security monitoring and logging. Implement security monitoring and logging to detect and respond to security events. Use tools that provide visibility into your cloud environment and alert you to potential security threats.
6. Use cloud-native security tools. Use cloud-native security tools and services such as security groups, network ACLs, and security information and event management (SIEM) to secure your cloud environment.
7. Develop an incident response plan. Develop an incident response plan that outlines how you will respond to security incidents, including containment, investigation, and remediation.

By following these best practices, you can effectively implement security measures in the cloud and protect your applications, data, and infrastructure from cyber threats.

The post The Modern Data Ecosystem: Leverage Content Delivery Network (CDN) appeared first on Unravel.

By Jason Bloomberg, Managing Partner, Intellyx
Part 4 of the Demystifying Data Observability Series for Unravel Data

In the first three articles in this four-post series, my colleague Jason English and I explored DataOps observability, the connection between DevOps and DataOps, and data-centric FinOps best practices.

In this concluding article in the series, I’ll explore DataOps resiliency – not simply how to prevent data-related problems, but also how to recover from them quickly, ideally without impacting the business and its customers.

Observability is essential for any kind of IT resiliency – you can’t fix what you can’t see – and DataOps is no exception. Failures can occur anywhere in the stack, from the applications on down to the hardware. Understanding the root causes of such failures is the first step to fixing, or ideally preventing, them.

The same sorts of resiliency problems that impact the IT environment at large can certainly impact the data estate. Even so, traditional observability and incident management tools don’t address specific problems unique to the world of data processing.

In particular, DataOps resiliency must address the problem of toxic workloads.

Understanding Toxic Workloads

Toxic data workloads are as old as relational database management systems (RDBMSs), if not older. Anyone who works with SQL on large databases knows there are some queries that will cause the RDBMS to slow dramatically or completely grind to a halt.

The simplest example: SELECT * FROM TRANSACTIONS where the TRANSACTIONS table has millions of rows. Oops! Your resultset also has millions of rows!

JOINs, of course, are more problematic, because they are difficult to construct, and it’s even more difficult to predict their behavior in databases with complex structures.

Such toxic workloads caused problems in the days of single on-premises databases. As organizations implemented data warehouses, the risks compounded, requiring increasing expertise from a scarce cadre of query-building experts.

Today we have data lakes as well as data warehouses, often running in the cloud where the meter is running all the time. Organizations also leverage streaming data, as well as complex data pipelines that mix different types of data in real time.

With all this innovation and complexity, the toxic workload problem hasn’t gone away. In fact, it has gotten worse, as the nuances of such workloads have expanded.

Breaking Down the Toxic Workload

Poorly constructed queries are only one of the causes of a modern toxic workload. Other root causes include:

• Poor quality data – one table with NULL values, for example, can throw a wrench into seemingly simple queries. Expand that problem to other problematic data types and values across various cloud-based data services and streaming data sources, and small data quality problems can easily explode into big ones.
• Coding issues – Data engineers must create data pipelines following traditional coding practices – and whenever there’s coding, there are software bugs. In the data warehouse days, tracking down toxic workloads usually revealed problematic queries. Today, coding issues are just as likely to be the root cause.
• Infrastructure issues – Tracking down the root causes of toxic workloads means looking everywhere – including middleware, container infrastructure, networks, hypervisors, operating systems, and even the hardware. Just because a workload runs too slow doesn’t mean it’s a data issue. You have to eliminate as many possible root causes as you can – and quickly.
• Human issues – Human error may be the root cause of any of the issues above – but there is more to this story. In many cases, root causes of toxic workloads boil down to a shortage of appropriate skills among the data team or a lack of effective collaboration within the team. Human error will always crop up on occasion, but a skills or collaboration issue will potentially cause many toxic workloads over time.

The bottom line: DataOps resiliency includes traditional resiliency challenges but extends to data-centric issues that require data observability to address.

Data Resiliency at Mastercard

Mastercard recently addressed its toxic workload problem on Hadoop, as well as Impala, Spark, and Hive.

The payment processor has petabytes of data across hundreds of nodes, as well as thousands of users who access the data in an ad hoc fashion – that is, they build their own queries.

Mastercard’s primary issue was poorly constructed queries, a combination of users’ inexperience as well as the complexity of the required queries.

In addition, the company faced various infrastructure issues, from overburdened data pipelines to maxed-out storage and disabled daemons.

All these problems led to application failures, system slowdowns and crashes, and resource bottlenecks of various types.

To address these issues, Mastercard brought in Unravel Data. Unravel quickly identified hundreds of unused data tables. Freeing up the associated resources improved query performance dramatically.

Mastercard also uses Unravel to help users tune their own query workloads as well as automate the monitoring of toxic workloads in progress, preventing the most dangerous ones from running in the first place.

Overall, Unravel helped Mastercard improve its mean time to recover (MTTR) – the best indicator of DataOps Resiliency.

The Intellyx Take

The biggest mistake an organization can make around DataOps observability and resiliency is to assume these topics are special cases of the broader discussion of IT observability and resiliency.

In truth, the areas overlap – after all, infrastructure issues are often the root causes of data-related problems – but without the particular focus on DataOps, many problems would fall through the cracks.

The need for this focus is why tools like Unravel’s are so important. Unravel adds AI optimization and automated governance to its core data observability capabilities, helping organizations optimize the cost, performance, and quality of their data estates.

DataOps resiliency is one of the important benefits of Unravel’s approach – not in isolation, but within the overall context for resiliency that is so essential to modern IT.

Copyright © Intellyx LLC. Unravel Data is an Intellyx customer. None of the other organizations mentioned in this article is an Intellyx customer. Intellyx retains final editorial control of this article. No AI was used in the production of this article.

The post DataOps Resiliency: Tracking Down Toxic Workloads appeared first on Unravel.

Here at Unravel, we are all about data observability. We live it. We breathe it. And increasingly, others are recognizing the vital role it plays in not only controlling costs, but helping companies realize and optimize their investments in cloud data projects. It’s become such a critical component of the data space that CRN created a new Data Observability category in this year’s BigData 100 list. We are excited, therefore, to announce that not only have we been named to the list for the fifth consecutive year, but we were among the inaugural group of Coolest Data Observability Companies of 2023.

CRN’s BigData 100 is an annual list that recognizes the technology vendors that go above and beyond by delivering innovation-driven products and services that solution providers can use to help organizations of all sizes gain the competitive advantages of becoming data-driven companies. The list identifies IT vendors that have consistently made technical innovation a top priority through their portfolios of big data management and integration tools; systems and platforms; and data operations and observability offerings.

We provide the market with the only platform that delivers AI-powered answers and precise recommendations for data teams. It’s no wonder that increasing numbers of data-driven enterprises rely on Unravel to help them bring their cloud costs under control. It’s safe to say that data observability has become a must-have for companies to succeed in today’s data-driven and cost-conscious environment.

Not only does data observability improve the speed and quality of data delivery, but it makes it easier for IT and business teams to work together and make smarter data-driven decisions, faster. By giving teams a unified view of data performance, cost, and quality across the entire organization, it breaks down silos and empowers individuals to make their own decisions on how to best reduce and/or optimize their cloud spend while still meeting SLAs.

Unravel provides data observability for the modern data stack. Learn more about how Unravel can help modern enterprises understand, troubleshoot, and optimize their data workloads.

The post Data Observability Is Big(Data 100) News appeared first on Unravel.

As usual, the latest release of Unravel (4.7.8.0) introduces new features and a slew of improvements and enhancements. As always, these new capabilities reflect the emerging needs of data team practitioners and their executive teams. Three of the new features are summarized below. Complete Release Notes for Unravel v4.7.8.0 can be found here.

The top three things any data-forward enterprise cares about are Performance, Cost, and Quality. That reliable results get delivered on time, every time, in the most cost-effective manner. Unravel’s strength has always been in performance observability, and we’ve taken the lead in cost governance for data workloads (DataFinOps). Now we bring in the third pillar: data quality.

Unravel integrates with Data Quality solutions

Unravel now integrates and correlates external data quality checks within its AI-driven insights. Unravel has taken the path of integrating data quality checks that run outside of Unravel—all the homegrown checks, assertions built in any number of excellent solution vendor tools—rather than building yet another data quality tool inside Unravel. 

The first data quality integration is with the open source leader, Great Expectations.

Now data teams have insights and details about performance, cost, and quality in a single pane of glass. No more jumping from tool to tool. And as different personas care about these different dimensions, everybody is working from the same single source of truth. With one click, drill down into quality check results, timelines, lineage, partitions, schema, usage, size, users, and more—right alongside AL recommendations and insights for performance and cost.

AutoActions for Amazon EMR help control cloud costs

AutoActions can now monitor EMR apps and clusters, and you can set the AutoAction policy to generate alerts when EMR exceeds a set dollar threshold, a running time threshold, or when it’s been idle for a long time.

Unravel gets EMR cost data while clusters are running, so you get a notification in real time that the cluster is exceeding a threshold that impacts costs—not after the fact.

You can get a list of all threshold violations across the various clusters, their cost, with drill-down details into each cluster.

Check out this video with a sample use case.

Get proactive budget-tracking alerts and notifications

Unravel can now proactively send notifications—via email or Slack—to specified individuals or groups whenever certain conditions are triggered. Perhaps most usefully, this feature gives both individual users and their managers a heads-up when projected costs are estimated to be at risk of exceeding budget (or have already done so). A common, reusable notification module can be used for multiple purposes, and a single notification may be used by multiple budgets.

Costs may be expressed in DBUs (for Databricks) or dollars (for Amazon EMR). To avoid alert storms and notification fatigue, only one alert is generated per month for budgets at risk, and one per month for exceeded budgets. (But if budgets are changed, alerts can be triggered again for the same month.)

Explore this feature further in this video.

The post What’s New In Unravel 4.7.8.0 appeared first on Unravel.

By Craig Wiley, Senior Director of Product Management, Databricks and Clinton Ford, Director of Product Marketing, Unravel Data

Introduction

Machine learning (ML) enables organizations to extract more value from their data than ever before. Companies who successfully deploy ML models into production are able to leverage that data value at a faster pace than ever before. But deploying ML models requires a number of key steps, each fraught with challenges:

• Data preparation, cleaning, and processing
• Feature engineering
• Training and ML experiments
• Model deployment
• Model monitoring and scoring

Figure 1. Phases of the ML lifecycle with Databricks Machine Learning and Unravel Data

Challenges at each phase

Data preparation and processing

Data preparation is a data scientist’s most time-consuming task. While there are many phases in the data science lifecycle, an ML model can only be as good as the data that feeds it. Reliable and consistent data is essential for training and machine learning (ML) experiments. Despite advances in data processing, a significant amount of effort is required to load and prepare data for training and ML experimentation. Unreliable data pipelines slow the process of developing new ML models.

Training and ML experiments

Once data is collected, cleansed, and refined, it is ready for feature engineering, model training, and ML experiments. The process is often tedious and error-prone, yet machine learning teams also need a way to reproduce and explain their results for debugging, regulatory reporting, or other purposes. Recording all of the necessary information about data lineage, source code revisions, and experiment results can be time-consuming and burdensome. Before a model can be deployed into production, it must have all of the detailed information for audits and reproducibility, including hyperparameters and performance metrics.

Model deployment and monitoring

While building ML models is hard, deploying them into production is even more difficult. For example, data quality must be continuously validated and model results must be scored for accuracy to detect model drift. What makes this challenge even more daunting is the breadth of ML frameworks and the required handoffs between teams throughout the ML model lifecycle– from data preparation and training to experimentation and production deployment. Model experiments are difficult to reproduce as the code, library dependencies, and source data change, evolve, and grow over time.

The solution

The ultimate hack to productionize ML is data observability combined with scalable, serverless, and automated ML model serving. Unravel’s AI-powered data observability for Databricks on AWS and Azure Databricks simplifies the challenges of data operations, improves performance, saves critical engineering time, and optimizes resource utilization.

Databricks Model Serving deploys machine learning models as a REST API, enabling you to build real-time ML applications like personalized recommendations, customer service chatbots, fraud detection, and more – all without the hassle of managing serving infrastructure.

Databricks + data observability

Whether you are building a lakehouse with Databricks for ML model serving, ETL, streaming data pipelines, BI dashboards, or data science, Unravel’s AI-powered data observability for Databricks on AWS and Azure Databricks simplifies operations, increases efficiency, and boosts productivity. Unravel provides AI insights to proactively pinpoint and resolve data pipeline performance issues, ensure data quality, and define automated guardrails for predictability.

Scalable training and ML experiments with Databricks

Databricks uses pre-installed, optimized libraries to build and train machine learning models. With Databricks, data science teams can build and train machine learning models. Databricks provides pre-installed, optimized libraries. Examples include scikit-learn, TensorFlow, PyTorch, and XGBoost. MLflow integration with Databricks on AWS and Azure Databricks makes it easy to track experiments and store models in repositories.

MLflow monitors machine learning model training and running. Information about the source code, data, configuration information, and results are stored in a single location for quick and easy reference. MLflow also stores models and loads them in production. Because MLflow is built on open frameworks, many different services, applications, frameworks, and tools can access and consume the models and related details.

Serverless ML model deployment and serving

Databricks Serverless Model Serving accelerates data science teams’ path to production by simplifying deployments and reducing mistakes through integrated tools. With the new model serving service, you can do the following:

• Deploy a model as an API with one click in a serverless environment.
• Serve models with high availability and low latency using endpoints that can automatically scale up and down based on incoming workload.
• Safely deploy the model using flexible deployment patterns such as progressive rollout or perform online experimentation using A/B testing.
• Seamlessly integrate model serving with online feature store (hosted on Azure Cosmos DB), MLflow Model Registry, and monitoring, allowing for faster and error-free deployment.

Conclusion

You can now train, deploy, monitor, and retrain machine learning models, all on the same platform with Databricks Model Serving. Integrating the feature store with model serving and monitoring helps ensure that production models are leveraging the latest data to produce accurate results. The end result is increased availability and simplified operations for greater AI velocity and positive business impact.

Ready to get started and try it out for yourself? Watch this Databricks event to see it in action. You can read more about Databricks Model Serving and how to use it in the Databricks on AWS documentation and the Azure Databricks documentation. Learn more about data observability in the Unravel documentation.

The post Solving key challenges in the ML lifecycle with Unravel and Databricks Model Serving appeared first on Unravel.

Most organizations spend at least 37% (sometimes over 50%) more than they need to on their cloud data workloads. A lot of costs are incurred down at the individual job level, and this is usually where there’s the biggest chunk of overspending. Two of the biggest culprits are oversized resources and inefficient code. But for an organization running 10,000s or 100,000s of jobs, finding and fixing bad code or right-sizing resources is shoveling sand against the tide. Too many jobs taking too much time and too much expertise. That’s why more and more organizations are progressing to the next logical step, to DataFinOps, and leveraging observability, automation, and AI to find–and ideally, fix–all those thousands of places where costs could and should be optimized. This is the stuff that without AI takes even experts hours or days (even weeks) to figure out. In a nutshell, DataFinOps empowers “self-service” optimization, where AI does the heavy lifting to show people exactly what they need to do to get cost right from the get-go.

DataFinOps and job-level “spending decisions”

One of the highly effective core principles of DataFinOps–the overlap and marriage between DataOps and FinOps–is the idea of holding every individual accountable for their own cloud usage and cost. Essentially, shift cost-control responsibility left, to the people who are actually incurring the expenses.

Now that’s a big change. The teams developing and running all the data applications/pipelines are, of course, still on the hook for delivering reliable results on time, every time–but now cost is right up there with performance and quality as a co-equal SLA. But it’s a smart change: let’s get the cost piece right, everywhere, then keep it that way.

At any given time, your organization probably has hundreds of people running 1,000s of individual data jobs in the cloud–building a Spark job, or Kafka, or doing something in dbt or Databricks. And the meter is always ticking, all the time. How high that meter runs depends a lot on thousands of individual data engineering “spending decisions” about how a particular job is set up to run. In our experience with data-forward organizations over the years, as much as 60% of their cost savings were found by optimizing things at the job level. 

Enterprises have 100,000s of places where cloud data spending decisions need to be made.

When thinking about optimizing cloud data costs at the job level, what springs to mind immediately is infrastructure. And running oversized/underutilized cloud resources is a big problem–one that the DataFinOps approach is especially good at tackling. Everything runs on a machine, sooner or later, and data engineers have to make all sorts of decisions about the number, size, type of machine they should request, how much memory to call for, and a slew of other configuration considerations. And every decision carries a price tag. 

There’s a ton of opportunity to eliminate inefficiency and waste here. But the folks making these spending decisions are not experts at making these decisions. Very few people are. Even a Fortune 500 enterprise could probably count on one hand the number of experts who can “right-size” all the configuration details accurately. There just aren’t enough of these people to tackle the problem at scale. 

But it’s not just “placing the infrastructure order” that drives up the cost of cloud data operations unnecessarily. Bad code costs money, but finding and fixing it takes–again–time and expertise.

So, we have the theory of FinOps kind of hitting a brick wall when it comes into contact with the reality of today’s data estate operations. We want to hold the individual engineers accountable for their cloud usage and spend, but they don’t have the information at their fingertips to be able to use and spend cloud resources wisely.

In fact, “getting engineers to take action on cost optimization” remains the #1 challenge according to the FinOps Foundation’s State of FinOps 2023 survey. But that stat masks just how difficult it is. And to get individuals to make the most cost-effective choices about configuring and running their particular jobs–which is where a lot of the money is being spent–it has to be easy for them to do the right thing. That means showing them what the right thing is. Otherwise, even if they knew exactly what they were looking for, sifting through thousands of logs and cloud vendor billing data is time-sucking toil. They don’t need more charts and graphs that, while showing a lot of useful information, still leave it to you to figure out how to fix things. 

That’s where DataFinOps comes in. Combining end-to-end, full-stack observability data (at a granular level), a high degree of automation, and AI capabilities, DataFinOps identifies a wide range of cost control opportunities–based on, say, what resources you have allocated vs. what resources you actually need to run the job successfully–then automatically provides a prescriptive AI-generated recommendation on what, where, how to make things more cost-effective.

Using AI to solve oversized resources and inefficient code at the job level

DataFinOps uses observability, automation, and AI to do a few things. First, it collects all sorts of information that is “hidden in plain sight”–performance metrics, logs, traces, events, and metadata from the dozens of components you have running in your modern data stack, at both the job (Spark, Kafka) and platform (Databricks, Snowflake, BigQuery, Amazon EMR, etc.) level; details from the cloud vendors about what is running, where, and how much it costs; details about the datasets themselves, including lineage and quality–and then stitches it all together into an easily understandable, correlated context. Next, DataFinOps throws some math at all that data to analyze usage and cost, hundreds of AI algorithms and ML models that have been well trained over the years. AI can do this kind of investigation, or detective work, faster and more accurately than humans. Especially at scale, when we’re looking at thousands and thousands of individual spending decisions.

Two areas where AI helps individuals cut to the chase and actually do something about eliminating overspending at the job level–which is where the biggest and longer-term cost savings opportunities can be found–are oversized resources and inefficient code.

Oversized resources are simply where somebody (or lots of somebodies) requested more resources from the cloud provider than are actually needed to run the job successfully. What DataFinOps AI does is analyze everything that’s running, pinpoint among the thousands of jobs where the number, size, type of resources you’re using is more than you need, calculate the cost implications, and deliver up a prescriptive recommendation (in plain English) for a more cost-effective configuration.

For example, in the screenshot below, the AI has identified more cost-effective infrastructure for this particular job at hand, based on real-time data from your environment. The AI recommendation specifies exactly what type of resource, with exactly how much memory and how many cores, would be less expensive and still get the job done. It’s not quite a self-healing system, but it’s pretty close. 

Code problems may be an even more insidious contributor to overspending. Workloads are constantly moving from on-prem Cloudera/Hadoop environments to the cloud. There’s been an explosion of game-changing technologies and platforms. But we’ve kind of created a monster: everybody seems to be using a different system, and it’s often different from the one they used yesterday. Everything’s faster, and there’s more of it. Everybody is trying to do code checks as best they can, but the volume and velocity of today’s data applications/pipelines make it a losing battle. 

DataFinOps pinpoints where in all that code there are problems causing cloud costs to rise unnecessarily. The ML model recognizes anti-patterns in the code–it’s learned from millions of jobs just like this one–and flags the issue.

In the example below, there’s a slow SQL stage: Query 7 took almost 6½ minutes. The AI/ML has identified this as taking too long, an anomaly or at least something that needs to be corrected. It performs an automated root cause analysis to point directly to what in the code needs to be optimized. 

Every company that’s running cloud data workloads is already trying to crack this nut of how to empower individual data engineers with this kind of “self-service” ability to optimize for costs themselves. It’s their spending decisions here at the job level that have such a big impact on overall cloud data costs. But you have to have all the data, and you have to have a lot of automation and AI to make self-service both easy and impactful.

AI recommendations show the individual engineers what to do, so being held accountable for cloud usage and cost is no longer the showstopper obstacle. From a wider team perspective, the same information can be rolled up, sliced-and-diced, into a dashboard that gives a clear picture on the overall state of cost optimization opportunities. You can see how you’re doing (and what still needs to be done) with controlling costs at the cluster or job or even individual user level.

Bottom line: The best way to control cloud data costs in a meaningful way, at scale and speed, with real and lasting impact, is the DataFinOps approach of full-stack observability, automation, and AI.

The post DataFinOps: Holding individuals accountable for their own cloud data costs appeared first on Unravel.

By Jason English, Principal Analyst, Intellyx
Part 3 in the Demystifying Data Observability Series, by Intellyx for Unravel Data

IT and data executives find themselves in a quandary about deciding how to wrangle an exponentially increasing volume of data to support their business requirements – without breaking an increasingly finite IT budget.

Like an overeager diner at a buffet who’s already loaded their plate with the cheap carbs of potatoes and noodles before they reach the protein-packed entrees, they need to survey all of the data options on the menu before formulating their plans for this trip.

In our previous chapters of this series, we discussed why DataOps needs its own kind of observability, and then how DataOps is a natural evolution of DevOps practices. Now there’s a whole new set of options in the data observability menu to help DataOps teams track the intersection of value and cost.

From ROI to FinOps

Executives can never seem to get their fill of ROI insights from IT projects, so they can measure bottom-line results or increase top-line revenue associated with each budget line item. After all, predictions about ROI can shape the perception of a company for its investors and customers.

Unfortunately, ROI metrics are often discussed at the start of a major technology product or services contract – and then forgotten as soon as the next initiative gets underway.

The discipline of FinOps burst onto the scene over the last few years, as a strategy to address the see-saw problem of balancing the CFO’s budget constraints with the CIO’s technology delivery requirements to best meet the current and future needs of customers and employees.

FinOps focuses on improving technology spending decisions of an enterprise using measurements that go beyond ROI, to assess the value of business outcomes generated through technology investments.

Some considerations frequently seen on the FinOps menu include:

• Based on customer demand or volatility in our consumption patterns, should we buy capacity on-demand or reserve more cloud capacity?
• Which FinOps tools should we buy, and what functionality should we build ourselves, to deliver this important new capability?
• Which cloud cost models are preferred for capital expenditures (capex) projects and operational expenditures (opex)?
• What is the potential risk and cost of known and unknown usage spikes, and how much should we reasonably invest in analysts and tools for preventative purposes?

As a discipline, FinOps has come a long way, building communities of interest among expert practitioners, product, business, and finance teams as well as solution providers through its own FinOps Foundation and instructional courses on the topic.

FinOps + DataOps = DataFinOps?

Real-time analytics and AI-based operational intelligence are enabling revolutionary business capabilities, enterprise-wide awareness, and innovative machine learning-driven services. All of this is possible thanks to a smorgasbord of cloud data storage and processing, cloud data lakes, cloud data warehouse, and cloud lakehouse options.

Unfortunately, the rich streams of data required for such sophisticated functionality bring along the unwanted side effect of elastically expanding budgetary waistbands, due to ungoverned cloud storage and compute consumption costs. Nearly a third of all data science projects go more than 40% over budget on cloud data, according to a recent survey–a huge delta between cost expectations and reality.

How can better observability into data costs help the organization wring more value from data assets without cutting into results, or risking cost surprises?

As it turns out, data has its own unique costs, benefits, and value considerations. Combining the disciplines of FinOps and DataOps – which I’ll dub DataFinOps just for convenience here – can yield a unique new set of efficiencies and benefits for the enterprise’s data estate.

Some of unique considerations of DataFinOps:

• Which groups within our company are the top spenders on cloud data analytics, and is anything anomalous about their spending patterns versus the expected budgets?
• What is the value of improving data performance or decreasing the latency of our data estate by region or geography, in order to improve local accuracy, reduce customer and employee attrition and improve retention?
• If we are moving to a multi-cloud, hybrid approach, what is an appropriate and realistic mix of reserved instances and spot resources for processing data of different service level agreements (SLAs)?
• Where are we paying excessive ingress / egress fees within our data estate? Would it be more cost effective to process data near the data or move our data elsewhere?
• How much labor do our teams spend building and maintaining data pipelines, and what is that time worth?
• Are cloud instances being intelligently right-sized and auto-scaled to meet demand?

Systems-oriented observability platforms such as DataDog and Dynatrace can measure system or service level telemetry, which is useful for a DevOps team looking at application-level cloud capacity and cost/performance ratios. Unfortunately these tools do not dig into enough detail to answer data analytics-specific FinOps questions.

Taming a market of data options

Leading American grocery chain Kroger launched its 84.51° customer experience and data analytics startup to provide predictive data insights and precision marketing for its parent company and other retailers, across multiple cloud data warehouses such as Snowflake and Databricks, using data storage in multiple clouds such as Azure and GCP.

Using the Unravel platform for data observability, they were able to get a grip on data costs and value across multiple data platforms and clouds without having to train up more experts on the gritty details of data job optimization within each system.

It’s still early days for this transformation, but a data cost reduction of up to 50% would go a long way toward extracting value from deep customer analytics, as transaction data volumes continue to increase by 2x or 3x a year as more sources come online.

The Intellyx Take

It would be nice if CFOs could just tell CIOs and CDOs to simply stop consuming and storing so much data, and have that reduce their data spend. But just like in real life, crash diets will never produce long-term results, if the ‘all-you-can-eat’ data consumption pattern isn’t changed.

The hybrid IT underpinnings of advanced data-driven applications evolves almost every day. To achieve sustainable improvements in cost/benefit returns on data, analysts and data scientists would have to become experts on the inner workings of each public cloud and data warehousing vendor.

DataFinOps practices should encourage data team accountability for value improvements, but more importantly, it should give them the data observability, AI-driven recommendations, and governance controls necessary to both contain costs, and stay ahead of the organization’s growing business demand for data across hybrid IT data resources and clouds.

©2023 Intellyx LLC. Intellyx is editorially responsible for the content of this document. At the time of writing, Unravel is an Intellyx customer. Image source: crayion.ai

The post DataFinOps: More on the menu than data cost governance appeared first on Unravel.

By Jason Bloomberg, President, Intellyx
Part 2 of the Demystifying Data Observability Series for Unravel Data

In part one of this series, fellow Intellyx analyst Jason English explained the differences between DevOps and DataOps, drilling down into the importance of DataOps observability.

The question he left open for this article: how did we get here? How did DevOps evolve to what it is today, and what parallels or differences can we find in the growth of DataOps?

DevOps Precursors

The traditional, pre-cloud approach to building custom software in large organizations separated the application development (‘dev’) teams from the IT operations (‘ops’) personnel responsible for running software in the corporate production environment.

In between these two teams, organizations would implement a plethora of processes and gates to ensure the quality of the code and that it would work properly in production before handing it to the ops folks to deploy and manage.

Such ‘throw it over the wall’ processes were slow and laborious, leading to deployment cycles many months long. The importance of having software that worked properly, so the reasoning went, was sufficient reason for such onerous delays.

Then came the Web. And the cloud. And enterprise digital transformation initiatives. All of these macro-trends forced enterprises to rethink their plodding software lifecycles.

Not only were they too slow to deliver increasingly important software capabilities, but business requirements would evolve far too quickly for the deployed software to keep up.

Such pressures led to the rise of agile software methodologies, cloud native computing, and DevOps.

Finding the Essence of DevOps

The original vision of DevOps was to pull together the dev and ops teams to foster greater collaboration, in hopes that software deployment cadences would accelerate while maintaining or improving the quality of the resulting software.

Over time, this ‘kumbaya’ vision of seamless collaboration itself evolved. Today, we can distill the essence of modern DevOps into these five elements:

• A cultural and organizational shift away from the ‘throw it over the wall’ mentality to greater collaboration across the software lifecycle
• A well-integrated, comprehensive automation suite that supports CI/CD activities, along with specialists who manage and configure such technologies, i.e., DevOps engineers
• A proactive, shift-left mentality that seeks to represent production behavior declaratively early in the lifecycle for better quality control and rapid deployment
• Full-lifecycle observability that shifts problem resolution to the left via better prediction of problematic behavior and preemptive mitigation of issues in production
• Lean practices to deliver value and improve efficiency throughout the software development lifecycle

Furthermore, DevOps doesn’t live in a vacuum. Rather, it is consistent with and supports other modern software best practices, including infrastructure-as-code, GitOps, and the ‘cattle not pets’ approach to supporting the production environment via metadata representations that drive deployment.

The Evolution of DataOps

Before information technology (IT), organizations had management of information systems (MIS). And before MIS, at the dawn of corporate computing, enterprises implemented data processing (DP).

The mainframes at the heart of enterprise technology as far back as the 1960s were all about processing data – crunching numbers in batch jobs that yielded arcane business results, typically dot-matrix printed on green and white striped paper.

Today, IT covers a vast landscape of technology infrastructure, applications, and hybrid on-premises and cloud environments – but data processing remains at the heart of what IT is all about.

Early in the evolution of DP, it became clear that the technologies necessary for processing transactions were different from the technology the organization required to provide business intelligence to line-of-business (LoB) professionals.

Enterprises required parallel investments in online transaction processing (OLTP) and online analytical processing (OLAP), respectively. OLAP proved the tougher nut to crack, because enterprises generated voluminous quantities of transactional data, while LoB executives required complex insights that would vary over time – thus taxing the ability of the data infrastructure to respond to the business need for information.

To address this need, data warehouses exploded onto the scene, separating the work of OLAP into two phases: transforming and loading data into the warehouses and supporting business intelligence via queries of the data in them.

Operating these early OLAP systems was relatively straightforward, centering on administering the data warehouses. In contrast, today’s data estate – the sum total of all the data infrastructure in a modern enterprise – is far more varied than in the early data warehousing days.

Motivations for DataOps

Operating this data estate has also become increasingly complex, as the practice of DataOps rises in today’s organizations.

Complexity, however, is only one motivation for DataOps. There are more reasons why today’s data estate requires it:

• Increased mission-criticality of data, as digital transformations rework organizations into digital enterprises
• Increased importance of real-time data, a capability that data warehouses never delivered
• Greater diversity of data-centric use cases beyond basic business intelligence
• Increased need for dynamic applications of data, as different LoBs need an ever-growing variety of data-centric solutions
• Growing need for operational cost predictability, optimization, and governance

Driving these motivations is the rise of AI, as it drives the shift from code-based to data-based software behavior. In other words, AI is more than just another data-centric use case. It repositions data as the central driver of software functionality for the enterprise.

The Intellyx Take

For all these reasons, DataOps can no longer follow the simplistic data warehouse administration pattern of the past. Today’s data estate is dynamic, diverse, and increasingly important, requiring organizations to take a full-lifecycle approach to collecting, transforming, storing, querying, managing, and consuming data.

As a result, DataOps requires the application of core DevOps practices along the data lifecycle. DataOps requires the cross-lifecycle collaboration, full-lifecycle automation and observability, and the shift-left mentality that DevOps brings to the table – only now applied to the enterprise data estate.

Thinking of DataOps as ‘DevOps for data’ may be too simplistic an explanation of the role DataOps should play. Instead, it might be more accurate to say that as data increasingly becomes the driver of software behavior, DataOps becomes the new DevOps.

Next up in part 3 of this series: DataFinOps: More on the menu than data cost governance

Copyright © Intellyx LLC. Unravel Data is an Intellyx customer. Intellyx retains final editorial control of this article.

The post The Evolution from DevOps to DataOps appeared first on Unravel.

By Jason English, Principal Analyst, Intellyx
Part 1 of the Demystifying Data Observability Series for Unravel Data

Don’t we already have DevOps?

DevOps was started more than a decade ago as a movement, not a product or solution category.

DevOps offered us a way of collaborating between development and operations teams, using automation and optimization practices to continually accelerate the release of code, measure everything, lower costs, and improve the quality of application delivery to meet customer needs.

Today, almost every application delivery shop naturally aspires to take flight with DevOps practices, and operate with more shared empathy and a shared commitment to progress through faster feature releases and feedback cycles.

DevOps practices also include better management practices such as self-service environments, test and release automation, monitoring, and cost optimization.

On the journey toward DevOps, teams who apply this methodology deliver software more quickly, securely, reliably, and with less burnout.

For dynamic applications to deliver a successful user experience at scale, we still need DevOps to keep delivery flowing. But as organizations increasingly view data as a primary source of business value, data teams are tasked with building and delivering reliable data products and data applications. Just as DevOps principles emerged to enable efficient and reliable delivery of applications by software development teams, DataOps best practices are helping data teams solve a new set of data challenges.

What is DataOps?

If “data is the new oil,” as pundits like to say, then data is also the most valuable resource in today’s modern data-driven application world.

The combination of commodity hardware, ubiquitous high-bandwidth networking, cloud data warehouses, and infrastructure abstraction methods like containers and Kubernetes creates an exponential rise in our ability to use data itself to dynamically compose functionality such as running analytics and informing machine learning-based inference within applications.

Enterprises recognized data as a valuable asset, welcoming the newly minted CDO (chief data officer) role to the E-suite, with responsibility for data and data quality across the organization. While leading-edge companies like Google, Uber and Apple increased their return on data investment by mastering DataOps, many leaders struggled to staff up with enough data scientists, data analysts, and data engineers to properly capitalize on this trend.

Progressive DataOps companies began to drain data swamps by pouring massive amounts of data (and investment) into a new modern ecosystem of cloud data warehouses and data lakes from open source Hadoop and Kafka clusters to vendor-managed services like Databricks, Snowflake, Amazon EMR, BigQuery, and others.

The elastic capacity and scalability of cloud resources allowed new kinds of structured, semi-structured, and unstructured data to be stored, processed and analyzed, including streaming data for real-time applications.

As these cloud resources quickly grew and scaled, they became a complex tangle of data sources, pipelines, dashboards, and machine learning models, with a variety of interdependencies, owners, stakeholders, and products with SLAs. Getting additional cloud resources and launching new data pipelines was easy, but operating them well required a lot of effort, and making sense of the business value of any specific component to prioritize data engineering efforts became a huge challenge.

Software teams went through the DevOps revolution more than a decade ago, and even before that, there were well-understood software engineering disciplines for design/build/deploy/change, as well as monitoring and observability. Before DataOps, data teams didn’t typically think about test and release cycles, or misconfiguration of the underlying infrastructure itself.

Where DevOps optimized the lifecycle of software from coding to release, DataOps is about the flow of data, breaking data out of work silos to collaborate on the movement of data from its inception, to its arrival, processing, and use within modern data architectures to feed production BI and machine learning applications.

DataOps jobs, especially in a cloudy, distributed data estate, aren’t the same as DevOps jobs. For instance, if a cloud application becomes unavailable, DevOps teams might need to reboot the server, adjust an API, or restart the K8s cluster.

If a DataOps-led application starts failing, it may show incorrect results instead of simply crashing, and cause leaders informed by faulty analytics and AI inferences to make disastrous business decisions. Figuring out the source of data errors and configuration problems can be maddeningly difficult, and DataOps teams may even need to restore the whole data estate – including values moving through ephemeral containers and pipelines – back to a valid, stable state for that point in time.

Why does DataOps need its own observability?

Once software observability started finding its renaissance within DevOps practices and early microservices architectures five years ago, we also started seeing some data management vendors pivoting to offer ‘data observability’ solutions.

The original concept of data observability was concerned with database testing, properly modeling, addressing and scaling databases, and optimizing the read/write performance and security of both relational and cloud back ends.

In much the same way that the velocity and automated release cadence of DevOps meant dev and ops teams needed to shift component and integration testing left, data teams need to tackle data application performance and data quality earlier in the DataOps lifecycle.

In essence, DataOps teams are using agile and other methodologies to develop and deliver analytics and machine learning at scale. Therefore they need DataOps observability to clarify the complex inner plumbing of apps, pipelines and clusters handling that moving data. Savvy DataOps teams must monitor ever-increasing numbers of unique data objects moving through data pipelines.

The KPIs for measuring success in DataOps observability include metrics and metadata that standard observability tools would never see: differences or anomalies in data layout, table partitioning, data source lineages, degrees of parallelism, data job and subroutine runtimes and resource utilization, interdependencies and relationships between data sets and cloud infrastructures – and the business tradeoffs between speed, performance and cost (or FinOps) of implementing recommended changes.

The Intellyx Take

A recent survey noted that 97 percent of data engineers ‘feel burned out’ at their current jobs, and 70 percent say they are likely to quit within a year! That’s a wakeup call for why DataOps observability matters now more than ever.

We must maintain the morale of understaffed and overworked data teams, when these experts take a long time to train and are almost impossible to replace in today’s tight technical recruiting market.

Any enterprise that intends to deliver modern DataOps should first consider equipping data teams with DataOps observability capabilities. Observability should go beyond the traditional metrics and telemetry of application code and infrastructure, empowering DataOps teams to govern data and the resources used to refine and convert raw data into business value as it flows through their cloud application estates.

Next up in part 2 of this series: Jason Bloomberg on the transformation from DevOps to DataOps!

©2022 Intellyx LLC. Intellyx retains editorial control of this document. At the time of writing, Unravel Data is an Intellyx customer. Image credit: Phil O’Driscoll, flickr CC2.0 license, compositing pins by author.

The post Why do we need DataOps Observability? appeared first on Unravel.

You’re unlikely to be able to build a business without data.

But how can you use it effectively?

There are so many ways you can use data in your business, from creating better products and services for customers, to improving efficiency and reducing waste.

Enter data observability. Using agile development practices, companies can create, deliver, and optimize data products, quickly and cost-effectively. Organizations can easily identify a problem to solve and then break it down into smaller pieces. Each piece is then assigned to a team that breaks down the work to solve the problem into a defined set of time – usually called a sprint – that includes planning, work, deployment, and review.

Marion Shaw, Head of Data and Analytics at Chaucer Group, and Unravel’s Ben Cooper presented on transforming data analytics to build better products and services and on making data analytics more efficient, respectively, at a recent Chief Disruptor virtual event.

Following on from the presentation, members joined a roundtable discussion and took part in a number of polls, in order to share their experiences. Here are just some examples of how companies have used data analytics to drive innovation:

• Improved payments. An influx of customer calls asking “where’s my money or payment?” prompted a company to introduce a “track payments” feature as a way of digitally understanding the payment status. As a result, the volume of callers decreased, while users of the new feature actually eclipsed the amount of original complaints, which proved there was a batch of customers who couldn’t be bothered to report problems but still found the feature useful. “If you make something easy for your customers, they will use it.“

• Cost reduction and sustainability: Moving from plastics to paper cups improved cost reduction and sustainability for one company, showing how companies can use their own data to make business decisions.

• New products: Using AI in drug discovery, data collaboration, to explore disease patterns can help pharmaceutical companies find new treatments for diseases with the potential for high returns. Cost of discovery is as expensive as using big data sets.

The key takeaways from the discussion were:

• Make it simple. When you make an action easy for your customers, they will use it.
• Lean on the data. If there isn’t data behind someone’s viewpoint, then it is simply an opinion.
• Get buy-in. Data teams need to buy into the usage of data—just because a data person owns the data side of things does not mean that they are responsible for the benefits or failings of it.

Using data analytic effectively with data observability is key. Companies across all industries are using data observability to create better products and services, reduce waste, and to improve productivity.

But data observability is no longer just about data quality or observing the condition of the data itself. Today it encompasses much more, and you can’t “borrow” your software teams’ observability solution. Discover more in our report DataOps Observability: The Missing Link for Data Teams.

The post Three Companies Driving Better Business Outcomes from Data Analytics appeared first on Unravel.

Uncontrolled cloud costs pose an enormous risk for any organization. The longer these costs go ungoverned, the greater your risk. Volatile, unforeseen expenses eat into profits. Budgets become unstable. Waste and inefficiency go unchecked. Making strategic decisions becomes difficult, if not impossible. Uncertainty reigns.

Everybody’s cloud bill continues to get steeper month over month, and the most rapidly escalating (and often the single largest) slice of the pie is cloud costs for data analytics—usually at least 40%. With virtually every organization asking for ever more data analytics, and more data workloads moving to the cloud, uncontrolled cloud data costs increasingly become a bigger part of the overall problem.

Data workloads are the fastest-growing cloud cost category and, if they’re not already, will soon be the #1 cloud expense.

All too often, business leaders don’t even have a clear understanding of where the money is going—or why—for cloud data expenditures, much less a game plan for bringing these costs under control.

Ungoverned cloud data usage and costs result in multiple, usually coexisting, business vulnerabilities.

Consider these common scenarios:

• You’re a C-suite executive who is ultimately responsible for cloud costs, but can’t get a good explanation from your data team leaders about why your Azure Databricks or AWS or Google Cloud costs in particular are skyrocketing.
• You’re a data team leader who’s on the hook to explain to the C-suite these soaring cloud data costs, but can’t. You don’t have a good handle on exactly who’s spending how much on what. This is a universal problem, no matter what platform or cloud provider: 70% of organizations aren’t sure what they spend their cloud budget on.
• You’re in Finance and are getting ambushed by your AWS (or Databricks or GCP) bill every month. Across companies of every size and sector, usage and costs are wildly variable and unpredictable—one survey has found that cloud costs are higher than expected for 6 of every 10 organizations.
• You’re a business product owner who needs additional budget to meet the organization’s increasing demand for more data analytics but don’t really know how much more money you’ll need. Forecasting—whether for Snowflake, Databricks, Amazon EMR, BigQuery, or any other cloud service—becomes a guessing game: Forrester has found that 80% of companies have difficulty predicting data-related cloud costs, and it’s becoming ever more problematic to keep asking for more budget every 3-6 months.
• You’re an Engineering/Operations team lead who knows there’s waste and inefficiency in your cloud usage but don’t really know how much or exactly where, much less what to do about it. Your teams are groping in the dark, playing blind man’s bluff. And it’s getting worse: 75% of organizations report that cloud waste is increasing.
• Enterprise architecture teams are seeing their cloud migration and modernization initiatives stall out. It’s common for a project’s three-year budget to be blown by Q1 of Year 2. And increasingly companies are pulling the plug. A recent report states that 81% of IT teams have been directed by the C-suite to cut or halt cloud spending. You find yourself paralyzed, unable to move either forward or back—like being stranded in a canoe halfway through a river crossing.
• Data and Finance VPs don’t know the ROI of their modern data stack investments—or even how to figure that out. But whether you measure it or not, the ROI of your modern data stack investments nosedives as costs soar. You find it harder and harder to justify your decision to move to Databricks or Snowflake or Amazon EMR or BigQuery. PricewaterhouseCoopers has found that over half (53%) of enterprises have yet to realize substantial value from their cloud investments.
• With seemingly no end in sight to escalating cloud costs, data executives may even be considering the radical step of sacrificing agility and speed-to-market gains and going back to on-prem (repatriation). The inability to control costs is a leading reason why 71% of enterprises expect to move all or some of their workloads to on-prem environments.

How’d we get into this mess?

The problem of ungoverned cloud data costs is universal and has been with us for a while: 83% of enterprises cite managing cloud spend as their top cloud challenge, and optimizing cloud usage is the top cloud initiative for the sixth straight year, according to the 2022 State of the Cloud Report.

Some of that is simply due to the increased volume and velocity of data and analytics. In just a few short years, data analytics has gone from a science project to an integral business-critical function. More data workloads are running in the cloud, often more than anticipated. Gartner has noted that overall cloud usage for data workloads “almost always exceeds initial expectations,” stating that workloads may grow 4X or more in the first year alone.

If you’re like most data-driven enterprises, you’ve likely invested millions in the innovative data platforms and cloud offerings that make up your modern data stack—Databricks, Snowflake, Amazon EMR, BigQuery, Dataproc, etc.—and have realized greater agility and go-to-market speed. But those benefits have come at the expense of understanding and controlling the associated costs. You’re also seeing your cloud costs lurch unpredictably upwards month over month. And that’s the case no matter which cloud provider(s) you’re on, or which platform(s) you’re running. Your modern data stack consumes ever more budget, and it never seems to be enough. You’re under constant threat of your cloud data costs jumping 30-40% in just six months. It’s become a bit like the Wild West, where everybody is spinning up clusters and incurring costs left and right, but nobody is in control to govern what’s going on.

The only thing predictable about modern data stack costs is that they seem to always go up. Source: Statistica

Many organizations that have been wrestling with uncontrolled cloud data costs have begun adopting a FinOps approach. Yet they are struggling to put these commonsensical FinOps principles into practice for data teams. They find themselves hamstrung by generic FinOps tools and are hitting a brick wall when it comes to actualizing foundational core capabilities.

So why are organizations having trouble implementing DataFinOps (FinOps for data teams)?

FinOps for data teams

Just as DevOps and DataOps are “cousin” approaches—bringing agile and lean methodologies to software development and data management, tackling similar types of challenges, but needing very distinct types of information and analyses to get there—FinOps and DataFinOps are related but different. In much the same way (and for similar reasons) DevOps observability built for web applications doesn’t work for data pipelines and applications, DataFinOps brings FinOps best practices to data management, taking the best of FinOps to help data teams measure and improve cost effectiveness of data pipelines and data applications.

DataOps + FinOps = DataFinOps

FinOps principles and approach

As defined by the FinOps Foundation, FinOps is “an evolving cloud financial management discipline and cultural practice that enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions.”

It’s important to bear in mind that FinOps isn’t just about lowering your cloud bill—although you will wind up saving money in a lot of areas, running things more cost-effectively, and being able to do more with less (or at least the same). It’s about empowering engineers and business teams to make better choices about their cloud usage and deriving the most value from their cloud investments.

There are a few underlying “north star” principles that guide all FinOps activities:

• Cloud cost governance is a team sport. Too often controlling costs devolves into Us vs. Them friction, frustration, and finger-pointing. It can’t be done by just one group alone, working with their own set of data (and their own tools) from their own perspective. Finance, Engineering, Operations, technology teams, and business stakeholders all need to be on the same page, pulling in the same direction to the same destination, working together collaboratively.
• Spending decisions are driven by business value. Not all cloud usage is created equal. Not everything merits the same priority or same level of investment/expenditure. Value to the business is the guiding criterion for making collaborative and intelligent decisions about trade-offs between performance, quality, and cost.
• Everyone takes ownership of their cloud usage. Holding individuals accountable for their own cloud usage—and costs—essentially shifts budget responsibility left, onto the folks who actually incur the expenses.This is crucial to controlling cloud costs at scale, but to do so, you absolutely must empower engineers and Operations with the self-service optimization capabilities to “do the right thing” themselves quickly and easily.
• Reports are accessible and timely. To make data-driven decisions, you need accurate, real-time data. The various players collaborating on these decisions all bring their own wants and needs to the table, and everybody needs to be working from the same information, seeing the issue(s) the same way—a single pane of glass for a single source of truth.Dashboards and reports must be visible, understandable, and practical for a wide range of people making these day-to-day cost-optimization spending decisions.

Applying FinOps to data management

Data teams can put these FinOps principles into practice with a three-phase iterative DataFinOps life cycle:

• Observability, or visibility, is understanding what is going on in your environment, measuring everything, tracking costs, and identifying exactly where the money is going.
• Optimization is identifying where you can eliminate waste and inefficiencies, take advantage of less-expensive cloud pricing options, or otherwise run your cloud operations more cost-efficiently—and then empowering individuals to actually make improvements.
• Governance is all about going from reactive problem-solving to proactive problem-preventing, sustaining iterative improvements, and implementing guardrails and alerts.

The DataFinOps life cycle: observability, optimization, and governance.

The essential principles and three-phase approach of FinOps are highly relevant to data teams. But just as DataOps requires a different set of information than DevOps, data teams need unique and data-specific details to achieve observability, optimization, and governance in their everyday DataFinOps practice.

What makes DataFinOps different—and difficult

The following challenges all manifest themselves in one way or another at each stage of the observability/optimization/governance DataFinOps life cycle. Be aware that while you can’t do anything until you have full visibility/observability into your cloud data costs, the optimization and governance phases are usually the most difficult and most valuable to put into action.

First off, just capturing and correlating the volume of information across today’s modern data stacks can be overwhelming. The sheer size and complexity of data applications/pipelines—all with multiple sub-jobs and sub-tasks processing in parallel—generates millions of metrics, metadata, events, logs, etc. Then everything has to be stitched together somehow in a way that makes sense to Finance and Data Engineering and DataOps and the business side.

Even a medium-sized data analytics operation has 100,000s of individual “data-driven spending decisions” to make. That’s the bad news. The good news is that this same complexity means there are myriad opportunities to optimize costs.

Second, the kinds of details (and the level of granularity) your teams need to make informed, intelligent DataFinOps decisions simply are not captured or visualized by cloud cost-management FinOps tools or platform-specific tools like AWS Cost Explorer, OverWatch, or dashboards and reports from GCP and Microsoft Cost Management. It’s the highly granular details about what’s actually going on in your data estate (performance, cost) that will uncover opportunities for cloud cost optimizations. But those granular details are scattered across dozens of different tools, technologies, and platforms.

Third, there’s usually no single source of truth. Finance, data engineers, DataOps, and the business product owners all use their own particular tools of choice to manage different aspects of cloud resources and spending. Without a common mechanism (the proverbial single pane of glass) to see and measure cost efficiency, it’s nearly impossible for any of them to make the right call.

Finally, you need to be able to see the woods for the trees. Seemingly simple, innocuous changes to a single cloud data application/pipeline can have a huge blast radius. They’re all highly connected and interdependent: the output of something upstream is the input to something else downstream or, quite likely, another application/pipeline for somebody somewhere else in the company. Everything must be understood within a holistic business context so that everybody involved in DataFinOps understands how everything is working as a whole.

Implementing DataFinOps in practice: A 6-step game plan

At its core, DataFinOps elevates cost as a first-class KPI metric, right alongside performance and quality. Most data team SLAs revolve around performance and quality: delivering reliable results on time, every time. But with cloud spend spiraling out of control, now cost must be added to reliability and speed as a co-equal consideration.

1. Set yourself up for success

Before launching DataFinOps into practice, you need to lay some groundwork around organizational alignment and expectations. The collaborative approach and the principle of holding individuals accountable for their cloud usage are a cultural sea-change. DataFinOps won’t work as a top-down mandate without buy-in from team members further down the ladder. Recognize that improvements will grow proportionally over time as your DataFinOps practice gains momentum. It’s best to adopt a crawl-walk-run approach, where you roll out a pilot project to discover best practices (and pitfalls) and get a quick win that demonstrates the benefits. Find a data analyst or engineer with an interest in the financial side who can be the flag-bearer, maybe take some FinOps training (the FinOps Foundation is an excellent place to start), and then dig into one of your more expensive workloads to see how to apply DataFinOps principles in practice. Similarly, get at least one person from the finance and business teams who is willing to get onboard and discover how DataFinOps works from their side.

Check out this 15-minute discussion on how JPMorgan Chase & Co, tackled FinOps at scale.

2. Get crystal-clear visibility into where the money is going

Before you can even begin to control your cloud costs, you have to understand—with clarity and precision—where the money is going. Most organizations have only hazy visibility into their overall cloud spend. The 2022 State of Cloud Cost Report states that gaining visibility into cloud usage is the single biggest challenge to controlling costs. The 2022 State of Cloud Cost Intelligence Report further finds that only 3 out of 10 organizations know exactly where their spend is going, with the majority either guesstimating or having no idea. And the larger the company, the more significant the cost visibility problem.

A big part of the problem are the cloud bills themselves. They’re either opaque or inscrutable; either way, they lack the business context that makes sense to your particular organization. Cloud vendor billing consoles (and third-party cost-management tools) give you only an aggregated view of total spend for different services (compute, storage, platform, etc.).

Cloud bills don’t answer questions like, Who spent on these services? Which department, teams, users are the top spenders?

Or you have to dedicate a highly trained expert or two to decode hundreds of thousands (millions, even) of individual billing line items and figure out who submitted which jobs (by department, team, individual user) for what purpose—who’s spending what, when, and why.

Decoding 1,000,000s of individual lines of billing data is a full-time job.

The lack of visibility is compounded by today’s heterogeneous multi-cloud, multi-platform reality. Most enterprise-scale organizations use a combination of different cloud providers and different data platforms, often the same platform on different cloud providers. While this is by no means a bad thing from an agility perspective, it does make comprehending overall costs across different providers even more difficult.

So, the first practical step to implementing DataFinOps is understanding your costs with clarity and precision. You need to be able to understand at a glance, in terms that make sense to you, which projects, applications, departments, teams, and individual users are spending exactly how much, and why.

You get precise visualization of cloud data costs through a combination of tagging and full-stack observability. Organizations need to apply some sort of cost-allocation tagging taxonomy to every piece of their data estate in order to categorize and track cloud usage. Then you need to capture a wide and deep range of granular performance details, down to the individual job level, along with cost information.

All this information is already available in different systems, hidden in plain sight. What you need is a way to pull it all together, correlating everything in a data model and mapping resources to the tagged applications, teams, individuals, etc.

Once you have captured and correlated all performance/cost details and have everything tagged, you can slice and dice the data to do a number of things with a high degree of accuracy:

• See who the “big spenders” are—which departments, teams, applications, users are consuming the most resources—so you can prioritize where to focus first on cost optimization, for the biggest impact.
• Track actual spend against budgets—again, by project or group or individual. Know ahead of time whether usage is projected to be on budget, is at risk, or has already gone over. Avoid “sticker shock” surprises when the monthly bill arrives.
• Forecast with accuracy and confidence. You can run regular reports that analyze historical usage and trends (e.g., peaks and valleys, auto-scaling) to predict future capacity needs. Base projections on data, not guesswork.
• Allocate costs with pinpoint precision. Generate cost-allocation reports that tell you exactly—down to the penny—who’s spending how much, where, and on what.

3. Understand why the costs are what they are

Beyond knowing what’s being spent and by whom, there’s the question of how money is being spent—specifically, what resources were allocated for the various workload jobs and how much they cost. Your data engineers and DataOps teams need to be able to drill down into the application- and cluster-level configuration details to identify which individuals are using cloud data resources, the size and number of resources they’re using, which jobs are being run, and what their performance looks like. For example, some data jobs are constrained by network bandwidth, while others are memory- or CPU-bound. Running those jobs on clusters and instance types may not maximize the return on your cloud data investments. If you think about cost optimization as a before-and-after exercise, your teams need a clear and comprehensive picture of the “before.”

With so many different cloud pricing models, instance types, platforms, and technologies available to choose from, you need everybody to have a fully transparent 360° view into the way expenses are being incurred at any given time so that the next step—cost optimization and collaborative decision-making about cloud spend—can happen intelligently.

4. Identify opportunities to do things more cost-effectively

If the observability phase gives you visibility into where costs are going, how, and why, the optimization phase is about taking that information and figuring out where you can eliminate waste, remove inefficiencies, and leverage different options that are less expensive but still meet business needs.

Most companies have very limited insight (if any) into where they’re spending more than they need to. Everyone knows on an abstract level that cloud waste is a big problem in their organization, but identifying concrete examples that can be remediated is another story. Various surveys and reports peg the amount of cloud budget going to waste between 30-50%, but the true percentage may well be higher—especially for data teams—because most companies are just guessing and tend to underestimate their waste. For example, after implementing cloud data cost optimization, most Unravel customers have seen their cloud spend lowered by 50-60%

An enterprise running 10,000s (or 100,000s) of data jobs every month has literally millions of decisions to make—at the application, pipeline, and cluster level—about where, when, and how to run those jobs. And each individual decision about resources carries a price tag.

Enterprises have 100,000s of places where cloud data spending decisions need to be made.

Only about 10% of cost-optimization opportunities are easy to see, for example, shutting down idle clusters (deployed but no-longer-in-use resources). The remaining 90% lie below the surface.

90% of cloud data cost-optimization opportunities are buried deep, out of immediate sight.

You need to go beyond knowing how much you are currently spending, to knowing how much you should be spending in a cost-optimized environment. You need insight into what resources are actually needed to run the different applications/pipelines vs. what resources are being used.

Data engineers are not wasting money intentionally; they simply don’t have the insights to run their jobs most cost-efficiently. Identifying waste and inefficiencies needs two things in already short supply: time and expertise. Usually you need to pull engineering or DataOps superstars away from what they’re doing—often the most complex or business-critical projects—to do the detective work of analyzing usage vs. actual need. This kind of cost-optimization analysis does get done today, here and there, on an ad hoc basis for a handful of applications, but doing so at enterprise scale is something that remains out of reach for most companies.

The complexity is overwhelming: for example, AWS alone has more than 200 cloud services and over 600 instance types available, and has changed its prices 107 times since its launch in 2006. For even your top experts, this can be laborious, time-consuming work, with continual trial-and-error, hit-or-miss attempts.

Spoiler alert: You need AI

AI is crucial to understanding where cost optimization is needed—at least, at enterprise scale and to make an impact. It can take hours, days, sometimes weeks, for even your best people to tune a single application for cost efficiency. AI can do all this analysis automatically by “throwing some math” at all the observability performance and cost data to uncover where there’s waste and inefficiencies.

• Overprovisioned resources. This is most often why budgets go off the rails, and the primary source of waste. The size and number of instances and containers is greater than necessary, having been allocated based on perceived need rather than actual usage. Right-sizing cloud resources alone can save an organization 50-60% of its cloud bill.
• Instance pricing options. If you understand what resources are needed to run a particular application (how that application interacts with other applications), the DataFinOps team can make informed decisions about when to use on-demand, reserved, or spot instances. Leveraging spot instances can be up to 90% less expensive than on-demand—but you have to have the insights to know which jobs are good candidates.
• Bad code. AI understands what the application is trying to do and can tell you that it’s been submitted in a way that’s not efficient. We’ve seen how a single bad join on a multi-petabyte table kept a job running all weekend and wound up costing the company over $500,000.
• Intelligent auto-scaling. The cloud’s elasticity is great but comes at a cost, and isn’t always needed. AI analyzes usage trends to help predict when auto-scaling is appropriate—or when rescheduling the job may be a more cost-effective option.
• Data tiering. You probably have petabytes of data. But you’re not using all of it all of the time. AI can tell you which datasets are (not) being used, applying cold/warm/hot labels based on age or usage, so you understand which ones haven’t been touched in months yet still sit on expensive storage. Moving cold data to less-expensive options can save 80-90% on storage costs.

5. Empower individuals with self-service optimization

Holding individual users accountable for their own cloud usage and cost is a bedrock principle of DataFinOps. But to do that, you also have to give them the insights and actionable intelligence for them to make better choices.

You have to make it easy for them to do the right thing. They can’t just be thrown a bunch or charts and graphs and metrics, and be expected to figure out what to do. They need quick and easy-to-understand prescriptive recommendations on exactly how to run their applications reliably, quickly, and cost-efficiently.

Ideally, this is where AI can also help. Taking the analyses of what’s really needed vs. what’s been requested, AI could generate advice on exactly where and how to change settings or configurations or code to optimize for cost.

Leveraging millions of data points and hundreds of cost and performance algorithms, AI offers precise, prescriptive recommendations for optimizing costs throughout the modern data stack.

6. Go from reactive to proactive

Optimizing costs reactively is of course highly beneficial—and desperately needed when costs are out of control—but even better is actively governing them. DataFinOps is not a straight line, with a beginning and end, but rather a circular life cycle. Observability propels optimization, whose outcomes become baselines for proactive governance and feed back into observability.

Governance is all about getting ahead of cost issues beforehand rather than after the fact. Data team leaders should implement automated guardrails that give a heads-up when thresholds for any business dimension (projected budget overruns, jobs that exceed a certain size, time, cost) are crossed. Alerts could be triggered whenever a guardrails constraint is violated, notifying the individual user—or sent up the chain of command—that their job will miss its SLA or cost too much money and they need to find less expensive options, rewrite it to be more efficient, reschedule it, etc. Or guardrail violations could trigger preemptive corrective “circuit breaker’ actions to kill jobs or applications altogether, request configuration changes, etc., to rein in rogue users, put the brakes on runaways jobs, and nip cost overruns in the bud.

Controlling particular users, apps, or business units from exceeding certain behaviors has a profound impact on reining in cloud spend.

Conclusion

The volatility, obscurity, and lack of governance over rapidly growing cloud data costs introduce a high degree of unpredictability—and risk—into your organization’s data and analytics operations. Expenses must be brought under control, but reducing costs by halting activities can actually increase business risks in the form of lost revenue, SLA breaches, or even brand and reputation damage. Taking the radical step of going back on-prem may restore control over costs but sacrifices agility and speed-to-market, which also introduces the risk of losing competitive edge.

A better approach, DataFinOps, is to use your cloud data spend more intelligently and effectively. Make sure that your cloud investments are providing business value, that you’re getting the highest possible ROI from your modern data stack. Eliminate inefficiency, stop the rampant waste, and make business-driven decisions—based on real-time data, not guesstimates—about the best way to run your workloads in the cloud.

That’s the driving force behind FinOps for data teams, DataFinOps. Collaboration between engineering, DataOps, finance, and business teams. Holding individual users accountable for their cloud usage (and costs).

But putting DataFinOps principles into practice is a big cultural and organizational shift. Without the right DataFinOps tools, you’ll find it tough sledding to understand exactly where your cloud data spend is going (by whom and why) and identify opportunities to operate more cost-efficiently. And you’ll need AI to help empower individuals to optimize for cost themselves.

Then you can regain control over your cloud data costs, restore stability and predictability to cloud budgets, be able to make strategic investments with confidence, and drastically reduce business risk.

The post Taming Cloud Costs for Data Analytics with FinOps appeared first on Unravel.

The 2023 Data Teams Summit (formerly DataOps Unleashed) was a smashing success, with over 2,000 participants from 1,600 organizations attending 23 expert-led breakout sessions, panel discussions, case studies, and keynote presentations covering a wide range of thought leadership and best practices.

There were a lot of sessions devoted to different strategies and considerations when building a high-performing data team, how to become a data team leader, where data engineering is heading, and emerging trends in DataOps (asset-based orchestration, data contracts, data mesh, digital twins, data centers center of excellence). And winding as a common theme throughout almost every presentation was that top-of-mind topic: FinOps and how to get control over galloping cloud data costs.

Some of the highlight sessions are available now on demand (no form to fill out) on our Data Teams Summit 2023 page. More are coming soon. 

There was a lot to see (and full disclosure: I didn’t get to a couple of sessions), but here are 3 sessions that I found particularly interesting.

Enabling strong engineering practices at Maersk

The fireside chat between Unravel CEO and Co-founder Kunal Agarwal and Mark Sear, Head of Data Platform Optimization at Maersk, one of the world’s largest logistics companies, is entertaining and informative. Kunal and Mark cut through the hype to simplify complex issues in commonsensical, no-nonsense language about:

• The “people problem” that nobody’s talking about
• How Maersk was able to upskill its data teams at scale
• Maersk’s approach to rising cloud data costs
• Best practices for implementing FinOps for data teams

Check out their talk here

Maximize business results with FinOps

Unravel DataOps Champion and FinOps Certified Practitioner Clinton Ford and FinOps Foundation Ambassador Thiago Gil explain how and why the emerging cloud financial management discipline of FinOps is particularly relevant—and challenging—for data teams. They cover:

• The hidden costs of cloud adoption
• Why observability matters
• How FinOps empowers data teams
• How to maximize business results 
• The state of production ML

See their session here

Situational awareness in a technology ecosystem

Charles Boicey, Chief Innovation Officer and Co-founder of Clearsense, a healthcare data platform company, explores the various components of a healthcare-centric data ecosystem and how situational awareness in the clinical environment has been transferred to the technical realm. He discusses:

• What clinical situational awareness looks like
• The concept of human- and technology-assisted observability
• The challenges of getting “focused observability” in a complex hybrid, multi-cloud, multi-platform modern data architecture for healthcare
• How Clearsense leverages observability in practice
• Observability at the edge

Watch his presentation here

How to get more session recordings on demand

1. To see other session recordings without any registration, visit the Unravel Data Teams Summit 2023 page. 
2. To see all Data Teams Summit 2023 recordings, register for access here.

And please share your favorite takeaways, see what resonated with your peers, and join the discussion on LinkedIn.  

The post 3 Takeaways from the 2023 Data Teams Summit appeared first on Unravel.

Data teams and their business-side colleagues now expect—and need—more from their observability solutions than ever before. Modern data stacks create new challenges for performance, reliability, data quality, and, increasingly, cost. And the challenges faced by operations engineers are going to be different from those for data analysts, which are different from those people on the business side care about. That’s where DataOps observability comes in.

But what is DataOps observability, exactly? And what does it look like in a practical sense for the day-to-day life of data application developers or data engineers or data team business leaders? 

In the Unravel virtual panel discussion What Is Data Observability? Sanjeev Mohan, principal with SanjMo and former Research Vice President at Gartner, lays out the business context and driving forces behind DataOps observability, and Chris Santiago, Unravel Vice President of Solutions Engineering, shows how different roles use the Unravel DataOps observability platform to address performance, cost, and quality challenges. 

Why (and what is) DataOps observability?

Sanjeev opens the conversation by discussing the top three driving trends he’s seeing from talking with data-driven organizations, analysts, vendors, and fellow leaders in the data space. Specifically, how current economic headwinds are causing reduced IT spend—except in cloud computing and, in particular, data and analytics. Second, with the explosion of innovative new tools and technologies, companies are having difficulty in finding people who can tie together all of these moving pieces and are looking to simplify this increasing complexity. Finally, more global data privacy regulations are coming into force while data breaches continue unabated. Because of these market forces, Sanjeev says, “We are seeing a huge emphasis on integrating, simplifying, and understanding what happens between a data producer and a data consumer. What happens between these two is data management, but how well we do the data management is what we call DataOps.”

Sanjeev presents his definition of DataOps, why it has matured more slowly than its cousin DevOps, and the kind(s) of observability that is critical to DataOps—data pipeline reliability and trust in data quality—and how his point of view has evolved to now include a third dimension: demonstrating the business value of data through BizOps and FinOps. Taken together, these three aspects (performance, cost, quality) give all the different personas within a data team the observability they need. This is what Unravel calls DataOps observability.

DataOps observability in practice with Unravel

Chris Santiago walks through how the various players on a data team—business leadership, application/pipeline developers and engineers, data analysts—use Unravel across the three critical vectors of DataOps observability: performance (reliability of data applications/pipelines), quality (trust in the data), and cost (value/ROI modern data stack investments).

Cost (value, ROI)

First up is how Unravel DataOps observability provides deep visibility and actionable intelligence into cost governance. As opposed to the kind of dashboards that cloud providers themselves offer—which are batch-processed aggregated summaries—Unravel lets you drill down from that 10,000-foot view into granular details to see exactly where the money is going. Chris uses a Databricks chargeback report example, but it would be similar for Snowflake, EMR, or GCP. He shows how with just a click, you can filter all the information collected by Unravel to see with granular precision which workspaces, teams, projects, even individual users, are consuming how many resources across your entire data estate—in real time. 

From there, Chris demonstrates how Unravel can easily set up budget tracking and automated guardrails for, say, a user (or team or project or any other tagging category that makes sense to your particular business). Say you want to track usage by the metric of DBUs; you set the budget/guardrail at a predefined threshold and get real-time status insight into whether that user is on track or is in danger of exceeding the DBU budget. You can set up alerts to get ahead of usage instead of getting notified only after you’ve blown a monthly budget.

See more self-paced interactive product tours here

Performance (reliability)

Chris then showed how Unravel DataOps observability helps the people who are actually on the hook for data pipeline reliability, making sure everything is working as expected. When applications or pipelines fail, it’s a cumbersome task to hunt down and cobble together all the information from disparate systems (Databricks, Airflow, etc.) to understand the root cause and figure out the next steps. Chris shows how Unravel collects and correlates all the granular details about what’s going on and why from a pipeline view. And then how you can drill down into specific Spark jobs. From a single pane of glass, you have all the pertinent logs, errors, metrics, configuration settings, etc., from Airflow or Spark or any other component. All the heavy lifting has been done for you automatically.

But where Unravel stands head and shoulders above everyone else is its AI-powered analysis. Automated root cause analysis diagnoses why jobs failed in seconds, and pinpoints exactly where in a pipeline something went wrong. For a whole class of problems, Unravel goes a step further and provides crisp, prescriptive recommendations for changing configurations or code to improve performance and meet SLAs. 

See more self-paced interactive product tours here

Data quality (trust)

Chris then pivots away from the processing side of things to look at observability of the data itself—especially how Unravel provides visibility into the characteristics of data tables and helps prevent bad data from wreaking havoc downstream. From a single view, you can understand how large tables are partitioned, size of the data, who’s using the data tables (and how frequently), and a lot more information. But what may be most valuable is the automated analysis of the data tables. Unravel integrates external data quality checks (starting with the open source Great Expectations) so that if certain expectations are not met—null values, ranges, number of final columns—Unravel flags the failures and can automatically take user-defined actions, like alert the pipeline owner or even kill a job that fails a data quality check. At the very least, Unravel’s lineage capability enables you to track down where the rogue data got introduced and what dependencies are affected.

Whether it’s engineering teams supporting data pipelines, developers themselves making sure they hit their SLAs, budget owners looking to control costs, or business leaders who funded the technology looking to realize full value—everyone who constitutes a “data team”—DataOps observability is crucial to ensuring that data products get delivered reliably on time, every time, in the most cost-efficient manner.

The Q&A session

As anybody who’s ever attended a virtual panel discussion knows, sometimes the Q&A session is a snoozefest, sometimes it’s great. This one is the latter. Some of the questions that Sanjeev and Chris field:

• If I’m migrating to a modern data stack, do I still need DataOps observability, or is it baked in?
• How is DataOps observability different from application observability?
• When should we implement DataOps observability?
• You talked about FinOps and pipeline reliability. What other use cases come up for DataOps observability?
• Does Unravel give us a 360 degree view of all of the tools in the ecosystem, or does it only focus on data warehouses like Snowflake?

To jump directly to the Q&A portion of the event, click on the video image below.

The post Panel recap: What Is DataOps observability? appeared first on Unravel.

The Data Teams Summit 2023 is just around the corner!

This year, on January 25, 2023, we’re taking the peer-to-peer empowerment of data teams one step further, transforming DatOps Unleashed into Data Teams Summit to better reflect our focus on the people—data teams—who unlock the value of data.

Data Teams Summit is an annual, full-day virtual conference, led by data rockstars at future-forward organizations about how they’re establishing predictability, increasing reliability, and creating economic efficiencies with their data pipelines.

Join us for sessions on:

• DataOps best practices
• Data team productivity and self-service
• DataOps observability
• FinOps for data teams
• Data quality and governance
• Data modernizations and infrastructure

The peer-built agenda is packed, with over 20 panel discussions and breakout sessions. Here’s a sneak peek at some of the most highly anticipated presentations:

Keynote Panel: Winning strategies to unleash your data team

Great data outcomes depend on successful data teams. Every single day, data teams deal with hundreds of different problems arising from the volume, velocity, variety—and complexity—of the modern data stack.

Learn best practices and winning strategies about what works (and what doesn’t) to help data teams tackle the top day-to-day challenges and unleash innovation.

Breakout Session: Maximize business results with FinOps

As organizations run more data applications and pipelines in the cloud, they look for ways to avoid the hidden costs of cloud adoption and migration. Teams seek to maximize business results through cost visibility, forecast accuracy, and financial predictability.

In this session, learn why observability matters and how a FinOps approach empowers DataOps and business teams to collaboratively achieve shared business goals. This approach uses the FinOps Framework, taking advantage of the cloud’s variable cost model, and distributing ownership and decision-making through shared visibility to get the biggest return on their modern data stack investments.

See how organizations apply agile and lean principles using the FinOps framework to boost efficiency, productivity, and innovation.

Breakout Session: Going from DevOps to DataOps

DevOps has had a massive impact on the web services world. Learn how to leverage those lessons and take them further to improve the quality and speed of delivery for analytics solutions.

Ali’s talk will serve as a blueprint for the fundamentals of implementing DataOps, laying out some principles to follow from the DevOps world and, importantly, adding subject areas required to get to DataOps—which participants can take back and apply to their teams.

Breakout Session: Becoming a data engineering team leader

As you progress up the career ladder for data engineering, responsibilities shift as you start to become more hands-off and look at the overall picture rather than a project in particular.

How do you ensure your team’s success? It starts with focusing on the team members themselves.

In this talk, Matt Weingarten, a lead Data Engineer at Disney Streaming, will walk through some of his suggestions and best practices for how to be a leader in the data engineering world.

Attendance is free! Sign up here for a free ticket

The post Sneak Peek into Data Teams Summit 2023 Agenda appeared first on Unravel.

What are the (rapidly) evolving trends in the data space? And how is Unravel responding—especially for FinOps? What’s on the product roadmap for 2023? How can you get the most out of your Unravel deployment today?

Unravel leadership presented a virtual year-end update to talk about emerging industry developments, give a sneak peek into what’s in store for 2023, and share our customers’ best practices on how they’re using Unravel to maximize business value.

Here are some of the highlights, with video clips.

The rapid rise of FinOps

CEO Kunal Agarwal discusses his vision for Unravel and how cost governance and cost optimization has become a top challenge for data teams running workloads in the cloud. Some 60% of organizations encounter cloud cost overruns. Some of that is due to increased workloads, but another big reason is that companies just have difficulty understanding how to run more efficiently. But that kind of “actionable intelligence” is the foundation of FinOps—engineering, finance, and business teams making collaborative data-driven decisions about spending.

It’s the same kind of insights—at both a granular and holistic level—you need to tackle performance issues and/or to migrate to the cloud reliably and efficiently. Kunal shows how customers are using Unravel to realize tangible benefits to optimize performance and cost, and recaps our recent $50 million Series D funding

Unravel roadmap for 2023

Vice President of Product Eric Chu gives a glimpse into what’s ahead for the Unravel platform. Over the next year, you’ll see Unravel expand its capabilities to include integration with data quality solutions, expand its coverage on the modern data stack, and introduce additional FinOps observability, intelligence, and automation features. Specifically:

• Unravel will support Snowflake
• Databricks SQL and Unity Catalog support
• Amazon EMR Serverless, as well as Glue and Kubernetes
• Integration with data quality solutions, starting with Great Expectations
• Integration with ServiceNow, PagerDuty, Jira, and more
• Top-down FinOps dashboards, as well as enhanced AI insights

We are collaborating with customers now to get design input and user feedback, enabling early access to preview features.

Unlocking value with Unravel

There are probably features in Unravel you already have access to but may not be leveraging to their fullest extent. Chris Santiago, Unravel VP of Solutions Engineering, digs into some hidden treasures and best practices that other customers use on a daily basis.

Chris walks through a FinOps-focused use case demo to show how Unravel capabilities helps at each stage of the cost-optimization lifecycle process: reporting (visualize, track, allocate), self-service insights (reduce, improve, empower), and guardrails & automation (control, prevent).

His demo takes us from a bird’s-eye view overview of Databricks costs, then walks through the various Unravel reports you can access to drill down into get precise insights—cost breakdown, user and usage, node downsizing recommendations, event analysis—into where the spend is going at a granular level, where there’s waste, where optimization can have the greatest impact, etc. Then Chris touches on how Unravel’s  AI recommendations fit in, as well as run through how to set up automated guardrails and real-time budget-tracking alerts to proactively control cloud costs. 

The demo is specific to Databricks, but there are dozens of reports available for any environment Unravel supports. Some of them are in beta, so reach out to your technical account manager (TAM) for access to or training on these reports. In the meantime, explore these key Unravel features.

New professional services offering

Unravel’s Global Head of Customer Success, Matt Sangar, introduces our new professional services to help you leverage Unravel’s full capabilities. Whether it’s improving application performance, lowering cloud costs, reducing MTTR and support ticket volume, or improving data team productivity, Unravel professional services can help you get the most out of Unravel quickly. He has added two new roles to your Customer Success team, an Unravel engineer and a project manager, that will work with you as an extension of your team.

Expanded training options

Unravel is also introducing new on-site training enablement for in-person learning, at basic, advanced, and custom level, for Admin and Ops teams, AppDev and Support teams, and FinOps—budget management training for Finance counterparts. 

Sometimes you might need extra help, or there are other teams who want/need to learn about Unravel. We now offer half- or full-day Unravel Data Days for on-site, in-person (or Zoom) workshops. We are also introducing office hours, when users can sit down with Unravel engineers to troubleshoot apps in real time.

For more information about our new training programs, professional services, access to beta reports, or anything else, please contact your TAM and we’ll take care of you!

The post 5 highlights from the Unravel Roadmap 2023 preview appeared first on Unravel.

Unravel is recognized as a Representative Vendor in the DataOps Market in the 2022 Gartner Market Guide for DataOps Tools.

Data teams are struggling to keep pace with the increased volume, velocity, variety—and complexity—of their data applications/pipelines. They are facing many of the same (generic) challenges that software teams did 10+ years ago. Just as DevOps helped streamline web application development and make software teams more productive, DataOps aims to do the same thing for data applications.

According to a Gartner strategic planning assumption from this Market Guide, “by 2025, a data engineering team guided by DataOps practices and tools will be 10 times more productive than teams that do not use DataOps.”

The report states that “the DataOps market is highly fragmented.” This Gartner Market Guide analyzes the DataOps market and explains the various capabilities of DataOps tools, paints a picture of the DataOps tool landscape, and offers recommendations.

Our understanding on DataOps and some of the key points we took away from the Gartner Market Guide:

The way data teams are doing things today isn’t working.

One of the key findings in the Gartner report analysis reveals that “a DataOps tool is a necessity to reduce the use of custom solutions and manual efforts around data pipeline operations. Buyers seek DataOps tools to streamline their data operations.” In our opinion, manual effort/custom solutions require prohibitively vast amounts of time and expertise—both of which are already in short supply within enterprise data teams. 

No single DataOps tool does everything.

DataOps covers a lot of ground. Gartner defines the core capabilities of DataOps tools as orchestration (including connectivity, workflow automation, data lineage, scheduling, logging, troubleshooting and alerting), observability (monitoring live/historic workflows, insights into workflow performance and cost metrics, impact analysis), environment management (infrastructure as code, resource provisioning, environment repository templates, credentials management), deployment automation, and test automation. We think it’s clear that no one tool does it all. The report recommends, “Follow the decision guidelines and avoid multibuy circumstances by understanding the diverse market landscape and focusing on a desired set of core capabilities.”

DataOps tools break down silos.

A consequence of modern data stack complexity is that disparate pockets of experts all run their own particular tools of choice in silos. Communication and collaboration break down, and you wind up with an operational Tower of Babel that leads to missed SLAs, friction and finger-pointing, and everybody spending way too much time firefighting. DataOps tools are designed specifically to avoid this unsustainable situation.

Choose DataOps tools that give a single pane of glass view.

The Gartner report recommends to “prioritize DataOps tools that give you a ‘single pane of glass’ for diverse data workloads across heterogeneous technologies with orchestration, lineage, and automation capabilities.” The modern data stack is a complex collection of different systems, platforms, technologies, and environment. Most enterprises use a combination of them all. You need a DataOps tool that works with all kinds of workloads in this heterogenous stack for different capabilities and different reasons—which can be boiled down to the three dimensions of performance, cost, and quality.

We are excited that Gartner recognized Unravel as a Representative Vendor in the DataOps market, and we couldn’t agree more with their Market Guide for DataOps Tools recommendations.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The post Our 4 key takeaways from the 2022 Gartner^® Market Guide for DataOps Tools appeared first on Unravel.

As most organizations that have started to run a lot of data applications and pipelines in the cloud have found out, it’s really easy for things to get really expensive, really fast. It’s not unusual to see monthly budget overruns of 40% or more, or for companies to have burned through their three-year data cloud budget by early in year two. Consequently, we’re seeing that cloud migration projects and modernization initiatives are stalling out. Plans to scale up usage of modern platforms (think: Databricks and Snowflake, Amazon EMR, Google BigQuery and Dataproc, Azure) are hitting a wall. 

Cloud bills are usually an organization’s biggest IT expense, and the sheer massive size of data workloads is driving most of the cloud bill. Many companies are feeling ambushed by their monthly cloud bills, and simply understanding where the money is going is a challenge—let alone figuring out where and how to rein in those costs and keeping them under control. Capacity/budget forecasting becomes a guessing game. There’s a general lack of individual accountability for waste/abuse of cloud resources. It feels like the Wild West, where everybody is spinning up instances left and right, without enough governance or control over how cloud resources are being used. 

Enter the emerging discipline of FinOps. Sometimes called cloud cost management or cloud optimization, FinOps is an evolving cloud financial management practice that, in the words of the FinOps Foundation, “enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions.”

The principles behind the FinOps lifecycle

It’s important to bear in mind that a FinOps approach isn’t just about slashing costs—although you almost invariably will wind up saving money. It’s about empowering data engineers and business teams to make better choices about their cloud usage and derive the most value from their modern data stack investments.

Controlling costs consists of three iterative phases along a FinOps lifecycle:

• Observability: Getting visibility into where the money is going, measuring what’s happening in your cloud environment, understanding what’s going on in a “workload-aware” context 
• Optimization: Seeing patterns emerge where you can eliminate waste, removing inefficiencies, actually making things better
• Governance: Going from reactive problem-solving to proactive problem-preventing, sustaining iterative improvements, automating guardrails, enabling self-service optimization

Each phase builds upon the former, to create a virtuous cycle of continuous improvement and empowerment for individual team members—regardless of expertise—to make better decisions about their cloud usage while still hitting their SLAs. In essence, this shifts the budget left, pulling accountability for controlling costs forward.

How Unravel puts FinOps lifecycle principles into practice

Unravel helps turn this conceptual FinOps framework into practical cost governance reality. You need four things to make this FinOps lifecycle work—all of which Unravel is uniquely able to provide:

1. You need to capture the right kind of details at a highly granular level from all the various systems in your data stack—horizontally and vertically—from the application down to infrastructure and everything in between. 
2. All this deep information needs to be correlated into a “workload-aware” business context—cost governance is not just an infrastructure issue, and you need to get a holistic understanding of how everything works together: apps, pipelines, users, data, as well as infrastructure resources.
3. You need to be able to automatically identify opportunities to optimize—oversized resources, inefficient code, data tiering—and then make it easy for engineers to implement those optimizations.
4. Go from reactive to proactive—not just perpetually scan the horizon for optimization opportunities and respond to them after the fact, but leverage AI to predict capacity needs accurately, implement automated governance guardrails to keep things under control, and even launch corrective actions automatically.

Observability: Understanding costs in context

The first step to controlling costs is to understand what’s happening in your cloud environment—who’s spending what, where, why. The key here is to measure everything with precision and in context. Cloud vendor billing consoles (and third-party cost management tools) give you only an aggregated view of total spend on various services (compute, storage, platform), and monthly cloud bills can be pretty opaque with hundreds of individual line items that, again, don’t go any deeper than service type. There’s nothing that tells you which applications are running, who’s submitting them, which datasets are actually touched, and other contextual details.

To counter such obscurity, many organizations employ some sort of cost-allocation tagging taxonomy to categorize and track resource usage. If you already have such a classification in place, Unravel can easily adopt it without having to reinvent the wheel. If you don’t, it’s simple to implement such an approach in Unravel.

Two things to consider when looking at tagging capabilities: 

• How deep and atomic is the tagging? 
• What’s the frequency?

Unravel lets you apply tags at a highly granular level: by department, team, workload, application, even down to the individual job (or sub-part of a job) or specific user. And it happens in real time—you don’t have to wait around for the end of the monthly billing cycle. 

These twin capabilities of capturing and correlating highly granular details and delivering real-time information are not merely nice-to-haves, but must-haves when it comes to the practical implementation of the FinOps lifecycle framework. They enable Unravel to:

• Track actual spend vs. budget. Know ahead of time whether workload, user, cluster, etc., usage is projected to be on target, is at risk, or has already gone over budget. Preemptively prevent budget overruns rather than getting hit with sticker shock at the end of the month.

Check out the 90-second Automated Budget Tracking interactive demo

•  Identify the big spenders. Know which projects, teams, applications, tables, queue, users, etc. are consuming the most cloud resources.

• Understand trends and patterns. Visualize how the cost of clusters changes over time. Understand seasonality-driven peaks and troughs—are you using a ton of resources only on Saturday night? Same time every day? When is there quiet time?—to identify opportunities for improvement.
  

• Implement precise chargebacks/showbacks. Automatically generate cost-allocation reports down to the penny. Pinpoint who’s spending how much, where, and why. Because Unravel captures all the deep, nitty-gritty details about what’s running across your data stack—horizontally and vertically—and correlates them all in a “workload-aware” business context, you can finally solve the problem of allocating costs of shared resources.
  

Check out the 90-second Chargeback Report interactive demo

• Forecast capacity with confidence. Run regularly scheduled reports that analyze historical usage and trends to predict future needs.
  

The observability phase is all about empowering data teams with visibility, precise allocation, real-time budgeting information, and accurate forecasting for cost governance. It provides a 360° view of what’s going on in your cloud environment and how much it’s costing you.

Optimization: Having AI do the heavy lifting

If the observability phase tells you what’s happening and why, the optimization phase is about taking that information to identify where you can eliminate waste, remove inefficiencies, or leverage different instance types that are less expensive without sacrificing SLAs.

In theory, that sounds pretty obvious and straightforward. In practice, it’s anything but. First of all, figuring out where there’s waste is no small task. In an enterprise that runs tens (or hundreds) of thousands of jobs every month, there are countless decisions—at the application, pipeline, and cluster level—to make about where, when, and how to run those jobs. And each individual decision carries a price tag. 

And the people making those decisions are not experts in allocating resources; their primary concern and responsibility is to make sure the job runs successfully and meets its SLA.  Even an enterprise-scale organization could probably count the number with such operational expertise on one hand. 

Most cost management solutions can only identify idle clusters. Deployed but no-longer-in-use resources are an easy target but represent only 10% of potential savings. The other 90% lie below the surface.

Digging into the weeds to identify waste and inefficiencies is a laborious, time-consuming effort that these already overburdened experts don’t really have time for.

AI is a must

Yet generating and sharing timely information about how and where to optimize so that individuals can assume ownership and accountability for cloud usage/cost is a bedrock principle of FinOps. This is where AI is mandatory, and only Unravel has the AI-powered insights and prescriptive recommendations to empower data teams to take self-service optimization action quickly and easily.

What Unravel does is take all the full-stack observability information and “throw some math at it”—machine learning and statistical algorithms—to understand application profiles and analyze what resources are actually required to run them vs. the resources that they’re currently consuming. 

This is where budgets most often go off the rails: overprovisioned (or underutilized) clusters and jobs due to instances and containers that are based on perceived need rather than on actual usage. What makes Unravel unique in the marketplace is that its AI not only continuously scours your data environment to identify exactly where you have allocated too many or oversized resources but gives you crisp, prescriptive recommendations on precisely how to right-size the resources in question.

Check out the 90-second AI Recommendations interactive demo

But unoptimized applications are not solely infrastructure issues. Sometimes it’s just bad code. Inefficient or problematic performance wastes money, too. We’ve seen how a single bad join on a multi-petabyte table kept a job running all weekend and wound up costing the company over $500,000. Unravel can prevent this from happening in the first place; the AI understands what your app is trying to do and can tell you this app submitted in this fashion is not efficient—pointing you to the specific line of code causing problems.

Check out the 90-second Code-Level Insights interactive demo

Every cloud provider has auto-scaling options. But what should you auto-scale, to what, and when? Because Unravel has all this comprehensive, granular data in “workload-aware” context, the AI understands trends and usage and access to help you predict and see the days of the week and times of day when auto-scaling is appropriate (or find better times to run jobs). Workload heatmaps based on actual usage make it easy to visualize when, where, and how to scale resources intelligently.

You can save 80-90% on storage costs through data tiering, moving less frequently used data to less expensive options. Most enterprises have petabytes of data, and they’re not using all of it all of the time. Unravel shows which datasets are (not) being used, applying cold/warm/hot labels based on age or usage, so you understand which ones haven’t been touched in months yet still sit on expensive storage.

The optimization phase is where you take action. This is the hardest part of the FinOps lifecycle to put into practice and where Unravel shines. The enormity and complexity of today’s data applications/pipelines comprise hundreds of thousands of places where waste, inefficiencies, or better alternatives exist. Rooting them out and then getting the insights to optimize require the two things no company has enough of: time and expertise. Unravel automatically identifies where you could do better—at the user, job, cluster level—and tells you the exact parameters to apply that would improve costs.

Governance: Going from reactive to proactive

Reducing costs reactively is good. Controlling them proactively is better. Finding and “fixing” waste and inefficiencies too often relies on the same kind of manual firefighting for FinOps that bogs down data engineers who need to troubleshoot failed or slow data applications. In fact, resolving cost issues usually relies on the same handful of scarce experts who have the know-how to resolve performance issues. In many respects, cost and performance are flip sides of the same coin—you’re looking at the same kind of granular details correlated in a holistic workload context, only from a slightly different angle. 

AI and automation are crucial. Data applications are simply too big, too complex, too dynamic for even the most skilled humans to manage by hand. What makes FinOps for data so difficult is that the thousands upon thousands of cost optimization opportunities are constantly recasting themselves. Unlike software applications, which are relatively static products, data applications are fluid and ever-changing. Data cloud cost optimization is not a straight line with a beginning and end, but rather a circular loop: the AI-powered insights used to gain contextual full-stack observability and actually implement cost optimization are harnessed to give everyone on the data team at-a-glance understanding of their costs, implement automated guardrails and governance policies, and enable non-expert engineers to make expert-level optimizations via self-service. After all, the best way to spend less time firefighting is to avoid there being a fire in the first place.

• With Unravel, you can set up customizable governance policy-based automated guardrails that set boundaries for any business dimension (looming budget overruns, jobs that exceed size/time/cost thresholds, etc.). Controlling particular users, apps, or business units from exceeding certain behaviors has a profound impact on reining in cloud spend.
  
• Proactive alerts can be triggered whenever a guardrail constraint is violated. Alerts can be sent to the individual user—or sent up the chain of command—to let them know their job will miss its SLA or cost too much money and they need to find less expensive options, rewrite it to be more efficient, reschedule it, etc. 
• Governance policy violations can even trigger preemptive corrective actions. Unravel can automatically take “circuit breaker” remediation to kill jobs or applications (even clusters), request configuration changes, etc., to put the brakes on rogue users, runaway jobs, overprovisioned resources, and the like.
  

The governance phase of the FinOps lifecycle deals with getting ahead of cost issues before the fact rather than afterwards. Establishing control in a highly distributed environment is a challenge for any discipline, but Unravel empowers individual users with self-service capabilities that enables them to take individual accountability for their cloud usage, automatically alerts on potential budget-busting, and can even take preemptive corrective action without any human involvement. It’s not quite self-healing, but it’s the next-best thing—the true spirit of AIOps.

The post How Unravel Helps FinOps Control Data Cloud Costs appeared first on Unravel.

DBTA recently hosted a roundtable webinar with four industry experts on “Unlocking the Value of Cloud Data and Analytics.” Moderated by Stephen Faig, Research Director, Unisphere Research and DBTA, the webinar featured presentations from Progress, Ahana, Reltio, and Unravel.

You can see the full 1-hour webinar “Unlocking the Value of Cloud Data and Analytics” below.

Here’s a quick recap of what each presentation covered.

Todd Wright, Global Product Marketing Manager at Progress, in his talk “More Data Makes for Better Analytics” showed how Progress DataDirect connectors let users get to their data from more sources securely without adding a complex software stack in between. He quoted former Google research director Peter Norvig (now fellow at the Stanford Institute for Human-Centered Artificial Intelligence) about how more data beats clever algorithms: “Simple models and a lot of data trump more elaborate models based on less data.” He then outlined how Progress DataDirect and its Hybrid Data Pipeline platform uses standard-based connectors to expand connectivity options of BI and analytics tools, access all your data using a single connector, and make on-prem data available to cloud BI and analytics tools without exposing ports or implementing costly and complex VPN tunnels. He also addressed how to secure all data sources behind your corporate authentication/identity to mitigate the risk of exposing sensitive private information (e.g., which tables and columns to expose to which people) and keep tabs on data usage for auditing and compliance.

Rachel Pedreschi, VP of Technical Services at Ahana, presented “4 Easy Tricks to Save Big Money on Big Data in the Cloud.” She first simplified the cloud data warehouse into its component parts (data on disk + some kind of metadata + a query layer + system that authenticates users and allows them to do stuff). Then she broke it down to see where you could save some money. Starting at the bottom, the storage layer, she said data lakes are a more cost-effective way of providing data to users throughout the organization. At the metadata level, Hive Metastore or AWS Glue are less expensive options. For authentication, she mentioned Apache Ranger or AWS Lake Formation. But what about SQL? For that she has Presto, an open source project that came out of Facebook as a successor to Hive. Presto is a massively scalable distributed in-memory system that allows you write queries not just against data lake files but also other databases as well. She calls this collectively the Open SQL Data Lakehouse. Ahana is an AWS-based service that gets you up and running with this Open Data Lakehouse in 30 minutes. 

Mike Frasca, Field Chief Field Technology Officer at Reltio, discussed the value and benefits of a modern master data management platform. He echoed previous presenters’ points about how the volume, velocity, and variety of data have become almost overwhelming, especially given how highly siloed and fragmented data is today. Data teams spend most of their time getting the data ready for insights—data consolidation, aggregation, and cleansing; synchronizing and standardizing data, ensuring data quality, timeliness, and accuracy, etc.—rather than actually delivering insights from analytics. He outlined the critical functions a master data management (MDM) platform should provide to deliver precision data. Entity management automatically unifies data into a dynamic enterprise source of truth, including context-aware master records. Data quality management continuously validates, cleans, and standardizes the data via custom validation rules. Data integration receives input from and distributes mastered data to any application or data store in real time and at high volumes. He emphasized that what characterizes a “modern” MDM is the ability to access data in real time—so you’re not making decisions based on data that’s a week or a month old. Cloud MDM extends the capabilities to relationship management, data governance, and reference data management. He wound up his presentation with compelling real-life examples of customer efficiency gains and effectiveness, including Forrester TEI (total economic impact) results on ROI.

Chris Santaigo, VP of Solutions Engineering at Unravel, presented how AI-enabled optimization and automatic insights help unlock the value of cloud data. He noted how nowadays every company is a data company. If they’re not leveraging data analytics to create strategic business advantages, they’re falling behind. But he illustrated how the complexity of the modern data stack is slowing companies down. He broke down the challenges into three C’s: cost, where enterprises are experiencing significant budget overruns; resource constraints—not infrastructure resources, but human resources—the talent gap and mismatch between supply and demand for expertise; and complexity of the stack, where 87% of data science projects never make it into production because it’s so challenging to implement them.

But it all comes down to people—all the different roles on data teams. Analysts, data scientists, engineers on the application side; architects, operations teams, FinOps, and business stakeholders on the operations side. Everybody needs to be working off a “single source of truth” to break down silos, enable collaboration, eliminate finger-pointing, and empower more self-service. Software teams have APM to untangle this complexity—for web apps. But data apps are a totally different beast. You need observability designed specifically for data teams. You could try to stitch together the details you need for performance, cost, data quality from a smorgasbord of point tools or solutions that do some of what you need (but not all), but that’s time-consuming and misses the holistic view of how everything works together so necessary to connect the dots when you’re looking to troubleshoot faster, control spiraling cloud costs, automate AI-driven optimization (for performance and cost), or migrate to the cloud on budget and on time. That’s exactly where Unravel comes in.

Check out the full webinar, including attendee Q&A here!

The post 3-Minute Recap: Unlocking the Value of Cloud Data and Analytics appeared first on Unravel.

This blog was originally published by Unravel CEO Kunal Agarwal on LinkedIn in September 2022.

I was chatting with Sanjeev Mohan, Principal and Founder of SanjMo Consulting and former Research Vice President at Gartner, about how the emergence of DataOps is changing people’s idea of what “data observability” means. Not in any semantic sense or a definitional war of words, but in terms of what data teams need to stay on top of an increasingly complex modern data stack. While much ink has been spilled over how data observability is much more than just data profiling and quality monitoring, until only very recently the term has pretty much been restricted to mean observing the condition of the data itself. 

But now DataOps teams are thinking about data observability more comprehensively as embracing other “flavors” of observability like application and pipeline performance, operational observability into how the entire platform or system is running end-to-end, and business observability aspects such as ROI and—most significantly—FinOps insights to govern and control escalating cloud costs.

That’s what we at Unravel call DataOps observability.

Data teams are getting bogged down

Data teams are struggling, overwhelmed by the increased volume, velocity, variety, and complexity of today’s data workloads. These data applications are simultaneously becoming ever more difficult to manage and ever more business-critical. And as more workloads migrate to the cloud, team leaders are finding that costs are getting out of control—often leading to migration initiatives stalling out completely because of budget overruns.

The way data teams are doing things today isn’t working.

Data engineers and operations teams spend way too much time firefighting “by hand.” Something like 70-75% of their time is spent tracking down and resolving problems through manual detective work and a lot of trial and error. And with 20x more people creating data applications than fixing them when something goes wrong, the backlog of trouble tickets gets longer, SLAs get missed, friction among teams creeps in, and the finger-pointing and blame game begins.

This less-than-ideal situation is a natural consequence of inherent process bottlenecks and working in silos. There are only a handful of experts who can untangle the wires to figure out what’s going on, so invariably problems get thrown “over the wall” to them. Self-service remediation and optimization is just a pipe dream. Different team members each use their own point tools, seeing only part of the overall picture, and everybody gets a different answer to the same problem. Communication and collaboration among the team breaks down, and you’re left operating in a Tower of Babel.

Accelerating next-gen DataOps observability

These problems aren’t new. DataOps teams are facing some of the same general challenges as their DevOps counterparts did a decade ago. Just as DevOps united the practice of software development and operations and transformed the application lifecycle, today’s data teams need the same observability but tailored to their unique needs. And while application performance management (APM) vendors have done a good job of collecting, extracting, and correlating details into a single pane of glass for web applications, they’re designed for web applications and give data teams only a fraction of what they need.

System point tools and cloud provider tools all provide some of the information data teams need, but not all. Most of this information is hidden in plain sight—it just hasn’t been extracted, correlated, and analyzed by a single system designed specifically for data teams.

That’s where Unravel comes in.

Data teams need what Unravel delivers—observability designed to show data application/pipeline performance, cost, and quality coupled with precise, prescriptive fixes that will allow you to quickly and efficiently solve the problem and get on to the real business of analyzing data. Our AI-powered solution helps enterprises realize greater return on their investment in the modern data stack by delivering faster troubleshooting, better performance to meet service level agreements, self-service features that allow applications to get out of development and into production faster and more reliably, and reduced cloud costs.

I’m excited, therefore, to share that earlier this week, we closed a $50 million Series D round of funding that will allow us to take DataOps observability to the next level and extend the Unravel platform to help connect the dots from every system in the modern data stack—within and across some of the most popular data ecosystems. 

Unlocking the door to success

By empowering data teams to spend more time on innovation and less time firefighting, Unravel helps data teams take a page out of their software counterparts’ playbook and tackle their problems with a solution that goes beyond observability—to not just show you what’s going on and why, but actually tell you exactly what to do about it. It’s time for true DataOps observability.

To learn more about how Unravel Data is helping data teams tackle some of today’s most complex modern data stack challenges, visit: www.unraveldata.com. 

The post Get Ready for the Next Generation of DataOps Observability appeared first on Unravel.

This year’s Big Data LDN (London) was huge. Over 150 exhibitors, 300 expert speakers across 12 technical and business-led conference theaters. It was like being the proverbial kid in a candy store, and I had to make some tough decisions on which presentations to attend (and which I’d miss out on).

One that looked particularly promising was “The Great Data Debate 2022,” panel discussion, hosted by industry analyst and conference chair Mike Ferguson with panelists Benoit Dageville, Co-founder and President of Products at Snowflake, Shinji Kim, Select Star Founder and CEO, Chris Gladwin, Ocient CEO, and Tomer Shiran, Co-founder and Chief Product Officer of Dremio. You can watch the 1 hour Great Data Debate 2022 recording below.

The panel covered a lot of ground, everything from the rise of component-based development, how the software development approach has gate-crashed the data and analytics world, the challenges around integrating all the new tools, best-of-breed vs. single platform, the future of data mesh, metadata standards, data security and governance, and much more.

Sometimes the panelists agreed with each other, sometimes not, but the discussion was always lively. The parts I found most interesting revolved around migrating to the cloud and controlling costs once there.

Moderator Mike Ferguson opened up the debate by asking the panelists how the current economic climate has changed companies’ approach—whether they’re accelerating their move to the cloud, focusing more on cost reduction or customer retention, etc.

All the panelists agreed that more companies are increasingly migrating workloads to the cloud. Said Benoit Dageville: “We’re seeing an acceleration to moving to the cloud, both because of cost—you can really lower your cost—and because you can do much more in the cloud.” 

Chris Gladwin added that the biggest challenge among hyperscale companies is that “they want to grow faster and be more efficient.” Shinji Kim echoed this sentiment, though from a different viewpoint, saying that many organisations are looking at how they want to structure the team—focusing more effort on automation or tooling to make everyone more productive in their own role. Tomer Shiran made the point that “a lot of customers now are leveraging data to either save money or increase their revenue. And there’s more focus on people asking if the path of spending they’re on with current data infrastructure is sustainable for the future.”

We at Unravel are also seeing an increased focus on making data teams more productive and on leveraging automation to break down silos, promote more collaboration, reduce toilsome troubleshooting, and accelerate the DataOps lifecycle. But piggy-backing on Tomer’s point: While the numbers certainly bear out that more workloads are indeed moving to the cloud, we are seeing that among more mature data-driven organisations—those that already have 2-3 years of experience running data workloads in the cloud under their belt—migration initiatives are “hitting a wall” and stalling out. Cloud costs are spiraling out of control, and companies find themselves burning through their budgets with little visibility into where the spend is going or ability to govern expenses.

As Mike put it: “As an analyst, I get to talk to CFOs and a lot of them have no idea what the invoice is going to be like at the end of the month. So the question really is, how does a CFO get control over this whole data and analytics ecosystem?”

Chris was first to answer. “In the hyperscale segment, there are a lot of things that are different. Every customer is the size of a cloud, every application is the size of a cloud. Our customers have not been buying on a per usage basis—if you’re hammering away all day on a cluster of clusters, you want a price based on the core. They want to know in advance what it’s going to cost so they can plan for it. They don’t want to be disincented from using the platform more and more because it’ll cost more and more.”

Benoit offered a different take: “Every organisation wants to become really data-driven, and it pushes a lot of computation to that data. I believe the cloud and its elasticity is the most cost-effective way to do that. And you can do much more at lower costs. We have to help the CFO and the organisation at large understand where the money is spent to really control [costs], to define budget, have a way to charge back to the different business units, and be very transparent to where the cost is going. So you have to have what we call cost governance. And we tell all our customers when they start [using Snowflake] that they have to put in place guardrails. It’s not a free lunch.”

Added Shinji: “It’s more important than ever to track usage and monitor how things are actually going, not just as a one-time cost reduction initiative but something that actually runs continuously.”

Benot summed it up by saying, “Providing the data, the monitoring, the governance of costs is a very big focus for all of us [on the panel], at different levels.”

It’s interesting to hear leaders from modern data stack vendors as diverse as Snowflake, Select Star, and Dremio emphasise the need for automated cost governance guardrails. Because nobody does cloud cost governance for data applications and pipelines better than Unravel.

Check out the full The Great Data Debate 2022 panel discussion.

The post Reflections on “The Great Data Debate 2022” from Big Data London appeared first on Unravel.

Chief Data & Analytics Officer UK (CDAO UK) is the United Kingdom’s premier event for senior data and analytics executives. The three-day event, with more than 200 attendees and 50+ industry-leading speakers, was packed with case studies, thought leadership, and practical advice around data culture, data quality and governance, building a data workforce, data strategy, metadata management, AI/MLOps, self-service strategies, and more.

Catherine King of Corinium Global Intelligence interviews Chris Santiago, Unravel Data VP of Solutions Engineering

Chris Santiago, Unravel VP of Solutions Engineering, sat down with Catherine King from event host Corinium Global Intelligence to discuss what’s top-of-mind for data and analytics leaders.

Here are the highlights from their 1-on-1 interview.

Catherine: What are you hearing in the market at the moment? What are people coming up to you and having a chat with you about today?

Chris: I think that, big picture-wise, there are a lot of things being talked about that are very insightful. But some of the stuff that hasn’t really been talked about are the things that people don’t want to talk about. They have this grand vision, but how are they going to get there? How are they going to execute? How are they going to have the processes in place, the right people hired—that sort of thing—to take advantage of data and all the great technology that’s out there and actually execute on the vision?

Catherine: I think you’ve hit the nail on the head there. I think the technology piece, which was so prevalent a few years ago—do we have the right tech, the right tools, to go out and do these things?—that’s almost been ticked off and done. Now it’s actually, do you have the processes in place to achieve it. So from your perspective, what would you like to see businesses do differently?

Chris: The technology has made leaps and bounds over the last few years. If you think about big data people who came from the Hadoop world, one of the challenges of that stack was that it did require expertise. Companies back then struggled to get the ROI on performance, on getting some sort of business value at the end of the day. Fast-forward to today, especially with the advancements in the cloud, they have solved a lot of the challenges. Ease of use? I can just literally just log in, click a button, and have an environment. Storage is cheaper. A lot of the problems back then have been solved with today’s technology.

I do think that the one thing that hasn’t been 100% solved yet, though, is the people, the skills. Right now, the things technology is not necessarily addressing directly are: Do we have the right skill set? Do we have the right people? As more folks are using these newer technologies, the gap in skills to do things the right way and best practices are not being directly addressed. The way that most customers are handling it right now is that they’ll bring in the experts, consultants like Accenture, Avanade, Deloitte, etc.

In order to achieve the true ROI in these data stacks, it’s adjusting the people-problem.

Catherine: What do you see coming in the next 12 months?

Chris: I think that if we look at the industry as a whole, a lot of the technology is still considered new(ish). You have a lot of folks who are still using DevOps tools. So they’re trying to work with observability tools that are focused on the issues that aren’t necessarily what data teams want. So I think in the next six to twelve months we’re going to have a proliferation of observability trying to solve this problem specifically for data teams. Because they’re different problems [than software application issues]. You can’t use the same [APM] tooling for folks who are running Databricks or Snowflake. There are going to be different problems, different challenges.

Obviously, Unravel is in that space now, but I do think the industry will recognize more and more that this is actually not just a small problem, but a major problem. Companies are starting to realize that [observability designed for data teams] is not a nice-to-have anymore, it’s a must-have—and needs to be addressed right now.

Everybody has a vision. Everybody has an idea of what they want to do with data, whether it’s having that strategic business advantage or getting insights that they didn’t know about—everybody trying to do really cool stuff—but everybody always seems to forget about how to execute. There’s lots of interesting talks about how we’re going to measure things, what KPIs we’re going to be tracking, what methodologies we should have in place—all great stuff—but if you truly want to be successful, it’s all about execution and [doing] the stuff people don’t want to talk about. That’s what will set up companies to be successful or not with their data initiatives, getting into the weeds and solving these hard challenges.

Check out the full interview with Chris from CDAO UK

The post The Data Challenge Nobody’s Talking About: An Interview from CDAO UK appeared first on Unravel.

As organizations invest ever more heavily in modernizing their data stacks, data teams—the people who actually deliver the value of data to the business—are finding it increasingly difficult to manage the performance, cost, and quality of these complex systems.

Data teams today find themselves in much the same boat as software teams were 10+ years ago. Software teams have dug themselves out the hole with DevOps best practices and tools—chief among them full-stack observability.

But observability for data applications is a completely different animal from observability for web applications. Effective DataOps demands observability that’s designed specifically for the different challenges facing different data team members throughout the DataOps lifecycle.

That’s data observability for DataOps.

In this white paper, you’ll learn:

• What is data observability for DataOps?
• What data teams can learn from DevOps
• Why observability designed for web apps is insufficient for data apps
• What data teams need from data observability for DataOps
• The 5 capabilities data observability for DataOps

Download the white paper here.

The post Data Observability: The Missing Link for Data Teams appeared first on Unravel.

Join us at an event near you or attend virtually to discover our DataOps observability platform, discuss your challenges with one of our DataOps experts, go under the hood to check out platform capabilities, and see what your peers have been able to accomplish with Unravel.

September 21-22: Big Data LDN (London) 

Big Data LDN is the UK’s leading free-to-attend data & analytics conference and exhibition, hosting leading data and analytics experts, ready to arm you with the tools to deliver your most effective data-driven strategy. Stop by the Unravel booth (stand #724) to see how Unravel is observability designed specifically for the unique needs of today’s data teams. 

Register for Big Data LDN here

And be sure to stop by the Unravel booth at 5PM on Day 1 for the Data Drinks Happy Hour for drinks and snacks (while supplies last!)

October 5-6: AI & Big Data Expo – North America (Santa Clara) 

The world’s leading AI & Big Data event returns to Santa Clara as a hybrid in-person and virtual event, with more than 5,000 attendees expected to join from across the globe. The expo will showcase the most cutting-edge technologies from 250+ speakers sharing their unparalleled industry knowledge and real-life experiences, in the forms of solo presentations, expert panel discussions and in-depth fireside chats.

Register for AI & Big Data Expo here

And don’t miss Unravel Co-Founder and CEO Kunal Agarwal’s feature presentation on the different challenges facing different AI & Big Data team members and how multidimensional observability (performance, cost, quality) designed specifically for the modern data stack can help.

October 10-12: Chief Data & Analytics Officers (CDAO) Fall (Boston)

The premier in-person gathering for data & analytics leaders in North America, CDAO Fall offers focus tracks on data infrastructure, data governance, data protection & privacy; analytics, insights, and business intelligence; and data science, artificial intelligence, and machine learning. Exclusive industry summit days for data and analytics professionals in Financial Services, Insurance, Healthcare, and Retail/CPG.

Register for CDAO Fall here

October 14: DataOps Observability Conf India 2022 (Bengaluru)

India’s first DataOps observability conference, this event brings together data professionals to collaborate and discuss best practices and trends in the modern data stack, analytics, AI, and observability.

Join leading DataOps observability experts to:

• Understand what DataOps is and why it’s important
• Learn why DataOps observability has become a mission-critical need in the modern data stack
• Discover how AI is transforming DataOps and observability

Register for DataOps Observability Conf India 2022 here

November 1-3: ODSC West (San Francisco)

The Open Data Science Conference (ODSC) is essential for anyone who wants to connect to the data science community and contribute to the open source applications it uses every day. A hybrid in-person/virtual event, ODSC West features 250 speakers, with 300 hours of content, including keynote presentations, breakout talk sessions, hands-on tutorials and workshops, partner demos, and more. 

Register for ODSC West here

Sneak peek into what you’ll see from Unravel 

Want a taste of what we’ll be showing? Check out our 2-minute guided-tour interactive demos of our unique capabilities. Explore features like:

• Full-stack data pipeline observability
• Automated root cause analysis, at both the job and pipeline level
• Job-level AI optimization recommendations
• Automated cloud cluster optimizations
• Pinpoint-precise chargeback reports
• Automated budget tracking
• Proactive cost governance alerts
• Cloud migration workload fit reports
• Automated cluster discovery

Explore all our interactive guided tours here.

The post Unravel Goes on the Road at These Upcoming Events appeared first on Unravel.