Building ETL Pipelines With Generative AI

<h2>Summary</h2> Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">dataengineeringpodcast.com/datafold</a></li> <li>You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to <a href="https://www.dataengineeringpodcast.com/materialize" rel="nofollow">dataengineeringpodcast.com/materialize</a> today to get 2 weeks free!</li> <li>As more people start using AI for projects, two things are clear: It’s a rapidly advancing field, but it’s tough to navigate. How can you get the best results for your use case? Instead of being subjected to a bunch of buzzword bingo, hear directly from pioneers in the developer and data science space on how they use graph tech to build AI-powered apps. . Attend the dev and ML talks at NODES 2023, a free online conference on October 26 featuring some of the brightest minds in tech. Check out the agenda and register at <a href="https://neo4j.com/nodes" rel="nofollow">Neo4j.com/NODES</a>.</li> <li>Your host is Tobias Macey and today I'm interviewing Jay Mishra about the applications for generative AI in the ETL process</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>What are the different aspects/types of ETL that you are seeing generative AI applied to? <ul> <li>What kind of impact are you seeing in terms of time spent/quality of output/etc.?</li> </ul></li> <li>What kinds of projects are most likely to benefit from the application of generative AI?</li> <li>Can you describe what a typical workflow of using AI to build ETL workflows looks like? <ul> <li>What are some of the types of errors that you are likely to experience from the AI?</li> <li>Once the pipeline is defined, what does the ongoing maintenance look like?</li> <li>Is the AI required to operate within the pipeline in perpetuity?</li> </ul></li> <li>For individuals/teams/organizations who are experimenting with AI in their data engineering workflows, what are the concerns/questions that they are trying to address?</li> <li>What are the most interesting, innovative, or unexpected ways that you have seen generative AI used in ETL workflows?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on ETL and generative AI?</li> <li>When is AI the wrong choice for ETL applications?</li> <li>What are your predictions for future applications of AI in ETL and other data engineering practices?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/jaymishra/" rel="nofollow">LinkedIn</a></li> <li><a href="https://twitter.com/MishraJay" rel="nofollow">@MishraJay</a> on Twitter</li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.astera.com/" rel="nofollow">Astera</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_vault_modeling" rel="nofollow">Data Vault</a></li> <li><a href="https://en.wikipedia.org/wiki/Star_schema" rel="nofollow">Star Schema</a></li> <li><a href="https://openai.com/" rel="nofollow">OpenAI</a></li> <li><a href="https://en.wikipedia.org/wiki/Generative_pre-trained_transformer" rel="nofollow">GPT == Generative Pre-trained Transformer</a></li> <li><a href="https://en.wikipedia.org/wiki/Record_linkage" rel="nofollow">Entity Resolution</a></li> <li><a href="https://en.wikipedia.org/wiki/LLaMA" rel="nofollow">LLAMA</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow">Materialize</a>: <a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow"> You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing. Go to <u>[materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)</u> today and get 2 weeks free!</a></li><li><a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">Datafold</a>: <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow"> This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!</a></li><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li><li><a href="https://neo4j.com/nodes" rel="nofollow">Neo4J</a>: <a href="https://neo4j.com/nodes" rel="nofollow"> NODES 2023 is a free online conference focused on graph-driven innovations with content for all skill levels. Its 24 hours are packed with 90 interactive technical sessions from top developers and data scientists across the world covering a broad range of topics and use cases. The event tracks: - Intelligent Applications: APIs, Libraries, and Frameworks – Tools and best practices for creating graph-powered applications and APIs with any software stack and programming language, including Java, Python, and JavaScript - Machine Learning and AI – How graph technology provides context for your data and enhances the accuracy of your AI and ML projects (e.g.: graph neural networks, responsible AI) - Visualization: Tools, Techniques, and Best Practices – Techniques and tools for exploring hidden and unknown patterns in your data and presenting complex relationships (knowledge graphs, ethical data practices, and data representation) Don’t miss your chance to hear about the latest graph-powered implementations and best practices for free on October 26 at NODES 2023. Go to <u>[Neo4j.com/NODES](https://Neo4j.com/NODES)</u> today to see the full agenda and register!</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

00:51:36

Powering Vector Search With Real Time And Incremental Vector Indexes

<h2>Summary</h2> The rapid growth of machine learning, especially large language models, have led to a commensurate growth in the need to store and compare vectors. In this episode Louis Brandy discusses the applications for vector search capabilities both in and outside of AI, as well as the challenges of maintaining real-time indexes of vector data.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">dataengineeringpodcast.com/datafold</a></li> <li>You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to <a href="https://www.dataengineeringpodcast.com/materialize" rel="nofollow">dataengineeringpodcast.com/materialize</a> today to get 2 weeks free!</li> <li>If you’re a data person, you probably have to jump between different tools to run queries, build visualizations, write Python, and send around a lot of spreadsheets and CSV files. Hex brings everything together. Its powerful notebook UI lets you analyze data in SQL, Python, or no-code, in any combination, and work together with live multiplayer and version control. And now, Hex’s magical AI tools can generate queries and code, create visualizations, and even kickstart a whole analysis for you – all from natural language prompts. It’s like having an analytics co-pilot built right into where you’re already doing your work. Then, when you’re ready to share, you can use Hex’s drag-and-drop app builder to configure beautiful reports or dashboards that anyone can use. Join the hundreds of data teams like Notion, AllTrails, Loom, Mixpanel and Algolia using Hex every day to make their work more impactful. Sign up today at <a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow">dataengineeringpodcast.com/hex</a> to get a 30-day free trial of the Hex Team plan!</li> <li>Your host is Tobias Macey and today I'm interviewing Louis Brandy about building vector indexes in real-time for analytics and AI applications</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you describe what vector search is and how it differs from other search technologies? <ul> <li>What are the technical challenges related to providing vector search?</li> <li>What are the applications for vector search that merit the added complexity?</li> </ul></li> <li>Vector databases have been gaining a lot of attention recently with the proliferation of LLM applications. Is a dedicated database technology required to support vector indexes/vector search queries? <ul> <li>What are the use cases for native vector data types that are separate from AI?</li> </ul></li> <li>With the increasing usage of vectors for data and AI/ML applications, who do you typically see as the owner of that problem space? (e.g. data engineers, ML engineers, data scientists, etc.)</li> <li>For teams who are investing in vector search, what are the architectural considerations that they need to be aware of? <ul> <li>How does it impact the data pipeline strategies/topologies used?</li> </ul></li> <li>What are the complexities that need to be addressed when updating vector data in a real-time/streaming fashion? <ul> <li>How does that influence the client strategies that are querying that data?</li> </ul></li> <li>What are the most interesting, innovative, or unexpected ways that you have seen vector search used?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on vector search applications?</li> <li>When is vector search the wrong choice?</li> <li>What do you see as future potential applications for vector indexes/vector search?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/lbrandy/" rel="nofollow">LinkedIn</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://rockset.com/" rel="nofollow">Rockset</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/rockset-serverless-analytics-episode-101/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.datastax.com/guides/what-is-a-vector-index" rel="nofollow">Vector Index</a></li> <li><a href="https://www.datastax.com/guides/what-is-vector-search" rel="nofollow">Vector Search</a> <ul> <li><a href="https://rockset.com/videos/vector-search-architecture/" rel="nofollow">Rockset Implementation Explanation</a></li> </ul></li> <li><a href="https://en.wikipedia.org/wiki/Vector_space" rel="nofollow">Vector Space</a></li> <li><a href="https://en.wikipedia.org/wiki/Euclidean_distance" rel="nofollow">Euclidean Distance</a></li> <li><a href="https://en.wikipedia.org/wiki/Online_analytical_processing" rel="nofollow">OLAP == Online Analytical Processing</a></li> <li><a href="https://en.wikipedia.org/wiki/Online_transaction_processing" rel="nofollow">OLTP == Online Transaction Processing</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li><li><a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow">Materialize</a>: <a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow"> You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing. Go to <u>[materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)</u> today and get 2 weeks free!</a></li><li><a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow">Hex</a>: <a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow"> Hex is a collaborative workspace for data science and analytics. A single place for teams to explore, transform, and visualize data into beautiful interactive reports. Use SQL, Python, R, no-code and AI to find and share insights across your organization. Empower everyone in an organization to make an impact with data. Sign up today at dataengineeringpodcast.com/hex to get a 30-day free trial of the Hex Team plan!</a></li><li><a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">Datafold</a>: <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow"> This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

00:59:16

Building Linked Data Products With JSON-LD

<h2>Summary</h2> A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">dataengineeringpodcast.com/datafold</a></li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to <a href="https://www.dataengineeringpodcast.com/materialize" rel="nofollow">dataengineeringpodcast.com/materialize</a> today to get 2 weeks free!</li> <li>If you’re a data person, you probably have to jump between different tools to run queries, build visualizations, write Python, and send around a lot of spreadsheets and CSV files. Hex brings everything together. Its powerful notebook UI lets you analyze data in SQL, Python, or no-code, in any combination, and work together with live multiplayer and version control. And now, Hex’s magical AI tools can generate queries and code, create visualizations, and even kickstart a whole analysis for you – all from natural language prompts. It’s like having an analytics co-pilot built right into where you’re already doing your work. Then, when you’re ready to share, you can use Hex’s drag-and-drop app builder to configure beautiful reports or dashboards that anyone can use. Join the hundreds of data teams like Notion, AllTrails, Loom, Mixpanel and Algolia using Hex every day to make their work more impactful. Sign up today at <a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow">dataengineeringpodcast.com/hex</a> to get a 30-day free trial of the Hex Team plan!</li> <li>Your host is Tobias Macey and today I'm interviewing Brian Platz about using JSON-LD for building linked-data products</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you describe what the term "linked data product" means and some examples of when you might build one? <ul> <li>What is the overlap between knowledge graphs and "linked data products"?</li> </ul></li> <li>What is JSON-LD? <ul> <li>What are the domains in which it is typically used?</li> <li>How does it assist in developing linked data products?</li> </ul></li> <li>what are the characteristics that distinguish a knowledge graph from </li> <li>What are the layers/stages of applications and data that can/should incorporate JSON-LD as the representation for records and events? <ul> <li>What is the level of native support/compatibiliity that you see for JSON-LD in data systems?</li> </ul></li> <li>What are the modeling exercises that are necessary to ensure useful and appropriate linkages of different records within and between products and organizations?</li> <li>Can you describe the workflow for building autonomous linkages across data assets that are modelled as JSON-LD?</li> <li>What are the most interesting, innovative, or unexpected ways that you have seen JSON-LD used for data workflows?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on linked data products?</li> <li>When is JSON-LD the wrong choice?</li> <li>What are the future directions that you would like to see for JSON-LD and linked data in the data ecosystem?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/brianplatz/" rel="nofollow">LinkedIn</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://flur.ee/" rel="nofollow">Fluree</a></li> <li><a href="https://json-ld.org/" rel="nofollow">JSON-LD</a></li> <li><a href="https://en.wikipedia.org/wiki/Knowledge_graph" rel="nofollow">Knowledge Graph</a></li> <li><a href="https://en.wikipedia.org/wiki/Adjacency_list" rel="nofollow">Adjacency List</a></li> <li><a href="https://www.w3.org/RDF/" rel="nofollow">RDF == Resource Description Framework</a></li> <li><a href="https://en.wikipedia.org/wiki/Semantic_Web" rel="nofollow">Semantic Web</a></li> <li><a href="https://ogp.me/" rel="nofollow">Open Graph</a></li> <li><a href="https://schema.org/" rel="nofollow">Schema.org</a></li> <li><a href="https://en.wikipedia.org/wiki/Semantic_triple" rel="nofollow">RDF Triple</a></li> <li><a href="https://www.fda.gov/industry/fda-data-standards-advisory-board/identification-medicinal-products-idmp" rel="nofollow">IDMP == Identification of Medicinal Products</a></li> <li><a href="https://spec.edmcouncil.org/fibo/" rel="nofollow">FIBO == Financial Industry Business Ontology</a></li> <li><a href="https://www.w3.org/OWL/" rel="nofollow">OWL Standard</a></li> <li><a href="https://en.wikipedia.org/wiki/NP-hardness" rel="nofollow">NP-Hard</a></li> <li><a href="https://en.wikipedia.org/wiki/Forward_chaining" rel="nofollow">Forward-Chaining Rules</a></li> <li><a href="https://www.w3.org/TR/shacl/" rel="nofollow">SHACL == Shapes Constraint Language)</a></li> <li><a href="https://en.wikipedia.org/wiki/Zero-knowledge_proof" rel="nofollow">Zero Knowledge Cryptography</a></li> <li><a href="https://www.w3.org/TR/turtle/" rel="nofollow">Turtle Serialization</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow">Materialize</a>: <a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow"> You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing. Go to <u>[materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)</u> today and get 2 weeks free!</a></li><li><a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow">Hex</a>: <a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow"> Hex is a collaborative workspace for data science and analytics. A single place for teams to explore, transform, and visualize data into beautiful interactive reports. Use SQL, Python, R, no-code and AI to find and share insights across your organization. Empower everyone in an organization to make an impact with data. Sign up today at dataengineeringpodcast.com/hex to get a 30-day free trial of the Hex Team plan!</a></li><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li><li><a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">Datafold</a>: <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow"> This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

01:02:15

An Overview Of The Sate Of Data Orchestration In An Increasingly Complex Data Ecosystem

<h2>Summary</h2> Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">dataengineeringpodcast.com/datafold</a></li> <li>You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to <a href="https://www.dataengineeringpodcast.com/materialize" rel="nofollow">dataengineeringpodcast.com/materialize</a> today to get 2 weeks free!</li> <li>Your host is Tobias Macey and today I'm welcoming back Nick Schrock to talk about the state of the ecosystem for data orchestration</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by defining what data orchestration is and how it differs from other types of orchestration systems? (e.g. container orchestration, generalized workflow orchestration, etc.)</li> <li>What are the misconceptions about the applications of/need for/cost to implement data orchestration? <ul> <li>How do those challenges of customer education change across roles/personas?</li> </ul></li> <li>Because of the multi-faceted nature of data in an organization, how does that influence the capabilities and interfaces that are needed in an orchestration engine?</li> <li>You have been working on Dagster for five years now. How have the requirements/adoption/application for orchestrators changed in that time?</li> <li>One of the challenges for any orchestration engine is to balance the need for robust and extensible core capabilities with a rich suite of integrations to the broader data ecosystem. What are the factors that you have seen make the most influence in driving adoption of a given engine?</li> <li>What are the most interesting, innovative, or unexpected ways that you have seen data orchestration implemented and/or used?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on data orchestration?</li> <li>When is a data orchestrator the wrong choice?</li> <li>What do you have planned for the future of orchestration with Dagster?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://twitter.com/schrockn" rel="nofollow">@schrockn</a> on Twitter</li> <li><a href="https://www.linkedin.com/in/schrockn" rel="nofollow">LinkedIn</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://dagster.io/" rel="nofollow">Dagster</a></li> <li><a href="https://graphql.org/" rel="nofollow">GraphQL</a></li> <li><a href="https://kubernetes.io/" rel="nofollow">K8s == Kubernetes</a></li> <li><a href="https://airbyte.com/" rel="nofollow">Airbyte</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/airbyte-open-source-data-integration-episode-173/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://hightouch.com/" rel="nofollow">Hightouch</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/hightouch-customer-data-warehouse-episode-168/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://airflow.apache.org/" rel="nofollow">Airflow</a></li> <li><a href="https://www.prefect.io" rel="nofollow">Prefect</a></li> <li><a href="https://flyte.org/" rel="nofollow">Flyte</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/flyte-data-orchestration-machine-learning-episode-291/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.getdbt.com/" rel="nofollow">dbt</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/dbt-data-analytics-episode-81/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph" rel="nofollow">DAG == Directed Acyclic Graph</a></li> <li><a href="https://temporal.io/" rel="nofollow">Temporal</a></li> <li><a href="https://docs.dagster.io/concepts/assets/software-defined-assets" rel="nofollow">Software Defined Assets</a></li> <li><a href="https://dataform.co/" rel="nofollow">DataForm</a></li> <li><a href="https://gradientflow.com/2022-workflow-orchestration-survey/" rel="nofollow">Gradient Flow State Of Orchestration Report 2022</a></li> <li><a href="https://mlops.community/mlops-is-mostly-data-engineering/" rel="nofollow">MLOps Is 98% Data Engineering</a></li> <li><a href="https://datahubproject.io/" rel="nofollow">DataHub</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/datahub-metadata-management-episode-147/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://open-metadata.org/" rel="nofollow">OpenMetadata</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/openmetadata-universal-metadata-layer-episode-237/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://atlan.com/" rel="nofollow">Atlan</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/atlan-data-team-collaboration-episode-179/" rel="nofollow">Podcast Episode</a></li> </ul></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li><li><a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow">Materialize</a>: <a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow"> You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing. Go to <u>[materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)</u> today and get 2 weeks free!</a></li><li><a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">Datafold</a>: <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow"> This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

01:01:25

Eliminate The Overhead In Your Data Integration With The Open Source dlt Library

<h2>Summary</h2> Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. The challenge is that most of those options are complex to operate and exist in their own silo. The dlt project was created to eliminate overhead and bring data integration into your full control as a library component of your overall data system. In this episode Adrian Brudaru explains how it works, the benefits that it provides over other data integration solutions, and how you can start building pipelines today.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to <a href="https://www.dataengineeringpodcast.com/materialize" rel="nofollow">dataengineeringpodcast.com/materialize</a> today to get 2 weeks free!</li> <li>This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">dataengineeringpodcast.com/datafold</a></li> <li>Your host is Tobias Macey and today I'm interviewing Adrian Brudaru about dlt, an open source python library for data loading</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you describe what dlt is and the story behind it? <ul> <li>What is the problem you want to solve with dlt?</li> <li>Who is the target audience?</li> </ul></li> <li>The obvious comparison is with systems like Singer/Meltano/Airbyte in the open source space, or Fivetran/Matillion/etc. in the commercial space. What are the complexities or limitations of those tools that leave an opening for dlt?</li> <li>Can you describe how dlt is implemented?</li> <li>What are the benefits of building it in Python?</li> <li>How have the design and goals of the project changed since you first started working on it?</li> <li>How does that language choice influence the performance and scaling characteristics?</li> <li>What problems do users solve with dlt?</li> <li>What are the interfaces available for extending/customizing/integrating with dlt?</li> <li>Can you talk through the process of adding a new source/destination?</li> <li>What is the workflow for someone building a pipeline with dlt?</li> <li>How does the experience scale when supporting multiple connections?</li> <li>Given the limited scope of extract and load, and the composable design of dlt it seems like a purpose built companion to dbt (down to the naming). What are the benefits of using those tools in combination?</li> <li>What are the most interesting, innovative, or unexpected ways that you have seen dlt used?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on dlt?</li> <li>When is dlt the wrong choice?</li> <li>What do you have planned for the future of dlt?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/data-team/?originalSubdomain=de" rel="nofollow">LinkedIn</a></li> <li><a href="https://join.slack.com/t/dlthub-community/shared_invite/zt-1slox199h-HAE7EQoXmstkP_bTqal65g" rel="nofollow">Join our community to discuss further</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://dlthub.com/" rel="nofollow">dlt</a> <ul> <li><a href="https://dlthub.com/success-stories/harness/" rel="nofollow">Harness Success Story</a></li> </ul></li> <li><a href="https://dlthub.com/product/" rel="nofollow">Our guiding product principles</a></li> <li><a href="https://dlthub.com/docs/dlt-ecosystem" rel="nofollow">Ecosystem support</a></li> <li><a href="https://dlthub.com/docs/getting-started/build-a-data-pipeline" rel="nofollow">From basic to complex, dlt has many capabilities</a></li> <li><a href="https://www.singer.io/" rel="nofollow">Singer</a></li> <li><a href="https://airbyte.com/" rel="nofollow">Airbyte</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/airbyte-open-source-data-integration-episode-173/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://meltano.com/" rel="nofollow">Meltano</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/meltano-data-integration-episode-141/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.matillion.com/" rel="nofollow">Matillion</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/matillion-cloud-data-integration-episode-286/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.fivetran.com/" rel="nofollow">Fivetran</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/fivetran-data-replication-episode-93/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://duckdb.org/" rel="nofollow">DuckDB</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/duckdb-in-process-olap-database-episode-270/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.openapis.org/" rel="nofollow">OpenAPI</a></li> <li><a href="https://martinfowler.com/articles/data-monolith-to-mesh.html" rel="nofollow">Data Mesh</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/data-mesh-revisited-episode-250/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://sqlmesh.com/" rel="nofollow">SQLMesh</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/sqlmesh-open-source-dataops-episode-380" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://airflow.apache.org/" rel="nofollow">Airflow</a></li> <li><a href="https://dagster.io/" rel="nofollow">Dagster</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/dagster-data-platform-big-complexity-episode-239/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.prefect.io/" rel="nofollow">Prefect</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/prefect-workflow-engine-episode-86/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://github.com/z3z1ma/alto" rel="nofollow">Alto</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li><li><a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow">Materialize</a>: <a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow"> You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing. Go to <u>[materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)</u> today and get 2 weeks free!</a></li><li><a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">Datafold</a>: <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow"> This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

00:42:12

Building An Internal Database As A Service Platform At Cloudflare

<h2>Summary</h2> Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you <em>are</em> a cloud service? In this episode Vignesh Ravichandran explains how his team at Cloudflare provides PostgreSQL as a service to their developers for low latency and high uptime services at global scale. This is an interesting and insightful look at pragmatic engineering for reliability and scale.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">dataengineeringpodcast.com/datafold</a></li> <li>You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to <a href="https://www.dataengineeringpodcast.com/materialize" rel="nofollow">dataengineeringpodcast.com/materialize</a> today to get 2 weeks free!</li> <li>Your host is Tobias Macey and today I'm interviewing Vignesh Ravichandran about building an internal database as a service platform at Cloudflare</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by describing the different database workloads that you have at Cloudflare? <ul> <li>What are the different methods that you have used for managing database instances?</li> </ul></li> <li>What are the requirements and constraints that you had to account for in designing your current system?</li> <li>Why Postgres?</li> <li>optimizations for Postgres <ul> <li>simplification from not supporting multiple engines</li> </ul></li> <li>limitations in postgres that make multi-tenancy challenging</li> <li>scale of operation (data volume, request rate</li> <li>What are the most interesting, innovative, or unexpected ways that you have seen your DBaaS used?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on your internal database platform?</li> <li>When is an internal database as a service the wrong choice?</li> <li>What do you have planned for the future of Postgres hosting at Cloudflare?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/vigneshravichandran28/" rel="nofollow">LinkedIn</a></li> <li><a href="https://viggy28.dev/" rel="nofollow">Website</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.cloudflare.com/" rel="nofollow">Cloudflare</a></li> <li><a href="https://www.postgresql.org/" rel="nofollow">PostgreSQL</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/postgresql-with-jonathan-katz-episode-42/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.postgresql.org/docs/current/datatype-net-types.html" rel="nofollow">IP Address Data Type in Postgres</a></li> <li><a href="https://www.cockroachlabs.com/" rel="nofollow">CockroachDB</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/cockroachdb-with-peter-mattis-episode-35/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.citusdata.com/" rel="nofollow">Citus</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/citus-data-with-ozgun-erdogan-and-craig-kerstiens-episode-13/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.yugabyte.com/" rel="nofollow">Yugabyte</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/yugabytedb-planet-scale-sql-episode-115/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://github.com/sorintlab/stolon" rel="nofollow">Stolon</a></li> <li><a href="https://www.postgresql.org/docs/current/app-pgrewind.html" rel="nofollow">pg_rewind</a></li> <li><a href="https://www.pgbouncer.org/" rel="nofollow">PGBouncer</a></li> <li><a href="https://www.youtube.com/watch?v=HIOo4j-Tiq4" rel="nofollow">HAProxy Presentation</a></li> <li><a href="https://etcd.io/" rel="nofollow">Etcd</a></li> <li><a href="https://patroni.readthedocs.io/en/latest/" rel="nofollow">Patroni</a></li> <li><a href="https://www.postgresql.org/docs/current/pgupgrade.html" rel="nofollow">pg_upgrade</a></li> <li><a href="https://en.wikipedia.org/wiki/Edge_computing" rel="nofollow">Edge Computing</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li><li><a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow">Materialize</a>: <a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow"> You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing. Go to <u>[materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)</u> today and get 2 weeks free!</a></li><li><a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">Datafold</a>: <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow"> This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

01:01:09

Harnessing Generative AI For Creating Educational Content With Illumidesk

<h2>Summary</h2> Generative AI has unlocked a massive opportunity for content creation. There is also an unfulfilled need for experts to be able to share their knowledge and build communities. Illumidesk was built to take advantage of this intersection. In this episode Greg Werner explains how they are using generative AI as an assistive tool for creating educational material, as well as building a data driven experience for learners.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">dataengineeringpodcast.com/datafold</a></li> <li>You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to <a href="https://www.dataengineeringpodcast.com/materialize" rel="nofollow">dataengineeringpodcast.com/materialize</a> today to get 2 weeks free!</li> <li>Your host is Tobias Macey and today I'm interviewing Greg Werner about building IllumiDesk, a data-driven and AI powered online learning platform</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you describe what Illumidesk is and the story behind it?</li> <li>What are the challenges that educators and content creators face in developing and maintaining digital course materials for their target audiences?</li> <li>How are you leaning on data integrations and AI to reduce the initial time investment required to deliver courseware?</li> <li>What are the opportunities for collecting and collating learner interactions with the course materials to provide feedback to the instructors?</li> <li>What are some of the ways that you are incorporating pedagogical strategies into the measurement and evaluation methods that you use for reports?</li> <li>What are the different categories of insights that you need to provide across the different stakeholders/personas who are interacting with the platform and learning content?</li> <li>Can you describe how you have architected the Illumidesk platform?</li> <li>How have the design and goals shifted since you first began working on it?</li> <li>What are the strategies that you have used to allow for evolution and adaptation of the system in order to keep pace with the ecosystem of generative AI capabilities?</li> <li>What are the failure modes of the content generation that you need to account for?</li> <li>What are the most interesting, innovative, or unexpected ways that you have seen Illumidesk used?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on Illumidesk?</li> <li>When is Illumidesk the wrong choice?</li> <li>What do you have planned for the future of Illumidesk?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/wernergreg/" rel="nofollow">LinkedIn</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.illumidesk.com/" rel="nofollow">Illumidesk</a></li> <li><a href="https://en.wikipedia.org/wiki/Generative_artificial_intelligence" rel="nofollow">Generative AI</a></li> <li><a href="https://www.pinecone.io/learn/vector-database/" rel="nofollow">Vector Database</a></li> <li><a href="https://en.wikipedia.org/wiki/Learning_Tools_Interoperability" rel="nofollow">LTI == Learning Tools Interoperability</a></li> <li><a href="https://scorm.com/scorm-explained/" rel="nofollow">SCORM</a></li> <li><a href="https://xapi.com/overview/" rel="nofollow">XAPI</a></li> <li><a href="https://en.wikipedia.org/wiki/Prompt_engineering" rel="nofollow">Prompt Engineering</a></li> <li><a href="https://en.wikipedia.org/wiki/GPT-4" rel="nofollow">GPT-4</a></li> <li><a href="https://en.wikipedia.org/wiki/LLaMA" rel="nofollow">LLama</a></li> <li><a href="https://www.anthropic.com/" rel="nofollow">Anthropic</a></li> <li><a href="https://fastapi.tiangolo.com/" rel="nofollow">FastAPI</a></li> <li><a href="https://www.langchain.com/" rel="nofollow">LangChain</a></li> <li><a href="https://docs.celeryq.dev/en/stable/" rel="nofollow">Celery</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow">Materialize</a>: <a href="https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access" rel="nofollow"> You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing. Go to <u>[materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access)</u> today and get 2 weeks free!</a></li><li><a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">Datafold</a>: <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow"> This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!</a></li><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

00:54:52

Unpacking The Seven Principles Of Modern Data Pipelines

<h2>Summary</h2> Data pipelines are the core of every data product, ML model, and business intelligence dashboard. If you're not careful you will end up spending all of your time on maintenance and fire-fighting. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data. In this episode Ariel Pohoryles explains what they are and how they work together to increase your chances of success.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">dataengineeringpodcast.com/datafold</a></li> <li>Your host is Tobias Macey and today I'm interviewing Ariel Pohoryles about the seven principles of modern data pipelines</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by defining what you mean by a "modern" data pipeline?</li> <li>At Rivery you published a white paper identifying seven principles of modern data pipelines: <ul> <li>Zero infrastructure management</li> <li>ELT-first mindset</li> <li>Speaks SQL and Python</li> <li>Dynamic multi-storage layers</li> <li>Reverse ETL & operational analytics</li> <li>Full transparency</li> <li>Faster time to value</li> </ul></li> <li>What are the applications of data that you focused on while identifying these principles?</li> <li>How do the application of these principles influence the ability of organizations and their data teams to encourage and keep pace with the use of data in the business?</li> <li>What are the technical components of a pipeline infrastructure that are necessary to support a "modern" workflow?</li> <li>How do the technologies involved impact the organizational involvement with how data is applied throughout the business?</li> <li>When using managed services, what are the ways that the pricing model acts to encourage/discourage experimentation/exploration with data?</li> <li>What are the most interesting, innovative, or unexpected ways that you have seen these seven principles implemented/applied?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working with customers to adapt to these principles?</li> <li>What are the cases where some/all of these principles are undesirable/impractical to implement?</li> <li>What are the opportunities for further advancement/sophistication in the ways that teams work with and gain value from data?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/ariel-pohoryles-88695622/" rel="nofollow">LinkedIn</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://rivery.io/" rel="nofollow">Rivery</a></li> <li><a href="https://rivery.io/downloads/7-principles-modern-data-pipeline-lp/" rel="nofollow">7 Principles Of The Modern Data Pipeline</a></li> <li><a href="https://en.wikipedia.org/wiki/Extract,_load,_transform" rel="nofollow">ELT</a></li> <li><a href="https://rivery.io/blog/what-is-reverse-etl-guide-for-data-teams/" rel="nofollow">Reverse ETL</a></li> <li><a href="https://chiefmartec.com/2023/05/2023-marketing-technology-landscape-supergraphic-11038-solutions-searchable-on-martechmap-com/" rel="nofollow">Martech Landscape</a></li> <li><a href="https://www.forbes.com/sites/bernardmarr/2022/01/18/what-is-a-data-lakehouse-a-super-simple-explanation-for-anyone/?sh=54d5c4916088" rel="nofollow">Data Lakehouse</a></li> <li><a href="https://www.databricks.com/" rel="nofollow">Databricks</a></li> <li><a href="https://www.snowflake.com/en/" rel="nofollow">Snowflake</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow">Datafold</a>: <a href="https://www.dataengineeringpodcast.com/datafold" rel="nofollow"> This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!</a></li><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

00:47:02

Quantifying The Return On Investment For Your Data Team

<h2>Summary</h2> As businesses increasingly invest in technology and talent focused on data engineering and analytics, they want to know whether they are benefiting. So how do you calculate the return on investment for data? In this episode Barr Moses and Anna Filippova explore that question and provide useful exercises to start answering that in your company.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>Your host is Tobias Macey and today I'm interviewing Barr Moses and Anna Filippova about how and whether to measure the ROI of your data team</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>What are the typical motivations for measuring and tracking the ROI for a data team? <ul> <li>Who is responsible for collecting that information?</li> <li>How is that information used and by whom?</li> </ul></li> <li>What are some of the downsides/risks of tracking this metric? (law of unintended consequences)</li> <li>What are the inputs to the number that constitutes the "investment"? infrastructure, payroll of employees on team, time spent working with other teams?</li> <li>What are the aspects of data work and its impact on the business that complicate a calculation of the "return" that is generated?</li> <li>How should teams think about measuring data team ROI? </li> <li>What are some concrete ROI metrics data teams can use? <ul> <li>What level of detail is useful? What dimensions should be used for segmenting the calculations?</li> </ul></li> <li>How can visibility into this ROI metric be best used to inform the priorities and project scopes of the team? </li> <li>With so many tools in the modern data stack today, what is the role of technology in helping drive or measure this impact? </li> <li>How do your respective solutions, Monte Carlo and dbt, help teams measure and scale data value? </li> <li>With generative AI on the upswing of the hype cycle, what are the impacts that you see it having on data teams? <ul> <li>What are the unrealistic expectations that it will produce?</li> <li>How can it speed up time to delivery? </li> </ul></li> <li>What are the most interesting, innovative, or unexpected ways that you have seen data team ROI calculated and/or used?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on measuring the ROI of data teams?</li> <li>When is measuring ROI the wrong choice?</li> </ul> <h2>Contact Info</h2> <ul> <li>Barr <ul> <li><a href="https://www.linkedin.com/in/barrmoses/" rel="nofollow">LinkedIn</a></li> </ul></li> <li>Anna <ul> <li><a href="https://www.linkedin.com/in/annafilippova" rel="nofollow">LinkedIn</a></li> </ul></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.montecarlodata.com/" rel="nofollow">Monte Carlo</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/monte-carlo-observability-data-quality-episode-155" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.getdbt.com/" rel="nofollow">dbt</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/dbt-data-analytics-episode-81" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.snowflake.com/webinar/thought-leadership/jet-blue-and-monte-carlos/" rel="nofollow">JetBlue Snowflake Con Presentation</a></li> <li><a href="https://generativeai.net/" rel="nofollow">Generative AI</a></li> <li><a href="https://en.wikipedia.org/wiki/Large_language_model" rel="nofollow">Large Language Models</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

01:01:52

Strategies For A Successful Data Platform Migration

<h2>Summary</h2> All software systems are in a constant state of evolution. This makes it impossible to select a truly future-proof technology stack for your data platform, making an eventual migration inevitable. In this episode Gleb Mezhanskiy and Rob Goretsky share their experiences leading various data platform migrations, and the hard-won lessons that they learned so that you don't have to.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at <a href="https://www.dataengineeringpodcast.com/rudderstack" rel="nofollow">dataengineeringpodcast.com/rudderstack</a></li> <li>Modern data teams are using Hex to 10x their data impact. Hex combines a notebook style UI with an interactive report builder. This allows data teams to both dive deep to find insights and then share their work in an easy-to-read format to the whole org. In Hex you can use SQL, Python, R, and no-code visualization together to explore, transform, and model data. Hex also has AI built directly into the workflow to help you generate, edit, explain and document your code. The best data teams in the world such as the ones at Notion, AngelList, and Anthropic use Hex for ad hoc investigations, creating machine learning models, and building operational dashboards for the rest of their company. Hex makes it easy for data analysts and data scientists to collaborate together and produce work that has an impact. Make your data team unstoppable with Hex. Sign up today at <a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow">dataengineeringpodcast.com/hex</a> to get a 30-day free trial for your team!</li> <li>Your host is Tobias Macey and today I'm interviewing Gleb Mezhanskiy and Rob Goretsky about when and how to think about migrating your data stack</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>A migration can be anything from a minor task to a major undertaking. Can you start by describing what constitutes a migration for the purposes of this conversation?</li> <li>Is it possible to completely avoid having to invest in a migration?</li> <li>What are the signals that point to the need for a migration? <ul> <li>What are some of the sources of cost that need to be accounted for when considering a migration? (both in terms of doing one, and the costs of not doing one)</li> <li>What are some signals that a migration is not the right solution for a perceived problem?</li> </ul></li> <li>Once the decision has been made that a migration is necessary, what are the questions that the team should be asking to determine the technologies to move to and the sequencing of execution?</li> <li>What are the preceding tasks that should be completed before starting the migration to ensure there is no breakage downstream of the changing component(s)?</li> <li>What are some of the ways that a migration effort might fail?</li> <li>What are the major pitfalls that teams need to be aware of as they work through a data platform migration?</li> <li>What are the opportunities for automation during the migration process?</li> <li>What are the most interesting, innovative, or unexpected ways that you have seen teams approach a platform migration?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on data platform migrations?</li> <li>What are some ways that the technologies and patterns that we use can be evolved to reduce the cost/impact/need for migraitons?</li> </ul> <h2>Contact Info</h2> <ul> <li>Gleb <ul> <li><a href="https://www.linkedin.com/in/glebmezh/" rel="nofollow">LinkedIn</a></li> <li><a href="https://twitter.com/glebmm" rel="nofollow">@glebmm</a> on Twitter</li> </ul></li> <li>Rob <ul> <li><a href="https://www.linkedin.com/in/robertgoretsky/" rel="nofollow">LinkedIn</a></li> <li><a href="https://github.com/RobGoretsky" rel="nofollow">RobGoretsky</a> on GitHub</li> </ul></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don't forget to check out our other shows. <a href="https://www.pythonpodcast.com" rel="nofollow">Podcast.__init__</a> covers the Python language, its community, and the innovative ways it is being used. The <a href="https://www.themachinelearningpodcast.com" rel="nofollow">Machine Learning Podcast</a> helps you go from idea to production with machine learning.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com" rel="nofollow">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you've learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com" rel="nofollow">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557" rel="nofollow">Apple Podcasts</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.datafold.com/" rel="nofollow">Datafold</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/datafold-proactive-data-quality-episode-205/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://www.informatica.com/" rel="nofollow">Informatica</a></li> <li><a href="https://airflow.apache.org/" rel="nofollow">Airflow</a></li> <li><a href="https://www.snowflake.com/en/" rel="nofollow">Snowflake</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/snowflakedb-cloud-data-warehouse-episode-110/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://aws.amazon.com/redshift/" rel="nofollow">Redshift</a></li> <li><a href="https://www.eventbrite.com/" rel="nofollow">Eventbrite</a></li> <li><a href="https://www.teradata.com/" rel="nofollow">Teradata</a></li> <li><a href="https://cloud.google.com/bigquery" rel="nofollow">BigQuery</a></li> <li><a href="https://trino.io/" rel="nofollow">Trino</a></li> <li><a href="https://aws.amazon.com/emr/" rel="nofollow">EMR == Elastic Map-Reduce</a></li> <li><a href="https://en.wikipedia.org/wiki/Shadow_IT" rel="nofollow">Shadow IT</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/shadow-it-data-analytics-episode-121" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://mode.com/" rel="nofollow">Mode Analytics</a></li> <li><a href="https://cloud.google.com/looker/" rel="nofollow">Looker</a></li> <li><a href="https://en.wikipedia.org/wiki/Sunk_cost" rel="nofollow">Sunk Cost Fallacy</a></li> <li><a href="https://github.com/datafold/data-diff" rel="nofollow">data-diff</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/data-diff-open-source-data-integration-validation-episode-303/" rel="nofollow">Podcast Episode</a></li> </ul></li> <li><a href="https://github.com/tobymao/sqlglot" rel="nofollow">SQLGlot</a></li> <li>[Dagster](dhttps://dagster.io/)</li> <li><a href="https://www.getdbt.com/" rel="nofollow">dbt</a></li> </ul> The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug" rel="nofollow">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/" rel="nofollow">The Freak Fandango Orchestra</a> / <a href="http://creativecommons.org/licenses/by-sa/3.0/" rel="nofollow">CC BY-SA</a></p>Sponsored By:</p><ul><li><a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow">Hex</a>: <a href="https://www.dataengineeringpodcast.com/hex" rel="nofollow"> Hex is a collaborative workspace for data science and analytics. A single place for teams to explore, transform, and visualize data into beautiful interactive reports. Use SQL, Python, R, no-code and AI to find and share insights across your organization. Empower everyone in an organization to make an impact with data. Sign up today at [dataengineeringpodcast.com/hex](https://www.dataengineeringpodcast.com/hex} and get 30 days free!</a></li><li><a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow">Rudderstack</a>: <a href="https://www.rudderstack.com/product/profiles/?utm_source=data_eng_podcast&utm_medium=podcast&utm_campaign=CMPGN_173_P&utm_content=None&utm_term=%7Bkeyword%7D&utm_referrer=data_eng_podcast&raid=afdb0d1c0f9e89f23c581928d8122f5d" rel="nofollow"> Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)</a></li></ul><a href="https://dataengineering.supercast.com/" rel="payment">Support Data Engineering Podcast</a></p>

01:09:52