Category Data

Data Intensive Applications with Martin Kleppmann

http://traffic.libsyn.com/sedaily/dataintensive_edited_fixed.mp3Podcast: Play in new window | Download A new programmer learns to build applications using data structures like a queue, a cache, or a database. Modern cloud applications are built using more sophisticated tools like Redis, Kafka, or Amazon S3. These tools do multiple things well, and often have overlapping functionality. Application architecture becomes less straightforward. The applications we are building today are data-intensive rather than compute-intensive. Netflix needs to

Continue reading…

RealmDB with Brian Munkholm

http://traffic.libsyn.com/sedaily/realmdb_edited.mp3Podcast: Play in new window | Download Expectations for mobile apps have gone up steadily since the iPhone was released. But the choice of databases built for mobile apps has remained limited mostly to SQLite. RealmDB was created as a new option for mobile developers on iOS, Android, or any other mobile platform.   Realm is not just a database. It is a database platform, offering a variety of systems

Continue reading…

Columnar Data: Apache Arrow and Parquet with Julien Le Dem and Jacques Nadeau

http://traffic.libsyn.com/sedaily/columnardata_edited_fixed.mp3Podcast: Play in new window | Download Column-oriented data storage allows us to access all of the entries in a database column quickly and efficiently. Columnar storage formats are mostly relevant today for performing large analytics jobs. For example, if you are a bank, and you want to get the sum of all of the financial transactions that took place on your system in the last week, you don’t want

Continue reading…

Data Engineering with Pete Soderling

http://traffic.libsyn.com/sedaily/hakkalabs_edited.mp3Podcast: Play in new window | Download In the last five years, companies started hiring data engineers. A data engineer creates the systems that manage and access the huge volumes of data that are accumulating on cheap cloud servers. As the saying goes, “it’s more expensive to throw out the data than to store it.” Pete Soderling joins the show to discuss the rise of the data engineer, and how

Continue reading…

Database as a Service with Eliot Horowitz

http://traffic.libsyn.com/sedaily/mongoservice_editedfixed1.mp3Podcast: Play in new window | Download Eight years ago, MongoDB was an internal project at 10gen, a company that was trying to build a platform-as-a-service out of open-source components. The team at 10gen realized that the platform-as-a-service play would be too complex, and difficult to build. Since MongoDB was the most valuable component of that project, they narrowed their focus to this new document-oriented database. In today’s episode, MongoDB

Continue reading…

Database Choices and Uber with Markus Winand

http://traffic.libsyn.com/sedaily/uber_database_edited.mp3Podcast: Play in new window | Download When Uber’s engineering team published a blog post about moving to MySQL from Postgres, Markus Winand started receiving lots of email. Markus writes about databases on his blog “Use The Index, Luke,” a guide to database performance for developers. The people emailing Markus wanted to know–if Postgres doesn’t work well for Uber, is it safe to use for anyone? Markus wrote a detailed

Continue reading…

Uber’s Postgres Problems with Evan Klitzke

http://traffic.libsyn.com/sedaily/Uber_DBs.mp3Podcast: Play in new window | Download When a company switches the relational database it uses, you wouldn’t expect the news of the switch to go viral. Most engineers are not interested in the subtle differences between MySQL and Postgres, right?   Uber recently switched from having Postgres as its main relational database to using MySQL. Evan Klitzke wrote a detailed blog post about the migration, and post got very

Continue reading…

Relational Databases with Craig Kerstiens

http://traffic.libsyn.com/sedaily/RelationalDBs.mp3Podcast: Play in new window | Download Relational databases are used by most applications. MySQL, Postgres, Microsoft SQL Server, and other products implement the core features of a relational database in different ways. A developer who has never studied this space in detail may not know the differences between these databases, and in this episode we describe some tradeoffs that relational databases can make. Craig Kerstiens is an engineer at

Continue reading…

Peter Bailis on the Data Community’s Identity Crisis

http://traffic.libsyn.com/sedaily/database_crisis_edited_fixed.mp3Podcast: Play in new window | Download Breakthroughs in modern data research tend to come from companies like Google, Facebook, and Amazon, with projects like MapReduce, Cassandra, and Dynamo.   Twenty years ago, this types of breakthroughs would be happening in academia, which causes today’s guest Peter Bailis to ask: is the academic data community having an identity crisis?   Peter is an assistant professor at Stanford University, where he

Continue reading…

Apache Arrow with Uwe Korn

http://traffic.libsyn.com/sedaily/arrow_edited_fixed.mp3Podcast: Play in new window | Download In a typical data analytics system, there are a variety of technologies interacting. HDFS for storing files, Spark for distributed machine learning, pandas for data analysis in Python–each of these different technologies has a different format for how data is represented.   Serialization and deserialization between these different formats causes significant latency across the overall system. Apache Arrow is a tool for improving

Continue reading…

Cassandra Data Modeling with Jon Haddad

http://traffic.libsyn.com/sedaily/Cassandra_jon_haddad_Edited.mp3Podcast: Play in new window | Download Data modeling is the process of creating relationships and rules about objects, so that we can decide how to store them in a database. Data modeling defines how we store and query our database systems.   Today’s episode features a discussion of data modeling in Cassandra with Jon Haddad, an evangelist at Datastax. The distributed nature of Cassandra creates some unique rules around

Continue reading…

Cassandra Compliant ScyllaDB with Dor Laor

http://traffic.libsyn.com/sedaily/ScyllaDB_Edited.mp3Podcast: Play in new window | Download Apache Cassandra is a distributed database that can handle large amounts of data with no single point of failure. Since 2008, Cassandra has been widely adopted and the software and the community around it have grown steadily. A software developer interacting with Cassandra uses CQL, the Cassandra Query Language. ScyllaDB is another open-source database that has been created to be totally compatible with

Continue reading…

Scaling PostgreSQL with Citus Data’s Ozgun Erdogan

http://traffic.libsyn.com/sedaily/Citus_Data.mp3Podcast: Play in new window | Download Ten years ago, databases were much simpler. Most companies would only have one or two types of databases in production. Today, the age of one-size-fits-all is over. Companies have multiple databases to deal with different types of use cases, and databases have become distributed to multiple nodes in order to be scalable. Ozgun Erdogan of Citus Data joins the show to give us

Continue reading…

Kafka, Storm, and Cassandra: Keen IO’s Analytics Architecture with Dan Kador

http://traffic.libsyn.com/sedaily/Keen_Edited_3.mp3Podcast: Play in new window | Download The process of building a software project requires us to make so many architectural decisions. Which programming languages should be used? Which cloud service provider? Which database? A newer type of building block is the analytics platform. Companies need to track events, aggregate metrics, and change the user’s experience based on aggregated data. Dan Kador is a co-founder of Keen IO, and he

Continue reading…

Netflix’s Data Pipeline with Steven Wu

http://traffic.libsyn.com/sedaily/Netflix.mp3Podcast: Play in new window | Download At Netflix, 500 billion events and 1.3 petabytes of data are ingested by the system per day.  This includes video viewing activities, error logs, and performance events. On today’s episode, we dive deep into the data pipeline of Netflix, and how it evolved from their 1.0 version to the modern 2.0 version. Before listening to this episode, check out the blog post that

Continue reading…

Crate.io and Distributed SQL with Jodok Batlogg

http://traffic.libsyn.com/sedaily/Crate.mp3Podcast: Play in new window | Download Distributed databases are difficult to operate, and Crate.io wants to change that. Crate is a fast, scalable, easy-to-use SQL database that is built to run in containerized environments. An average software company runs several databases–MySQL for relational store, MongoDB for a document database, HDFS for blob storage and data warehouse, elastic search for search. On today’s show, Jodok Batlogg from Crate discuss the

Continue reading…

Azure Stream Analytics with Santosh Balasubramanian

http://traffic.libsyn.com/sedaily/azure_stream_final.mp3Podcast: Play in new window | Download Microsoft has built a suite of technologies on top of its Azure infrastructure as a service. Today, we discuss Azure Stream Analytics, a real-time event processing engine developed at Microsoft. Azure Streaming allows for constant querying of incoming data streams, and my guest Santosh Balasubramanian discusses Azure and the movement from batch processing to stream processing. Sponsors Hired.com is the job marketplace for

Continue reading…

Spark and Cassandra with Tim Berglund

http://traffic.libsyn.com/sedaily/Cassandra_Spark_Edited_2.mp3Podcast: Play in new window | Download Apache Spark is a framework for fast, distributed, in-memory analysis. Apache Cassandra is a distributed database management system that provides high availability and fast throughput. Today, we are collecting fast, big data streams from user behavior, smart phones and sensors, and the disk checkpointing of and query language of Hadoop MapReduce is no longer adequate.   Tim Berglund from Datastax came on Software

Continue reading…

Azure Event Hubs and Kafka with Dan Rosanova

http://traffic.libsyn.com/sedaily/Eventhubs_Edited.mp3Podcast: Play in new window | Download Apache Kafka has become the most popular open-source solution for persistent replicated messaging in the Hadoop ecosystem. But some software engineers who are working with “big data” don’t want to deal with the configuration and set up of Kafka. One way to side step this problem is to go with a managed solution, like Microsoft Azure Event Hubs. Dan Rosanova is today’s guest.

Continue reading…

CockroachDB with Ben Darnell

http://traffic.libsyn.com/sedaily/cockroachdb_Edited.mp3Podcast: Play in new window | Download “Eventual consistency is really kind of a marketing term from some of these NoSQL systems – it’s not really consistent in any strong sense of the term.” Google has published papers on distributed systems such as BigTable, Chubby, and the Google File System. During this episode, we focus on a product that takes inspiration from Google’s Spanner project, a database that is built

Continue reading…