EPISODE 1963

[INTRODUCTION]

[0:00:00] ANNOUNCER: Many real-world processes produce data as a continuous stream rather than as isolated records. Sensor readings, financial markets, and application telemetry all generate data this way. This kind of time series data has a distinctive shape. It's written far more often than it is updated, it accumulates continuously, and it is usually queried across ranges of time. Time series databases are built specifically for this kind of workload. 

TimescaleDB is an open -source database from Tiger Data that adds time series capabilities to PostgreSQL. It's implemented as a PostgreSQL extension, so it introduces new functionality while preserving standard PostgreSQL behavior and SQL. This lets a single PostgreSQL-based system handle both transactional and analytical workloads without splitting data across multiple tools. 

Brandon Purcell is the Director of Product Management at TigerData. In this episode, Brandon joins Kevin Ball to discuss why time series data breaks conventional databases, how hypertables and hypercores scale Postgres, zero-copy database forking for agent-based workflows, and much more. 

Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc. 

[INTERVIEW]

[0:01:51] KB: Brandon, welcome to the show. 

[0:01:53] BP: Yeah, thanks for having me, Kevin. 

[0:01:54] KB: Yeah, I'm excited to learn a little bit more about what you're up to. So let's start with you. Can you give us a quick overview of your background and then how you ended up at Tiger and what you're doing there? 

[0:02:04] BP: Yeah, no, absolutely. I've been working in the tech industry for a very long time, spent my early years in the military, even writing code, and then started working at Adobe soon after that, working in a number of different roles there, and met a couple of guys where we built a startup. The startup was called SpatialKey. The background in my area and experience in Postgres goes back almost 20 years. We built a geospatial analytics platform that was built on top of Postgres, primarily initially focusing across a broad range, horizontal, but then switching that into insurance and everything from underwriting, to event response and portfolio management type use cases. And probably after about 13 years, we sold the company and I spent some time, took some time off afterwards and decided to come back and actually take a very different role. I was doing most of the leadership. I was the CTO of the company and really wanted to get back into the tech side of it and product. And I love Postgres, and so ended up coming over to Tiger Data because I believe they have a great product and it really takes Postgres to the next level. 

[0:03:08] KB: That's a good intro. So can you tell us a little bit about the product at Tiger and what is it that it handles that vanilla Postgres doesn't? 

[0:03:16] BP: No, absolutely. And so just kind of taking a step back, you have TigerData as the company and the organization. And for most that are familiar with Timescale, which is the product TimescaleDB that Tiger Data has built, it's an extension of Postgres. And so by an extension, it's not a fork, it's an actually extension. And so it's offered as an open source product. 

And the challenge is a lot of times when you have repeated time series type data readings, you end up growing that data pretty extensively. And oftentimes, as that grows in vanilla Postgres, you start to run into challenges. And those challenges yield themselves as read ingest or ingest bottlenecks. Read start to slow down as your data starts to bloat. You run into challenges with an index that's growing with a single table. Time range queries get challenged. And so TimescaleDB addresses that by building this extension. 

And we'll talk about it in a number of the technologies that sit below the surface that address that. But from a developer perspective, it's the same SQL, no differences at all, but there's some intricate workings of the database that makes it easier to use and actually scale a lot better. And so from an offering perspective, we have a couple of different offerings within that, but most people start with the open source product and we also have our Tiger Cloud solution as well. 

[0:04:30] KB: Yeah, so let's talk a little bit about time series and what are the problem domains and use cases. If you're someone who's coming from a web background, maybe you've only ever dealt with relational classic REST objects and things like that. What are the types of problems that get you into saying, "Hey, actually, I need a different data structure here. I need something that looks like a time series." 

[0:04:49] BP: People typically do start with Postgres. I think 60% of the world. You start with a relational database, Postgres being the one that's probably used the most by developers. But you end up having a particular device or set of devices that are emitting readings. And that can be across a number of different areas as well. Manufacturing or robotics, medical devices, finance and crypto are also used there where you have a number of different transactions, smart buildings and industrial systems. So there's a number of different areas. And I think it's a growing area as well as we look at what we call the physical world. We have solar, energy, any number of things that are all emitting readings on a periodic basis. And the number of devices within that is growing as well. So you end up with hundreds of thousands or even millions of devices. 

We have one customer that has 22 million devices and they're all emitting readings every few seconds, every second, in some even sub-second as well. So the volume of data that you're ingesting and the volume that's growing over time gets larger and larger, right? And so that challenges the databases in new and unique ways. 

The other piece of that is that you still need the transactional relational part of the database as well. And so as you start to grow, and oftentimes people start with Postgres and as you start to outgrow it, your query slows down, your ingest slows down. You have two options, really, right? You can separate it off into another solution. So you bifurcate with some type of analytical database, and then you retain your transactional database. Or you can adopt a system like TimescaleDB that gives you the best of both, where you have co-located into the same system, we can extend it. 

And this is with zero application changes at all. And we can talk about what that migration path looks like, but you're not having to make application changes, where if you bifurcate it, you have to manage multiple data pipelines between this other system. And typically, most of these systems have some evolution of SQL that is a little bit different. So you end up having to rewrite portions of your application. 

[0:06:48] KB: Yeah, that makes sense. I mean, I guess looking at that piece as an application developer, presumably the underlying data model looks a little bit different when you start to go to Timescale. Is there any difference at all? I need to know in terms of - 

[0:07:00] BP: Zero. Zero difference at all. And so you think about you're starting with the table, you have a reading, you have a timestamp, and then you have any number of attributes that are associated with that device. So that stays exactly the same as you move into Timescale. What happens under the covers is you have a concept of everyone understands tables in existing databases. We have something called a Hypertable, which takes that same concept and it starts with the partitioning, the automatic partitioning of that data. So breaking based upon those time steps and partitioning it in chunks that separate it over time. So you're not reading all of the data at once and there's a number of pieces that are around that. 

And so within those chunks as well, we have something called Hypercore, which allows us to have a hybrid row and column store. So typical database in Postgres, you have a row store and your data is organized in rows just like traditional databases. But below that as well, and this happens through policies and automatically and developers have control over this as well, data is migrated into a column store as well. And this is getting the value of compression. We've seen customers with 98% compression. Yeah. 

And that column store is, as we've evolved it over time, it used to be very immutable. And as we've moved and we improved that, it's very immutable as well. So the data can change within there as well. And so you get the benefit of the space savings. Typically, we see users with 90%. We've seen as high as 98% where you have really duplicated readings oftentimes. And then you also get the value of vectorized query acceleration as well. So that allows the underlying processor through a feature called SIMD to execute multiple batches against the column store at a time. 

And the result is you think compression is slower, right? But when you're compressing the data combined with a vectorization, we've seen acceleration in queries 10, to 100, to 1,000 times. Then beyond that a bit, we also - you think of normal B3 indexes that you have in a row or store, we have something called sparse indexes as well, which are like min-max and Bloom filter indexes that allow for execution of queries to pull from the metadata that is in these batches or in these chunks, so you're not reading all the way down to the row level and pulling data. 

So common, sum, query, first, last, things that you would have in this world would be much, much faster because they're reading from metadata versus from the underlying rows and having to decompress it. And so the combination of those things from a developer perspective is you migrate data over, there's a couple of commands you run to turn a normal table into hypertable or create a hypertable from scratch. But when you insert data or when data is migrated, it automatically, based upon those timestamps, partitions it in the appropriate chunk. 

And that partitioning extends beyond just time series. We have numeric partitioning as well. And also like UUID v7, which has a time-based component as well, we also have support for that as well. So your data ends up being broken up into these chunks. When you're querying it from a top level, it appears as one. And oftentimes one of the biggest values of that is when you have billions and billions of rows that maybe go over 10-plus years, if you're querying more near-term data, the query planner is only going to hit the chunks that are necessary for that. And so it's reducing that footprint that you're querying from. 

[0:10:30] KB: Yeah. No, it's interesting. This is bringing me back in more recent years have dealt with smaller scale data. But yeah, a decade ago was dealing with trying to - at that point, it was hundreds of millions of rows where we started to have to think about all these different things. And we're planning out partitioned tables and stuff. So drawing back on that, asking some of the questions I ran into then to see how you handle it. One of the big questions there is around is this all-time partition versus do you have other partitions? Is this non-time? I'm guessing from the name of this, this is all focused on time bucket based on day, month, year, or something along that? Or are there ways to utilize this underlying infrastructure for non-time-based partitioning? 

[0:11:10] BP: You can use additional partition schemes that come with Postgres. But for the most part, I find they're not necessary. And in the specific problem domain, people are typically querying by time as one attribute. And then if you're looking at in device world, by device or other data as well, and there's a concept that we use internally called segmentBy for how you segment the data into these batches, usually by a device ID, some type of tenant ID, user ID. And that allows the engine to group the data intelligently so that those batches compress better and also are more queryable and faster when you're querying the data. And that's a big part of that row store, column store hybrid approach where we're grouping those and creating those logical batches. 

[0:11:55] KB: Yeah, super interesting. Now, another thing that at that time was challenging was, yeah, if you wanted to join from this over into something else, because at that point I was dealing with partition logic at the app layer, you're down in Postgres. Can I just interact with other tables and joins and things like that how I might expect? 

[0:12:12] BP: Yeah, it works exactly. So think of it as, from a developer perspective, really no different than what your traditional experience would be in Postgres or any other relational database. And users typically have metadata that's associated with these devices, let's say, for example, or users. that are stored in traditional normal tables that aren't partitioned by time. And you can join directly against those tables as well. And that's a pretty common need and use case where if you're going to bifurcate that and have it across multiple systems, you have to do that at the application layer versus the database layer. And so you have to query it and then unify that back at the application layer where you can just write a query, join it across the two, aggregate the values you want, and then display that back to a user. 

[0:12:57] KB: That's fascinating. Because yeah, I remember that being a huge pain point once we got to the place we were having to partition is suddenly all these things that we used to be able to do at the SQL layer, we had to pull into the application. 

Is there any secret sauce to how you do that and keep it performant? Because under the hood, you've got all these different partitioned things. Do you plug in directly into the query engine? What does that look like to make that work? 

[0:13:19] BP: It's intercepting those queries, and the query planner is actually executing. And this is part of the extension. And I think really the difference in the value of Postgres with its extensibility. I used PostGIS for years, and I've been amazed at what you could do accomplish from a GIS perspective. And on TimescaleDB is using those same concepts with the extensibility of Postgres. And the ability to interact from a query planner automatically route to the appropriate request against the right chunks. And we've spent a lot of time optimizing that path as well. So you touch as few chunks as possible for your queries. And so you can have a lot of historical data. 

And under the covers, these chunks are really just tables and then there's a layer of metadata that Timescale is being able to use to organize that. And because of that too, your indexes, if you're talking about in the row store within those chunks, are broken up across those as well, which is one of the big challenges when you have a huge table. That index grows very substantially. And then you have an insert and has to update that index. 

And one of the tunable parameters that users have to figure out oftentimes, and we're making this easier and easier, is what is the appropriate chunk size, right? We come out of the box with seven days. But if you have a really high-volume data, you can maybe go down to even an hour, right? And so there's some different optimizations techniques around that. But for the most part from a developer perspective, it's pretty automatic and simple to get started and configured and get it right out of the box. 

[0:14:49] KB: That's interesting. So I'd love to kind of understand the underpinnings that you're giving those handles to. In particular, one angle into this, you mentioned people often start looking at this when they run into challenges at the ingest time. And then when we were talking about performance optimizations, you talked about query speed, which is read time. How does this help on the ingest time? And if I'm running into ingestion problems, does changing that bucket size help? Are there things I can tinker with what type of index I'm doing? Because I know, yeah, definitely in vanilla Postgres, some indexes are better for write-heavy applications versus not. What are the knobs and levers here? 

[0:15:23] BP: The same things really apply to Postgres. The difference really comes down to, I think, the chunk size as well. So you end up with the active, let's say, chunk that you're inserting, if you're just append-only type workload. You want to think about the size that that chunk is going to be as a row store before it's converted into the column store and compressed. And so the same kind of rules apply, but it allows you - because you have many chunks, you only have to worry about that one in terms of the indexes and that. 

And so what we've seen from that is, as you have a normal Postgres table and you ingest over time, you see this linear degradation of actually the amount of rows per second that you can ingest. But with TimescaleDB, if you're tuning things appropriately and you have the right chunk size as you work through that, you're not going to see a drop-off in that at all. 

And on ingest, the routing algorithm for how it goes to the appropriate chunk, so it allows you to backfill old historic data as well. And that's going to be routed to old chunks as well. And by default, it goes into the row store and then is converted for a policy into the column store. We do have the ability to even write directly to the column store. And that's something we're evolving over time. And what we found is you can, on nominal hardware, get to a million rows per second plus from an ingestion standpoint, which is very difficult to do in vanilla Postgres. And that's going to drop off pretty quickly as your data starts to grow. 

[0:16:54] KB: Yeah, for sure. So would it be fair to say like in vanilla Postgres, you see this linear degradation with the size of the table. In TimescaleDB, is that essentially the size of the chunk? 

[0:17:04] BP: It's pretty much flat, I've found, is what you can see. And I forget if it was a Microsoft study or another one that did an actual independent test and benchmark of it, and we see that line is fairly flat. And the chunk size does matter more for probably query performance than it does for ingest performance. Because most of your chunks, unless you set like 30 days or two months for one and you have a ton of data, it's not going to grow substantial like you would see tens of millions or hundreds of millions in a single one. You're going to start to spread those out. And so you don't run into those same problems that you would see in normal Postgres. 

[0:17:39] KB: Makes sense. So another thing that I start thinking about when I think about time series style data is aggregates, right? I want to look at things aggregated by day or by week or maybe many different versions of that. And if I'm coming from a vanilla Postgres world, maybe I have some materialized views that are computing that for me, or I have a stored procedure that's happening or some application logic that's running this and dumping it into a table. How does that work in the TimescaleDB world? 

[0:18:07] BP: So taking the same concept of materialized views, which are a commonly used concept in relational databases, we've built an extension of that called continuous aggregates, which behind the scenes really are a materialized view. It's creating an additional hyper table, but it's aggregating data based upon - we have a function called time bucket. And I always forget the equivalent in Postgres. It's basically bucketing based upon a time range. And those are self-updating as well, based upon policies as users you set for how often they're updated, how far back you want to update that. That allows you to have an aggregate level view of your data that is automatically self-updating when any changes to the hypertable occur. And users will even often take those and build those hierarchical. So they build off of one another. 

So let's say you had a device that was emitting every minute and you wanted to roll up that with five minute, one hour, one day, one week. And you could have four different aggregates that build on each other that are scheduled automatically that update. And that allows you to have - it will take what could be hundreds of billions of rows. And then breaking that down into individual time buckets and get sub-low millisecond query times to be able to query against them. And we have customers that are in our Tiger Cloud environment that have 20,000, 30,000 queries a second that are hitting billions of queries a day against what I would consider nominal hardware, not crazy specialized hardware. 

[0:19:40] KB: Yeah. Well, that maybe is a good opportunity to start talking about Tiger Cloud, right? Because everything we've talked about to date, I could get open source, I put a plugin in my running Postgres. And as you highlight, I don't need a very fancy hardware to run this. Why would I go to Tiger Cloud? What do you all add there? 

[0:19:56] BP: Yeah, so Tiger Cloud has a couple of different features around it. One is just the overall managed service, the ability to take the pain off the team to run a managed service, much like you would have with RDS on AWS, which I use pretty extensively before coming to Tiger Data for our geospatial product. But within that as well, there's a couple of other features beyond that. So the ability to have read replicas, HA replicas, all self-managed by their team, having a connection load balancer that sits on the outside. So if you need to scale up, if a single instance isn't going to suffice across many different instances, the ability to scale across those many different read replicas as well. 

One of the other features I think that's worth calling out that makes a big difference for users that want to retain a long history of data without keeping all of that data hot is tiered data. And what it does behind the scenes is it takes a table. And let's say you have 10 years of history and you want the first two years to be in hot queryable and then you want the other eight years to be tiered. And that tiers that off to S3 in the AWS case and then it ensures that in iceberg format behind the scenes. But the entire time range is still queryable as one table. 

So when you query the table, if you do the first two, if you're expanding outside of that first two years, it's actually hitting and combining those results together to allow you to query that full 10 years. Now, if you think about if you had to do that manually, you'd have to have some archiving scheme and data pipeline that would go and move that data around. And there's a lot of that. Here you set a tiering policy. And it automatically starts tiering that data based upon that date. So you can have a much longer history of retention for longer query periods. 

[0:21:44] KB: That's fascinating. Well, and that gets into - I mean, there's all sorts of different things that people do right now in terms of piping data into warehousing or analytics stores or lake houses or all these different things. Are you moving in that space as well? Or is this staying constrained to these like timescale bucket? 

[0:22:01] BP: No, we're also moving in that space with a product that is due for full release later this year. It's still in beta right now, which is Tiger Lake. And that's to allow that bi-directional connectivity with other teams often where you would have that data in iceberg format, pushing data to Snowflake, pushing data to Databricks, and ability to use those other processes. 

And like I said, that's bi-directional. Maybe we are bringing data in where Tiger data or timescale becomes your operational database for your dashboards where you need really fast responsiveness. Maybe you have a data science team or another analytical team that's consuming that using Snowflake, Databricks, or Athena in AWS, and the ability to have that live in both locations. 

And I've even seen it too for compliance reasons. Organizations need multiple copies of that data in multiple places. And so instead of having to build that data pipeline themselves and kind of make that automatic where you just enable this feature and it automatically where based on the data that you've selected, it automatically starts writing using Tiger Lake out to S3. And we also have the equivalent on Azure as well. 

[0:23:09] KB: Yeah, super interesting. So I'm going to go a little bit more hypothetical now because we're talking about these different things and I'm starting to see the picture that this looks like, right? It's a lot easier to manage them. I mean, my current situation, right? I've got stuff in Postgres and a few different places, and then we're shifting things into AWS and have this whole processing pipeline. How do you see this changing the way that people manage their data? 

[0:23:31] BP: Yeah, no, I think the goal is to make it easier so people aren't focusing on the plumbing and all the work around it and are focusing on the business problems and solving them and enabling really multiple teams to be effective using some of the same data. And the intent is, is you can get that right depending on where your source of record is or where you're ingesting. If you're bringing it directly into Postgres, into Tiger data, and then being able to use that downstream and other processes. Maybe a team writes their data directly to S3. And then that did hold in and operationally enabled through Tiger Data. 

It allows for flexibility for how you want to work and gesture data and bring it in, but then making it exposed and sort of democratizing that for the team broadly, but not having to worry about all the plumbing and all the work that needs to be done, which AI makes it easier from a development perspective. But if you don't have to manage to do that yourself, you're focusing on using the data and leveraging it both in your applications and your other downstream analytical processes. 

[0:24:30] KB: Yeah. So we've talked about some of the plumbing that makes this possible, right? You have the hypertables and the hybrid row column stuff. Are there any missing primitives? Are there primitives that you all are still working on that you feel like are needed to realize that vision? 

[0:24:45] BP: And by primitives, I mean, we really support everything that's in Postgres today, right? So it is vanilla Postgres with these extension to be able to then partition the data. If there are other - you mean from a data sync perspective or other areas? 

[0:25:01] KB: I'm just thinking like we're talking about a future that - I mean, as someone who increasingly is like, "I just want to see the answers." right? I don't want to think about the details. I really resonate with your vision for like, "We'll focus on the business problem." Let's focus there. Hypertable's TimescaleDB was arising out of a thing that was a gap. Right? There's a gap already. Do you feel like with this, all the pieces are there and it's about execution? Or are there core underlying things that you all are working on that are like, "Oh, actually, we also need -" I don't know. I keep going back to non-time related partitions because that's what I was dealing with back when I was dealing with the scope of data. But are there things on the roadmap where you're like, "Oh, this is really going to open another door?" 

[0:25:40] BP: So I think it's more around enabling others to use these different things and some of the tooling around this to make it easier to consume and get set up. There are some pieces of the learning curve, nd we've spent a lot of time building out our CLI and MCP, as well as something we call PGAI guide, which is a guide that will guide coding agents to leverage all of the capabilities and tools very easily. 

And so I've even personally just building some side projects, building a home project for all my data telemetry for my entire smart home, and then using MQTT protocol to push that in is I built a majority of that just giving it a description of what I wanted in the coding agent using the CLI and MCP. And didn't even have to touch anything on the database side. And it built everything from the schema, the designs, the hypertable design, the continuous aggregate design, all of those things very well guided to get you set up in just a few hours. Right? And so up and running. 

And then tying that back into my home assistant and then a couple other things on top of that. But being able to do that very quickly and enabling someone without having t - you understand Postgres. And it's good to understand the concepts. But I think as we get more and more down this road, you want to focus on building and not necessarily understanding every technical concept. 

But from a roadmap perspective, it's making these things easier and more turnkey and ability to scale higher and higher because the needs of - what we're seeing is we focus as a business in the physical world all of these industrial applications. It's a growing - from data centers to solar, any number of things. The number of devices and the amount of data being emitted is just exploding and the volume of it. And so one, making that easier. And two, making that easier to scale. So you have that long history of data without having to sacrifice the value of something like Postgres and the developer kind of productivity around it. 

[0:27:37] KB: Yeah, that's really interesting. It reminds me of, yeah, conversations I've had with people doing like test infrastructure for physical world stuff. The volume of data is unreal to somebody coming from a web background. 

[0:27:50] BP: Yeah, it's very different, because I built a lot of web applications in my time. And you basically have crud operations with reading and writing. And it's a different problem space when you're you have something that's emitting a lot of time series data. And it's an entirely set of different challenges. And oftentimes, people will look at an LLM or Google it and their specific problem, whether it's write ingest, challenges, read problems or queries, and stumble upon something like TimescaleDB. 

And I'll be honest, before I joined, I didn't know it existed as a product. So to me, it was a new product. And so it was cool to see how they took the extensibility of it and built this specialized time series database without sacrificing all of the other capabilities. 

[0:28:37] KB: Yeah. So that's kind of interesting. So from that educational standpoint, we talked about some use cases earlier. But if I'm a developer, what are the things that are going to cause me where I should be like, "Oh, that I should check out TimescaleDB?" What are the problem spaces that maybe I would ask my agent about? And I should just have my head in the space of, "Oh, this is probably a timescale or a time series problem." 

[0:28:59] BP: We see a lot of greenfield applications as the sort of the visibility and kind of people are more familiar with it. I think anytime - because you're starting with Postgres, so you have all your relational stuff. But if you're building an application where you have any time series data and you know over time it's going to grow, I don't think you can go wrong in starting with it. 

And whether that's even local and building locally, take you five minutes to stand up a Docker image and run it local and test it out. So it's very straightforward with a single line curl install. But if you know you're going to have data that's going to emit a number of time series values, it makes sense just to start with that. 

On the flip side, if you're talking about brownfield applications and customers are starting to run into this issue, typically you're going to just have a significant volume of data. And maybe you don't even have any query challenges or ingest issues yet, but just the footprint. And we've seen this - I did a webinar with CERN a couple of weeks ago, and they were using a previous database before and had migrated over to TimescaleDB. And one of the things they called out, they had like 70 terabytes of data for time series across. This is like the Hadron Collider and all the other devices in CERN. 

And by moving over and getting the value of compression, they reduced that footprint from 70 terabytes down to 5 terabytes. They were able to then actually run that on higher speed SSDs, NVMe SSDs. So they just got better performance by default on all of their data because that 70 terabyte footprint that was spread out over maybe more disk, they were able to get on faster disk as well. So the compression and reduction in footprint allows you to have - using more commoditized hardware and get better performance, both in terms of throughput with multiple users hitting it, as well as sort of you have any ingest challenges if you have high volume of data as well. 

[0:30:55] KB: It's interesting, again, to think of like, "Oh, the data problems that CERN has, right? They're doing ridiculous amounts of just stuff coming off of every one of those runs and they've got to deal with that. It's pretty different than those of us building web apps. I guess a lot of work now moving into the AI space as well. It feels like there's so many more data points that we're trying to integrate tracing and observability and things like that. Are there any features aimed at agentic use of these tools? You mentioned kind of exposing it in the developer ecosystem, but what about other uses? 

[0:31:30] BP: This is one of the nice things as well. You don't have to have a separate vector database. You have pgvector. And we also have an extension pgvector scale that scales out as you get to larger data sets. But you have your vector embeddings right alongside all of your other data that could be a part of this hypertable and time series data. So you have the value of a time series database with a transaction relational and the ability to have vector embeddings stored right alongside of that. 

And maybe that's what you were getting to in terms of the other primitives as well. That's a common use case where you need that. And we also have authored something called pg_textsearch, which is an open source extension for text search. And we've seen it scale alongside what you would see comparable to Elasticsearch. And you can build a true hybrid search using a combination of text search and vector embeddings, and then combining those two together. 

So we've seen it in the telemetry space where oftentimes you have other metadata or other textual descriptions that you would want to search against to then get back to the underlying readings and the values of that. And so you have all of those technologies combined into one database where you often have to maybe split that over a vector database, a time series database, and then your relational and then have to join that application layer across all of them. You get that in one place. And then on top of that, the value of compression. So you get the reduction in footprint from a data standpoint. 

[0:32:59] KB: Yeah, it's interesting. I feel like every time I interview someone from a database company, one of my questions is, "All right, why you and not Postgres?" And your answer is, Yes. Why not? Let's do it with Postgres." 

[0:33:09] BP: Yeah, no, it's one of the reasons I joined too, because I just have loved Postgres just because the extensibility of it. And one is an open source product. And then the community is amazing in terms of how many other extensions. Because you can install and have those other extensions right alongside. So you're not limited. And we talk about extending privatives and other areas. You can just add those other extensions in. 

[0:33:33] KB: Yeah, that makes sense. Well, let's talk a little bit about what's coming down the pipe. What do you have on the roadmap? What should we be watching out for as we look at Tiger Data and TimescaleDB and all these things? 

[0:33:46] BP: I think it's continuing to build on the Tiger Lake story with connectors into Snowflake, connectors into Databricks, longer term as well. Those are areas we'd like to continue to expand on. And from a developer perspective, our goal is to make this sort of out of the box, not having to do the tuning exercises, making continuous aggregates easier, making the developer experience really world-class. And that's from both the database and getting set up, but also our tooling, the CLI and MCP, those other tools around it, not only for setup and configuration. 

But one of the questions we often get from developers is, "How do I know I'm set up correctly?" Validation, being able to validate that it's configured properly and with guidance based upon our SA team that's helping guide customers to be able to scale that as well. 

So one of the other areas is the common agentic workflows as well, where you have agents working against databases. We're internally working on an evolution to our storage engine called Fluid Storage, which is sort of a zero-copy type storage that allows you to create forks without having to copy the underlying data. And then for any changes, it will diff from there as well. 

And combined with our CLI and MCP, the ability to - in your CI/CD process or your agentic process, quickly fork those databases off until versions of it. So you can take your production database in seconds or minutes, be able to fork that and then have an agent that can work against that data and then stand that up and tear it back down very easily. And so you may have different flows from an agentic standpoint that just allow you to work against the original data without having to worry about degradating performance in your production system, still using production data, and then not have to worry about destructive operations as well. 

[0:35:41] KB: I mean, honestly, I'd love that just from a QA standpoint, right? We have a tooling to easily spin up application forks to test branches or things like that. But anything data intensive is still a pain because you got to get a copy of your production data and you may mutate it, whatever. And that takes a lot of time in most environments today. 

[0:36:01] BP: Yeah. And so it's all the tooling around that, which we already have the ability from the command line today to fork your database. But the challenge there, if you have a fairly large database, you have to copy that. And it's going to take some time to copy that. And if you're looking at EC2 with EBS spec data, it can take a while to move that stuff around and set it up. 

This new fluid storage system we've been building is going to be a game changer in terms of the ability to fork much, much faster or even spin up read replicas if you needed to scale your system, for example, for peak loads where you can scale to have multiple read replicas that are load balance behind. And you can kind of scale to peak traffic and then reduce those back down if you chose to. 

[0:36:44] KB: Yeah, super interesting. I think this developer experience direction is actually an interesting one to talk about. Where are the rough edges today? You talked about a lot of things you want to improve there. If I was coming in and using this, where am I going to get stuck as a developer right now? 

[0:36:58] BP: It starts with getting your chunk tuning right, first and foremost. And you need to get a good amount of data in before you can probably get that exactly right. I think we have some smart defaults. And in our cloud environment, we actually have an engine or a solution that will help recommend chunk tuning. And we want to move that back to the CLI and MCP. 

The other challenge oftentimes is getting continuous aggregates right. And in particular, in the physical world, you have these data patterns that are a bit different than you see in append-only type workloads. You have devices that go offline that may - in some cases, we have a ship carrier. Their devices could go upline for two weeks to a month. And then all of a sudden, you have all of this batch of data that can go back two weeks or a month. 

And so you think about these materialized views, they now have to update an entire range of data over a long period of time. And that can be CPU intensive, as well as disk I/O intensive. And so getting those refresh policies right, getting all those configured correctly. And one of the other features we're adding in the future is to solve this exact problem is granular refresh, which you can think about a tenant, could be a device, could be a user, whatever, when you have that backfill, we have one device is only updating that device that changed and not the entire time range. The way that continuous aggregates work today is that you update that entire time range within the aggregates that you have. And our ability to more finally granular address that is going to be an important piece. 

[0:38:36] KB: Yeah, that's interesting. So when you're doing these changes and config changes, how hard is it to, for example - say I want to change chunk size. We talked about that. Imagining back to my old, manual partitioning days, doing it for new data, not a problem. Migrating all the old data, extremely intensive and challenging. Do you take care of that for us? What does that migration path look like? 

[0:39:00] BP: So when you change the chunk size, it is changing it only for new ones, but we have functions as well for splitting and combining chunks that you can call manually to do that. Now, the challenge there is that - and we're working on reducing the locking operations around this, is that you have challenges with the locking. If you have DDL or reads or other things going on during that time, there can be some challenges there. But from a developer standpoint, you just call these functions to split and combine these chunks. 

[0:39:30] KB: Got it. I mean, the locking raises an interesting thing. If I'm doing this manipulation of the underlying structure, or configuration or things like that, what tools do I have for managing that? I think about that and I'm like, "Oh, okay, maybe I want to maintain this as a duplicate for a while and then wait till I'm ready and then swap over so that I don't have to deal with I'm locking it so I can't do anything while I'm dumping this out." 

[0:39:56] BP: You mean at the hypertable level or individual chunks? 

[0:39:58] KB: Well, it could be either, honestly. I'm just thinking about, once again, you're automating all of these things that I used to do by hand. When I run into something that's not working the way I like it yet, how do I dig into that? What are the next level debug for, "Okay, my chunk size isn't right, and now I'm migrating it, and I want to be able to migrate it without any downtime." What can I do as an external developer? 

[0:40:21] BP: There's a number of observability features built in, starting with our console to be able to see that. We also have on Tiger Cloud exporters for a number of different third-party tools, Datadog, Prometheus, solutions like that, that provide deeper insight into exactly what's happening in the system as well. And we're continuing to expand those. If we talk about the roadmap, we have all the Postgres metrics. But even getting into fine grain metrics around what's happening in compression policies, CAG or continuous aggregate refresh policies. Being able to see what's happening really behind the scenes as well. 

And Postgres has a large number of metrics that you have that are exposed to you to really dig in to see what's happening in the system itself, as well as all of the SQL. The SQL is no different. So if you're familiar with Postgres, this is not a learning curve other than the specific areas around hypertables and continuous aggregates. 

[0:41:20] KB: Right. Well, and can I Postgres to query down a level below your sort of high level of abstraction? Query a particular chunk or something along those lines? 

[0:41:31] BP: There are functions that you can find in our API docs to access specific chunks for certain operations to see what's happening below the scenes. Those APIs are provided in our API docs. So if you really want to get into the weeds and dig into exactly how some of the sausage is made, it's there. And additionally, too, it's open source product. So you can go download the source code and take a look at it and see exactly all the APIs and how it's actually constructed. 

[0:42:01] KB: Yeah. No, that's awesome. That's a really nice kind of continuous story. For those of us who get nervous, trusting all of the different data providers out there. And I'm like, well, you work great by default, but what happens when something goes wrong? It's a nice extension of the Postgres story, right? Postgres is great for debuggability because it has all these internal tools and metrics. And so just having that there, that's great. 

Well, is there anything we haven't talked about yet that you think would be important to leave folks with? 

[0:42:28] BP: The other area we have, and I mentioned so far two offerings. I mentioned the open source offering with TimescaleDB, and I mentioned Tiger Cloud. And as we grow in the physical world, in industrial operations, we're also building a product called TimescaleDB Enterprise, which is an on-prem version of the open source solution that uses a number of technologies with its own console and platform to be able to stand up an HA environment with read replicas, with connection, load balancing, backups, automating a lot of the things that if you were going to build yourself, you'd have to start with the open source product, add SCD, Petroni, other technologies on top of that to build a true HA configuration. We're providing that out of the box as well. 

And we also, as part of that, have something called our Postgres Connector. For developers that are interested in trying out, let's say, TimescaleDB or Tiger Cloud, you can use our Postgres Connector to point it at your database, replicate the tables you want live, and it keeps it, and it will go through and play back the wall and then actually write the data and then keep those in sync. You can actually test it in parallel to your existing production database. And so that provides a really easy evaluation path. Let's say you don't want to actually download and do a pg_dump and pg_restore and take all the time to do that. You can just set up the pg connector. 

That's also leveraged in this Timescale Enterprise DB story where you have multiple levels. I'll give one example. We're working with data center operators where, within a building, they have these different pods. And there could be a bunch of different GPUs in the area. A pod rolls up to a building, rolls up to an enterprise. You can replicate data across multiple tiers using this connector as well. 

So you have your operational layer all the way down to that pod where you're monitoring, and maybe you have a group that's monitoring the entire industrial operation or the entire data center. We can replicate that data up. It's a number of different technologies that sit on top of TimescaleDB, the extensibility to run it on-premise with an HA configuration, as well as this tooling to make it easier to replicate data from vanilla Postgres or TimescaleDB to a TimescaleDB instance. 

[0:44:55] KB: I mean, that feels very, very nicely consistent with what you were talking about in terms of just wanting to kind of make the plumbing problem go away for folks where this is a problem we all have if we're managing lots of data or we're looking at putting data in different places. It's like, "How do I get it from here to there? How do I do that in a way that's not disrupting my operations?" Yeah, that makes a lot of sense. 

So if we were to circle back, people getting started, probably just like, if I want to install this locally, what version of Postgres do I need? Do I need to be on one of the most modern versions of Postgres to put TimescaleDB in? 

[0:45:30] BP: We've been 15 through 18. Recently, they released actually that just came out, and it's coming out on GitHub today, which is I think it's 2.29 deprecates PG15. So you got 16, 17 and 18. And we'll be adding 19 soon when it's coming out. We try to keep up to date on that. So you can just Google TimescaleDB GitHub. There's a one line command, curl command you can do to get running locally. You can also go to tigerdata.com, sign up for an evaluation. We have a free 30-day evaluation with a number of credits, so you can go ahead and get started and try that out. 

I recommend, too, using the tool, the CLI and MCP, as well, to get started. It's going to guide you through, and it's going to be through migration process, guide you through getting set up and configuring it. And what I've done is I've really just documented my use case. What do I want to build in a coding agent? And then use these tools. And it's going to use the best practices that we've put together in our documentation to build out the schema, to implement the features and get you up and running. And it can be a pretty quick turnaround to be able to do that. 

And then load a lot of data and really put it through its paces because I think you'd be pleasantly surprised. And then leverage the value of our column store to compress data, to reduce the data footprint. All of those things are pretty well sort of guided through our tooling as well. As on our console as well, we have different wizards and things to walk through it. 

Depending on if you're a developer that lives in the command line, stay there. And we have those tools to get you there. If you're less technical and you want to use our console and go through the Tiger Cloud and use that with our SaaS console, that's another path as well. So easy path to get started, but try it out. And I think people will find a lot of value in it, especially those that have the need for that time series data that's growing, which is I think a common path for people. 

[0:47:27] KB: I love the CLI and MCP is like the first weigh-in, right? Because increasingly, we're all using coding agents to explore all of these tools and treating that as a kind of first-class expected approach and saying, "Okay, how do we cater to that? How do we make that go on the rails rather than making assumptions it shouldn't?

[0:47:44] BP: Aand it's beyond even getting started. We're building further tooling to really debug and validate and show slow queries, things that you could optimize within a system as well. And I think I see even the ability to build independent agents that are monitoring your database, monitoring the system, making recommendations in real time, above and beyond just the initial development out of the box, all using the same tools behind the scenes. 

[END]