What is database seeding?
Database seeding is an essential yet often overlooked aspect of setting up a productive development environment. Despite the critical role quality data plays in software development, it’s frequently an afterthought.
In essence, database seeding is the practice of preparing a development database with data that ought to resemble the state of the data at the code’s runtime. So typically, this means seeding your local development database with data that looks like and behaves like production data.
Why use seed data?
Seed data, also known as generated or dummy data, is crucial in software development, offering an initial set of data for databases. It serves multiple purposes:
- Development: Provides consistent and representative datasets for developers.
- Testing: Ensures predictability when verifying features.
- Demonstration: Showcases your software's abilities with example content.
- Default Data: Includes necessary built-ins like country lists.
- Onboarding: Offers filled-out starting points for new users.
- Performance Testing: Helps simulate high-usage scenarios.
- Guides: Assists in tutorials and documentation.
How to obtain seed data
There are several popular methods for obtaining seed data:
Manual data insertion
The first and easiest option is to simply manually insert data into your development database by hand. This works if you only need a few rows of data, but anything more than this and it’s not particularly realistic to do so. The data won’t look or behave realistically, and will lack the sophistication and relationships of real data.
Seed script
Another popular option is to write a seed script, which uses a series of inputs to populate a database with data. The quality of your data is only as good as your seed script, and it’s hard to cover all the edge cases. Seed scripts are also difficult to maintain over time, and tend to produce data that’s not very realistic compared to real data.
Synthetic data generation
Synthetic data generation tools are another option, and are capable of producing large quantities of data. The quality of this data is dependent on both the tool and time spent configuring it, and while it’s possible to achieve high levels of realism, these tools are often expensive and complicated to configure.
Anonymized production data - pg_dump
Another method that is popular specifically because it offers very realistic seed data, is to grab a dump of data from production, using something like pg_dump. This is very risky though, as production data contains personally-identifiable information (PII) which needs to be anonymized before it can be safely used.
Using production data without anonymizing it is often in contravention of security best practice, not to mention privacy legislation like GDPR. As a result, access to production data is often tightly restricted in a team environment, and not every developer may even be able to access production data to begin with.
The difference between low and high quality seed data
When developers write or test code using seed data that is more realistic and more closely resembles the data in their production environment, they make fewer assumptions, which results in more realistic code that is higher quality, and has fewer bugs. Fewer bugs mean less rework, and less frustration.
If coding against better quality, more realistic seed data is so good, why don't developers do it all the time? There are several reasons why developers often compromise and code against poor quality seed data:
- Developers often believe they understand the structure and intricacies of production data, and believe that a simple seed script can adequately simulate the data. However, these assumptions are often incorrect, and validating them is difficult. This results in unforeseen challenges when shipping to production.
- Developers use seed data because there’s no compelling alternative - directly coding against actual production data isn't feasible. There are legitimate privacy concerns, regulatory restrictions, intricate workflow complications, and the challenge of managing massive database dumps make it an impractical solution. Production data and production schemas are also dynamic and constantly changing, and even if it’s possible to obtain a santized database dump this week, it can be inaccurate and unrepresentative the following week.
Snaplet for seed data
This is the problem we’ve set out to solve with Snaplet - how to seed a database with better data in a way that doesn’t require a lot of ongoing effort.
How you use Snaplet depends on you - there’s two key workflows that are dependent on whether you have production data already that you can access, or not.
Snaplet snapshot: If you have production data already, and you can access your production database, Snaplet can securely connect to your database, capture a subset of this data, anonymize any sensitive data (personally-identifiable information for instance) and output a snapshot that you can restore to any database to code or test against, locally or in the cloud. Snapshot includes an automated workflow that you can setup once and use to get high quality, production-realistic data to code against on an ongoing basis. Think of Snaplet snapshot as a better and safer pg_dump.
Snaplet seed:What if you don’t have access to a production database, or you’re on a greenfields project that has no data? What if you need to create specific data for your testing? Enter Snaplet seed, our automated data generation tool that helps you get better quality, more accurate seed data in a snap. Snaplet seed uses your schema to intelligently create more realistic seed data you can use for coding or tests. Think of Snaplet seed as a better and easier seed script.
Getting started with Snaplet
Snaplet offers a more accurate, realistic, and easier approach to seed your database, whether for local development or as a replacement for cumbersome seed scripts. Try Snaplet for a safer, better way to manage your database seeding needs.
