Generating salesforce big object test data means designing around index fields first and record volume second, because Big Objects skip standard relationship enforcement and query entirely through their defined index. Most teams treat Big Object test data like any other bulk load and get query timeouts or empty result sets in return. The fix isn't more records. It's matching the index structure before a single row goes in.
Big Objects exist for one reason: storing hundreds of millions of records without touching your standard data storage limits. Field History Archive, Data Retention Policy data, and custom archive objects all live here. But that storage model comes with trade-offs that directly change how you generate test data for them, and most of the usual rules for standard object test data don't apply.
How Big Objects Differ From Standard Objects in Test Data Terms
A standard object record gets inserted, updated, deleted, and queried through SOQL with full relationship support. A Big Object record gets inserted once and queried only through its defined index. There's no update in the traditional sense — you insert a new version keyed to the same index values, and there's no delete through the UI at all. Bulk deletion requires a batch job calling Database.deleteImmediate.
This changes your test data strategy immediately. You can't generate a record, run a test, then mutate that record and run another test. Every test scenario needs its own fully-formed record inserted fresh, with index values that won't collide with anything already loaded. Collision isn't a performance problem here. It's a correctness problem, because overlapping index values on a Big Object produce duplicate rows that your query logic was never built to filter.
Relationships are the other gap. Big Objects support lookup-style fields to standard objects, but there's no referential integrity enforcement and no cascade behavior. If you generate a Big Object record pointing to an Account ID that doesn't exist, Salesforce accepts it without complaint. That's convenient for speed and dangerous for test validity — your data will load clean and still be wrong.
Index Field Design Dictates Your Data Generation Order
Every Big Object has exactly one index, defined at creation time across up to four fields, and it cannot be changed afterward without deleting and recreating the object. Query performance depends entirely on filtering by that index in leading-field order, the same way a composite database index works. Generate test data without respecting that order and your queries will full-scan, even on an object built specifically to avoid full scans.
Say your index is defined on Account_Id__c, Event_Date__c, and Record_Type__c, in that order. A query filtering only on Event_Date__c won't use the index efficiently. Your test data generation has to account for this by distributing values realistically across the leading field first. If every test record shares one Account_Id__c value, you've built a dataset that can't validate index performance at all, regardless of how many rows you load.
I'd go further: test data for Big Objects should be generated with index cardinality as the primary design input, not record count. A thousand records spread across 50 distinct leading-index values tests query behavior far better than 50,000 records under a single value. Teams that chase raw volume without varying the index end up with datasets that look big and prove nothing.
Loading Big Object Records With Bulk API v2
Big Objects accept inserts through Bulk API v2 using the same job-based model as standard objects, with one meaningful difference: there's no upsert operation. Every load is an insert, and because Big Objects don't enforce uniqueness outside the index, duplicate protection has to happen before the file reaches the API.
A typical generation and load sequence looks like this:
- Pull the Big Object's field metadata and index definition through the Tooling API or Setup.
- Generate index field values first, deliberately varying the leading field to match expected production cardinality.
- Populate non-index fields with schema-compliant values, respecting field length, required status, and picklist constraints.
- Chunk the CSV or JSON payload into Bulk API v2 batches, typically 100,000 to 150,000 records depending on row width.
- Submit the job with the correct object API name and monitor for partial batch failures tied to malformed index values.
One detail trips up a lot of admins: Bulk API v2 jobs against Big Objects don't support the same monitoring granularity you get with standard objects in some org configurations. Row-level failure reporting can lag, so it pays to validate a small batch of 500 to 1,000 records first, confirm they're queryable through the index, and only then scale to the full volume.
Simulating Realistic Volume for Query Performance Testing
The whole point of putting data in a Big Object is querying it fast at scale, so test data that doesn't reach a representative volume defeats the purpose. If production will hold 80 million archived records, testing against 10,000 tells you nothing about whether your index design holds up under real load.
Here's a rough volume comparison worth planning around:
| Test Scenario | Minimum Useful Volume | Primary Risk if Underfilled |
|---|---|---|
| Index field selectivity check | 500K+ rows across varied index values | False confidence in query speed |
| Archive job performance (e.g., Field History) | 5M+ rows | Batch timeout not surfaced until production |
| Async SOQL validation | 1M+ rows | Query plan never triggers full-index scan path |
| UI component rendering against Big Object data | 10K-50K rows | Pagination and governor limit bugs missed |
Generating millions of rows by hand or through Apex loops isn't realistic — Apex governor limits and heap size make that approach fail long before you hit useful volume. This is where Bulk API v2's parallel processing matters: it's built to move high row counts without the per-transaction ceilings that Apex-based generation runs into, which makes it the only practical path to Big Object volumes that mean anything.
Common Failure Points When Generating Big Object Test Data
Index value collisions top the list. Two test records sharing identical values across every index field don't error out, they silently become indistinguishable from a query standpoint, and your test results start reporting record counts that don't match what you expect.
Field-level type mismatches are next. Big Object fields support a narrower set of data types than standard objects — no multi-select picklists, no roll-up summaries, limited text area options. Test data generated from a standard-object template without checking the Big Object's actual field types will fail on load with type coercion errors that are easy to miss in a large batch job.
Timestamp precision causes quieter problems. If your index includes a DateTime field, test data generated with identical or near-identical timestamps across thousands of records clusters your index unnaturally. Production data rarely lands with millisecond collisions; generated data often does, because naive generation scripts reuse a single timestamp or increment in large uniform steps.
Last, teams forget that Big Objects can't be queried with standard SOQL in all contexts — some require Async SOQL for full scans. Test data validation scripts written against standard SOQL assumptions will report false negatives, suggesting your data didn't load when it actually did.
Where SproutEzee Fits the Big Object Workflow
SproutEzee reads your org's schema directly, including custom Big Object definitions and their index field order, and builds generation logic around that index rather than treating every object the same way. For Big Objects specifically, it varies leading-index values deliberately instead of defaulting to sequential or repeated values, which matters more here than on almost any other object type.
Because the platform pushes data through Bulk API v2 natively, it handles the chunking and job sequencing that Big Object loads require without manual CSV splitting. That includes respecting field type constraints pulled straight from schema metadata, so a text area that's valid on a standard object but unsupported on a Big Object gets flagged before the job submits, not after a failed batch.
For architects validating archive strategies or Field History Archive migrations before a production cutover, that combination of schema-aware generation and bulk loading cuts the setup time from days of script-writing to a configuration pass against the actual org metadata.
Frequently Asked Questions
Can I use Apex to generate test data for Salesforce Big Objects?
You can, but it's a poor fit for anything beyond small validation sets because Apex governor limits and heap size cap how many records you can build and insert in a single transaction. Bulk API v2 bypasses those per-transaction ceilings and is the practical route for generating Big Object volumes in the hundreds of thousands or millions. Reserve Apex-based generation for quick smoke tests of a few hundred records.
Why does Big Object test data need to vary index field values deliberately?
Big Objects query exclusively through their defined index, and query performance depends on how selective the leading index field is across the dataset. If every test record shares the same leading-index value, queries can't exercise the index path the way they would against real, varied production data. Deliberately spreading values across the leading field is what makes performance test results meaningful.
Does referential integrity get enforced on Big Object lookup fields?
No. Big Object lookup-style fields accept any ID value, including one pointing to a record that doesn't exist, without raising an error. This means test data generation has to validate relationships at the generation stage rather than relying on Salesforce to reject bad references on load.
How do I delete or reset Big Object test data between test runs?
There's no standard delete through the UI for Big Object records. You need to run a batch job using Database.deleteImmediate against the specific records, typically identified by their index field values. Plan your test data generation with this in mind by using index values you can cleanly target for cleanup afterward.
What's the biggest mistake teams make generating Salesforce Big Object test data?
Treating record count as the main success metric instead of index cardinality and field type compliance. A dataset with a million rows under one index value tests nothing useful, while a smaller dataset with properly varied index values and correct field types gives far more reliable performance and validation results. Volume matters, but only once the structure underneath it is right.