Full Refresh - Overwrite
Overview
The Full Refresh modes are the simplest methods that Airbyte uses to sync data, as they always retrieve all available information requested from the source, regardless of whether it has been synced before. This contrasts with Incremental sync, which does not sync data that has already been synced before.
In the Overwrite variant, new syncs will destroy all data in the existing destination table and then pull the new data in. Therefore, data that has been removed from the source after an old sync will be deleted in the destination table.
Example Behavior
On the nth sync of a full refresh connection:
Replace existing data with new data. The connection does not create any new tables.
data in the destination before the sync:
| Languages |
|---|
| Python |
| Java |
new data:
| Languages |
|---|
| Python |
| Java |
| Ruby |
data in the destination after the sync:
| Languages |
|---|
| Python |
| Java |
| Ruby |
Note: This is how Singer target-bigquery does it.
In the future
We will consider making other flavors of full refresh configurable as first-class citizens in Airbyte. e.g. On new data, copy old data to a new table with a timestamp, and then replace the original table with the new data. As always, we will focus on adding these options in such a way that the behavior of each connector is both well documented and predictable.