Ingestion Methods
Last updated
Was this helpful?
Last updated
Was this helpful?
The entire contents of the table is pulled every time on a regular schedule (with a minimum frequency of 1 hour).
This is the least preferred method as it is accounts for a very large amount of data processed and is the heaviest performance hit on the server.
A column with a date or integer value is used to track data changed since the last scheduled query (usually every 5 to 60 minutes).
Your application is responsible for updating that column with a new timestamp or value any time the data in that row changes.
Typically, this method only tracks Insert and Update changes. Delete changes are often not tracked unless they are performed as Updates with null data for all columns.
It is highly recommended that you create an index on the column used as the incremental column. This is especially important on very large tables (500K+ rows) to ensure ingestion happens quickly.
A light-weight method built into some database engines using triggers that allows Verb to query only rows that have changed since the last query.
This removes the need for your application to update an incremental value on the updated row.
Generally, this tracks Insert, Update and Delete changes.
A powerful transaction log driven function built into some database engines that uses commit logs or table-valued functions to enable querying of data that has been changed, added, or removed.
As with CT, you will not need to maintain an incremental column.
Tracks Insert, Update and Delete changes.
