Ingestion Methods
- The entire contents of the table is pulled every time on a regular schedule (with a minimum frequency of 1 hour).
- This is the least preferred method as it is accounts for a very large amount of data processed and is the heaviest performance hit on the server.
- A column with a date or integer value is used to track data changed since the last scheduled query (usually every 5 to 60 minutes).
- Your application is responsible for updating that column with a new timestamp or value any time the data in that row changes.
- Typically, this method only tracks Insert and Update changes. Delete changes are often not tracked unless they are performed as Updates with null data for all columns.
- It is highly recommended that you create an index on the column used as the incremental column. This is especially important on very large tables (500K+ rows) to ensure ingestion happens quickly.
- A light-weight method built into some database engines using triggers that allows Verb to query only rows that have changed since the last query.
- This removes the need for your application to update an incremental value on the updated row.
- Generally, this tracks Insert, Update and Delete changes.
- A powerful transaction log driven function built into some database engines that uses commit logs or table-valued functions to enable querying of data that has been changed, added, or removed.
- As with CT, you will not need to maintain an incremental column.
- Tracks Insert, Update and Delete changes.
Last modified 2yr ago