Ingestion Methods

Available Table Ingestion Methods

Full Data Sync

  • The entire contents of the table is pulled every time on a regular schedule (with a minimum frequency of 1 hour).

  • This is the least preferred method as it is accounts for a very large amount of data processed and is the heaviest performance hit on the server.

Incremental Column Data Sync

  • A column with a date or integer value is used to track data changed since the last scheduled query (usually every 5 to 60 minutes).

  • Your application is responsible for updating that column with a new timestamp or value any time the data in that row changes.

  • Typically, this method only tracks Insert and Update changes. Delete changes are often not tracked unless they are performed as Updates with null data for all columns.

  • It is highly recommended that you create an index on the column used as the incremental column. This is especially important on very large tables (500K+ rows) to ensure ingestion happens quickly.

Change Tracking (CT)

  • A light-weight method built into some database engines using triggers that allows Verb to query only rows that have changed since the last query.

  • This removes the need for your application to update an incremental value on the updated row.

  • Generally, this tracks Insert, Update and Delete changes.

Change Data Capture (CDC)

  • A powerful transaction log driven function built into some database engines that uses commit logs or table-valued functions to enable querying of data that has been changed, added, or removed.

  • As with CT, you will not need to maintain an incremental column.

  • Tracks Insert, Update and Delete changes.

Last updated