Data Ingestion

Field

Sample

Description

ingestion:

drop_if_create:

True/False

Not used by these connectors

catalog_step

True/False

Creates a csv file with possible attributes from the client system. If set to True, the data pull does not occur. This property only needs to be set true in the first ingestion run to identify possible attributes.

staging:

True/False

Determines whether to pull data to autoid_staging or autoid_base keyspaces. If staging is set to True, the system pulls to autoid_staging keyspace; while set to False, it pulls to autoid_base keyspace. The autoid_staging keyspace lets us pull data without on-the-fly transformations, while also supporting pulling from multiple sources.

connector:

type:

csv, iiq, oim

Connector for type of source data. Currently only supporting three options.

batchsize:

1000

Integer. Specifies the number of rows or pages pulled as part of each batch.

timeout:

10

Integer representing seconds. Specifies the amount of time to wait before timeout.

change_reconciliation:

enabled:

True/False

Determines whether to pull data that filters only results that been modified after a certain date, specified by the time property.

time:

'2013-05-17T00:00:00Z','2020-05-17 00:00:00'

Specifies the data-time after which the data should be pulled. Note IIQ requires a time zone (first value).

features_mappings:

chief_yes_no

'User:CHIEF_YES_NO'

Value take from the csv generated in the catalog step. Note that chief_yes_no is not required, but is included here as an example. For reference, see Appendix: analytics-init-config.yml.