Expert Integration Tips: Setting Up Your Origin

Starting an integration project in StarfishETL? We have some tips to ensure your project runs with maximum effectiveness. In the first post of our Expert Integration Tips series, we'll talk about the various approaches you can use to set up your Origin connection.

Regardless which filtering system you set in place, remember that for optimal integration performance and safety, you should only process the records you NEED to process each time StarfishETL runs. As you determine the settings for your Origin connection (aka the system you are moving most of your data from) think about which of these four ways would make sense for limiting your records:

1. Filter by the Date of the Last StarfishETL Run

Depending on your integration needs, this approach may be ideal for your integration. However, be warned. If you use this approach, time changes could affect your outcomes. Take daylight savings time as an example. What happens if your integration server clock is adhering to daylight savings time, but the Origin server clock is not? Now you have each end of the integration trying to run an hour off, and possibly running records that shouldn’t be run, or missing records it should have caught. Don’t forget to sort by ascending date to keep everything in order.

2. Filter by the Date of the Last Record Processed

This approach tracks the date modified or date created of the last record you processed. To do this, it is crucial that you sort ascending by the date you are tracking. One thing to be aware of when using this strategy is the potential to run into issues if there are multiple records created or modified at the exact same time.

3. Filter for Records that are Marked as Unprocessed

For this to work, your Origin system must mark a record as “unprocessed.”  This can be accomplished by telling the Origin system to set the unprocessed field to true each time the record is modified.  When StarfishETL processes the record, have StarfishETL set the Origin record’s unprocessed field to false.

4. Run all records but check for changes in the record

If there is no date to track, you must run all records. You can cut down on the number of records actually run by using row hashing. Row hashing ensures StarfishETL only processes records that have changed. A “Row Hash Code” variable is available through the StarfishETL platform.

No matter which strategy you use for setting up your Origin, remember this one important rule: Only pull the fields you absolutely need. If you are reading Accounts, you may find 200 fields in that entity. Do you really need all 200 of those? If you only use 10 of them, be sure to set up your Origin to pull only those 10 fields. It can also be helpful for testing and diagnosing issues if you sort your Origin even if you aren’t using a filtering method that requires it.We often sort by date modified, date created, ID or Name.These are best practices for a safe and efficient integration. For more tips on integration, or to get help with your integration set up, contact us at info@starfishetl.com.