Integration 101: Introduction & Best Practices

The Internet of Things (IoT), the Cloud, AI, data lakes, automation, and every other 2019 IT buzzword all stem from the same belief: connected data is more powerful. The more transparency between your systems, the easier it is for your teams to collaborate, plan, analyze, and deliver. Integration is the lifeline that makes these initiatives a reality.

Businesses use integrated systems to transcend the limitations of a single source of truth and create a truly comprehensive view of their company. Throughout this series, we’ll explore the many components of integration and how they apply in business scenarios. To start, we’ll intro the types of data integration, use cases for common integrations, and best practices for successful integration.

The Types of Data Integration

Integration manages data in motion. The way you choose to integrate your systems depends on several factors:

• How many and what types of systems you are connecting
• How much data you are integrating
• How often you need to move the data
• The ultimate goal of the integration

Here are a few types of data integration to be aware of, and the use cases for each:

Batch Integration: This type of integration groups the data and sends it periodically (in batches) from the source to the target. Compressing the data in this way helps save network bandwidth. Data is collected, entered, processed, and then produced in the batch results for integration. Intervals can be set for daily, weekly, or monthly. Batch integration is fast because data passing to the destination doesn’t have to be processed immediately for the integration to continue running, making each transaction quicker. This style of integration is efficient for processing high volumes of data that must be loaded quickly, but whose transactions take place over time. For example, batch integration would be ideal for payroll and billing systems. Batch integration is also good for archiving or documenting, where it’s not crucial for the information to be updated right away.

Real-time: Real-time integration involves continual input. Unlike batch integration, the data load is not scheduled, but rather spread across the day to keep the flow as current as possible. This type of integration is best for businesses that want the data instantaneously available. For example, a credit card company may prefer real-time integration to detect fraudulent charges as soon as they happen.

Big Data: The term “big data” implies large volumes of data, which is exactly what this type of integration is referencing. Big data integration consolidates data first and then processes it across the source data, only integrating the results. If you have various types of data from various technologies, this may be the style of integration to consider. However, the process is much slower and may be more expensive than batch or real-time integration.

Data Virtualization: Data virtualization involves using several types of integration techniques to consolidate data that’s being pulled from various technologies/sources in different formats. Data virtualization’s main use is for analysis across historical snapshots of data. With real-time or batch integration, the data is constantly changing, making the analysis difficult. In a sense, this style is more meant for data warehousing than anything else.

Which Systems Should I Integrate?

Each type of integration has its own advantages but knowing which integrations you need depends on the goals of your business. Are you a call center that wants to improve customer service? Are you a public sector business looking for better ways to manage time and service communities? Are you a private enterprise that needs fast inventory assessments and monitored distribution? The integration possibilities are exponential. Some of the most common integrations occur between ERPs and CRMs, but other popular connections include:

• CRM and Email: Sales reps love this integration because it makes logging customer interactions faster and simpler.
• CRM and Social Media: This one is great for marketing teams, who can track which users engage with you and how.
• Marketing Automation and CRM: Marketing professionals appreciate the ability to maintain up-to-date marketing lists and segment audiences with minimal effort.
• ERP and Warehouse management: This integration is huge for helping warehouse managers track inventory, ordering, billing cycles, and more.
• ERP and eCommerce solutions: Companies turn to this integration to end the time-consuming task of manually entering orders from their website into their ERP.

No matter which systems you’re integrating, it’s important to go into the project knowing what your integration options are, and the best practices for both client and integrator to ensure success.

When Starting the Integration Process…

There are a couple of things to keep in mind going in to the integration process:

1. Integrations can be technically complexEvery integration is different because every business is unique. You need an integration to work the way you work, with proper fields and reportable data. Have an open discussion with your team and integration partner to understand the complexities of your specific integration to make sense of delivery timelines, special fields or functions, and the capital necessary for the desired outcomes.
2. Data quality is crucialA recent study by the Institute of Electrical and Electronics Engineers (IEEE) found data discovery (organizing your data stores and creating an inventory of the information you have) takes up more than 40% of the time spent on the average integration project. That means most integrating businesses don’t have a true understanding of the data they have and how they use it. It is so crucial to integrate with data that is clean, organized, and streamlined. Have you checked for duplicate entries and missing data? Do you know all your system customizations and where data resides within those? Any misalignments of data between system records?An iPaaS (or integration-platform-as-a-service) solution will usually offer the option for automated data cleaning — which can help accelerate this initial process — but the human element is still necessary to answer questions on the data’s usefulness and viability.
3. Business users drive value tooData integration systems were designed with IT professionals and developers in mind. However, it’s the business users who will make the greatest impact on core activities by using the integration, so their value cannot be forgotten. Assemble a team of these users and ask them to provide their perspectives throughout the integration project.

The right integration tool should give users more control over their data and be manageable for the “average” person.

Best Practices for Successful Integration

A successful integration requires the client and the integrator to work together. Each side must contribute their expertise to the process – the client needs to clearly communicate the needs and desired outcomes of the integration and the integrator needs to advise on the best approach and the most cost-effective options. As the client, keep these best practices in mind:

• DO create an internal integration team representing all stakeholders: data users, decision makers, IT dept, database management professionals, etc.
• DO allocate appropriate resources and budget
• DO ask your integrator how to deal with data quality & cleaning
• DO look for tools that both IT and business users can adapt to
• DO make sure your integrator offers proper security protocols
• DO choose a provider with a scalable platform for on-premise, hybrid, and Cloud deployments
• DO be prepared for the integration testing to cause some errors (your integrator will help you mitigate)
• DO make your integrator aware of any special fields or functions you want carried over
• DON’T think all integration solutions are the same
• DON’T expect overnight success; integration takes time and attention to detail
• DON’T skimp on testing
• DON’T start an integration project without a plan
• DON’T restrict your internal team to management only. Ask for input from your everyday users to help discover functionality gaps.
• DON’T try to integrate too much all at once. Prioritize based on your most pressing business goal. Once that integration is running smoothly, start thinking about the next goal.