Starting in Starfish ETL version 2.3, we've created a method to allow the creation of custom Connectors.
What is a Starfish Connector?
Simply put, a connector is one of the items on the Origin/Destination tabs. It allows Starfish to make a connection to a data source and read or write data to it. So for instance: SQL Server, Salesforce, SData, OLE DB are all connectors that are already included with Starfish. Connectors perform a common set of operations, allowing them all fit within the Starfish mapping and execution framework. To make something work with Starfish, it will need to be able to perform certain actions like lookup results and create or update records. Each connector does this differently because a SQL databases handles inserts differently than a Salesforce instance. However, the way these commands are called are always the same from Starfish - passing in a request and expecting a specific kind of response.
Why would I want to build my own connector?
Perhaps you'd like to set up an integration to a custom web service via SOAP or REST. You could build a native connector for a database platform that we don't yet support. A connector could be built to another ERP or CRM system, Legacy system or a COM-based interface. Here are a couple of more creative options.. You could treat your file system like a database, with tables representing folder and fields representing filename. Or treat a custom service call like a database by having tables represent your methods and fields represent the method parameters. So when the CreateRow() function is called, your method is called with specific parameters.
How do I build my own connector?
A Starfish Connector is built like a plugin. You'll start by downloading the SDK. This includes sample code and a project template for your new connector. The project built in Visual Studio 2010, as a C# Class object against the .NET 4.0 framework. You may convert this to VB.NET if you'd feel more comfortable with that.
Your class object inherits an Interface called IStarfishConnector. You'll set some properties and populate the code for about a dozen or some methods. When you're done, you build the project and place the resulting DLL in the Starfish bin folder. Next time you load Starfish Admin, it will automatically find your new connector and add the tab to both the Origin & Destination placeholders.
Here are a few examples of the methods you'll have to populate:
- GetTableList() - return a list of tables/entities available on the system
- GetColumnList() - given a table name, return a list of the columns/fields available
- GetQueryCount() - given a table name and set of criteria, how many rows are returned?
- CreateRow() - given a table name and list of values, create a new record
To find out more, download the SDK and look through the template project. Psuedocode is provided for each method, along with a definition of each parameter.
Included in the SDK is the full source code for our SQL Server connector for your reference. This connector utilizes the .NET Data.SqlClient namespace. Feel free to use this code as a starting point instead if you're implementing a connector based on an ADO.NET driver. Our MySQL and Oracle connectors use very similar code.
You may review the SDK Documentation online or in the downloaded file in .chm format.
NOTE: After downloading, you may have a problem reading the Starfish ETL Connector SDK.chm file. (message such as "Navigation to the webpage was canceled") To fix, right-click the file and go to Properties. Then click the "Unblock" button. Click OK and re-open.
If you have any questions about building connectors, possibilities or limitations, ownership, etc.. please feel free to post in the Connector Development category on this forum or email us directly.
Login or sign up for our forums to connect to the user community.