Python & ETL: A Great Match

written by
Danine Midura
authored on
November 15, 2019
header image

According to GitHub’s 2019 State of the Octoverse report, Python is now the second most popular programming language, overtaking the scripting behemoth, Java. Python is used for developing websites, Web applications, and GUI (graphical user interface) for desktops.

It’s popular among data scientists and is a great match with ETL (extract, transform, load) because it makes it easy to simplify complex ETL and data engineering tasks.

Python also is popular because of its robust Standard Library that offers an array of modules and allows users to build custom applications without writing additional code. The syntax rules Python uses place an emphasis on code readability, so it helps keep the code clean and easy to maintain over time.

Python supports a wide range of major platforms and systems (just like StarfishETL) so users can work in the operating system they prefer. Its open source framework and tools let the user optimize to his or her precise needs and cuts down on development time.

Python & StarfishETL

StarfishETL recently added Python to its library of scripting languages which already includes Javascript, C#, and VBScript.  Using Python with StarfishETL has some powerful advantages. You can:

  • Reference Python’s Standard Library or use your own modules to complete tasks in StarfishETL
  • Use global code to reference custom functions or global variables from any stage within your ETL job
  • Debug scripts from the syntax highlighting editor, with detailed logging options available
  • Connect to Cloud data services or pull data from local sources using the StarfishETL Ray
  • Use the built-in “Starfish” library to easily perform ETL specific tasks such as lookups, make execution flow decisions, and more
lookup

  • Perform common data translations tasks such as string manipulation, parsing, and type conversions 
  • Create global functions and access them anywhere throughout your ETL job execution. Reuse this code across different tasks and projects 
function field

Try Python scripting in your next StarfishETL project! Need help? Check out our KnowledgeBase or contact us.

Posted in:
ETL

Start a Project with us

Fill out the form below and we will contact you