By Rudderstack Team

How to load data from Magento to Redshift

Magento is one of the most popular e-commerce platforms. How can you combine your Magento data with other sources to gain new insights? Let’s see how to get the data we have on Magento to Redshift. This post will help you define a process or pipeline, for extracting your e-commerce related data from Magento and load it into Amazon Redshift for further analysis. We will see how to access and extract your data from Magento through its API and how to load it into Redshift.

This process requires you to write the code to get the data and ensure that this process will run every time new data are generated. Alternatively, to load your data from Magento to Redshift, you can use products like RudderStack that can automatically handle this kind of problem for you.

About Magento

Magento is an e-commerce platform built on open source technology that provides online merchants with a flexible shopping cart system and control over the look, content, and functionality of their online store. Magento offers powerful marketing, search engine optimization, and catalog-management tools. Some of its main characteristics are the following:

  • Feature-rich: Magento is very rich in functionality and offers an in-depth and powerful platform.
  • Powerful SEO: Magento is well known for its SEO capabilities. It offers out-of-the-box, SEO optimization for the stores hosted on it.
  • Magento: ensures that your store can seamlessly grow along with your business.
  • Flexibility: Its template-based architecture allows you to pretty customize everything.
  • Magento community edition is open source and free to use.
  • Security: Magento is built with security in its core.
  • User friendly: The administration area exposes a simple back end with intuitive navigation and well-organized store management features.
  • Community: Being open-source guarantees a healthy ecosystem to support and further develop the platform.

Magento offers three different versions of its platform:

  • Community edition: It’s the open-source and free version of Magento.
  • Enterprise edition: Magento Enterprise Edition is designed to empower merchants to rapidly innovate and deliver engaging experiences to customers across all channels and devices.
  • Enterprise cloud edition: The Enterprise Edition of the platform as a service delivered over the cloud.

About Amazon Redshift

Amazon Redshift is one of the most popular data warehousing solutions, part of the Amazon Web Services (AWS) ecosystem. It is a petabyte-scale, fully managed data warehouse as a service solution that runs on the cloud. It is SQL-based, and you can communicate with it as you would do with PostgreSQL. You can use the same driver although it would be better to use the drivers recommended by Amazon. You can connect either through JDBC or ODBC connections.

Extract your data from Magento

Magento exposes its platform through both a REST and a SOAP interface. Both can be used to pull data from it, which is also the scope of this article and interact with the platform. By using these interfaces, developers create rich applications and plugins for Magento. In this post, we will use the REST version of the Magento platform.

As a Web API following the RESTful architecture principles, it can be accessed through HTTP. As a RESTful API, interacting with it can be achieved by using tools like CURL or Postman or by using HTTP clients for your favorite language or framework. A few suggestions:

Magento does not publish official SDKs, but by using the SOAP interface, it is possible to automatically generate clients that can act as SDKs for your favorite language or platform. For example, in Java, you can create a client in Eclipse by providing the WSDL file that Magento exposes after you set up the platform.

Magento API Authentication

Magento is a self-hosted platform, unless you are using the Enterprise cloud edition, so you have much more control over its access than other solutions. Still, if you want to access its data through the REST API that it has, then you will have to use OAuth for authentication which Magento supports.

Magento rate limiting

As a platform hosted on your premises, it doesn’t impose any rate-limiting. In any case, as you would like to avoid stressing your e-commerce platform that is facing your customers, you should make sure that your pipeline process does not overstress your Magento installation. But this is completely at your discretion.

Endpoints and available resources

Magento exposes the following resources:

  • Products: Retrieve the list of products, create, update, delete a product.
  • Product categories: Retrieve the list of categories assigned to a product, assign and unassign the category from a product.
  • Product images: Retrieve the list of websites assigned to a product, assign, unassign a website to/from a product.
  • Customers: Retrieve the list of customers, create, delete a customer, and update the customer information.
  • Customer Addresses: Retrieve the list of customer addresses, create, update, and delete the customer address.
  • Inventory: Retrieve the list of stock items, update required stock items.
  • Sales Orders: Retrieve the list of sales orders with detailed information on order addresses, items, and comments.
  • Order Addresses: Retrieve information on the specified order comments.
  • Order Item: Retrieve information on specified order items.

The API is possible to return either JSON or XML responses. This is something that you can control by providing the appropriate Accept header content type.

For all the above resources we can request from the Magento platform to pull out a list of results with all the associated data. Ideally, we would like to pull all the data and make sure that we keep them up to date on our analytics platform of ch