By Rudderstack Team

How to load data from the Magento to Google BigQuery

This post helps you with loading your data from Magento to BigQuery. Suppose you are looking to get analytics-ready data without the manual hassle. In that case, you can integrate Magento to BigQuery with RudderStack, so you can focus on what matters, getting value out of your shop data.

Extract your data from Magento

Magento exposes its platform through both a REST and a SOAP interface. Both can be used to pull data from it, which is also the scope of this article and interact with the platform.

By using these interfaces, developers create rich applications and plugins for Magento. In this post, we will use the REST version of the Magento platform.

As a Web API following the RESTful architecture principles, it can be accessed through HTTP. As a RESTful API, interacting with it can be achieved using tools like CURL or Postman or by using HTTP clients for your favorite language or framework. A few suggestions:

Magento does not publish official SDKs but by using the SOAP interface it is possible to automatically generate clients that can act as SDKs for your favorite language or platform. For example, in Java, you can create a client in Eclipse by providing the WSDL file that Magento exposes after you set up the platform.

About Magento

Magento is an e-commerce platform built on open source technology that provides online merchants with a flexible shopping cart system, as well as control over the look, content, and functionality of their online store. Magento offers powerful marketing, search engine optimization, and catalog-management tools. Some of its main characteristics are the following:

  • Feature-rich. Magento is very rich in functionality and offers an in-depth and powerful platform.
  • Powerful SEO. Magento is well known for its SEO capabilities. It offers one of them, out-of-the-box, SEO optimization for the stores hosted on it.
  • Magento ensures that your store can seamlessly grow along with your business.
  • Flexibility. Its template-based architecture allows you to customize everything.
  • Magento community edition is open source and free to use.
  • Security. Magento is built with security at its core.
  • User-friendly. The administration area exposes a simple backend with intuitive navigation and well-organized store management features.
  • Community. Being open-source guarantees a healthy ecosystem to support and further develop the platform.

Magento offers three different versions of its platform:

  • Community edition. It’s the open-source and free version of Magento.
  • Enterprise edition. Magento Enterprise Edition is designed to empower merchants to rapidly innovate and deliver engaging experiences to customers across all channels and devices.
  • Enterprise cloud edition. The Enterprise Edition of the platform as a service delivered over the cloud.

Magento API Authentication

Magento is a self-hosted platform unless you are using the Enterprise cloud edition, so you have much more control over its access than other solutions, but if you want to access its data through the REST API that it has, then you will have to use OAuth for authentication which Magento supports.

Magento rate limiting

As you would like to avoid stressing your e-commerce platform facing your customers, you should make sure that your pipeline process does not overstress your Magento installation. As a platform hosted on your premises, it doesn’t impose any rate-limiting. But this is completely at your discretion.

Endpoints and available resources

Magento exposes the following resources:

  • Products. Retrieve the list of products, create, update, delete a product.
  • Product categories. Retrieve the list of categories assigned to a product, assign and unassign the category from a product.
  • Product images. Retrieve the list of websites assigned to a product, assign, unassign a website to/from a product.
  • Customers. Retrieve the list of customers, create, delete a customer, and update the customer information.
  • Customer Addresses. Retrieve the list of customer addresses, create, update, and delete the customer address.
  • Inventory. Retrieve the list of stock items, update required stock items.
  • Sales Orders. Retrieve the list of sales orders with detailed information on order addresses, items, and comments.
  • Order Addresses. Retrieve information on the specified order comments.
  • Order Items. Retrieve information on specified order items.

The API is possible to return either JSON or XML responses. This is something that you can control by providing the appropriate Accept headercontent type.

We can request all the above resources from the Magento platform to pull out a list of results with all the associated data, so ideally, we would like to pull all the data and make sure that we keep them up to date on our analytics platform of choice for further analysis. For this post, we will just see how we can pull data for one resource, the Sales Orders. The process is the same for all other resources.

https://magentohost/api/rest/orders

As a platform hosted by you, you need to replace the “magentohost” part of the URL with the actual URL of the host that has Magento running. The rest of the URL is the same as the above. The default response is in XML and looks like the following: