How to load data from the HubSpot to MS SQL Server
Extract your data from HubSpot
HubSpot’s APIs are following the REST architecture, which can be accessed through HTTP. As a RESTful API, interacting with it can be achieved by using tools like CURL or Postman or by using HTTP clients for your favorite language or framework.
- A few suggestions:
- Apache HttpClient for Java
- Spray-client for Scala
- Hyper for Rust
- Ruby rest-client
- Python http-client
Responses of the API are all in JSON, including errors although HTTP codes are also returned to indicate errors. HubSpot maintains some official SDKs for their APIs, like the hapipy client for python, also unofficial clients can be found.
- haPiHP PHP wrapper for the APIs
- Ruby Wrapper (unofficial)
HubSpot API Authentication
The HubSpot API allows two types of authentication. OAuth and API keys. HubSpot encourages the use of oAuth for any serious integration and suggests using basic authentication with API keys only for testing and rapid prototyping purposes.
HubSpot’s API Rate Limiting
The HubSpot public endpoints are powered by the same underlying technology that powers the core HubSpot application. As a result, HubSpot engineering closely monitors the usage of the public APIs to ensure a quality experience for users of the HubSpot application.
Below, you’ll find the limits by which a single integration (as identified by an access token) can consume the HubSpot public APIs.
- Integrations cannot consume the HubSpot API at a rate greater than 10 requests/second.
- Polling of HubSpot APIs should occur at intervals of 5 minutes or more.
- Total requests to the HubSpot APIs should not exceed 10,000 in a 24 hour period.
- Failed requests to the HubSpot APIs may not exceed 5% of total syncs.
- All data passed to HubSpot must be properly encoded, and use application/json formatting.
- Integrations should use HubSpot’s OAuth protocol.
- Integrators must store time-to-live (TTL) data for OAuth access tokens as well as refresh tokens. Unauthorized (401) requests are not a valid indicator that a new access token must be retrieved.
- Integrators should use their own public and documented APIs when working with the HubSpot APIs.
- We reserve the right to change or deprecate the APIs over time – we will provide developers ample notification in those cases.
Endpoints and available Resources
The HubSpot’s API is actually a collection of a large number of APIs. Something that makes sense if we consider that HubSpot is actually more than one product and each one of these products is quite complex on its own. The APIs are the following:
- Calendar API. Anything that has to do with calendars in HubSpot and their possible operations.
- Companies APIs. When we talk about business and customers we talk about companies, so here is your API for interacting with companies.
- Companies properties APIs. Companies are important for HubSpot and there’s a lot of functionality around them so there’s an auxiliary API just for working with properties of companies.
- Contacts APIs. Contacts are the fundamental building block to HubSpot – they store lead-specific data that makes it possible to leverage much of the functionality in HubSpot, from marketing automation, to lead scoring to smart content.
- Contact Lists APIs. API for managing the lists of your contacts.
- Contact properties APIs. Similar to companies, this API allows you to interact with the properties of your contacts.
- COS Blog API. Interact with Blogs throughs the HubSpot platform.
- COS Blog Authors API. API for interacting with the authors of your blogs.
- COS Blog Comments API. Exposes functionality regarding the comments of your APIs.
- COS Blog Posts API. Anything regarding the posts of your Blog.
- COS Blog Topics API. Manage the topics of your Blog.
- COS Domains API. Manage your Domains through this API.
- COS Files API. Operations related to file management inside HubSpot.
- COS Layouts API. Manage the layouts of your pages through this API.
- COS Page Publishing API. Operations about publishing content through the HubSpot platform.
- COS Sitemaps API. Operations about the management of sitemaps for the sites you create through the HubSpot platform.
- COS Templates API. API for managing the templates of your sites.
- COS URL Mappings API. Operations related to URL mappings for the sites inside HubSpot.
- Deals API. Anything that has to do with deals inside your CRM.
- Deal Pipelines API. Manage the sales pipelines through this API.
- Deals Properties API. Again, manage the properties of your deals.
- Email API. Anything related to emailing from within the HubSpot platform.
- Email Events API. Track and interact with events that happen inside emails.
- Engagements API. Anything relevant about customer engagement inside the HubSpot platform.
- Events API. Event handling for HubSpot.
- Forms API. Manage custom forms that you create inside the platform.
- Keywords API. Operations related to keywords for SEO.
- Owners API. Anything regarding the Owner.
- Social Media API. API for interacting with Social media through the HubSpot platform.
- Transactional Email API. The transactional email functionality of HubSpot.
- Workflows API. Define and manage sales and marketing workflows.
From all the above endpoints we can pull data out of the platform, so it is easy to understand the richness of data we can get from a platform like HubSpot. Let’s assume, as an example, that we want to get all the Deals data. By executing a GET request like this GET /deals/v1/deal/recent/modified we can get all the recently modified deals. The parameters that we can pass to the call are the following:
- count: for specifying the number of results per page of the response.
- offset: for paginating through all available results.
- since: a timestamp for defining from which exact time you would like to fetch data from.
As we said earlier, results from the HubSpot API are always in JSON, so if we successfully execute the above query we’ll get the following results back:
JSON
{"results": [{"portalId": 62515,"dealId": 1030663,"isDeleted": false,"associations": {"associatedVids": [27316],"associatedCompanyIds": [],"associatedDealIds": []},"properties": {"dealstage": {"value": "closedwon","timestamp": 1417686612442,"source": "API","sourceId": null,………
The API offers you the opportunity to get very granular data about your accounting activities and use it for analytic and reporting purposes.
About HubSpot
HubSpot is a suite of software offering two different customer-related products:
- HubSpot CRM. For Customer Relations Management.
- HubSpot Marketing. Marketing platform, focusing mainly on inbound marketing.
HubSpot CRM offers a CRM platform that is connected to all the different channels and touchpoints where the modern sales process takes place. Channels like:
- Landing pages
- phone calls
- social media
By using HubSpot CRM it is possible to control your content, channels, and marketing performance from one single platform, making it much easier to have a birds-eye view of your sales process. You can create customized views, it offers deal and task dashboards and it connects directly with HubSpot Marketing so you can effortlessly capture, score and hand off lead to your sales team.
HubSpot Marketing which was also the first product of HubSpot, offers the complete marketing funnel to its user, from attracting users to closing customers with less hassle and more control of the whole process. It focuses mainly on inbound marketing techniques related to the following:
- Blogging
- SEO
- Social Media
- Landing pages
- Email campaigns
It allows you to easily perform lead management, it supports marketing automation like triggering emails when specific events happen on your landing page. It offers rich analytics and of course, it integrates seamlessly with HubSpot CRM.
It is clear that the HubSpot products generate a large number of valuable data related to your business, product, and customers. While it offers analytic services, you might want to run some more engaged analysis with your HubSpot data or merge these data with other sources like your ticketing system, your transaction database, and logs, or with user-generated events from your product. HubSpot exposes a rich ecosystem of tools and APIs for interacting with its platform and it is possible by using these APIs to pull the data that are valuable to you from it.
HubSpot Data Preparation for MS SQL Server
As in every relational database, SQL Server requires a well-defined database schema before we start populating with data. Data is organized in schemas, which are distinct namespaces where database objects belong to.
The most common database objects are of course tables which have a number of columns with each one having a declared data type. MS SQL Server supports a large number of different data types. This gives us great flexibility in expressing the data that we have and at the same time optimizing our data warehouse.
When working with data coming from web services, where data is usually serialized in JSON, it is important to correctly map every data to the right data types. As changing the data types in the future is a process that might cost in downtime of your database, it is important to spend enough time thinking about the proper data type assignments.
For example, dates in JSON are just strings, but when storing date objects in a database, we can enhance analytics with great capabilities by transforming the raw string data into an appropriate date type. A typical strategy for loading data coming from HubSpot to the MS SQL Server database is to create a schema where you will map each API endpoint to a table. Each key inside the HubSpot API endpoint response should be mapped to a column of that table and you should ensure the right conversion to an SQL Server compatible data type.
Of course, you will need to ensure that as the data types from the HubSpot API might change, you will adapt your database tables accordingly, there’s no such thing as automatic data typecasting. After you have a complete and well-defined data model or schema for Microsoft SQL Server, you can move forward and start loading them into the database.
About Microsoft SQL Server
Microsoft SQL Server is one of the oldest and most mature database systems. Its first version was introduced about 28 years ago, in 1989, and Microsoft has been consistently supporting and extending the product until today.
So, it’s no surprise that MS SQL Server has one of the richest feature sets among the currently available database systems.
SQL Server is delivered in different editions or flavors. The most notable being the Enterprise edition which can manage databases as large as 524 petabytes utilizing up to 12 terabytes of memory and 640 CPU processors. A free and scaled-down version is called Express.
A Business Intelligence version focusing on use cases where BI is performed on-premises. This version is actually a bundle of different products, including the core database system, together with other Microsoft-related products that can be used for BI purposes like visualization and data management.
In addition, there are also plenty of specialized versions of the database like the Compact edition that can be used on small devices and of course the Azure version, which is the cloud-based edition of SQL Server. Microsoft SQL Server incorporates a modular architecture that can extend the database with additional services. Replication services can extend the database to a cluster version and thus help with scaling and fault tolerance.
The SQL Server Analysis Services, augment the database with OLAP and data mining capabilities, making the database ideal for the workloads that we care about in this guide.
Load data from HubSpot to MS SQL Server
As a feature-rich and mature product, MS SQL Server offers a large and diverse set of methods for loading data into a database. One way of importing data to your database is by using the SQL Server Import and Export Wizard. With it and through a visual interface you will be able to bulk load data using a number of data sources that are supported.
You can import data by another SQL Server, from an Oracle database, from Flat Files, from an Access Data Source, PostgreSQL, MySQL, and finally Azure Blob Storage. Especially if you are using a managed version of MS SQL Server on Azure, you should definitely consider utilizing the Azure Blob Storage connection.
In this way, you will be loading data as Blobs on Azure and your MS SQL Server database will sync with it through the Import and Export Wizard.
Another way for importing bulk data into an SQL Server, both on Azure and on-premises, is by using the bcp utility. This is a command-line tool that is built specifically for bulk loading and unloading of data from an MS SQL database.
Finally and for compatibility reasons, especially if you are managing databases from different vendors, you can BULK INSERT SQL statements.
In a similar way and as it happens with the rest of the databases, you can also use the standard INSERT statements, where you will be adding data row-by-row directly to a table. It is the most basic and straightforward way of adding data in a table but it doesn’t scale very well with larger data sets.
So for bulk datasets, you better consider one of the previous methods.
Updating your HubSpot data on MS SQL Server
As you will be generating more data on HubSpot, you will need to update your older data on an MS SQL Server database. This includes new records together with updates to older records that for any reason have been updated on HubSpot.
You will need to periodically check HubSpot for new data and repeat the process that has been described previously while updating your currently available data if needed. Updating an already existing row on a SQL Server table is achieved by creating UPDATE statements.
Another issue that you need to take care of is the identification and removal of any duplicate records on your database. Either because HubSpot does not have a mechanism to identify new and updated records or because of errors on your data pipelines, duplicate records might be introduced to your database.
In general, ensuring the quality of data that is inserted in your database is a big and difficult issue and MS SQL Server features like TRANSACTIONS can help tremendously, although they do not solve the problem in the general case.
The best way to load data from HubSpot to MS SQL Server
So far we just scraped the surface of what you can do with MS SQL Server and how to load data into it. Things can get even more complicated if you want to integrate data coming from different sources.
Are you striving to achieve results right now?
Instead of writing, hosting, and maintaining a flexible data infrastructure use RudderStack that can handle everything automatically for you.
RudderStack with one click integrates with sources or services, creates analytics-ready data, and syncs your HubSpot to MS SQL Server right away.
Don't want to go through the pain of direct integration? RudderStack's HubSpot integration makes it easy to send data from HubSpot to MS SQL Server.