Using Shopify's REST API cursor-based pagination with Rails

How to use the 2019-07 (and later) cursor-based pagination from Shopify's REST API with Rails.

Shopify recently changed the way it responds to paginatable results in their REST API. Previously, you could request the total count of objects and then use a 'times' loop to send that many requests for object data, using offsets and rounded division if you wanted to request objects in bulk.

Shopify has since replaced the above (known as 'page-based' pagination) with a new system 'cursor-based' pagination which is available in any API version after 2019-07 but was greatly expanded upon in API version 2019-10 (see all paginatable objects alongside API versions here: Making requests to paginated REST Admin API endpoints).

Cursor-based pagination responds with a 'link' header when a next page (or previous page) is available and you'll need to use that header to both check if you should send the next request, and if so where you should send the next request.

The following is a simplified example of cursor-based pagination in rails using the 'HTTParty' gem when you want to retrieve every product belonging to a shopify store. Here I set the limit to 250 objects per page as that is the maximum allowed, however it is possible to go lower if you need to.

# Set start URL  
product_limit = 250
products_url = "{product_limit.to_s}"

# Setup empty products array
products = [] 

# Loop all available pages and place all products in products array
next_page = true
while next_page
  product_response = HTTParty.get(
    headers: headers # Contains 'Authorization' and other standard headers, you'll need to set this up beforehand.

  # Push all products from response into products array
  product_response['products'].each do |product|
    products << product

  # Re-loop control
  if product_response.headers['link'] && product_response.headers['link'].include?('rel="next"')
    # if a next page exists set the url to the next page for the next loop
    products_url = product_response.headers['link'].remove('<').remove('>; rel="next"')
    # if a next page doesnt exist we tell the loop so it can end
    next_page = false

# The products array now contains all product objects from the Shopify store.