Migrating to Hygraph

#Overview

Migrating from any platform to another isn't a straightforward process, the same goes for a Headless CMS. Hygraph provides the SDK, APIs, and UI to make the process easier to create, and manage your schema and content.

The Hygraph schema and Content API are completely designed by you. This means any content types, and fields you add will immediately be available via the API. There are a few system content types, and system fields you get out of the box, but you can use as much, or as little as you need.

Before you begin creating any schema, or content, take a look at what content, and content types (referred to as "models" at Hygraph) you already have, and what they may look like in Hygraph.

Hygraph supports over a dozen field types, everything from regular strings, booleans, dates, polymorphic relations (Union Types) to remote field level resolvers that let you fetch content from other APIs.

You'll want to begin by deciding on the models, and fields you need.

Some users migrating prefer to take the time now to normalize their schema in a way that could not be done before, but this can often lead to manual intervention when importing content.

If you don't normalize this now, you can always use String, and JSON fields to represent as much of your data without any modification, however you'll lose some benefits with filtering that data at an API later if fields aren't set to their best field types.

This document covers the steps you need to follow to migrate to Hygraph, the tools you will need to use for it, a list of migration best practices, and tips on migrating different fields types, such as assets, rich text, and relations.

Our data migration flow follows four steps:

#1. Explore your current data and relations

This step requires you to look at the structure of the project you're migrating to Hygraph, and then consider Hygraph and its features.

Plan your schema

When you start thinking about how data should be modeled in your Hygraph project, you'll realize that you may be able to keep some structures as they are due to features in common, and you'll also find that there are structures that you may want to change in order to create an efficient schema that takes advantage of Hygraph's features.

How is your current project structured? Does it have the same structural elements that Hygraph offers to create schemas? Is there anything in your current project's structure that could benefit from using Hygraph features? How do the structures in your project relate to each other?

Look into these questions as you plan the structure of your Hygraph project, including how the structure that you currently have will need to change to use some of Hygraph features - such as components - where it makes sense.

We suggest that you look at your existing data, and export it to a format such as JSON, or CSV. You can use this export to inspect your existing content types, and field names, as well as any relations between data.

For the purposes of this article, we'll use a very basic CSV of authors:

oldId,firstName,lastName
1,Stephen,King
2,Frank,Herbert
3,Brian,Herbert
4,Kevin,Anderson
5,Agatha,Christie
6,Haruki,Murakami
7,Isaac,Asimov

#2. Create your new schema

Create your schema

Once the schema is planned, you can start the process of creating it in Hygraph. Bear in mind that you will first need to create the schema, and then the content.

You can create the schema using the UI or using the Management SDK to create your models, fields, enums, relations programmatically. Using the Management SDK is faster, more efficient, and it also lets you keep a track record of what's been done.

#Use the Management SDK

To create your schema using the Management SDK you will mostly use create and connect actions. Before we dive into a full example on how we'd use it in a migration, let's look into how we'd use it to create models and fields first.

You will be using createModel to create your models, like this:

client.createModel({
  apiId: '<your_api_id>',
  apiIdPlural: '<your_api_id_plural>',
  description: `<your_model_description>`,
  displayName: '<Your model name>',
});

So, if you wanted to create two models, called Author and Book, you'd do something like this:

client.createModel({
  apiId: 'Author',
  apiIdPlural: 'Authors',
  displayName: 'Author',
});

client.createModel({
  apiId: 'Book',
  apiIdPlural: 'Books',
  displayName: 'Book',
});

To create a simple field, you'll be using createSimpleField, like this:

client.createSimpleField({
  apiId: `<your_api_id>`,
  description: `<your_description>`,
  displayName: `<your_display_name>`,
  embeddableModels: `<embeddable_models>`,
  embedsEnabled: `<boolean>`,
  formConfig: `<form_config_json>`,
  formExtension: `<form_extension>`,
  formRenderer: `<form_renderer>`,
  isHidden: `<boolean>`,
  isList: `<boolean>`,
  isLocalized: `<boolean>`,
  isRequired: `<boolean>`,
  isTitle: `<boolean>`,
  isUnique: `<boolean>`,
  migrationValue: `<migration_value>`,
  modelApiId: `<model_api_id>`,
  parentApiId: `<parent_api_id>`,
  position: `<int>`,
  tableConfig: `<table_config_json>`,
  tableExtension: `<table_extension>`,
  tableRenderer: `<table_renderer>`,
  type: SimpleFieldType,
  validations: `<SimpleFieldValidationsInput>`,
  visibility: `<VisibilityTypes>`,
});

So, to create a simple string field in your Author's model to include the author's favorite pastime, you could add something like this:

client.createSimpleField({
  parentApiId: 'Author',
  type: SimpleFieldType.String,
  apiId: 'favoritePastime',
  displayName: 'Author Favorite Pastime',
  isRequired: true,
  visibility: VisibilityTypes.ReadWrite,
});

If your project uses component, enumeration, or remote source fields, you'll need to create those at the base schema level, before you add them to your models. You'll find information on how to create each one in the Management SDK examples for model and field creation document we mentioned above.

Let's now look into a full example:

In the example below, we'll import the @graphcms/management dependency, and create a model, and add some fields.

// migration.js
const { newMigration, FieldType } = require('@graphcms/management');

const migration = newMigration({ endpoint: '...', authToken: '...' });

const author = migration.createModel({
  apiId: 'Author',
  apiIdPlural: 'Authors',
  displayName: 'Author',
});

author.addSimpleField({
  apiId: 'firstName',
  displayName: 'First Name',
  type: FieldType.String,
});
author.addSimpleField({
  apiId: 'lastName',
  displayName: 'Last Name',
  type: FieldType.String,
});

migration.run();

Next, you'll need to run this with node. If you'd like to see the result or any errors of the migration, you can opt to run this migration in the foreground by passing true as the first argument to run().

const result = migration.run(foreground);

if (result.errors) {
  console.log(result.errors);
} else {
  console.log(result.name);
}

Depending on what filename you gave the migration, you can run node migration.js from the command line.

There are many other options available using the Management SDK, so it's recommended you read through the documentation, and understand everything that is available before using it.

#Use the UI

To create the schema using the UI, you will need to create the models, and then add fields to them.

Our getting started flow has a great section on creating models and adding fields to them.

#3. Map content & plan content migration

Once you create the structure of your project, you can start planning how to import your content. Migrating the content requires that you look at the content that you have and plan how it needs to be migrated taking into consideration the structure you just created for it.

For instance, the order of migration matters when content is related. i.e., you'd have to migrate your assets first so when you migrate the content entries, relations can be successfully established.

Once you have your plan laid out, you can start migrating the content.

#4. Import your existing content

This will be done using the Asset Upload API for assets, then the Content API for content.

API Endpoints

#Asset Upload API

Projects older than February 2024 use the Legacy asset system and will show an endpoint that allows uploading assets from your file system or from a remote URL. Newer projects use the Hygraph Asset Management system, which lets you upload assets via URL or file.

Hygraph supports uploading assets via HTTP. You'll need a Permanent Auth Token with Mutations API access enabled to upload by file, or URL.

Assets are treated just like any other content entry, so they are automatically bound to the environment, and authorization settings of your project.

Once this is set up, you can upload assets by file or by remote URL.

To upload by file you can use the following if your project uses the legacy asset system:

And if your project uses the Hygraph asset management system, you need to first create the asset via GraphQL mutation, like this:

And then upload the asset, like this:

URL="https://dev-1-assets-delivery-f78c5b5.s3.eu-central-1.amazonaws.com"
DATE="20240123T173451Z"
KEY="clr7o58jb00w701vwhpc69xda/upload/clr7o587k000j01uhul69p0rb/clr7o58ca00qi01vw7wzoguau/clrqmzdpn00id0bvucez7gnm9"
SIGNATURE="3f812cc0b05d59ee3d1efdd9dc046ed6edf595749c2763b85b923133a84e8d86"
ALGORITHM="AWS4-HMAC-SHA256"
POLICY="eyJleHBpcmF0aW9uIjoiMjAyNC0wMS0yM1QxOTo1OTo1MS45NDdaIiwiY29uZGl0aW9ucyI6W3siYnVja2V0IjoiZGV2LTEtYXNzZXRzLWRlbGl2ZXJ5LWY3OGM1YjUifSx7ImtleSI6ImNscjdvNThqYjAwdzcwMXZ3aHBjNjl4ZGEvdXBsb2FkL2NscjdvNTg3azAwMGowMXVodWw2OXAwcmIvY2xyN281OGNhMDBxaTAxdnc3d3pvZ3VhdS9jbHJxbXpkcG4wMGlkMGJ2dWNlejdnbm05In0seyJ4LWFtei1hbGdvcml0aG0iOiJBV1M0LUhNQUMtU0hBMjU2In0seyJ4LWFtei1jcmVkZW50aWFsIjoiQVNJQVZRUkUzVk1FMkJHWVZLT1AvMjAyNDAxMjMvZXUtY2VudHJhbC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMjQwMTIzVDE3MzQ1MVoifSx7IngtYW16LXNlY3VyaXR5LXRva2VuIjoiSVFvSmIzSnBaMmx1WDJWakVLci8vLy8vLy8vLy93RWFER1YxTFdObGJuUnlZV3d0TVNKSE1FVUNJRUF6SFQ2OUdEWm9CcXp5Nkw4bExDUDFEZnNBZ3FaM0phRms0M3ZvcGNGRUFpRUFrVTljUVNTSTFCRmV5TVZjTEI1QkFrVzg3NUE2M2hkWTVnNDgxcEY4QitNcTlBTUlZeEFGR2d3ek56a3hNRGd4TnpRMk1ERWlERkxzOHB2a0JHRU9aV3dSZVNyUkF4amJvVXRobDRtRUM3VHVzTHpFcXc1T1JUOUZRSStYYVVQWXZRZi9ST3ZNOExBVkFYVHQwMWZHT0JyUkVRSDRDckJLSVVtMDNtQWFPd05nQ2g1aGFlTDhmTzZvRUJ3aTFLa2ZuRWpNWmtQcVJma0krMDZhK0o0SVBYTmR0bEt6b0VVOVd4Uk9TMzEyTC8rMjBpUkpLNmNoWmhhK04zVXNJMzRYR0U0L2IzNzV0aXpsWUx0RHJYN0JiMXFIdFVNYlg3VWVUaVJaSDU4S1VwRjloK2QyWWk2bUhuL1lGcmJVaGpOcjVGL1pkL2FkdU5nOGpKU3liVkw5bTVZdWQvdlI3enMyNXdpYlpJK3BOQjFXcVZzRWtiN09EN2g1QjFlQkE5dnNMdzBKdVlFMmFnYm0xNjF0SjZVS3A5Tkd4YmVWaFM1TjQ5N2txV2MzN3pCT3ZxdWxUU1JHUW04V1QrNGx1YzZyd2VTT0lFeUxMaFowT1ZCSWM1Q0Nrc1YvaFAxT2dxeUdSNGdKTDlwa3RhUDl0encxaFhyUHJhUW1mRHhoaVgrUStaNitkSWFxalY2VGlwcVhqTk9jM2tFSFIwNmpydnlhOVFsRGxXaWd2eGRaTHkwZktzQW8vR0dBTlhlM2ZGSzhVclVObE4ySHFGeHpBYU9JNEJPMmU1VE1sUzA5MTZZTFZUN1Zxd3hHOHQ0NTlUTUFLcXpJMTJ4N2JTbHFYQ0lHUlc1anlsazhaTE1mbGRZS2t6eDN6dWNKZDYvYVN4Q1c1bE1pa2lGRjYrbW9WQW9kck1qeGhLZW42QnNRZ3ZMVW1kamFmaVROMkRERDhMK3RCanFsQVRzWGdYUmdZOUVJQzJMUXEyOU0xSGRlbmViY3I5YU5JbmFBTVFKRlpMUnl1bnM3dzk0MnhjaW1sSTYyak15RWFUczRZM3M3aTMwYVpqc1E2ampMK1hKMVJTQW5Ta2lndFJZbHVKSGFlaWJkRDMydEZDamZhY3ovZWF5eVhGcEgzekc5NHJRZUk1V0ZBWVZXRFZyWEN2WmJROG9SejV4OHN5ZFBBZWROR1VEOHYrMk5XWDc0NTBTcXdXc2Jta0dlekxoZ0pzNkxyOVhSSG5rTEljdGFoUDVoazg0Q3p3PT0ifV19"
CREDENTIAL="ASIAVQRE3VME2BGYVKOP/20240123/eu-central-1/s3/aws4_request"
SECURITY_TOKEN="IQoJb3JpZ2luX2VjEKr//////////wEaDGV1LWNlbnRyYWwtMSJHMEUCIEAzHT69GDZoBqzy6L8lLCP1DfsAgqZ3JaFk43vopcFEAiEAkU9cQSSI1BFeyMVcLB5BAkW875A63hdY5g481pF8B+Mq9AMIYxAFGgwzNzkxMDgxNzQ2MDEiDFLs8pvkBGEOZWwReSrRAxjboUthl4mEC7TusLzEqw5ORT9FQI+XaUPYvQf/ROvM8LAVAXTt01fGOBrREQH4CrBKIUm03mAaOwNgCh5haeL8fO6oEBwi1KkfnEjMZkPqRfkI+06a+J4IPXNdtlKzoEU9WxROS312L/+20iRJK6chZha+N3UsI34XGE4/b375tizlYLtDrX7Bb1qHtUMbX7UeTiRZH58KUpF9h+d2Yi6mHn/YFrbUhjNr5F/Zd/aduNg8jJSybVL9m5Yud/vR7zs25wibZI+pNB1WqVsEkb7OD7h5B1eBA9vsLw0JuYE2agbm161tJ6UKp9NGxbeVhS5N497kqWc37zBOvqulTSRGQm8WT+4luc6rweSOIEyLLhZ0OVBIc5CCksV/hP1OgqyGR4gJL9pktaP9tzw1hXrPraQmfDxhiX+Q+Z6+dIaqjV6TipqXjNOc3kEHR06jrvya9QlDlWigvxdZLy0fKsAo/GGANXe3fFK8UrUNlN2HqFxzAaOI4BO2e5TMlS0916YLVT7VqwxG8t459TMAKqzI12x7bSlqXCIGRW5jylk8ZLMfldYKkzx3zucJd6/aSxCW5lMikiFF6+moVAodrMjxhKen6BsQgvLUmdjafiTN2DDD8L+tBjqlATsXgXRgY9EIC2LQq29M1Hdenebcr9aNInaAMQJFZLRyuns7w942xcimlI62jMyEaTs4Y3s7i30aZjsQ6jjL+XJ1RSAnSkigtRYluJHaeibdD32tFCjfacz/eayyXFpH3zG94rQeI5WFAYVWDVrXCvZbQ8oRz5x8sydPAedNGUD8v+2NWX7450SqwWsbmkGezLhgJs6Lr9XRHnkLIctahP5hk84Czw=="

curl --request POST \
  --url $URL
  --form X-Amz-Date=$DATE
  --form key=$KEY
  --form X-Amz-Signature=$SIGNATURE
  --form X-Amz-Algorithm=$ALGORITHM
  --form policy=$POLICY
  --form X-Amz-Credential=$CREDENTIAL
  --form X-Amz-Security-Token=$SECURITY_TOKEN
  --form file=@./test.jpg

Click here to learn more about the Hygraph asset management system

You can also upload files by providing a remote URL, instead of a file. If your project uses the legacy asset system, this is how you would do it:

And if your project uses the Hygraph asset management system, you would create the asset in the same way we showed you for the Upload by file option, and then do this:

mutation test {
  createAsset(
    data: {
			uploadUrl:"https://images.unsplash.com/photo-1682687218147-9806132dc697"
    }
  ) {
    id
    url
  }
}

Click here to learn more about the Hygraph asset management system

#Content API

To find the Asset Upload API in your Hygraph project, go to Project settings > Access > API Access > Endpoints.

To migrate your content you'll be using mutations, which can even be nested. So, before you can begin creating any content within Hygraph, you'll want to inspect the auto-generated GraphQL mutations for your models that Hygraph automatically created as soon as you created those models.

Since the data living in the old CMS is mostly of a different shape and structure than it currently needs to be, it's quite difficult to create a 1:1 mapping, and import from your existing database to Hygraph. Because of this, you will want to consider tweaking your current dataset to match your new input types.

The following script shows a simple import example using the CSV sample authors file we shared earlier.

We've added comments to the code to explain each step:

// Import necessary libraries
const { GraphQLClient, gql } = require('graphql-request'); // Importing graphql-request library and gql tag
const csvToJson = require('csvtojson'); // Importing csvtojson library

require('dotenv').config(); // Importing environment variables from .env file via the dotenv NPM package


// Initialize GraphQL client with endpoint and authorization headers
const client = new GraphQLClient(process.env.HYGRAPH_ENDPOINT, {
    headers: {
        authorization: `Bearer ${process.env.HYGRAPH_TOKEN}`,
    },
});


// Function to create a GraphQL mutation based on provided data
function createMutation(data) {
  // Create the mutation
  const mutation = gql`mutation MyMutation {
    createAuthor(data: {
        firstName: "${data.firstName}", 
        lastName: "${data.lastName}", 
        oldId: "${data.oldId}"
      }) {
        id
    }
  }`;
  
  return mutation;
}

// Asynchronous function to run the migration process
async function run() {
    // Retrieve data from a CSV file and convert it to JSON format
    const data = await csvToJson().fromFile('./data.csv');

    // Generate an array of GraphQL mutations based on the retrieved data
    const mutations = data.map((item) => {
        return createMutation(item)
    });


// Execute each mutation after a timed delay
mutations.forEach((item, index) => {
  setTimeout(() => {
    
    console.log(`Running mutation ${index + 1} of ${mutations.length}`)
    // Make a request to the GraphQL endpoint using the generated mutation
    client.request(item).then((response) => {
        console.log(response)
        idsToUpdate.push(response.id)
    });// Store the retrieved ID for update
  }, (index + 1) * 1000); // Delay each iteration by (index + 1) seconds
});

}



// Running the migration process
run();

#Migrating content

This section contains additional help to migrate certain content types.

#Rich text

When migrating content into Hygraph, you might find that the way we save rich text is not the same your previous service does. Hygraph stores rich text as an Abstract Syntax Tree (AST) behind the scenes, which is based on Slate.

Migrating rich text to Hygraph requires two steps:

1. Conversion: Before transferring the content, you'll need to convert the way your current system stores rich text to match our AST format. Using our converter tool will ensure compatibility with our structure.
2. GraphQL Create Mutation: Once the content has been converted into the AST format, you can use a create mutation to migrate it.

Here's an example of a create mutation for rich text:

#Relations

To migrate relations, you can do a create mutation and then a nested mutation.

You would first do a create mutation for a Book with a nested create mutation for an Author, for example. You'd do this for the first book, then for the rest of the books by that author you just do a connect mutation, to connect the rest fo the books to that author that's already been created:

Alternatively, you could create all author entries, then all book entries - both using a create mutation - then use an update mutation to connect them. You can also use an update many mutation to connect them as well.

#Migration best practices

• Plan your migration before you carry it out.
• Look into Hygraph's capabilities and recreate your schema optimizing efficiency using our features (i.e. components).
• Migrate items in the right order for relations to work. i.e. migrate assets before the content entries that contain them.
• Figure out what is the most important / used content in your project vs. the supporting content, so you can ensure all the really important content can be moved over first.
• Try not to overload the mutations by doing everything in a single one. For instance, if you have a model that connects to ten other models, it's not generally a good idea to try to create all models and their content at once. Try to space mutations out to keep them running smoothly. Check out our documentation on rate limits to learn more.