# Github Sync

When Github Syncing is enabled on your project, Pipedream will serialize your workflows and synchronize changes to a GitHub repo.

Capabilities include:

  • Bi-directional GitHub sync (push and pull changes)
  • Edit in development branches
  • Track commit and merge history
  • Link users to commits
  • Merge from Pipedream or create PRs and merge from GitHub
  • Edit in Pipedream or use a local editor and synchronize via GitHub (e.g., edit code, find and replace across multiple steps or workflows)
  • Organize workflows into projects with support for nested folders

TIP

Follow our quickstart guide to start building projects on Pipedream using GitHub Sync.

# Getting Started

# Create a new project and enable GitHub Sync

A project may contain one or more workflows and may be further organized using nested folders. Each project may be synchronized to a single GitHub repo.

  • Go to https://pipedream.com/projects
  • Create a new project
  • Enter a project name and check the box to Configure GitHub Sync
    • To use OAuth
      • Select a connected account, GitHub scope and repo name
      • Pipedream will automatically create a new, empty repo in GitHub
      • Enabling Github on a Pipedream project
    • To use Deploy Keys
      • Create a new repo in GitHub
      • Follow the instructions to configure the deploy key
      • Test your setup and create a new project
      • Enabling Github sync with a Deploy Key

# Create a branch to edit a project

Branches are required to make changes

All changes to resources in a project must be made in a development branch.

Examples of changes include creating, editing, deleting, enabling, disabling and renaming workflows. This also includes changing workflow settings like concurrency, VPC assignment and auto-retries.

To edit a git-backed project you must create a development branch by clicking Edit > Create Branch

Creating a new git backed development branch in a workflow

Next, name the branch and click Create:

Enter your desired branch name, and click create to create the branch

To exit development mode without merging to production, click Exit Development Mode:

Exiting the branch without committing the changes

Your changes will be saved to the branch, if you choose to revisit them later.

# Merge changes to production

Once you've committed your changes, you can deploy your changes by merging them into the production branch through the Pipedream UI or GitHub.

When you merge a Git-backed project to production, all modified resources in the project will be deployed. Multiple workflows may be deployed, modified, or deleted in production through a single merge action.

# Merge via the Pipedream UI

To merge changes to production, click on Merge to production:

Click on the Merge to production button in the top right of the UI to merge the changes

Pipedream will present a diff between the development branch and the production. Validate your changes and click Merge to production to complete the merge:

In the confirmation modal, click Merge to production to confirm the changes

# Create a Pull Request in Github

To create a pull request in GitHub, either choose Open GitHub pull request from the git-actions menu in Pipedream or in GitHub:

Opening a PR request in Pipedream

You can also review and merge changes directly from GitHub using the standard pull request process.

Pull request reviews cannot be required

PR reviews cannot be required. That feature is on the roadmap for the Business tier.

# Commit changes

To commit changes without merging to production, select Commit Changes from the Git Actions menu:

Select commit changes from the dropdown menu to make a commit

You can review the diff and enter a commit message:

Preview your changes and approve them with a commit message

# Pull changes and resolve conflicts

If remote changes are detected, you'll be prompted to pull the changes:

Click the Pull to pull in the latest changes to your current branch

Pipedream will attempt to automatically merge changes. If there are conflicts, you will be prompted to manually resolve it:

Example of a commit that requires manual resolution to continue with the commit

# Move existing workflows to projects

Not available for v1 workflows

Legacy (v1) workflows are not supported in projects. Follow this guide to migrate your v1 workflows to v2 workflows.

First, select the workflow(s) you want to move from the workflows listing page (opens new window) and click Move in the top action menu:

Select your workflows you'd like to transfer to a project, then click the Move button in the top right

Then, select the project to move the selected workflows to:

Select which project to move the selected workflows into in the dropdown in the top right of the screen

Undeployed changes are automatically assigned a development branch

If any moved workflows have undeployed changes, those changes will staged in a branch prefixed with undeployed-changes (e.g., undeployed-changes-27361).

# Use the changelog

The changelog tracks all git activity (for projects with GitHub sync enabled). If you encounter an error merging your project, go to the changelog and explore the log details to help you troubleshoot issues in your workflows:

Opening the changelog of commits on the left hand menu

# Local development

Projects that use GitHub sync may be edited outside of Pipedream. You can edit and commit directly via GitHub’s UI or clone the repo locally and use your preferred editor (e.g., VSCode).

To test external edits in Pipedream:

  1. Commit and push local changes to your development branch in GitHub
  2. Open the project in Pipedream’s UI and load your development branch
  3. Use the Git Actions menu to pull changes from GitHub

# Known Issues

Below are a list of known issues that do not currently have solutions, but are in progress:

  • Project branches on Pipedream cannot be deleted.
  • If a workflow uses an action that has been deprecated, merging to production will fail.
  • Legacy (v1) workflows are not supported in projects.
  • Self-hosted GitHub Server instances are not yet supported. Please contact us for help (opens new window).
  • Workflow attachments are not supported

# Github Enterprise Cloud

If your repository hosted on an Github Enterprise account, you can allow the static Pipedream IP address to sync your project changes.

Follow the directions here (opens new window) to add an IP address.

Then add this static IP address 3.214.142.179 to allow Pipedream to sync changes.

Github Sync is available on Business and above plans

To use this public IP address and connect to Github Enterprise Cloud hosted repositories, you'll need to have a Pipedream Business plan. View our plans (opens new window).

# Frequently Asked Questions

# How are Pipedream workflows synchronized to Github?

Pipedream will serialize your project's workflows and their configuration into a standard YAML format for storage in Github.

Then Pipedream will commit your changes to your connected Github account.

# Do you have a definition of this YAML?

Not yet, please stay tuned!

# Can I sync multiple workflows to a single Github Repository?

Yes, projects are synced to a single Github Repository which allows you to store multiple workflows into a single Github Repository for easier organization and management.

# Can I use this feature to develop workflows locally?

Yes, you can use the Github Syncing feature to develop your workflows from YAML files checked into your Pipedream connected Github Repository.

Then pushing changes to the production branch will trigger a deploy for your Pipedream workflows.

# Why am I seeing an error about "private auth mismatch" when trying to merge a branch to production?

Private Auth Mismatch

This error occurs when both of the below conditions are met:

  1. The referenced workflow is using a connected account that's not shared with the entire workspace
  2. The change was merged from outside the Pipedream UI (via github.com or locally)

Since Pipedream can't verify the person who merged that change should have access to use the connected account in a workflow in this case, we block these deploys.

To resolve this error:

  1. Make sure all the connected accounts in the project's workflows are accessible to the entire workspace
  2. Re-trigger a sync with Pipedream by making a nominal change to the workflow from outside the Pipedream UI (via github.com or locally), then merge that change to production

# Can I sync an existing GitHub Repository with workflows to a new Pipedream Project?

No, at this time it’s not possible because of how resources are connected during the bootstrapping process from the workflow YAML specification. However, this is on our roadmap, please subscribe to this issue (opens new window) for the latest details.

# How does the production branch work?

Anything merged to the production branch will be deployed to your production workflows on Pipedream.

From a design perspective, we want to let you manage any branching strategy on your end, since you may be making commits to the repo outside of Pipedream. Once we support managing Pipedream workflows in a monorepo, where you may have other changes, we wanted to use a branch that didn’t conflict with a conventional main branch (like main or master).

In the future, we also plan to support you changing the default branch name.

← File Stores →