Pipeline Management in Python

Learn how to create, update, and manage real-time data pipelines using the GlassFlow Python SDK

Personal Access Token

The Personal Access Token (PAT) is required for managing your GlassFlow account and accessing broader administrative capabilities beyond individual pipelines. This is available in your profile section of the GlassFlow App

Key Features of Personal Access Tokens

  • Full Account Access: Unlike pipeline-specific tokens, a PAT allows management of all pipelines, spaces, and other account-related settings.
  • Secure & User-Specific: PATs are tied to a user account and provide authentication for API access at a higher privilege level.

Get Pipeline

Retrieve pipeline details by pipeline ID. This returns a Pipeline object, allowing you to inspect and manage your pipeline programmatically.
📌 Full API reference GlassFlowClient.get_pipeline

List Pipelines

You can fetch details about all your pipelines. If needed, you can filter by space_id to narrow results to a specific space.
📌 Full API reference GlassFlowClient.list_pipelines

Update Pipeline

You can retrieve a pipeline using its pipeline_id. The returned Pipeline object allows you to modify and update:

  • The pipeline name
  • Transformation code
  • Requirements
  • Source and sink configurations

Once the update method is called, the pipeline is updated instantly, and the new parameters take effect immediately.
📌 Full API reference Pipeline.update

Pause a Pipeline

To temporarily stop a pipeline from processing new events, you can set its state to paused.

  • A paused pipeline remains deployed but does not consume new events.
  • Events sent while the pipeline is paused are queued and will be processed once the state is changed back to running.
  • This is useful for updating transformation logic or debugging issues

Modify Transformation

Update the transformation function by specifying a new transformation file. You can also update environment variables for the transformation function.

Delete Pipeline

The Pipeline object has a delete method to delete a pipeline.
📌 Full API reference Pipeline.delete

Pipeline Access Token

Pipeline Access Token enables you to send-receive events to a pipeline without the need of the personal access token. You can get the Pipeline Access Token from the Pipeline object that is returned from the get_pipeline() method.