Getting Started with GlassFlow
GlassFlow is a serverless platform that allows you to build, deploy, and manage real-time data pipelines entirely in Python.
GlassFlow eliminates the complexity of managing infrastructure, letting you focus on writing business logic for your data transformations.
Quickstart
The following quickstart video shows how to get started with GlassFlow. In less than 10 minutes, build your first streaming pipeline.
Features
-
Data Transformations in Python: Define your transformation function in plain Python using any Python library. GlassFlow will run the transformation function for every event in the pipeline in real-time
-
Built-in Message Broker: GlassFlow comes with an integrated message broker, powered by NATS. You can connect directly to GlassFlow via REST APIs, Python SDK or through integrations.
-
Fully Managed: GlassFlow is a fully managed service running in the cloud. You do not need to provision or manage any infrastructure. All pipelines created are deployed and ready to process events instantaneously
-
Managed Connectors and Integrations: With managed connectors to several data sources and data sinks, GlassFlow easily fits into your existing stack by pulling and pushing data automatically to other systems.
-
CI/CD Pipelines: GlassFlow enables seamless integration with GitHub, allowing users to build, deploy, and maintain their data pipelines using GitHub Actions.
-
Python SDK: All operations in GlassFlow can be done via the GlassFlow Python SDK, including creating and managing pipelines, creating access tokens, sending and receiving events, logging and monitoring. GlassFlow also has a developer friendly web UI to manage all operations via a visual interface.
-
Built-in Monitoring and Logging: With built-in monitoring and logging, you always stay updated on your pipeline's status. Easily debug any faults or get alerted on any data anomalies in real-time.
-
Data Reprocessing: Every failed event is stored in a dead-letter queue and can be reprocessed indefinitely if needed.
-
Bring Your Own Cloud (BYOC): GlassFlow offers a BYOC option for all major cloud providers so your data can stay within your environment.
Core Concepts
-
Pipeline: A pipeline is the backbone of GlassFlow, defining the flow of data from sources to transformations and then to destinations (sinks).
-
Space: A Space represents an isolated environment within GlassFlow, where you can organize and manage pipelines, data sources, and sinks.
-
Data Transformation: Data transformation is the process of modifying, enriching, or cleaning data as it moves through a pipeline.
-
Data Source: A Data Source is where your data originates, such as an API, database, or file storage.
-
Data Sink: A Data Sink is the destination where processed data is sent, such as another API, storage service, or database.