Google Data pipelines API
Browse all Google APIs41
AI ReadinessFoundational (D-)41/100
Official vendor OpenAPI document · agent-readyAnalyticsData Pipelinesoauth28 EndpointsREST
For Agents
Create, list, run, and stop scheduled Dataflow pipelines through a managed scheduling layer. Lets agents launch recurring data analytics jobs without wiring up Cloud Scheduler.
Use for: I need to schedule a Dataflow batch pipeline to run every night at 02:00, Trigger an on-demand run of a recurring data pipeline, Stop a streaming Dataflow pipeline, List all jobs produced by a pipeline in the last 7 days
Not supported: Does not author Beam code, manipulate running Dataflow jobs directly, or schedule non-Dataflow targets - use for scheduling and lifecycle of Dataflow-template-backed pipelines only.
The Google Cloud Data pipelines API offers a thin scheduling layer over Dataflow templates, letting teams create, list, run, and stop recurring data analytics pipelines. It exposes 8 endpoints scoped to pipelines and their job history, with a built-in cron-style schedule on each pipeline. Pipelines are backed by Dataflow batch and streaming jobs, but this API removes the need to schedule them through Cloud Scheduler or Composer.
Install Jentic One Beta
Connect the Data pipelines API to your agent
Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Data pipelines API, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.
Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.
1
Step 1: Jentic One Host machine
# On the machine that will host your Jentic One instance:
curl -fsSL "https://jentic.com/install.sh?src=apis&api=%2Fapis%2Fgoogleapis.com%2Fdatapipelines" | sh2
Step 2: Agent machine
# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL "https://jentic.com/install.sh?src=apis&api=%2Fapis%2Fgoogleapis.com%2Fdatapipelines" | sh
jentic register # connects your agent to your Jentic One instanceJentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.
Capabilities
What an agent can do with Data pipelines API.
Create scheduled batch pipelines backed by Dataflow templates
Trigger an on-demand run of a pipeline regardless of its schedule
Stop a streaming pipeline cleanly through the API
List historical jobs produced by a recurring pipeline
Update pipeline parameters, schedule, and worker configuration
Inspect each pipeline's last execution status and error reason
Use Cases
Patterns agents use Data pipelines API for, with concrete tasks.
★ Scheduled Dataflow Batch Pipelines
Schedule recurring Dataflow batch jobs from a managed cron without standing up Cloud Scheduler or Composer. The API stores the template path, parameters, and schedule on a Pipeline resource, and creates a Dataflow job each tick. Agents can list past job runs and inspect failure causes from a single resource. Setup typically takes minutes once a template URL is available.
Create a pipeline 'nightly-export' running gs://acme/templates/export.json with parameters region=us-central1, scheduled at '0 2 * * *' UTC
On-Demand Pipeline Runs
Trigger a pipeline run outside its normal schedule for backfills, ad-hoc reports, or operator-driven reruns after a failure. The API's run endpoint creates a fresh Dataflow job from the pipeline's template and parameters, and returns the resulting Dataflow job reference for downstream monitoring. This avoids re-deploying the template just to run it once.
Run pipeline 'nightly-export' on demand and return the resulting Dataflow job ID for the operator to track
Pipeline Lifecycle Stop and Resume
Stop a running streaming pipeline, update its parameters or schedule, and recreate it cleanly without orphaning Dataflow jobs. The API's stop endpoint drains the underlying Dataflow job, freeing slots and stopping further charges. This supports cost-control automations that pause non-critical pipelines outside business hours.
Stop pipeline 'real-time-clicks' to drain the underlying Dataflow streaming job, then update its workerCount and recreate
AI Agent Pipeline Scheduler
An AI agent can schedule, run, and stop recurring Dataflow pipelines through Jentic without writing OAuth or template-management code. Jentic search returns the matching createPipeline, runPipeline, or stopPipeline operation, the agent loads the schema, and Jentic executes against datapipelines.googleapis.com using vault-stored credentials.
Use Jentic to search 'create a scheduled dataflow pipeline', load the createPipeline schema, and execute it with the template, parameters, and cron schedule
Key Endpoints
8 endpoints — the google cloud data pipelines api offers a thin scheduling layer over dataflow templates, letting teams create, list, run, and stop recurring data analytics pipelines.
METHOD
PATH
DESCRIPTION
POST
/v1/{+parent}/pipelines
Create a scheduled Dataflow pipeline
GET
/v1/{+parent}/pipelines
List pipelines under a project location
POST
/v1/{+name}:run
Trigger an on-demand pipeline run
POST
/v1/{+name}:stop
Stop a running streaming pipeline
GET
/v1/{+parent}/jobs
List historical jobs of a pipeline
POST
/v1/{+parent}/pipelines
Create a scheduled Dataflow pipeline
GET
/v1/{+parent}/pipelines
List pipelines under a project location
POST
/v1/{+name}:run
Trigger an on-demand pipeline run
POST
/v1/{+name}:stop
Stop a running streaming pipeline
GET
/v1/{+parent}/jobs
List historical jobs of a pipeline
Jentic AI Readiness Score
This API is usable in Jentic One now. Its AI-readiness score against Jentic's framework shows where it stands today and where improvements would make it even easier for agents to use.
D
Data pipelines API
- Foundational (D-)41/100
70
Foundational Compliance56
Developer Experience & Jentic Compatibility18
AI-Readiness & Agent Experience94
Agent Usability63
Security63
AI DiscoverabilityPowered by JenticScoring Framework 1.0.0 | Scoring Engine 0.4.0
Show dimension breakdown
70
Foundational Compliance
Base layer of spec validity and structural soundness.
Grade: A-Signals: 4
81%
Lint Results
Aggregated quality score from linter diagnostics, weighted by severity.
100%
Resolution Completeness
Percentage of `$ref` references that resolve successfully.
0%
Specification Validity
Checks whether the API description parses successfully and conforms to its declared specification (e.g., OpenAPI).
100%
Structural Integrity
Structural correctness score based on schema issues using logarithmic dampening.
56
Developer Experience & Jentic Compatibility
Clarity, completeness, and ingestion readiness for developers and tooling.
Grade: CSignals: 4
0%
Example Density
How richly the API is illustrated with examples.
100%
Example Validity
Percentage of examples that conform to their schemas.
25%
Response Coverage
Percentage of operations with complete response definitions (success, client error, server error).
100%
Tooling Readiness
Health of API ingestion, bundling, and resolution within Jentic pipelines.
18
AI-Readiness & Agent Experience
Semantic breadth, depth, and agent comprehension for AI systems.
Grade: FSignals: 4
74%
Description Coverage
Coverage of descriptions across API elements.
0%
Error Standardization
Coverage of RFC 9457 Problem Details for error responses.
0%
OperationId Quality
Coverage, uniqueness, and casing consistency of operationIds for AI inference.
0%
Summary Coverage
Coverage of summaries across operations/tags/info.
94
Agent Usability
Functional utility, complexity comfort, and AI orchestration readiness.
Grade: A+Signals: 1
94%
Complexity Comfort
Agent comfort level based on API operational and structural complexity.
63
Security
Trust, risk posture, and security compliance.
Grade: B-Signals: 1
62%
Authentication Strength
Average quality of security schemes based on authentication method strength (weakest link for OAuth2).
63
AI Discoverability
Findability, semantic richness, and reasoning readiness.
Grade: B-Signals: 1
62%
Descriptive Richness
Clarity and depth of descriptions across API elements.
Score it yourself
Every API in the directory is allowlisted, so you can re-score it with no key required.
npx @jentic/api-scorecard-cli score <openapi-url>Why Jentic?
What agents get from Jentic-routed access to this vendor.
Setup
Wiring the Data pipelines API by hand means setting up Google OAuth, refreshing short-lived scoped tokens rather than holding client secrets in code, and addressing pipeline resources on datapipelines.googleapis.com. Through Jentic you install once, import the Data pipelines API from the API Directory, store the Google credential once, and your agent calls it.
Permission scoping
The API carries the parent and pipeline name in the URL path (/v1/{parent}/pipelines and /v1/{name}:run), so a rule can pin your agent to one pipeline. You choose the operations it may call, so it can list and run pipelines while stop is not included unless you add it.
Credential isolation
Your Google OAuth credential is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
Intent-based discovery
Agents search Jentic by intent such as 'schedule a Dataflow pipeline', and Jentic returns the matching Data pipelines operation with its input schema so the agent calls the right endpoint without browsing the reference docs.
Related APIs
Alternatives and complements available in the Jentic catalogue.
FAQs
Specific to using Data pipelines API through Jentic.
What authentication does the Data pipelines API use?
The Data pipelines API uses OAuth 2.0 with the cloud-platform scope. Through Jentic the OAuth client and refresh tokens are stored in your Jentic One instance and the agent receives short-lived scoped tokens, so raw Google credentials never enter the agent context.
Can I run on-demand jobs with the Data pipelines API?
Yes. POST /v1/{+name}:run starts a fresh Dataflow job from the pipeline's template and parameters, independent of its cron schedule. The response includes the Dataflow job reference so the agent can follow the job through the Dataflow API.
What are the rate limits for the Data pipelines API?
Google enforces standard Cloud quotas on datapipelines.googleapis.com: per-project rate limits on read/write calls and Dataflow's underlying quota on concurrent jobs per region. Quotas are visible in the Cloud Console under IAM and admin, quotas, filtered to datapipelines.googleapis.com.
How do I create a scheduled pipeline through Jentic?
Search Jentic for 'create a scheduled dataflow pipeline', load the schema for POST /v1/{+parent}/pipelines, and execute with the template URL, parameters, schedule (cron string), and pipelineSources. Jentic returns the created Pipeline resource with its scheduleInfo set.
Is the Data pipelines API free?
API calls are free; pipeline runs are billed under standard Dataflow pricing (vCPU, memory, and PD-SSD seconds for batch and streaming jobs). There is no separate scheduling charge from Data pipelines itself.
How do I stop a streaming pipeline cleanly?
Call POST /v1/{+name}:stop with the pipeline resource name. The service issues a drain on the underlying Dataflow streaming job, allowing in-flight elements to complete before the workers shut down. The pipeline state moves to PAUSED.
Can I limit what my agent is allowed to do with the Data pipelines API?
Yes. Because you run Jentic One yourself, your own rules decide which operations and credentials the agent may use, and this API makes scoping straightforward since the parent and pipeline name travel in the URL path (/v1/{parent}/pipelines and /v1/{name}:run). You can pin the agent to a single pipeline and choose exactly which operations it may call, for example allowing it to list and run pipelines while withholding the stop operation. The Google OAuth credential is held by your instance and injected only at execution time, so the agent never sees the raw secret.
GET STARTED