Batch Processing Example

This example demonstrates KDeps' items iteration feature for processing multiple items in parallel or sequentially.

Overview

Items iteration allows you to process arrays of data efficiently:

• Process multiple API requests
• Transform collections of data
• Aggregate results from multiple sources
• Parallel or sequential execution

Features Demonstrated

1. Items Iteration - Process array elements with items: "{{...}}"
2. Conditional Skip - Skip resources based on conditions
3. Result Aggregation - Combine results from multiple iterations
4. Context Variables - Access {{item}} in iterations

Running the Example

# Start the server
kdeps run examples/batch-processing/workflow.yaml

# In another terminal, send a batch request
curl -X POST http://localhost:16395/process \
  -H "Content-Type: application/json" \
  -d @test-request.json

Example Request

{
  "items": [
    {"org": "kubernetes", "repo": "kubernetes"},
    {"org": "docker", "repo": "docker"},
    {"org": "golang", "repo": "go"}
  ]
}

The workflow will:

1. Fetch GitHub repo data for each item (parallel execution)
2. Transform each result to extract star counts
3. Aggregate all results into a final summary

How Items Iteration Works

1. Declare Items Array

resources:
  - id: process-item
    items: "{{input.items}}"  # Array from request
    config:
      # ... configuration that uses {{item}}

2. Access Current Item

Inside an items-iterating resource, use {{item}} to access the current element:

url: "https://api.example.com/{{item.id}}"

3. Chain Iterations

You can iterate over previous results:

resources:
  - id: first
    items: "{{input.data}}"
    # ... processes each item

  - id: second
    dependsOn: ["first"]
    items: "{{first}}"  # Iterate over first's results
    # ... processes each result from first

Use Cases

API Batch Requests

Process multiple API endpoints in parallel:

items: "{{input.user_ids}}"
config:
  url: "https://api.example.com/users/{{item}}"

File Processing

Process multiple files:

items: "{{ctx.files}}"
config:
  command: "convert"
  args: ["{{item.path}}", "-resize", "800x600", "{{item.path}}.thumb.jpg"]

Data Transformation

Transform array elements:

items: "{{input.records}}"
config:
  prompt: "Summarize this record: {{item}}"

Parallel LLM Calls

Process multiple prompts in parallel:

items: "{{input.questions}}"
config:
  model: "llama3.2"
  prompt: "Answer this question: {{item}}"

Advanced Patterns

Conditional Processing

Skip items based on conditions:

items: "{{input.items}}"
skip: "{{item.status == 'processed'}}"

Nested Iterations

Process multidimensional data:

# First level
- id: process-categories
  items: "{{input.categories}}"

  # Second level (processes each category's items)
  - id: process-items-in-category
    items: "{{item.items}}"

Result Filtering

Filter results before aggregation:

- id: aggregate
  dependsOn: ["process-items"]
  config:
    # Only aggregate successful results
    data: "{{filter(process-items, 'success == true')}}"

Performance Notes

• Items are processed in parallel by default
• Use dependsOn to force sequential processing
• Large arrays are handled efficiently
• Results are collected automatically

Next Steps

• See complex-workflow for advanced dependencies
• See http-advanced for API request patterns
• Combine with LLM for batch AI processing