VoltSP Quickstart with YAML

Prerequisites

VoltSP 1.7+ installed and on your PATH
Kafka broker running on localhost:9092 (required for Steps 4–5)
VoltDB running on localhost:21212 (required for Step 5)

1 What We're Building

Source → Processor → Sink

Welcome! In this quickstart you'll build a VoltSP pipeline entirely in YAML — no Java, no compilation, no JAR files.

VoltSP pipelines follow a simple three-stage pattern:

Source

→

Processor

→

Sink

A Source produces data. Processors transform each record. A Sink consumes the output. You define all three in one YAML file and run it with a single command.

Tip: Start with the basic pipeline in Step 2. We'll walk through it and extend it step by step, ending with a full Kafka-to-VoltDB flow.

2 Basic Pipeline

Collection → Processor → Stdout

This pipeline has a collection source that emits five greeting strings, a simple processor that appends a newline (stdout doesn't add one automatically), and a stdout sink.

Structure

Every YAML pipeline needs four fields:

Structure

version: 1              # Required - must be 1
name: "pipeline-name"   # Required - pipeline identifier
source: {}              # Required - where data comes from
sink: {}                # Required - where data goes

The collection source is a fixed list of strings — perfect for testing. The stdout sink prints each record to the console.

Pipeline YAML

Save the following as hello-pipeline.yaml:

hello-pipeline.yaml

YAML

version: 1
name: "hello-pipeline"

source:
  collection:
    elements:
      - "Hello, Alice!"
      - "Hello, Bob!"
      - "Hello, Charlie!"
      - "Hello, Diana!"
      - "Hello, Eve!"

processors:
  - javascript:
      script: |
        function process(input) {
          return input + "\n";
        }

sink:
  stdout: {}

Running It

Run the pipeline

Shell

voltsp hello-pipeline.yaml

Expected Output

Hello, Alice!
Hello, Bob!
Hello, Charlie!
Hello, Diana!
Hello, Eve!

3 Add a Processor

Collection → JavaScript → Stdout

Now let's transform the data. Replace the processors section to convert each greeting to uppercase:

Replace the processors block

YAML

processors:
  - javascript:
      script: |
        function process(input) {
          return input.toUpperCase() + "\n";
        }

Processors are an array — you can chain multiple. Each receives the output of the previous stage. YAML pipelines support JavaScript, Python, and Ruby processors.

Expected Output

Output

HELLO, ALICE!
HELLO, BOB!
HELLO, CHARLIE!
HELLO, DIANA!
HELLO, EVE!

Filtering

Return null from a processor to drop the record:

Filter example

YAML

processors:
  - javascript:
      script: |
        function process(input) {
          if (input.includes("Eve")) return null;
          return input.toUpperCase() + "\n";
        }

Tip: Try editing the processor to apply different transformations. The pipeline uses whatever YAML is in the file when you run it.

4 Kafka Source

Kafka → JavaScript → Stdout

Now let's switch from a static collection to a live Kafka topic. This requires changing both the source and the processor — Kafka delivers records as Java KafkaRequest objects (not plain strings), so the processor must decode the ByteBuffer.

Save the following as kafka-pipeline.yaml:

Complete Kafka pipeline

YAML

version: 1
name: "kafka-pipeline"

source:
  kafka:
    bootstrapServers:
      - "localhost:9092"
    topicNames:
      - "greetings"
    groupId: "quickstart-group"
    startingOffset: "EARLIEST"
    properties:
      fetch.max.wait.ms: "100"
      fetch.min.bytes: "1"
      max.poll.records: "1"

processors:
  - javascript:
      script: |
        function process(input) {
          var Charset = Java.type(
            "java.nio.charset.StandardCharsets");
          var str = Charset.UTF_8
            .decode(input.getValue()).toString();
          return str.toUpperCase() + "\n";
        }

sink:
  stdout: {}

Key differences from the collection pipeline:

source.kafka — connects to a Kafka broker and subscribes to topics
input.getValue() — extracts the raw ByteBuffer from the KafkaRequest
Charset.UTF_8.decode() — converts the ByteBuffer to a string
The pipeline stays running, waiting for new messages (unlike collection which exits)

Test It

First, create the Kafka topic and run the pipeline:

Create topic and run pipeline

Shell

kafka-topics.sh --create --topic greetings --bootstrap-server localhost:9092

voltsp kafka-pipeline.yaml

In another terminal, send messages to the greetings topic:

Send Kafka messages

Shell

kafka-console-producer.sh --topic greetings \
  --bootstrap-server localhost:9092
> Hello, World!
> Hi from Kafka!
> Greetings, VoltSP!

Pipeline Output

HELLO, WORLD!
HI FROM KAFKA!
GREETINGS, VOLTSP!

Tip: Deploy the pipeline, then send messages from another terminal. You'll see uppercase output in the pipeline console.

5 VoltDB Sink

Kafka → JavaScript → VoltDB

Finally, replace stdout with a VoltDB stored procedure sink. This requires two changes:

Add a resources block that defines a VoltDB client connection
Replace the sink with a voltdb-procedure sink that references the resource

Resources Block

Add this before the source section:

Add resources block (before source)

YAML

resources:
  - name: "voltdb"
    voltdb-client:
      servers:
        - "localhost:21212"

Sink Block

Replace the sink section:

Replace the sink block

YAML

sink:
  voltdb-procedure:
    voltClientResource: "voltdb"

The voltdb-procedure sink calls a stored procedure for each record. The processor must return a VoltProcedureRequest Java object.

Update the Processor

The processor must create a VoltProcedureRequest using Java interop. Use Java.type() to import the class, decode the Kafka ByteBuffer to a string, then construct the request:

Replace the processors block

YAML

processors:
  - javascript:
      script: |
        var VoltProcedureRequest = Java.type(
          "org.voltdb.stream.plugin.volt.api.VoltProcedureRequest");
        var Charset = Java.type(
          "java.nio.charset.StandardCharsets");
        function process(input) {
          var str = Charset.UTF_8
            .decode(input.getValue()).toString();
          var parts = str.split("|");
          var sender = parts[0] || "unknown";
          var recipient = parts[1] || "unknown";
          var message = parts[2] || str;
          console.log(
            "[GREETING] " + sender +
            " -> " + recipient +
            ": " + message);
          var params = Java.to(
            [sender, recipient, message,
             java.lang.System.currentTimeMillis()],
            "java.lang.Object[]");
          return new VoltProcedureRequest(
            "InsertGreeting", params);
        }

VoltDB Setup

Before deploying, create the greetings table and stored procedure in VoltDB:

Create greetings table (run in sqlcmd)

SQL

CREATE TABLE greetings (
  sender VARCHAR(64) NOT NULL,
  recipient VARCHAR(64) NOT NULL,
  message VARCHAR(256) NOT NULL,
  greeting_time TIMESTAMP DEFAULT NOW
);
PARTITION TABLE greetings ON COLUMN sender;

CREATE PROCEDURE InsertGreeting
  PARTITION ON TABLE greetings COLUMN sender
AS
  INSERT INTO greetings (sender, recipient, message, greeting_time)
  VALUES (?, ?, ?, TO_TIMESTAMP(MILLISECOND, ?));

Complete Pipeline

Save the following as kafka-to-voltdb.yaml:

Full Kafka → VoltDB pipeline

YAML

version: 1
name: "kafka-to-voltdb"

resources:
  - name: "voltdb"
    voltdb-client:
      servers:
        - "localhost:21212"

source:
  kafka:
    bootstrapServers:
      - "localhost:9092"
    topicNames:
      - "greetings"
    groupId: "quickstart-group"
    startingOffset: "EARLIEST"
    properties:
      fetch.max.wait.ms: "100"
      fetch.min.bytes: "1"
      max.poll.records: "1"

processors:
  - javascript:
      script: |
        var VoltProcedureRequest = Java.type(
          "org.voltdb.stream.plugin.volt.api.VoltProcedureRequest");
        var Charset = Java.type(
          "java.nio.charset.StandardCharsets");
        function process(input) {
          var str = Charset.UTF_8
            .decode(input.getValue()).toString();
          var parts = str.split("|");
          var sender = parts[0] || "unknown";
          var recipient = parts[1] || "unknown";
          var message = parts[2] || str;
          console.log(
            "[GREETING] " + sender +
            " -> " + recipient +
            ": " + message);
          var params = Java.to(
            [sender, recipient, message,
             java.lang.System.currentTimeMillis()],
            "java.lang.Object[]");
          return new VoltProcedureRequest(
            "InsertGreeting", params);
        }

sink:
  voltdb-procedure:
    voltClientResource: "voltdb"

Test It

Deploy the pipeline, then send pipe-delimited messages (sender|recipient|message) to the greetings Kafka topic:

Send test messages

Shell

kafka-console-producer.sh --topic greetings --bootstrap-server localhost:9092
> Alice|Bob|Hello from the stream!
> Charlie|Diana|VoltSP is working!

Verify the data arrived in VoltDB:

Query after sending messages

SQL

SELECT * FROM greetings ORDER BY greeting_time DESC;

Expected Result

SENDER   RECIPIENT  MESSAGE                      GREETING_TIME
-------- ---------- ---------------------------- --------------------------
Alice    Bob        Hello from the stream!       2024-02-06 12:34:56.789000
Charlie  Diana      VoltSP is working!           2024-02-06 12:34:57.123000

Note: This is the same Kafka → VoltDB pattern used in production at hundreds of thousands of events per second.

6 Deploy Pipeline

Kafka → JavaScript → VoltDB

Your pipeline is ready! Run it with a single command:

Deploy the pipeline

Shell

voltsp kafka-to-voltdb.yaml

What Happens on Deploy

The YAML is validated for correct structure
VoltSP parses source, processors, and sink definitions
The pipeline starts consuming from the configured source
Each record flows through processors and into the sink

What You Learned

YAML pipeline structure — version, name, source, processors, sink
Collection source — fixed data for testing
JavaScript processors — inline transformation logic
Kafka source — consume from real topics
VoltDB sink — write to stored procedures
Deploy — one command, no compilation needed

Ready for more? Switch to the Advanced tutorial to learn Java pipelines with the full VoltSP SDK, type safety, and production patterns.

VoltSP Quickstart (YAML) — Quick Start Guide

Prerequisites

Structure

Pipeline YAML

Running It

Expected Output

Filtering

Test It

Resources Block

Sink Block

Update the Processor

VoltDB Setup

Complete Pipeline

Test It

What Happens on Deploy

What You Learned