Spark Structured Streaming in Apache Spark 2.X version

From the Spark 2.x release onwards , Structured Streaming API was included . Built on Spark SQL library, It makes easier to apply SQL query(Data frame API ) or scala operations(Dataset API) on streaming data.

Architecture of Structured Streaming:

Structured Streaming works on the same platform like Spark Streaming of polling the data after some duration. It is based on the trigger interval, but some differences are there in structured streaming and makes it similar to real streaming. No batch concept is present in Structured Streaming. The received data in a trigger is appended to the continuously flowing data stream. Each row of the data stream is processed and the result is updated into the unbounded result table. The output result (updated, new result only, or all the results) depends on the output mode of the operations (Complete, Update, Append) we set.

i. Complete Mode

When updated result table will be written to the external sink, that mode is Complete Mode.

ii. Append Mode

In append mode only the new rows will be appended to the result table since the last trigger and we will write those rows to the external sink. This mode is applicable when existing rows in result table are not expected to change.

iii. Update Mode

In this mode, only the rows that were updated in the result table since the last trigger will be written to the external sink. Unlike complete mode, this only outputs the rows that have changed since the last trigger and it will be as same as Append mode when query doesn’t contain aggregations.

It is scalable ,fault-tolerant stream processing engine and provides end to end guarantees of data delivery.

It uses checkpointing and also applies two conditions to recover from any error:

1. The source must be replayable.

2. The sinks must support idempotent operations to support reprocessing in case of failures.

With restricted sinks, Spark Structured Streaming always provides end-to-end, exactly once semantics.

See you in next blog!!

Originally published at https://itechshree.blogspot.com.