streaming_job

Functions

get_config()
write_to_postgres(batch_df, batch_id, cfg)	Utile function to write the batch into the sql table using the connection
main()

Module Contents

streaming_job.get_config()

streaming_job.write_to_postgres(batch_df: pyspark.sql.DataFrame, batch_id: int, cfg: dict)

Utile function to write the batch into the sql table using the connection

Parameters:

• batch_df (DataFrame) – Spark Dataframe to write into the DB

• batch_id (int) – a unique identifier for the batch

• cfg (dict) – DB connection configuration

Returns:

execution status

Return type:

int

streaming_job.main()