streaming_job

Functions

get_config()

write_to_postgres(batch_df, batch_id, cfg)

Utile function to write the batch into the sql table using the connection

main()

Module Contents

streaming_job.get_config()
streaming_job.write_to_postgres(batch_df: pyspark.sql.DataFrame, batch_id: int, cfg: dict)

Utile function to write the batch into the sql table using the connection

Parameters:

  • batch_df (DataFrame) – Spark Dataframe to write into the DB

  • batch_id (int) – a unique identifier for the batch

  • cfg (dict) – DB connection configuration

Returns:

execution status

Return type:

int

streaming_job.main()