transformation.utils_helper

Submodules

• transformation.utils_helper.io_hepler

Functions

readFromDb(→ pyspark.sql.DataFrame)	Read the currect untreated day data from the database to be transformed
writeToDb(df, conf)	Write cleaned DataFrame to the database

Package Contents

transformation.utils_helper.readFromDb(session: pyspark.sql.SparkSession, conf: dict) → pyspark.sql.DataFrame

Read the currect untreated day data from the database to be transformed Args :

• session (SparkSession) : the intial spark session with the db driver to use to read from database

• cong (dict) : the dictionary that contain the connection parametres

transformation.utils_helper.writeToDb(df: pyspark.sql.DataFrame, conf: dict)

Write cleaned DataFrame to the database :param - df: The cleaned dataframe we want to write to the database :type - df: DataFrame :param - conf: the dictionary that contain the connection parametres :type - conf: dict