transformation.utils_helper.io_hepler

Functions

readFromDb(→ pyspark.sql.DataFrame)

Read the currect untreated day data from the database to be transformed

writeToDb(df, conf)

Write cleaned DataFrame to the database

Module Contents

transformation.utils_helper.io_hepler.readFromDb(session: pyspark.sql.SparkSession, conf: dict) → pyspark.sql.DataFrame

Read the currect untreated day data from the database to be transformed Args :

  • session (SparkSession) : the intial spark session with the db driver to use to read from database

  • cong (dict) : the dictionary that contain the connection parametres

transformation.utils_helper.io_hepler.writeToDb(df: pyspark.sql.DataFrame, conf: dict)

Write cleaned DataFrame to the database :param - df: The cleaned dataframe we want to write to the database :type - df: DataFrame :param - conf: the dictionary that contain the connection parametres :type - conf: dict