transforms
==========

.. py:module:: transforms


Functions
---------

.. autoapisummary::

   transforms.parse_kafka
   transforms.apply_transforms


Module Contents
---------------

.. py:function:: parse_kafka(df: pyspark.sql.DataFrame, schema: pyspark.sql.types.StructType) -> pyspark.sql.DataFrame

   Util function to parse the json message based on a given schema
   :param df: A stream based dataframe from the kafka stream read
   :type df: DataFrame
   :param schema: schema definition
   :type schema: StructType

   :returns: parsed json of the stream dataframe
   :rtype: df (DataFrame)


.. py:function:: apply_transforms(df: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame

   Util apply basic transformation to the DataFrame

   :param df: flawed Spark DataFrame
   :type df: DataFrame

   :returns: rectified DataFrame
   :rtype: parsed (DataFrame)