This is an experiment that you have to be very careful with, as you deal with open fire.
Choose a regular tea bag (not a triangular bag). Cut off the tea bag just below the closure and pour the contents. Fold up the empty bag to get a long, narrow, hollow cylinder of paper. The cylinder is placed upright on the ground and the top is lit.
What happens is that the fire spreads downwards and when it reaches the lower part of the ”vehicle” the remains of the bag lift due to the rising hot air.
The explanation is that during combustion the surrounding air is heated. Then the temperature increases, expands the air and thus gets lower density. The lower-density hot air rises upward and then gives rise to an airflow. The tea bag paper is drawn into the air stream as soon as the weight has dropped sufficiently. The weight decreases as the paper is burned.