pyspark.sql.SparkSession.builder.config

builder.config(key=None, value=None, conf=None, *, map=None)

Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession’s own configuration.

New in version 2.0.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

keystr, optional

a key name string for configuration property

valuestr, optional

a value for configuration property

confSparkConf, optional

an instance of SparkConf

map: dictionary, optional

a dictionary of configurations to set

New in version 3.4.0.

Returns

SparkSession.Builder

See also

SparkConf

Examples

For an existing SparkConf, use conf parameter.

>>> from pyspark.conf import SparkConf
>>> conf = SparkConf().setAppName("example").setMaster("local")
>>> SparkSession.builder.config(conf=conf)
<pyspark.sql.session.SparkSession.Builder...

For a (key, value) pair, you can omit parameter names.

>>> SparkSession.builder.config("spark.some.config.option", "some-value")
<pyspark.sql.session.SparkSession.Builder...

Set multiple configurations.

>>> SparkSession.builder.config(
...     "spark.some.config.number", 123).config("spark.some.config.float", 0.123)
<pyspark.sql.session.SparkSession.Builder...

Set multiple configurations using a dictionary.

>>> SparkSession.builder.config(
...     map={"spark.some.config.number": 123, "spark.some.config.float": 0.123})
<pyspark.sql.session.SparkSession.Builder...