1.1.4 Spark Session Timeouts and Auto Scaling

What is the default idle session timeout for Spark in Fabric?

20 minutes. After this period of inactivity, the session is terminated and resources are released.

How do you change the Spark session timeout?

In Workspace Settings > Spark Settings for workspace-wide, or in %%configure / session configuration for a specific notebook session.

How does Spark auto-scaling work in Fabric?

Fabric dynamically adds or removes executor nodes based on pending task queues and executor utilization, within the min/max node range of the pool.

What triggers Fabric to scale up Spark executors?

When there are pending tasks queued waiting for resources, indicating the current executor count is insufficient for the workload.

What triggers Fabric to scale down Spark executors?

When executors are idle with no pending tasks for a sustained period, Fabric removes excess nodes to free up capacity.

What is the impact of a longer session timeout on capacity?

Longer timeouts keep sessions alive during idle periods, consuming capacity units even when no work is being done.

What is Spark session queuing?

When all available Spark resources in the capacity are in use, new session requests are placed in a queue and start when resources become available.

What happens if a queued Spark session exceeds the queue timeout?

The session request fails with a capacity error. The user must retry when resources are available.

What is capacity burst in Fabric Spark?

A feature that allows temporary compute usage above the base capacity allocation for short spikes, with smoothing over a 24-hour window.

How do you monitor active Spark sessions?

Through the Monitoring Hub in the Fabric portal, which displays all active/recent Spark sessions, their duration, resource usage, and status.