Handle lack of DSM slots in parallel btree build, take 2.

Commit 74618e77 added a new check intended to fix a bug, but put
it in the wrong place so that parallel btree build was always
disabled.  Do the check after we've actually tried to create
a DSM segment.  Back-patch to 11, like the earlier commit.

Reviewed-by: Peter Geoghegan
Discussion: https://postgr.es/m/CAH2-WzmDABkJzrNnvf%2BOULK-_A_j9gkYg_Dz-H62jzNv4eKQTw%40mail.gmail.com

src/backend/access/nbtree/nbtsort.c

@@ -1332,14 +1332,6 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 	pcxt = CreateParallelContext("postgres", "_bt_parallel_build_main",
 								 request);
 
-	/* If no DSM segment was available, back out (do serial build) */
-	if (pcxt->seg == NULL)
-	{
-		DestroyParallelContext(pcxt);
-		ExitParallelMode();
-		return;
-	}
-
 	scantuplesortstates = leaderparticipates ? request + 1 : request;
 
 	/*
@@ -1383,6 +1375,16 @@ _bt_begin_parallel(BTBuildState *buildstate, bool isconcurrent, int request)
 	/* Everyone's had a chance to ask for space, so now create the DSM */
 	InitializeParallelDSM(pcxt);
 
+	/* If no DSM segment was available, back out (do serial build) */
+	if (pcxt->seg == NULL)
+	{
+		if (IsMVCCSnapshot(snapshot))
+			UnregisterSnapshot(snapshot);
+		DestroyParallelContext(pcxt);
+		ExitParallelMode();
+		return;
+	}
+
 	/* Store shared build state, for which we reserved space */
 	btshared = (BTShared *) shm_toc_allocate(pcxt->toc, estbtshared);
 	/* Initialize immutable state */