mirror of https://github.com/raandree/NTFSSecurity
Browse Source
Decision 24 (proposed): the matrix lab, what its deployment and the runs showed, and what the maintainer decides. The context, patterns, progress, and deployment notes carry the lessons: the Authz regime of a domain member, the stale Kerberos S4U state of a re-created account, the evaluation client, and the integration of the stacked branches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: AI Assistant <ai@example.com>pull/119/head
Raimund Andree
2 days ago
6 changed files with 402 additions and 22 deletions
@ -0,0 +1,136 @@ |
|||
--- |
|||
status: proposed |
|||
date: 2026-10-09 |
|||
last-verified: 2026-10-10 |
|||
owner: shared |
|||
source: agent decisions under the maintainer's delegation of 2026-10-09 (Handoff 2); the scope follows Decision 21, phase 3 |
|||
--- |
|||
|
|||
# Decision 24: The operating-system matrix lab |
|||
|
|||
- Context: Decision 21 requires the live tests on more operating systems, |
|||
"such as a Windows 11 client and Server 2019 and 2022 file servers", and the |
|||
published package must pass them. Every machine of `WindowsAccessControlLab` |
|||
is Server 2025. Handoff 2 asks for the maintainer's approval of scope and |
|||
topology before new VMs. On 2026-10-09 at 21:21 UTC the maintainer, going to |
|||
bed, wrote "you can do whatever is required with the lab" and told the agent |
|||
to decide and report later. The agent took that as the approval for the |
|||
minimal matrix below and for nothing broader. It is the agent's decision, so |
|||
the status stays `proposed` until the maintainer confirms it. |
|||
- Choice: |
|||
1. Cells: a Windows 11 client with each file server (Server 2019 Datacenter |
|||
10.0.17763.1217, Server 2022 Datacenter 10.0.20348.4773, Server 2025 |
|||
Datacenter 10.0.26100.32690), the module in Windows PowerShell 5.1 and |
|||
PowerShell 7 in every cell. The client was planned as Windows 11 Pro |
|||
26H1 (10.0.28000.1836). It cannot keep a secure channel to the Server 2025 |
|||
domain controller (see "What the deployment showed"), so the domain client |
|||
is `OSWin11E`, Windows 11 Enterprise Evaluation 22H2 (10.0.22621.525), and |
|||
the 26H1 machine `OSWin11` stays in the lab outside the domain for runs of |
|||
the module's own tests. The reference cell of Decision 20 (Server 2025 |
|||
client and file server in `WindowsAccessControlLab`) stays as it is. |
|||
Server 2019 and 2022 as clients are extra cells, to run the module on the |
|||
older .NET Framework builds (Server 2019 has 4.7.2). |
|||
2. Topology: a separate AutomatedLab lab `NtfsSecurityOsMatrixLab` with its |
|||
own internal switch (`192.168.12.0/24`) and its own forest `osmatrix.net`: |
|||
`OSDC1` (Server 2025, root domain controller), `OSFile19`, `OSFile22`, |
|||
`OSFile25`, `OSWin11E`, and `OSWin11`. `Deploy-OsMatrixLab.ps1` deploys it |
|||
with the maintainer's AutomatedLab and the VM path `V:\AutomatedLab-VMs`; |
|||
`Add-OsMatrixMachine.ps1` adds a machine to the deployed lab; the |
|||
payloads of the existing lab (PowerShell 7.6.3 and Pester 5.7.1 from the |
|||
host, because the VMs have no internet) come from |
|||
`Complete-OsMatrixLab.ps1`. |
|||
3. Case 9 (accounts of other domains and forests) needs trusts to the |
|||
forests of the existing lab, so the matrix cells run with |
|||
`-ForeignDomainController @()`; the existing lab keeps that case. |
|||
4. The controller of the repository runs in every cell with `-LabName`, |
|||
`-DomainController`, `-FileServer`, and `-Client`. The matrix showed three |
|||
defects of its setup and removal, fixed in `7d47316`: a recursive delete |
|||
fails with "The directory is not empty" on Windows Server 2019 (and the |
|||
stderr line ended the script before any retry, because `2>&1` under `Stop` |
|||
is terminating in Windows PowerShell 5.1), `Get-LocalGroupMember` fails on |
|||
an orphaned SID, and a vanished profile failed the client cleanup. |
|||
5. The module's own behavior tests run on every machine as well |
|||
(`Run-MatrixLocalSuite.ps1`), elevated and as a basic user, in both |
|||
editions, as scheduled tasks so that the token matches a CI runner. This |
|||
found three defects of the module, each fixed in its own commit on |
|||
`ai/quality-gate-lab-matrix`: `Get-NTFSInheritance -SecurityDescriptor` |
|||
and `Get-NTFSEffectiveAccess -ServerName ''` (`962887a`), and |
|||
`Get-NTFSEffectiveAccess` for a user who isn't an administrator on a |
|||
computer in a domain (`fdd7a8b`, with a live test for the ServerAdmin |
|||
role). The maintainer decides which of them belong to rc7. |
|||
- Why a separate lab: AutomatedLab 5.61 refuses to add machines to an |
|||
imported lab, and defining a lab under an existing name would overwrite the |
|||
metadata of its 13 machines. A separate lab leaves every shared machine, |
|||
switch, domain, and account untouched, which Handoff 2 requires. Inside the |
|||
new lab, a machine can be added with `Import-LabDefinition`, |
|||
`Add-LabMachineDefinition`, and `Export-LabDefinition`, followed by the steps |
|||
that `Install-Lab` runs for one machine; `Add-OsMatrixMachine.ps1` does this |
|||
after it copies the lab metadata. |
|||
- Cost and rollback: six VMs (4 GB for the domain controller and both clients, |
|||
3 GB for each file server), four new base images, measured at 17.8 GB for |
|||
the differencing disks and 42.4 GB for the base images (about 60 GB on `V:`; |
|||
my first estimate of 100 GB was too high). The deployment added twelve lines |
|||
to the hosts file of the host, which `Remove-Lab` removes. To remove the |
|||
matrix, run `Remove-Lab -Name NtfsSecurityOsMatrixLab` from AutomatedLab; |
|||
nothing else depends on it. No existing machine, checkpoint, or lab was |
|||
changed. A copy of the lab metadata from before the sixth machine is in |
|||
`C:\ProgramData\AutomatedLab\Backups` (administrators only). |
|||
- What the deployment showed (the agent's decisions D11 to D23 of the night |
|||
log, each reversible): |
|||
- The base image of a Server 2019 or a Windows 11 22H2 machine had an empty |
|||
EFI system partition: the `bcdboot` of the Server 2025 host fails with |
|||
exit code 193 on their boot files, and AutomatedLab ignores the exit code, |
|||
so the generation 2 machine fails with Hyper-V event 18603. The images of |
|||
Server 2022 and Windows 11 26H1 are fine. `Repair-OsMatrixBoot.ps1` runs |
|||
the `bcdboot` of the image itself on the differencing disk of the one |
|||
machine and starts it. |
|||
- The AutomatedLab driver sat in its file-server job wait with idle remote |
|||
runspaces after all features were installed, so I stopped it and ran the |
|||
rest by script. |
|||
- Windows 11 26H1 (10.0.28000.1836) joins the domain but cannot keep the |
|||
Netlogon secure channel to the Server 2025 domain controller |
|||
(10.0.26100.32690). The client calls `NetrLogonGetCapabilities` with query |
|||
level 2, which the protocol document describes as a check of the flags the |
|||
client sent; the controller answers `STATUS_ACCESS_DENIED` (level 1 and |
|||
`NetrServerAuthenticate` succeed), and the client denies the channel |
|||
(`NlConfirmRequestedCapabilities: denying access ... 0xc0000022`). |
|||
`Test-ComputerSecureChannel -Repair` can't help. Windows 11 22H2 against the |
|||
same controller works (secure channel, Kerberos, readiness). This is an |
|||
environment finding about two Microsoft builds, not about NTFSSecurity; the |
|||
maintainer may want to know it for his own labs. |
|||
- The Windows 11 Enterprise Evaluation 22H2 image (`OSWin11E`) is in |
|||
notification mode from its first day and shuts down an hour after every |
|||
start (`wlms.exe`, 0xC004F009 "grace time expired"): its install time was |
|||
recorded on a clock about seven hours ahead, which was then corrected. One |
|||
of its two rearms didn't help. After an unplanned shutdown its machine |
|||
account password no longer matched (domain logons fail with 0xC000018D, |
|||
`nltest /sc_verify` says `ERROR_INVALID_PASSWORD`); |
|||
`Test-ComputerSecureChannel -Repair` with the lab account fixed it, and |
|||
`/sc_verify` kept showing the old status afterwards, so test a domain |
|||
session instead. A run on this machine has to stay under an hour from its |
|||
start. |
|||
- A profile of an account that a probe's scheduled task used stayed loaded on |
|||
one server until it restarted; the probe now uses a new account name for |
|||
every run. |
|||
- The fixture of the live controller deleted its accounts after a cell and |
|||
created them again with the same names for the next. Windows then returns |
|||
the SID and the groups of the deleted account for a Kerberos S4U logon on |
|||
the domain controller and the file server for more than seven minutes, so |
|||
`Get-NTFSEffectiveAccess` returned no access for the new account in some |
|||
cells (Windows Server 2022, Admin role), for the baseline and the final |
|||
candidate alike. This looked like a regression of the module until a loop |
|||
probe showed both builds failing the same way. The controller now gives a |
|||
new fixture a new name for the account of case 3 (`1dec389`); the record |
|||
has the evidence. |
|||
- Result: [the record](../../Tests/Lab/Acceptance-2026-10-10-os-matrix.md). The |
|||
final candidate (`fdd7a8b`) passes the module's suite on all five machines |
|||
and the host in all four configurations, and the live cells (see the record). |
|||
- Open: the maintainer confirms or changes the matrix and decides whether to |
|||
keep the VMs after 5.0.0. Local `-ModulePath` runs are validation; the gate |
|||
needs the published package in every cell (Handoff 3, stage D). The newest |
|||
Windows 11 build that can join a Server 2025 domain here is 22H2; a domain |
|||
cell with 26H1 needs a newer domain controller build or a fix of the |
|||
mismatch. The maintainer also decides which of the three module fixes belong |
|||
to rc7 (each is its own commit, `git revert` removes it), and whether the |
|||
evaluation client stays (it needs a start shortly before every run) or is |
|||
replaced by a client with a license that doesn't expire. |
|||
@ -0,0 +1,135 @@ |
|||
--- |
|||
status: current |
|||
last-verified: 2026-10-10 |
|||
owner: software-engineer |
|||
source: release gates of 5.0.0 (lab acceptance, OS matrix, publication plan), repository evidence |
|||
--- |
|||
|
|||
# Deployment notes |
|||
|
|||
## Publish the next prerelease (rc7) |
|||
|
|||
State on 2026-10-09: #116 (rc7, head `d25647d`, base `master`), #117 (head |
|||
`f11ff41`, base `ai/release-5.0.0-rc7`), and #118 (draft, head `83149ee`, |
|||
base `ai/quality-gate-coverage`) are open and green. A local merge in that |
|||
order, each with a merge commit (Decision 15), ends in exactly the tree of |
|||
#118 (`b1dc006`), without conflicts. The manifest says `5.0.0` with |
|||
`Prerelease = 'rc7'`, and `$publishedVersions` in `Tests/Repository.Tests.ps1` |
|||
lists the versions up to rc6, as it must before rc7 is published. |
|||
|
|||
The branch `ai/quality-gate-lab-matrix` (local until the maintainer pushes it) |
|||
is stacked on #118. It holds three fixes of the module (`962887a`, `fdd7a8b`; |
|||
each is its own commit and `git revert` removes it), the kit of the |
|||
operating-system matrix, the changes of the live controller, and the record |
|||
(Decision 24). rc7 contains the module fixes only if the branch is merged after |
|||
#118 and before the tag; otherwise they go to the next prerelease. The |
|||
maintainer decides; open it as a draft with base `ai/quality-gate-paths`. |
|||
|
|||
1. Merge #116, then #117, then #118 into `master`, each with **Create a merge |
|||
commit**. Deleting the merged head branch makes GitHub retarget the next |
|||
pull request to `master`; otherwise change its base. Wait for green CI on |
|||
the retargeted pull request. |
|||
2. Tag the merge commit on `master` with `5.0.0-rc7` and push the tag. The |
|||
`release` job checks the tag against the manifest and builds nothing new: |
|||
it publishes the package that the `build` job tested. Approve the |
|||
deployment of the `powershell-gallery` environment if it asks. |
|||
3. After the publication, add `5.0.0-rc7` to `$publishedVersions` with the |
|||
next change that goes to `master`. |
|||
|
|||
## Accept a published package |
|||
|
|||
Local `-ModulePath` runs are validation; the gate needs the published bytes. |
|||
|
|||
1. `Tests/Lab/Acceptance/Test-PublishedRelease.ps1 -Version <version> |
|||
-OutputPath <folder>` (read-only): tag and commit on `master`, the CI run |
|||
of the tag, the Gallery's SHA-512 against the downloaded nupkg (ordinal, |
|||
case-sensitive base64), the nupkg against the GitHub zip file by file, and |
|||
the identity of the manifest. Dry run on rc6: all checks passed. |
|||
2. `Tests/Lab/Invoke-NTFSSecurityLabTest.ps1 -Version <version>` in the |
|||
existing lab, both editions, and `Tests/Lab/Acceptance/Run-MatrixSequence.ps1 |
|||
-Version <version>` for each cell of the matrix (Decision 24). Check every |
|||
role from the result files with `Validate-LabResults.ps1`, never from the |
|||
marker `DONE` of the controller. |
|||
3. Remove the fixture and check the end state independently with |
|||
`Test-MatrixCleanup.ps1`, which takes the lab name and the machine names |
|||
(`-LabName WindowsAccessControlLab -DomainController F1ADC1, F1BDC1, F2DC1, |
|||
F3DC1 -Machine F1AFile1, F1AFile2` for the existing lab). |
|||
4. If the published binary changes, repeat the cells; never combine runs of |
|||
different binaries into one matrix. |
|||
|
|||
## Lab lessons |
|||
|
|||
- AutomatedLab 5.61 can't add machines to an imported lab (`Add-LabMachineDefinition` |
|||
throws "Lab is already imported"), and `New-LabDefinition` under an existing |
|||
name overwrites its metadata. New machines go into a new lab with its own |
|||
switch and domain, and `-LabName` of the controller selects it. |
|||
- `Install-Lab -NetworkSwitches -BaseImages` creates the switch and the base |
|||
images first; the base images of Server 2019, Server 2022, and Windows 11 Pro |
|||
took about two to four minutes each from the ISO files. AutomatedLab |
|||
adds records to the hosts file of the host, which `Remove-Lab` removes. |
|||
- The VMs have no internet: take PowerShell 7 and Pester 5.7.1 from the host |
|||
(`Copy-LabFileItem`, `Install-LabSoftwarePackage`). |
|||
- AutomatedLab ignores the exit code of `bcdboot` when it builds a base image. |
|||
The Server 2019 image that it built on this Server 2025 host got an empty |
|||
EFI system partition: the `bcdboot` of the host fails with exit code 193, |
|||
"Failure when attempting to copy boot files", on the 2019 boot files, and the |
|||
VM failed to boot (Hyper-V event 18603, "failed to boot an operating |
|||
system"; no memory demand, no IP, heartbeat `NoContact`). Check the EFI |
|||
system partition of a new base image before the first VM: mount the image |
|||
read-only (`Mount-DiskImage -Access ReadOnly`) and look for |
|||
`EFI\Microsoft\Boot\bootmgfw.efi` (the images of Server 2022 and Windows 11 |
|||
Pro had 140 and 149 files). Repair a VM, not the base: stop the VM, mount its |
|||
own differencing disk, run the `bcdboot.exe` of the image (`D:\Windows\System32\bcdboot.exe |
|||
D:\Windows /s H: /f UEFI`), copy `bootmgfw.efi` to `EFI\Boot\bootx64.efi`, |
|||
dismount, and start the VM. A changed base image would invalidate its |
|||
differencing disks. |
|||
- The tool output of the agent masks text that looks like a secret, such as |
|||
`-Password $password`, in what it shows. Test such a line by parsing the |
|||
file, and don't repair it from the displayed text. |
|||
- A script that a detached process runs needs its own log, an exit marker, and |
|||
an end-state check of its own; verify cleanup from the end state, not from |
|||
its marker. |
|||
- Extend a deployed lab with one machine like this: in a process that never ran |
|||
`Import-Lab`, call `Import-LabDefinition`, `Add-LabMachineDefinition`, and |
|||
`Export-LabDefinition`, then `New-LabBaseImages` and `New-LabVM -Name <machine>`. |
|||
`Install-Lab` has no per-machine selector, and `Add-LabMachineDefinition` |
|||
throws as soon as `Get-Lab` returns a lab. `Add-OsMatrixMachine.ps1` does it |
|||
after it copies the lab metadata to `C:\ProgramData\AutomatedLab\Backups`. |
|||
- Windows 11 22H2 (10.0.22621) has the empty EFI system partition problem too |
|||
(`bcdboot` exit code 193 on the host); `Repair-OsMatrixBoot.ps1` repairs it. |
|||
After `Mount-VHD` the host gives the NTFS partition a letter on its own; don't |
|||
assign a second one. |
|||
- Run AutomatedLab processes one after the other. Two `Import-Lab` calls at the |
|||
same time corrupt each other (XML errors, "No machines imported"). |
|||
- Check the secure channel of every domain client before the first run |
|||
(`Test-MatrixReadiness.ps1`). Windows 11 26H1 (10.0.28000.1836) joins a |
|||
Server 2025 domain (10.0.26100.32690) but loses the channel: the client asks |
|||
`NetrLogonGetCapabilities` for query level 2, the domain controller answers |
|||
`0xC0000022`, and the client denies the channel. Rejoining doesn't help. |
|||
- A process that starts from a remoting session has every privilege enabled |
|||
and no credentials of its own, so tests that expect disabled privileges fail |
|||
(eight per edition). Run the suite of a VM as a scheduled task with a batch |
|||
logon at the highest run level (`Register-ScheduledTask -RunLevel Highest |
|||
-User -Password`): that token matches a CI runner. A restricted token (a basic |
|||
user) can't run Pester's NUnit export, because it asks WMI for the |
|||
environment, so write the JSON summary first. |
|||
- In Windows PowerShell 5.1, `$PSScriptRoot` is empty in a parameter default of a |
|||
script that runs with `-File`; compute it in the body. `-File` passes an array |
|||
as one string, so split on commas. With `$ErrorActionPreference = 'Stop'`, a |
|||
line that a native command writes to stderr and that `2>&1` redirects is a |
|||
terminating error; let the command write its errors to stdout. |
|||
- `Get-LocalGroupMember` fails with "Failed to compare two elements in the array" |
|||
when the group holds an orphaned SID. Add members with `Add-LocalGroupMember` |
|||
and ignore `MemberExistsException`; read and remove members with |
|||
`net localgroup <name>` and `net localgroup <name> <SID> /delete`. The SIDs |
|||
of a deleted account can't be found afterwards, so keep `fixture-sids.json` |
|||
from before the removal of the organizational unit. |
|||
- An account that is deleted and created again with the same name keeps its old |
|||
SID and groups in Kerberos S4U logons on the domain controller and member |
|||
servers for more than seven minutes, and nothing flushes it (see |
|||
`techContext.md`). The controller names the account of case 3 anew for each |
|||
new fixture; a script of your own that recreates accounts needs unique names |
|||
too. |
|||
- Restart the evaluation client (`OSWin11E`) right before a sequence or a suite, |
|||
not before several: it shuts down an hour after each start. The restart takes |
|||
about two and a half minutes and may need the repair of the secure channel. |
|||
Loading…
Reference in new issue