StadyShellNoIRFilter
Filtering inward infrared does not work on a complete shell. It may work somewhat with an "off axis" leaky shell with radially offset perforations to a rear shell.
For now, ignore the IR effects of planets. They will capture light and IR over various bands, and emit IR, but in equilibrium, they do not change the outbound energy flux.
Assume a filter (for some band of IR) with an emissivity of ε, between zero and one. Assume the complete shell contains a reflected flux of F in equilibrium, and the captured solar flux is L. Then the black-body heat emitted outwards from the shell is L, and the IR heat flux emitted inwards is εL .
The inward-emitted heat flux has no place to go - the Sun and planets are too small to intercept much of it. So F will build up until the shell is absorbing the same amount of flux as leaving: εF = εL . So, F = L, the same condition as with ε=1 and no infrared filtering.
Bilevel shell
What if the shell is not continuous, but has large holes that can emit infrared? Can we still let out the IR so the solar system stays cooler. One way to do that is to construct a pair of 50% shells at two different radii, inner radius R_1 and outer radius R_2 , with segment sizes large enough to minimize edge effects from the finite size of the Sun. The size of the edge effects E are the pinhole projection of the sun on the outer sphere:
E = R_S (R_2 - R_1) / R_1
The Sun's radius is R_S = 696,340 km = 0.465e-3 AU . If (WAG) R_1 = 45 AU and R_2 = 55 AU, then E = 1.03e-3 AU.
Assume the outer segments are circles with radius R_G, positioned above circular holes in the inner shell. They probably should be arranged as vertices of an icosahedral geodesic grid, and with spacing-to-radius set for minimum inside radiation density, but that optimization can come later. For now, assume a square grid with round holes, center-to-center spacing at 4 times the radius.
Assuming thinsats cannot be thinner than 50% of the normal size, then some sunlight will escape past the edges of the outer segments.