I think we worked out the math/geometry of the spectrometer.
First thing to note is that there is an ideal wavelength range where 100% of the light hits the spectrometer's CCD. This is because the focusing mirror is not big enough to capture all diffracted light outside the range of 45 to 75 degrees (as measured in the first picture).
The formula for the wavelength (λ) at each CCD pixel (p), λ=λ(p), depends on the angle of the collimated light leaving the collimating mirror (γ), the adjustable angle of the diffraction grid (δ) and focusing mirror (α) the pixel positon (p0) opposite the center of the focusing mirror and on the distance between them (f2). These parameters shown in the second picture where they are defined by the dashed black lines which are either parallel or perpendicular to the CCD surface. The formula is,
λ = d { sin(γ-δ) + cos(δ+2α+arctan[(p0-p)/f2]) }
Where d is the line width of the diffraction grating (for 1800 lines per mm, d = 555.6nm). Note that when fitting the equation the 3 adjustable angles only represent two free parameters. For fitting purposes one can simply use,
λ = d { A + cos(B+arctan[(C-p)/D]) }
and obtain the constants A, B, C, D from the fit which each have a physical meaning. This should be an alternative to the 3rd order polynomial approximation that the software uses whose fitted coefficients don't have a clear meaning (at least to me).
The third picture is an example of a setup that works well for a 532nm laser. The parameters used to generate the plot are:
γ = 20°, d = 555.6nm, p0 = 1240, f2 = 2700, δ = 3°, α = 13°.
The last two parameters (diffraction grid and focusing mirror angles) where adjusted while the others were kept constant to find this optimal setting, which I think is how one would adjust a focused spectrometer. Increasing (decreasing) δ moves the chart down (up) and increasing (decreasing) α = 13° moves the chart to the right (left).
Disclaimer: This is all just math at this point and has not been verified experimentally yet. There could be errors, but I've cross-checked the derivation several ways so I'm hoping it is correct. I can also post a working spreadsheet to tinker with the parameters, and/or the formula derivation if anyone is interested in that.