First, let me congratulate you on choosing some really great speakers. I own both the Dirty Weekend Specials (in the office system) and also the Soul Superfly in the main system.
So, what you have are somewhat specialized speakers because they are an easy load on the amplifier and very high efficiency. They are also very revealing of the electronics that drive them which means that getting the most out of them requires pairing down the electronics to the minimum. If we consider that every component (tube, transistor, resistor, capacitor, volume control, interconnect cable, jack & plug) that the signal goes through alters it in some way (adds noise and distortion), even slightly, then we can begin to appreciate that the less we manipulate the signal, the cleaner the output to the speakers. Another way of saying it is that the signal will never be more perfect than it is at the output of the CD player or phono stage, so the goal is simply to amplify that signal to the point that it drives the speakers to listening levels.
So what does that mean in the real world? Since all that each of us can do is comment on our own experiences, I will tell you what I have done. For the main system using the Soul Superfly speakers the entire signal chain from the output of the CD player is: cable/jack - volume control - driver/voltage amp tube - EML 320B output tube - output transformer- cable - Zu speakers. OK, let's decipher that. The signal from the CD player goes by interconnect to the volume control on the amplifier; that attenuated signal is fed to the tube that amplifies the voltage necessary to drive the output tube to clipping; the amplified signal is fed to the grid of the output tube by direct coupling (no capacitor); the output tube works into the single ended output transformer which converts the high impedance of the triode into the low impedance (16 ohms) that the speaker needs.
Consider the signal chain: one interconnect, volume control, one driver tube, one output tube, output transformer, cable, speakers. That's the entire signal chain from CD player to speakers. There are no superfluous stages, no buffers, no extra stages of amplification that will simply be attenuated in the volume control, and all that adds up to a lack of noise and distortion in the signal chain.
So that's my recommendation: reduce the number of amplification stages, reduce the number of components the signal goes through, reduce the number of interconnects and with each reduction of an unnecessary component you are making the signal cleaner and clearer. These speakers sing with a high quality source into a single ended triode amp that is two stages: driver tube and output tube with a volume control at the input. Naturally, direct coupling between the two stages is preferred since that eliminates one capacitor in the signal chain.
I would suggest you experiment. If the volume control on the CD player can attenuate the signal in the digital domain without corrupting the signal, then use that directly into your Class D amp with the volume control maxed. Now try maxing the volume on the CD player and using the volume control on the Class D amp. Which sounds better? Eventually, you should get a two stage SET amp to try since they can be extremely transparent.