First, do not buy a new design for at least a year or two. As you may know, the XDS 45 has been recalled and Springfield had no fix for the gun. They finally have a fix. Other guns have had recalls when they were released. This does not do you much good if you don't have the gun for awhile.
A gun that is 5.4" tall is a full sized gun. It is more difficult to conceal, as you well know. The key to carrying this gun is a heavy gun belt, a holster with forward cant and two spare mags on the other side to balance the rig. I have found that the Glock 19 size is very versatile at 4.75" tall. The Glock 26 at 4.17" tall is very, very concealable and any gun with a similar height will be just as good. The Browning High Power in 9x19mm is also exceptionally concealable due to the shape of the grip. The single stack guns conceal better, but six or seven shots goes very, very quickly.
The problem with going too small is the guns become difficult to shoot due to short sight radius and shorter grip. Capacity is also limited. The effects of recoil can be exaggerated. I think you will find guns such as the Glock 26, Kahr K9, Walther PPS, S&W M&P C, and Ruger SR9 C are very concealable while allowing the shooter sufficient gun to shoot well. Also, many of these guns accept full sized magazines or have longer magazines. You can use some sort of spacer to support more of your hand. I recommend against the very small guns such as the Kahr PM9, SIG P238, Colt Mustang, Ruger LCP and all small revolvers. Even the SIG P938 may be a bit small.
I typically carry my Glock 26 with a short 12 round magazine in the gun and at least one spare Glock 19 magazine with A&G Grip Adapter. The gun works well for 90% of my carry needs. It carries well strong side, in the appendix position, in a large pocket, or with SmartCarry.