Don't sit on your wallet, and lean your seat a little further back, like 1 click. The only thing sitting on a pillow is doing for your lower back is letting you rock your hips. If you realize your hips are tilted more forward or back on the pillow, you can use the seat height adjuster to better fit your body. Lean your seat a little further back if you sit very upright. The only support your seat can provide to your lower back will be while it is resting against it.
It is probably your wallet, though. Sitting on a pillow will help offset some of the wedge that drives into your hips.