I think that the answer is dependent on the level of technical talent that is already on-board. If you have an engineering focused squad, you can get away with rudimentary understanding. To your description a "non-technical CEO" is just that. Of course, whenever there is a gap in the team, it is generally the CEO's job to wear different types of hats as the weather changes. One of those hats may be a hacker.