Optimizations add complexity.
Visualize two curves: increasing performance, decreasing understandability.
Different projects have different sweet spots. This depends on team composition, the time the project will live in production, etc.
Decide wisely!