How do you handle audit logs in your systems?

I'm working on designing audit logging for a system that needs to track actions like data changes, user access, and administrative operations. The goal is to ensure traceability, support compliance, and assist with incident response.

I'm curious how others handle this in production:

- What data do you log?

- How do you structure audit logs (JSON, text, DB records)?

- How do you ensure logs are immutable/tamper-evident?

- Do you store them separately from application logs?

- What tooling or patterns have worked well (or poorly) for you?

Any war stories, best practices, or pitfalls to avoid would be really appreciated.