Hi,
This is likely to be caused by the 'parental' nature of the relationship between records and their activities. By default this allows the owner of the parent record to have access to the child records as though they own them as well.
If you want to change this, you need to go into the relationship from the parent to the letters/tasks/emails etc. and change from 'Parental' to 'Configurable Cascading' change the 'Reparent' to 'Cascade None'
This will break the relationship so that security will be honoured on the child records irrespective of their parent.
hth