Currently, there is no way to get the list of recent contacts or starred contacts for a specific user.
This limits the ability to develop integrations that takes into consideration the actual people a user communicates with.
We should have API to get Recent Contacts and Starred Contacts, possible add this to the /chat/users/me/contacts endpoint as an additional type parameter.
An alternative is to read the user Chat Message Events (Webhook) and use this information to construct a list of recent contacts, but in many cases, an App will only need to get a list of users and not read user Massages.