If you want to set the alternate key to detecting duplicate record based on lookup it was not earlier. If you want to have duplicated detection based on you would not be able to that you have to write a plugin to get that result.
You can now achieve a duplicate detection rule by defining the lookup as an alternate key. You can create an alternate key by following the below steps
- To define the alternate keys, go to Go to Settings> Customizations
- Choose Customize the System–>Components ->Entities –> Entity–>Keys. Choose New.
3. On the form, fill in the required fields (Display Name and Name) and then choose and add the fields to the key. Save the key. In the example shown here, we used the ParentId field in the alternate key definition.
Hope this helps while defining the alternate key with lookup/entity reference