From Annotations to Interactions - Putting it altogether