Well, in the first line, the word "to" shouldn't be there. You don't use the infinitive "to have" in that context because the pronoun "I" modifies the verb.
In the second line, yes it should be "broken" rather than "broke" because you are using the present perfect tense ("you have").
So there's your little grammar lesson for the day. ;)