No, you cannot legitmately deduct anything from the deposit because it is the tenant's money and remains so until a deposit scheme adjudicator decides otherwise. Since you did not protect it, you cannot appeal to an adjudicator. You should return the money you have kept back and hope the 'acquaintance' doesn't sue you for non-compliance with deposit protection, as advised above.
Yes you could sue your T for the cost of cleaning the property - the agency's photographs might help, but in the circumstances (ie with no proper inventory in place) I wouldn't bother.
The dog is not going to carry much weight - most tenancy agremeents (including the WHS one I think) say 'no pets except with consent, not to be unreasonably witheld'. If the dog has wrecked the carpets etc then you could in theory ask for extra for the cleaning and de-fleaing - but you would be relying on your T's goodwill here, as you have no protected deposit to fall back on.
You need to chalk this loss up to experience and learn to be more businesslike about letting.
Oh, and sack your agent. Thery should have told you all this at the outset.
I have dreamed in my life, dreams that have stayed with me ever after, and changed my ideas; they have gone through and through me, like wine through water, and altered the colour of my mind. Emily Bronte