No, you cannot legitmately deduct anything from the deposit because it is the tenant's money and remains so until a deposit scheme adjudicator decides otherwise. Since you did not protect it, you cannot appeal to an adjudicator. You should return the money you have kept back and hope the 'acquaintance' doesn't sue you for non-compliance with deposit protection, as advised above.
Yes you could sue your T for the cost of cleaning the property - the agency's photographs might help, but in the circumstances (ie with no proper inventory in place) I wouldn't bother.
The dog is not going to carry much weight - most tenancy agremeents (including the WHS one I think) say 'no pets except with consent, not to be unreasonably witheld'. If the dog has wrecked the carpets etc then you could in theory ask for extra for the cleaning and de-fleaing - but you would be relying on your T's goodwill here, as you have no protected deposit to fall back on.
You need to chalk this loss up to experience and learn to be more businesslike about letting.
Oh, and sack your agent. Thery should have told you all this at the outset.
How is education supposed to make me feel smarter? Besides, every time I learn something new, it pushes some old stuff out of my brain. Remember when I took that home winemaking course, and I forgot how to drive? Homer Simpson