No, you cannot legitmately deduct anything from the deposit because it is the tenant's money and remains so until a deposit scheme adjudicator decides otherwise. Since you did not protect it, you cannot appeal to an adjudicator. You should return the money you have kept back and hope the 'acquaintance' doesn't sue you for non-compliance with deposit protection, as advised above.
Yes you could sue your T for the cost of cleaning the property - the agency's photographs might help, but in the circumstances (ie with no proper inventory in place) I wouldn't bother.
The dog is not going to carry much weight - most tenancy agremeents (including the WHS one I think) say 'no pets except with consent, not to be unreasonably witheld'. If the dog has wrecked the carpets etc then you could in theory ask for extra for the cleaning and de-fleaing - but you would be relying on your T's goodwill here, as you have no protected deposit to fall back on.
You need to chalk this loss up to experience and learn to be more businesslike about letting.
Oh, and sack your agent. Thery should have told you all this at the outset.
'Pause you who read this, and think for a moment of the long chain of iron or gold, of thorns or flowers, that would never have bound you, but for the formation fo the first link on one memorable day'. Charles Dickens, Great Expectations