What tools to send push notifications and in-app messages do you use? Which engagement type works better?

I think push vs in-app are two different things, like email and telephone call. Push reaches people who are not using your app, but require OS-level permission. In-app gives you much more flexibility but they only reach people who are using your app (obviously!).

As to whether in-house or external SDK is best, it depends what you are trying to do and whether an external system handles it. For a typical retail environment the SDKs might be easiest--it's a well-established use-case. For something needing a more customized algorithm, an in-house system might work as well as an external SDK (for less money).
There are simple options to save you from starting completely from scratch, from PHP toolkits to Amazon SNS. It depends on what (if any) server environment you have...
Hope that helps!

Both, you must mix and match to keep the consumer guessing.. It sucks to just use one method and beat it into the the prospect.. I currently use Urban Airship and a smaller entity for these type of messaging.. My. Most viral and highest ctr were a mixture of push notifications that lead to media within app that where content gated and once unlocked released not only the media, but also a coupon.. You could update the coupon with a serial number that these pushes would provide.. #SecretSauce

How long is a piece of string?

There are tons of different mobile messaging platforms available--some basic and others with steep learning curves. It all depends on what you need and what your budget can handle. Parse, Mixpanel and Urban Airship are the big ones, but there many niche players for gaming, retail, etc..

If you're a newbie, stick with a platform that offers an SDK instead of hard coding it yourself. This is the fastest way with the shortest learning curve, and many offer free trials and educational resources.

As for format, in-app and push messaging have their roles and one is not better than the other. For example, a weather app might issue a push every morning at 7am, and then offer in-app messaging for specific details and advertising offers. They each have their place. You should also not overlook email and text messaging, which are hugely effective at reviving lapsed users.

So it all depends on what you need. Stick with 3rd party services and experiment. Let me know if you have any further questions.