Very well written, and I applaud the effort. But personally I don't care for the "magical" aura that writers tend to give ANNs - to me, they are simply (non-linear) function approximators that have a nice fitting algorithm. They work well for some problems and poorly for others. Also, beware of over-fitting - ANNs tend to be parameter-heavy, although there are approaches to prune the connections.
I think this problem would be significantly less if there was no"neural" in "neural network." That said that one word has probably brought a significant number of eyes to the subject that would have looked elsewhere with out it.
I'd go further - the "nice fitting algorithm" is to minimise the error on the training set as a function of the weight parameters, and one obvious way to (locally) minimise that is gradient descent + the chain rule.
The math / applied math machinery is all incredibly general, and a useful way to think about many machine learning algorithms.
Not to say that applied math is the only valuable perspective to think about, there's clearly also statistical and computational views as well. E.g. we're trying to approximate a function that we can never directly evaluate (error on the samples we havent seen yet)
I believe svantana was merely trying to differentiate ANNs from magic, not ANNs from other ML algorithms. Indeed you are correct that machine learning as a whole could be seen as function fitting.
