Neural networks are often used to approximate functions defined over high-dimensional data spaces (e.g. text data, genomic data, multi-sensor data). Such approximation tasks are usually difficult due to the curse of dimensionality and improved methods are needed to deal with them effectively and efficiently. Since the data generally resides on a lower dimensional manifold various methods have been proposed to project the data first into a lower dimension and then build the neural network approximation over this lower dimensional projection data space. Here we follow this approach and combine it with the idea of weak learning through the use of random projections of the data. We show that random projection of the data works well and the approximation errors are smaller than in the case of approximation of the functions in the original data space. We explore the random projections with the aim to optimize this approach.