...
The Generalized Neural Network (GNN) bundle, an ECL framework, synergizes the parallel processing power of HPCC Systems with the Neural Network capabilities of Keras and TensorFlow, offering ECL programmers the benefits of distributed computing while retaining a single-node programming paradigm. Our work addresses the challenges of parallelizing neural network training, recognizing that, in certain scenarios, superior results can be
achieved by training on a single high-powered node (e.g., with GPU) or a limited number of nodes. To enhance network performance and accuracy, we pursued two distinct approaches. Firstly, we optimized neural network training by strategically setting a limit on the number of nodes used for training, thereby reducing communication overhead. Through the implementation of this controlled node setting, performance improvements were achieved.
Secondly, we investigated an alternative multi-node approach to network training, varying the starting points across multiple nodes and averaging the results. This technique shows promises yielding improved predictions compared to a single-network setup. Our research contributes to the evolution of the GNN bundle, empowering more efficient and accurate neural network training within distributed environments. It sheds light on the intricacies of parallelizing neural networks, emphasizing the significance of selecting the most suitable training approach based on the unique characteristics of the neural network model and the available computing resources. As a result, our findings provide valuable insights to guide the implementation of neural network training strategies for optimal outcomes.
...