Tuning parameters Sample Clauses

Tuning parameters. Like other deep learning models (or statistical models), the proposed methods and the algorithm used to solve neural networks involve a number of parameters that are specified beforehand. This section briefly discusses the functions of these parameters and their impact on the randomness and efficiency of constructed models. Learning rate is a hidden parameter inside the learning process of neural network. The magnitude of learning rate decides how large the ”step size” is for each update step. The default learning rate automatically decreases proportionally from a max value (0.025) to a mim value (0.001) during the training process of one dataset. This is problematic when one want to train the model sequentially and obtain a global model. In Distributed NCE, we are able to adjust this parameter since both data sizes can be obtained when collecting vocabularies from different sites. The learning rate decreases proportional to data size as the way used in global model. On the contrary, the learning rates in naive updates and dropout updates cannot be adjusted since the size of the second data set is usually unknown when applying these two methods. Iteration is a parameter controls the number of times the model is trained itera- tively over the entire data. Default value for iteration used in Word2Vec is 5. More iterations result in more stable results but too many iterations would also dramat- ically increase computing time. Figure S3 demonstrates the PTK performance of Distributed NCE using different data partitions and various iterations. When num- ber of iterations increases, the performance of unequal partitions stabilizes but the performance of equal partitions decreases. This happens because when applying Dis- tributed NCE, model is trained with the first dataset repeatedly for Niter times and then trained with the second dataset repeatedly for Niter times. But when applying global model, model is trained with the whole dataset repeatedly for Niter times. Un- equal patitions tend to have one large dataset which dominates the training process and thus have xxxxxxx performance than equal partitions.