* optimize: Refactor gradient convergence and remove DefaultSettings
The current API design makes it easy to make a mistake in not using the DefaultSettings. This change makes the zero value of Settings do the 'right thing'. The remaining setting that is used by the DefaultSettings is to change the behavior of the GradientTolerance. This was necessary because gradient-based Local methods (BFGS, LBFGS, CG, etc.) typically _define_ convergence by the value of the gradient, while Global methods (CMAES, GuessAndCheck) are defined by _not_ converging when the gradient is small. The problem is to have two completely different default behaviors without knowing the Method. The solution is to treat a very small value of the gradient as a method-based convergence, in the same way that a small spread of data is a convergence of CMAES. Thus, the default behavior, from the perspective of Settings, is never to converge based on the gradient, but all of the Local methods will converge when a value close to the minimum is found. This default value is set to a very small value, such that users should not want a smaller value. A user can thus still set a (more reasonable) convergence value through settings.
Fixes 677.
* Improve parameters when dimension of NelderMead is one. In particular the shrink parameter is set to 0, which causes the algorithm to fail to converge.
Fixes#542.
* Add test for 1-D NelderMead
* optimize: Remove Local function
This change removes the Local function. In order to do so, this changes the previous LocalGlobal wrapper to LocalController to allow Local methods to be used as a Global optimizer. This adds methods to all of the Local methods in order to implement GlobalMethod, and changes the tests accordingly. The next commit will fix all of the names
* optimize: Change Settings to allow InitialLocation
This modifies Settings to allow specifying an initial location and properties of the function (value, gradient, etc.). This allows to work with local optimizers that are seeded with initial settings. This has two fields that must be specified, InitX and InitValues. Ideally this would only be one location, but the difficulty is that the default value of the function is 0. We either must require the user to specify it is set (in this case that InitValues is non-zero), or require the user to change the default value away if it is not set. The former seems much safer.
* optimize: remove Local implementation and replace with a call to Global
This PR starts the process described in #482. It removes the existing Local implementation, replacing with a function that wraps Method to act as a GlobalMethod. This PR also adds a hack to fix an inconsistency with FunctionConverge between Global and Local (and a TODO to make it not a hack in the future)
* optimize: completely overhaul Global
The previous implementation of Global was a minefield for incorrectly implementing global optimization methods. It was very difficult to correctly implement methods (both of the provided methods were incorrect), and the resulting code is very ugly. This commit switches to use channels to communicate, allowing a more clear ordering of concurrent code. This also enables better shutdown of methods.
In addition to the main fix of Global, this refactors the two Global methods to use the updated interface, and makes some small improvements that were previously not possible. In addition, there are some small cleanups of Local to better match between the two calls.
If anyone has been curious about what is meant by 'Don't communicate by sharing memory, share memory by communicating' this is it, and why.
* respond to PR comments
* make constants
* simplify termination logic
* optimize: simplify stats collection
* overhaul documentation and respond to PR comments
* implement PR requests
* clean up cmaes
