* Add unit tests for TokenProcessor functionality
* Add trace stubs for token processor tests
* Increase token processor test coverage
* Clean up imports in test_token_processor.py
Remove unnecessary path manipulation in test file.
* Cleanup: Remove unused imports in test_token_processor
Removed unused imports from the test file.
* Add trace_carrier to task in test cases
Added trace_carrier attribute to task in multiple test cases to ensure proper handling of trace information.
* Refine token processor tests for safe coverage
* Expand postprocess coverage
* Add ZMQ logprob parsing test
---------
Co-authored-by: CSWYF3634076 <wangyafeng@baidu.com>
Co-authored-by: Tao Luo <luotao02@baidu.com>
3aee5c4bf5