How to Calculate Token Score (Token / minute) to check if the CPU is strong enough?

[waqaszs]

Describe the feature

Hi, your unity bindings for Llama.cpp is great. I'm wondering is there any equivilant method in this SDK which I can use to calculate token score? This is how this android based library doing this.

suspend fun myCustomBenchmark(): Flow<String> = flow {
        try {
            withTimeout(30.seconds) { // Set timeout to 2 minutes
                when (val state = threadLocalState.get()) {
                    is State.Loaded -> {
                        val ncur = IntVar(completion_init(state.context, state.batch, "Write an article on global warming in 1000 words", nlen))
                        while (ncur.value <= nlen) {
                            val str = completion_loop(state.context, state.batch, state.sampler, nlen, ncur)
                            if (str == null) {
                                _isSending.value = false
                                _isCompleteEOT.value = true
                                break
                            }
                            if (stopGeneration) {
                                break
                            }
                            emit(str)
                        }
                        kv_cache_clear(state.context)
                    }
                    else -> {
                        _isSending.value = false
                    }
                }
            }
        } catch (e: Exception) {
            // Handle timeout or any other exceptions if necessary
            if (e is kotlinx.coroutines.TimeoutCancellationException) {
                println("Benchmark timed out after 2 minutes.")
            }
        } finally {
            _isSending.value = false
        }
    }.flowOn(runLoop)

Does your library also has the equivilant method to check benchmarks (token per minute)? Thanks.

[3inary]

Just noticed that the Debug Log Level has output like this:

slot print_timing: id 0 | task 2008 |
prompt eval time = 183.52 ms / 27 tokens ( 6.80 ms per token, 147.12 tokens per second)
eval time = 1389.76 ms / 55 tokens ( 25.27 ms per token, 39.58 tokens per second)
total time = 1573.28 ms / 82 tokens

Maybe this could be accesed?

[amakropoulos]

Yes, the completion can return a response json as outlined in llama.cpp here that contains timings.
To enable that, this line

LLMUnity/Runtime/LlamaLib/LLM.cs

Line 189 in 63b79c5

result = llamaLib.LLM_Completion(llm, prompt ?? string.Empty, callback, idSlot);

needs to be changed to:
result = llamaLib.LLM_Completion(llm, prompt ?? string.Empty, callback, idSlot, true);
and then process the result json.
This is low priority in my mind because the info can be seen in the debug messages, and haven't found the time to work on it (open to PRs 🫣)

[cch1rag]

Yes, the completion can return a response json as outlined in llama.cpp here that contains timings. To enable that, this line

LLMUnity/Runtime/LlamaLib/LLM.cs

Line 189 in 63b79c5

result = llamaLib.LLM_Completion(llm, prompt ?? string.Empty, callback, idSlot);

needs to be changed to:
result = llamaLib.LLM_Completion(llm, prompt ?? string.Empty, callback, idSlot, true);
and then process the result json.
This is low priority in my mind because the info can be seen in the debug messages, and haven't found the time to work on it (open to PRs 🫣)

Opened a PR for this here: #399. Let me know if this looks good or if you'd like any changes.