2014-03-20 15:50:26 +04:00
// Copyright 2014 The Gogs Authors. All rights reserved.
2019-06-26 19:12:38 +03:00
// Copyright 2019 The Gitea Authors. All rights reserved.
2022-11-27 21:20:29 +03:00
// SPDX-License-Identifier: MIT
2014-03-20 15:50:26 +04:00
package admin
import (
2014-03-22 15:42:24 +04:00
"fmt"
2021-04-05 18:30:52 +03:00
"net/http"
2014-03-22 15:42:24 +04:00
"runtime"
2023-06-03 17:03:41 +03:00
"sort"
2014-03-22 15:42:24 +04:00
"time"
2014-03-21 11:27:59 +04:00
2022-08-25 05:31:57 +03:00
activities_model "code.gitea.io/gitea/models/activities"
2016-11-10 19:24:48 +03:00
"code.gitea.io/gitea/modules/base"
"code.gitea.io/gitea/modules/context"
2023-06-29 13:03:20 +03:00
"code.gitea.io/gitea/modules/graceful"
2023-06-03 17:03:41 +03:00
"code.gitea.io/gitea/modules/json"
2023-06-29 13:03:20 +03:00
"code.gitea.io/gitea/modules/log"
2016-11-10 19:24:48 +03:00
"code.gitea.io/gitea/modules/setting"
2021-10-21 19:10:49 +03:00
"code.gitea.io/gitea/modules/updatechecker"
2021-01-26 18:36:53 +03:00
"code.gitea.io/gitea/modules/web"
2021-11-16 16:30:11 +03:00
"code.gitea.io/gitea/services/cron"
2021-04-06 22:44:05 +03:00
"code.gitea.io/gitea/services/forms"
2023-06-29 13:03:20 +03:00
repo_service "code.gitea.io/gitea/services/repository"
2014-03-20 15:50:26 +04:00
)
2014-06-22 21:14:03 +04:00
const (
Rewrite queue (#24505)
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 14:49:59 +03:00
tplDashboard base . TplName = "admin/dashboard"
Improve queue & process & stacktrace (#24636)
Although some features are mixed together in this PR, this PR is not
that large, and these features are all related.
Actually there are more than 70 lines are for a toy "test queue", so
this PR is quite simple.
Major features:
1. Allow site admin to clear a queue (remove all items in a queue)
* Because there is no transaction, the "unique queue" could be corrupted
in rare cases, that's unfixable.
* eg: the item is in the "set" but not in the "list", so the item would
never be able to be pushed into the queue.
* Now site admin could simply clear the queue, then everything becomes
correct, the lost items could be re-pushed into queue by future
operations.
3. Split the "admin/monitor" to separate pages
4. Allow to download diagnosis report
* In history, there were many users reporting that Gitea queue gets
stuck, or Gitea's CPU is 100%
* With diagnosis report, maintainers could know what happens clearly
The diagnosis report sample:
[gitea-diagnosis-20230510-192913.zip](https://github.com/go-gitea/gitea/files/11441346/gitea-diagnosis-20230510-192913.zip)
, use "go tool pprof profile.dat" to view the report.
Screenshots:
![image](https://github.com/go-gitea/gitea/assets/2114189/320659b4-2eda-4def-8dc0-5ea08d578063)
![image](https://github.com/go-gitea/gitea/assets/2114189/c5c46fae-9dc0-44ca-8cd3-57beedc5035e)
![image](https://github.com/go-gitea/gitea/assets/2114189/6168a811-42a1-4e64-a263-0617a6c8c4fe)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-11 10:45:47 +03:00
tplCron base . TplName = "admin/cron"
tplQueue base . TplName = "admin/queue"
Rewrite queue (#24505)
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 14:49:59 +03:00
tplStacktrace base . TplName = "admin/stacktrace"
tplQueueManage base . TplName = "admin/queue_manage"
2023-06-03 17:03:41 +03:00
tplStats base . TplName = "admin/stats"
2014-06-22 21:14:03 +04:00
)
2014-03-22 15:42:24 +04:00
var sysStatus struct {
2023-04-11 02:01:20 +03:00
StartTime string
2014-03-22 15:42:24 +04:00
NumGoroutine int
// General statistics.
MemAllocated string // bytes allocated and still in use
MemTotal string // bytes allocated (even if freed)
MemSys string // bytes obtained from system (sum of XxxSys below)
Lookups uint64 // number of pointer lookups
MemMallocs uint64 // number of mallocs
MemFrees uint64 // number of frees
// Main allocation heap statistics.
HeapAlloc string // bytes allocated and still in use
HeapSys string // bytes obtained from system
HeapIdle string // bytes in idle spans
HeapInuse string // bytes in non-idle span
HeapReleased string // bytes released to the OS
HeapObjects uint64 // total number of allocated objects
// Low-level fixed-size structure allocator statistics.
// Inuse is bytes used now.
// Sys is bytes obtained from system.
StackInuse string // bootstrap stacks
StackSys string
MSpanInuse string // mspan structures
MSpanSys string
MCacheInuse string // mcache structures
MCacheSys string
BuckHashSys string // profiling bucket hash table
GCSys string // GC metadata
OtherSys string // other system allocations
// Garbage collector statistics.
NextGC string // next run in HeapAlloc time (bytes)
LastGC string // last run in absolute time (ns)
PauseTotalNs string
PauseNs string // circular buffer of recent GC pause times, most recent at [(NumGC+255)%256]
NumGC uint32
}
func updateSystemStatus ( ) {
2023-04-11 02:01:20 +03:00
sysStatus . StartTime = setting . AppStartTime . Format ( time . RFC3339 )
2014-03-22 17:21:57 +04:00
2014-03-22 15:42:24 +04:00
m := new ( runtime . MemStats )
runtime . ReadMemStats ( m )
sysStatus . NumGoroutine = runtime . NumGoroutine ( )
sysStatus . MemAllocated = base . FileSize ( int64 ( m . Alloc ) )
sysStatus . MemTotal = base . FileSize ( int64 ( m . TotalAlloc ) )
sysStatus . MemSys = base . FileSize ( int64 ( m . Sys ) )
sysStatus . Lookups = m . Lookups
sysStatus . MemMallocs = m . Mallocs
sysStatus . MemFrees = m . Frees
sysStatus . HeapAlloc = base . FileSize ( int64 ( m . HeapAlloc ) )
sysStatus . HeapSys = base . FileSize ( int64 ( m . HeapSys ) )
sysStatus . HeapIdle = base . FileSize ( int64 ( m . HeapIdle ) )
sysStatus . HeapInuse = base . FileSize ( int64 ( m . HeapInuse ) )
sysStatus . HeapReleased = base . FileSize ( int64 ( m . HeapReleased ) )
sysStatus . HeapObjects = m . HeapObjects
sysStatus . StackInuse = base . FileSize ( int64 ( m . StackInuse ) )
sysStatus . StackSys = base . FileSize ( int64 ( m . StackSys ) )
sysStatus . MSpanInuse = base . FileSize ( int64 ( m . MSpanInuse ) )
sysStatus . MSpanSys = base . FileSize ( int64 ( m . MSpanSys ) )
sysStatus . MCacheInuse = base . FileSize ( int64 ( m . MCacheInuse ) )
sysStatus . MCacheSys = base . FileSize ( int64 ( m . MCacheSys ) )
sysStatus . BuckHashSys = base . FileSize ( int64 ( m . BuckHashSys ) )
sysStatus . GCSys = base . FileSize ( int64 ( m . GCSys ) )
sysStatus . OtherSys = base . FileSize ( int64 ( m . OtherSys ) )
sysStatus . NextGC = base . FileSize ( int64 ( m . NextGC ) )
sysStatus . LastGC = fmt . Sprintf ( "%.1fs" , float64 ( time . Now ( ) . UnixNano ( ) - int64 ( m . LastGC ) ) / 1000 / 1000 / 1000 )
2014-03-22 17:21:57 +04:00
sysStatus . PauseTotalNs = fmt . Sprintf ( "%.1fs" , float64 ( m . PauseTotalNs ) / 1000 / 1000 / 1000 )
sysStatus . PauseNs = fmt . Sprintf ( "%.3fs" , float64 ( m . PauseNs [ ( m . NumGC + 255 ) % 256 ] ) / 1000 / 1000 / 1000 )
2014-03-22 15:42:24 +04:00
sysStatus . NumGC = m . NumGC
}
2016-11-21 06:21:24 +03:00
// Dashboard show admin panel dashboard
2016-03-11 19:56:52 +03:00
func Dashboard ( ctx * context . Context ) {
2014-08-28 18:29:00 +04:00
ctx . Data [ "Title" ] = ctx . Tr ( "admin.dashboard" )
ctx . Data [ "PageIsAdminDashboard" ] = true
2021-10-21 19:10:49 +03:00
ctx . Data [ "NeedUpdate" ] = updatechecker . GetNeedUpdate ( )
ctx . Data [ "RemoteVersion" ] = updatechecker . GetRemoteVersion ( )
2020-02-26 01:54:13 +03:00
// FIXME: update periodically
updateSystemStatus ( )
ctx . Data [ "SysStatus" ] = sysStatus
2020-10-08 19:43:15 +03:00
ctx . Data [ "SSH" ] = setting . SSH
2021-04-05 18:30:52 +03:00
ctx . HTML ( http . StatusOK , tplDashboard )
2020-02-26 01:54:13 +03:00
}
// DashboardPost run an admin operation
2021-01-26 18:36:53 +03:00
func DashboardPost ( ctx * context . Context ) {
2021-04-06 22:44:05 +03:00
form := web . GetForm ( ctx ) . ( * forms . AdminDashboardForm )
2020-02-26 01:54:13 +03:00
ctx . Data [ "Title" ] = ctx . Tr ( "admin.dashboard" )
ctx . Data [ "PageIsAdminDashboard" ] = true
updateSystemStatus ( )
ctx . Data [ "SysStatus" ] = sysStatus
2014-05-06 21:47:47 +04:00
// Run operation.
2020-05-17 02:31:38 +03:00
if form . Op != "" {
2023-06-29 13:03:20 +03:00
switch form . Op {
case "sync_repo_branches" :
go func ( ) {
if err := repo_service . AddAllRepoBranchesToSyncQueue ( graceful . GetManager ( ) . ShutdownContext ( ) , ctx . Doer . ID ) ; err != nil {
log . Error ( "AddAllRepoBranchesToSyncQueue: %v: %v" , ctx . Doer . ID , err )
}
} ( )
ctx . Flash . Success ( ctx . Tr ( "admin.dashboard.sync_branch.started" ) )
default :
task := cron . GetTask ( form . Op )
if task != nil {
go task . RunWithUser ( ctx . Doer , nil )
ctx . Flash . Success ( ctx . Tr ( "admin.dashboard.task.started" , ctx . Tr ( "admin.dashboard." + form . Op ) ) )
} else {
ctx . Flash . Error ( ctx . Tr ( "admin.dashboard.task.unknown" , form . Op ) )
}
2014-05-06 21:47:47 +04:00
}
}
2020-07-05 22:38:03 +03:00
if form . From == "monitor" {
Improve queue & process & stacktrace (#24636)
Although some features are mixed together in this PR, this PR is not
that large, and these features are all related.
Actually there are more than 70 lines are for a toy "test queue", so
this PR is quite simple.
Major features:
1. Allow site admin to clear a queue (remove all items in a queue)
* Because there is no transaction, the "unique queue" could be corrupted
in rare cases, that's unfixable.
* eg: the item is in the "set" but not in the "list", so the item would
never be able to be pushed into the queue.
* Now site admin could simply clear the queue, then everything becomes
correct, the lost items could be re-pushed into queue by future
operations.
3. Split the "admin/monitor" to separate pages
4. Allow to download diagnosis report
* In history, there were many users reporting that Gitea queue gets
stuck, or Gitea's CPU is 100%
* With diagnosis report, maintainers could know what happens clearly
The diagnosis report sample:
[gitea-diagnosis-20230510-192913.zip](https://github.com/go-gitea/gitea/files/11441346/gitea-diagnosis-20230510-192913.zip)
, use "go tool pprof profile.dat" to view the report.
Screenshots:
![image](https://github.com/go-gitea/gitea/assets/2114189/320659b4-2eda-4def-8dc0-5ea08d578063)
![image](https://github.com/go-gitea/gitea/assets/2114189/c5c46fae-9dc0-44ca-8cd3-57beedc5035e)
![image](https://github.com/go-gitea/gitea/assets/2114189/6168a811-42a1-4e64-a263-0617a6c8c4fe)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-11 10:45:47 +03:00
ctx . Redirect ( setting . AppSubURL + "/admin/monitor/cron" )
2020-07-05 22:38:03 +03:00
} else {
ctx . Redirect ( setting . AppSubURL + "/admin" )
}
2014-03-20 15:50:26 +04:00
}
Improve queue & process & stacktrace (#24636)
Although some features are mixed together in this PR, this PR is not
that large, and these features are all related.
Actually there are more than 70 lines are for a toy "test queue", so
this PR is quite simple.
Major features:
1. Allow site admin to clear a queue (remove all items in a queue)
* Because there is no transaction, the "unique queue" could be corrupted
in rare cases, that's unfixable.
* eg: the item is in the "set" but not in the "list", so the item would
never be able to be pushed into the queue.
* Now site admin could simply clear the queue, then everything becomes
correct, the lost items could be re-pushed into queue by future
operations.
3. Split the "admin/monitor" to separate pages
4. Allow to download diagnosis report
* In history, there were many users reporting that Gitea queue gets
stuck, or Gitea's CPU is 100%
* With diagnosis report, maintainers could know what happens clearly
The diagnosis report sample:
[gitea-diagnosis-20230510-192913.zip](https://github.com/go-gitea/gitea/files/11441346/gitea-diagnosis-20230510-192913.zip)
, use "go tool pprof profile.dat" to view the report.
Screenshots:
![image](https://github.com/go-gitea/gitea/assets/2114189/320659b4-2eda-4def-8dc0-5ea08d578063)
![image](https://github.com/go-gitea/gitea/assets/2114189/c5c46fae-9dc0-44ca-8cd3-57beedc5035e)
![image](https://github.com/go-gitea/gitea/assets/2114189/6168a811-42a1-4e64-a263-0617a6c8c4fe)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-11 10:45:47 +03:00
func CronTasks ( ctx * context . Context ) {
ctx . Data [ "Title" ] = ctx . Tr ( "admin.monitor.cron" )
ctx . Data [ "PageIsAdminMonitorCron" ] = true
2015-08-17 21:19:29 +03:00
ctx . Data [ "Entries" ] = cron . ListTasks ( )
Improve queue & process & stacktrace (#24636)
Although some features are mixed together in this PR, this PR is not
that large, and these features are all related.
Actually there are more than 70 lines are for a toy "test queue", so
this PR is quite simple.
Major features:
1. Allow site admin to clear a queue (remove all items in a queue)
* Because there is no transaction, the "unique queue" could be corrupted
in rare cases, that's unfixable.
* eg: the item is in the "set" but not in the "list", so the item would
never be able to be pushed into the queue.
* Now site admin could simply clear the queue, then everything becomes
correct, the lost items could be re-pushed into queue by future
operations.
3. Split the "admin/monitor" to separate pages
4. Allow to download diagnosis report
* In history, there were many users reporting that Gitea queue gets
stuck, or Gitea's CPU is 100%
* With diagnosis report, maintainers could know what happens clearly
The diagnosis report sample:
[gitea-diagnosis-20230510-192913.zip](https://github.com/go-gitea/gitea/files/11441346/gitea-diagnosis-20230510-192913.zip)
, use "go tool pprof profile.dat" to view the report.
Screenshots:
![image](https://github.com/go-gitea/gitea/assets/2114189/320659b4-2eda-4def-8dc0-5ea08d578063)
![image](https://github.com/go-gitea/gitea/assets/2114189/c5c46fae-9dc0-44ca-8cd3-57beedc5035e)
![image](https://github.com/go-gitea/gitea/assets/2114189/6168a811-42a1-4e64-a263-0617a6c8c4fe)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-11 10:45:47 +03:00
ctx . HTML ( http . StatusOK , tplCron )
2019-11-30 17:40:22 +03:00
}
2023-06-03 17:03:41 +03:00
func MonitorStats ( ctx * context . Context ) {
ctx . Data [ "Title" ] = ctx . Tr ( "admin.monitor.stats" )
ctx . Data [ "PageIsAdminMonitorStats" ] = true
bs , err := json . Marshal ( activities_model . GetStatistic ( ) . Counter )
if err != nil {
ctx . ServerError ( "MonitorStats" , err )
return
}
statsCounter := map [ string ] any { }
err = json . Unmarshal ( bs , & statsCounter )
if err != nil {
ctx . ServerError ( "MonitorStats" , err )
return
}
statsKeys := make ( [ ] string , 0 , len ( statsCounter ) )
for k := range statsCounter {
if statsCounter [ k ] == nil {
continue
}
statsKeys = append ( statsKeys , k )
}
sort . Strings ( statsKeys )
ctx . Data [ "StatsKeys" ] = statsKeys
ctx . Data [ "StatsCounter" ] = statsCounter
ctx . HTML ( http . StatusOK , tplStats )
}