Kopia

This release focuses on improvements to the repository format, stability and performance.

The new repository index format enables efficient append-only operation where delete permission is not required, removes the need for quick maintenance and enables internal content-level compression.

See https://kopia.io/docs/upgrade/ for upgrade instructions.

Kopia now also supports repository password change (new repositories only).

Big thanks to @pawitp for contributing major performance improvement to splitters (#1251), which improves large file upload performance by up to 30%.

Providers

• azure: added support for using SAS Tokens instead of storage keys (#1093)
• rclone: wait for rclone transfers to finish when closing (#1201)
• sftp: added automatic reconnect and connection pooling (#1195)
• sftp: ensure key file and known hosts are absolute paths (#1092)
• s3: point-in-time support for s3 versioned stores (#1259)
• webdav & and rclone support for atomic writes (#1206)
• webdav: disable server-side compression (#1100)

Repository features

• new index format supporting internal compression and does not require quick maintenance for performance reasons
• splitter: major performance improvements (#1251)
• implemented epoch-based index manager (#1174)
• internal logging - low-level logs are now stored in the repository.
• support for content-level compression (#1076)
• added ‘lz4’ and ‘deflate’ methods (#1077)
• added sharding of indexes on write (#1176)
• repository password change support (#1197)
• limit the duration of kopia.repository caching to 15 minutes (#1196)
• encryption: removed old, non-authenticated encryption methods (#979)
• big performance improvement for WriteContent with repo server (#1182)
• disabled quick maintenance for new repositories

CLI

• ‘kopia content verify’ performance improvements (#1120)
• add a flag to not shallowrestore small files (#1126)
• added ‘kopia repo set-parameters’ to override mutable parameters (#1148)
• added ‘kopia repository validate-provider` (#1205)
• additional ‘index inspect’ flags (#1231)
• changed ‘kopia snapshot verify —verify-files-percent’ to float (#1210)
• fixed ETA estimation of ‘snapshot verify’ (#1213)
• force-hash: Allow fractional percentages (#1209)
• include parameters in maintenance info JSON output (#981)
• optimized ‘kopia index recover’ by leveraging partial read and parallelism (#1094)
• support for shallow restore (#725)
• tagging of kopia snapshots and listing of snapshots by tag (#1030)
• logging: switched file log timestamps to be UTC by default, otherwise it’s hard to correlate logs (#1240)

Policy Changes

• Fix weekly retention labels not being assigned correctly (#1211)
• policy: path validation (#1006)

UI Changes

• do not attempt running maintenance if the current user is not the maintenance owner, to avoid producing error in the Tasks tab (#1010)
• fixed Estimate not honoring the defined policies (#1002)
• for read-only repositories start a read-only source manager (#1009)
• open .gitignore help in a new window (#1016)
• show a spinner when saving/deleting policy (#1018)

Security

• security: switched to github.com/golang-jwt/jwt/v4 to fix upstream security issue (#1235)

Notable internal Changes

• automatically refresh indexes on read instead of relying on RefreshPeriodically() loop (#1243)
• ensure that we always consistently pick content.Info amongst entries with identical time and deleted flag (#1239)
• refactored content.Info to be an interface and switched index parsing to be lazy (#1008)
• refactored own writes cache and list cache into blob.Storage wrappers (#1133)
• few subtle threading bugs uncovered by stress test and rewrote the test to be model-based (#1157)
• reworked memory management to avoid allocating large chunks of memory for most buffers
• ci: Allow to compile Kopia cli on OpenBSD (#983)
• upgrade golang to 1.17
• testing: Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

Changelog

8b2b91f9 content: fixed repo upgrade version (#1286) 06a99738 cli: include parameters in maintenance info JSON output (#981) d98b0ede endurance: rewrote test to be more stable (#1285) 5941d631 ci: increased test timeout to 15 minutes 135c307c ci: fixed rclone installation 0d3c8cfd testing: report reason why rclone test is being skipped 7e68d8e4 Consolidated format version flags (#1284) 87dbf44d Fix link in _index.md (#1134) 2721fba2 Upgraded remaining UI deps (#1283) 6c83cc6a ui: upgraded all NPM dependencies for htmlui/ and app/ (#1278) 296a5198 build(deps): bump github.com/aws/aws-sdk-go from 1.40.34 to 1.40.37 (#1277) 562c596c build(deps): bump gocloud.dev from 0.23.0 to 0.24.0 (#1273) 1077cc24 blob/s3: point-in-time support for s3 versioned stores (#1259) cc692c40 ci: delete golang installation on armhf before installing a new one (#1276) 0eedca83 build(deps-dev): bump electron-notarize from 1.0.1 to 1.1.0 in /app (#1262) 56edb042 content: switched defaults to use v2 index format and epoch-based indexes (#1275) 59b5a44c Makefile: refactored build tools installation (#1274) 06451aa7 upgrade golang to 1.17 + various dependencies (#1256) 082785af ci: run stress test on kopia/kopia repo (#1257) 310f4510 ci: avoid double workflow executions on PRs (#1258) 9cebffc6 Fix endurance test (#1254) 74074192 maintenance: check for clock skew at the beginning of maintenance (#1253) 0578e2e5 splitter: optimize performance (#1251) 16aeb843 Fixed for rare stress test failure caused by use-after-free (#1250) a203fa41 gather: MakeContiguous support for arbitrary chunk sizes (#1247) 9e182f13 linter: upgraded to 1.42.0 (#1246) 35d0f31c huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244) 4676e9e4 content: automatically refresh indexes on read instead of relying on RefreshPeriodically() loop (#1243) bece46d5 logging: switched file log timestamps to be UTC by default, otherwise it’s hard to correlate logs (#1240) 1e10e852 content: ensure that we always consistently pick content.Info amongst entries with identical time and deleted flag (#1239) 0830de9c build(deps): bump github.com/klauspost/compress from 1.12.2 to 1.13.3 (#1237) ff19caab build(deps): bump github.com/aws/aws-sdk-go from 1.40.12 to 1.40.17 (#1236) 4aacad25 server: switched to github.com/golang-jwt/jwt/v4 to fix upstream security issue (#1235) e3e7cae4 content: removed EpochMarkerIndexBlobPrefix from allIndexBlobPrefixes (#1232) 1238a0db cli: additional ‘index inspect’ flags (#1231) b96623f8 epoch: simplified TestIndexEpochManager_DeletionFailing test (#1230) fcd79b40 build(deps): bump electron-log from 4.3.5 to 4.4.1 in /app (#1220) eae75218 build(deps): bump google.golang.org/api from 0.50.0 to 0.52.0 (#1229) 67165cae build(deps): bump github.com/prometheus/client_golang (#1226) 17b43060 build(deps): bump postcss from 8.3.5 to 8.3.6 in /htmlui (#1217) c04e3be5 build(deps-dev): bump electron-notarize from 1.0.0 to 1.0.1 in /app (#1219) ecd9b021 build(deps): bump github.com/zeebo/blake3 from 0.1.1 to 0.2.0 (#1223) 140a86ba build(deps): bump github.com/google/uuid from 1.2.0 to 1.3.0 (#1222) 5b8485b8 build(deps): bump github.com/aws/aws-sdk-go from 1.39.4 to 1.40.12 (#1224) d6d9a1fb Maintenance improvements for epoch-based index structures (#1225) d77e771d provider test: fixed typo cfdbc407 cli: Fixed ETA estimation of ‘snapshot verify’ (#1213) 6205b556 Fix weekly retention labels not being assigned correctly (#1211) 6c5bf42d cli: changed ‘kopia snapshot verify —verify-files-percent’ to float (#1210) 317d5c34 force-hash: Allow fractional percentages (#1209) 4f93a177 WebDAV and Rclone atomic writes (#1206) e42cc6cc Added ‘kopia repository validate-provider` (#1205) 7dd9e332 ci: fixed syntax e2e0cb37 ci: changed continue-on-error: true to if: succeeded() || failed() bdd53f05 ci: install provider test dependencies separately 49d8a327 testing: split provider tests into separate steps 33cf3229 ci: run provider test more often to get better stability signal adbf2131 build(deps): bump @testing-library/react in /htmlui (#1169) 9df5e5f5 rclone: updated wait logic 9fd6a89f testing: possible fix for TestS3StorageMinioSTS 8c2e4ff7 testing: simplified S3 test to fix error caused by minio behavior change 1a6f0ba4 testing: possible fix for TestS3StorageMinioSTS 4c9b9e52 Test cleanup improvements (#1203) 42f02780 s3: cleaned up test 4c2f52a2 Rclone and testing improvements (#1202) 928e1d30 rclone: wait for rclone transfers to finish when closing (#1201) 7b70446b ci: switch license NPM check to ‘onlyAllow’ instead of ‘failOn’ 42e5661b ci: added license check using github.com/frapposelli/wwhrd (#1200) b2cc840f build(deps): bump github.com/pierrec/lz4 (#1188) 51c54033 build(deps): bump github.com/Azure/azure-storage-blob-go (#1194) 730ba7b9 Repository password change support (#1197) f15c76ef build(deps): bump google.golang.org/api from 0.48.0 to 0.50.0 (#1191) bffe8b37 repo: limit the duration of kopia.repository caching to 15 minutes (#1196) 83b4dee3 serialize client auth (#232) (#1198) 7cabed85 sftp: added automatic reconnect and connection pooling (#1195) f605f526 Upgraded bootstrap to v5 and react-bootstrap accordingly (#1192) 369dea9b build(deps): bump cloud.google.com/go/storage from 1.15.0 to 1.16.0 (#1193) 3d596a8c build(deps): bump google.golang.org/protobuf from 1.26.0 to 1.27.1 (#1189) 10aa2143 build(deps): bump github.com/pkg/sftp from 1.13.0 to 1.13.2 (#1186) ed88b779 build(deps): bump github.com/gofrs/flock from 0.8.0 to 0.8.1 (#1190) 6eb0064f build(deps): bump github.com/sanity-io/litter from 1.3.0 to 1.5.1 (#1171) f14e8651 build(deps): bump google.golang.org/grpc from 1.38.0 to 1.39.0 (#1172) eb3b0612 build(deps): bump github.com/aws/aws-sdk-go from 1.38.69 to 1.39.4 (#1180) 071a306b dependabot: switched to monthly cadence to reduce noise f702e65a deps: removed dependency on github.com/minio/minio/pkg, replaced with command line tool (#1185) f95ef881 testing: fixed typo a1aa6ae3 tests: install minio mc tool (#1184) 1ef3d243 repo: big performance improvement for WriteContent with repo server (#1182) 61af6399 cli: fixed profiling flags (#1181) 831ffcbf Bump github.com/studio-b12/gowebdav (#1179) 47bc640c epoch: handling of canceled context on refresh (#1178) 62ad437b Implemented epoch-based index manager (#1174) cc9b0e83 content: added sharding of indexes on write (#1176) 5642a8a5 refactor: extracted complete blob set functions to separate package (#1175) a9170275 build(deps): bump postcss from 8.2.15 to 8.3.5 in /htmlui (#1162) 47131489 build(deps): bump github.com/google/go-cmp from 0.5.5 to 0.5.6 (#1123) 7ac0cb31 build(deps): bump google.golang.org/api from 0.46.0 to 0.48.0 (#1125) 6d192fa9 build(deps): bump github.com/aws/aws-sdk-go from 1.38.41 to 1.38.69 (#1159) 9ceb302c build(deps): bump @testing-library/jest-dom in /htmlui (#1164) e64d5b8e Fixed few subtle threading bugs uncovered by stress test and rewrote the test to be model-based (#1157) d73e0d60 robustness: reduce memory consumption (#1155) 474ef1c9 Multiplaces (#1154) d6d97deb content: re-added missing lock accidentally removed by #1138 (#1156) 85cfbb1a cli: added ‘kopia repo set-parameters’ to override mutable parameters (#1148) 0756dee6 More epoch manager work (#1147) 9e059a12 upgraded linter to 1.41.0 (#1144) 5b04a1c4 Added documentation with storage tier advice (#1143) 8b0296cd Misc index blob manager refactorings (#1138) ede09c15 listcache: added missing FlushCaches() method which fixed test flakes 6277fa27 content: refactored own writes cache and list cache into blob.Storage wrappers (#1133) e3782295 site: fix privacy policy link in footer (#1132) 17ba77f5 Index manager refactor (#1131) 42c59cb6 Fixed bug in pattern matching for patterns containing a star. (#1129) 7735fcc5 epoch: added epoch manager + unit tests (#1128) 46a6cc3f blob: minor improvements + test coverage (#1127) 93979818 Add a flag to not shallowrestore small files (#1126) 6fc3bf1a Support for shallow restore (#725) b7c84630 cli: ‘kopia content verify’ performance improvements (#1120) c6accb85 Misc small and trivial changes (#1119) 53589d9e content: fixed minor data race 6b646f7e Content manager cleanups (#1118) f094b976 build(deps-dev): bump concurrently from 6.1.0 to 6.2.0 in /app (#1108) c01e7323 build(deps): bump github.com/fatih/color from 1.11.0 to 1.12.0 (#1103) 87f1f528 cli: added tests for benchmark subcommands d84c8843 Added content manager internal logging (#1116) 4b251bda mechanical: added ctx parameter to repo.{Direct}WriteSession callback (#1114) f5eb12e8 content: introduced content.Crypter (#1112) 3a15f666 webdav: disable server-side compression (#1100) 6811f1fa cli: optimized ‘kopia index recover’ by leveraging partial read and parallelism (#1094) e15a7947 azure: added support for using SAS Tokens instead of storage keys (#1093) 76490dc3 sftp: ensure key file and known hosts are absolute paths (#1092) 40510c04 Support for content-level compression (#1076) 99b7a6e5 unrelated cleanups from PR #1076 (#1091) ba5eb741 robustness: use testing.T for parallel robustness tests (#1051) 57c3ba51 testing: exclude additional test-only packages from code coverage b5db8b09 trivial: fix typo 2faed2bc sftp: run tests as part of unit tests (#1086) 227b2f1b ci: change codecov thresholds (#1085) 5179ad2c cli: test + misc improvements (#1083) 19ad8082 build(deps): bump google.golang.org/grpc from 1.37.0 to 1.37.1 (#1080) 488ab940 build(deps): bump github.com/aws/aws-sdk-go from 1.38.36 to 1.38.41 (#1084) 5ed3a3ac build(deps): bump gocloud.dev from 0.22.0 to 0.23.0 (#1082) a699cc4d build(deps): bump github.com/fatih/color from 1.10.0 to 1.11.0 (#1079) 90a11d46 NPM package upgrades (#1078) 42be5cd4 build(deps): bump react-table from 7.6.3 to 7.7.0 in /htmlui (#1036) d77149bc build(deps): bump github.com/klauspost/compress from 1.11.13 to 1.12.2 (#1025) 56b5f6bb build(deps): bump @testing-library/jest-dom in /htmlui (#1038) 79251b86 build(deps): bump react-bootstrap from 1.4.3 to 1.6.0 in /app (#1064) a59ee47e build(deps): bump electron-updater from 4.3.5 to 4.3.8 in /app (#1041) cb9ad453 build(deps-dev): bump electron from 11.2.1 to 12.0.7 in /app (#1063) 38d01d4c compression: added ‘lz4’ and ‘deflate’ methods (#1077) 34257add testing: disable OneDrive test which is currently broken c017019f rclone: fixed test cleanup (#1075) aeca7331 testing: added onedrive backend for rclone tests (#1074) 862298f9 testing: provider test fixes (#1073) 30ca3e2e Upgraded linter to 1.40.1 (#1072) fcd507a5 Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059) 41931f21 repo: refactored password persistence (#1065) b844ce6b build(deps): bump electron-log from 4.3.4 to 4.3.5 in /app (#1039) b5c3855c build(deps): bump github.com/aws/aws-sdk-go from 1.38.30 to 1.38.36 (#1061) e5fecb18 build(deps): bump google.golang.org/api from 0.45.0 to 0.46.0 (#1060) dec9d9a6 build(deps): bump github.com/pkg/profile from 1.5.0 to 1.6.0 (#1062) 281a7fcc e2e test refactoring (#1058) 8a216778 cli: final steps to remove last global variables for password and globalPasswordFromToken (#1056) 9e861c9e Implemented index v2 builder and parser (#1028) a461d767 cli: plumbed through ‘textOutput’ which controls stdout/stderr writers (#1053) 6c5d58bf build(deps): bump github.com/aws/aws-sdk-go from 1.38.26 to 1.38.30 (#1047) d2288c44 cli: major refactoring (#1046) a56e099f Benchmark: Remove superfluous = in printout (#1043) 1a7a016d Update full maintenance automation explanation (#1042) dd41296f Tagging of kopia snapshots and listing of snapshots by tag (#1030) d2b81693 robustness: add tests for kopia server repos (#1029) 5fcf3895 Add documentation for nginx reverse proxy (#1033) 170d2a21 site: fixed Google indexing cc0dd86f build(deps): bump cloud.google.com/go/storage from 1.14.0 to 1.15.0 (#1024) 11d764dc build(deps): bump github.com/aws/aws-sdk-go from 1.38.25 to 1.38.26 (#1026) df430371 Refactored content.Info to be an interface and switched index parsing to be lazy (#1008) 5dffad96 build(deps): bump google.golang.org/api from 0.44.0 to 0.45.0 (#1022) 2593cdeb build(deps): bump github.com/aws/aws-sdk-go from 1.38.22 to 1.38.25 (#1021) dd3933f1 Grammar fix. (#1023) 0fb4ba41 ui: show a spinner when saving/deleting policy (#1018) a5e1ecbb ui: open .gitignore help in a new window (#1016) d81f3f11 ci: do not upload artifacts from self-hosted runners, fixes build race (#1015) d290c0a9 ui: do not attempt running maintenance if the current user is not the maintenance owner, to avoid producing error in the Tasks tab (#1010) 70a83b38 kopia-ui: for read-only repositories start a read-only source manager (#1009) 069299f2 Policy path validation (#1006) 62fab592 ui: fixed Estimate not honoring the defined policies (#1002) 3f29141c User must install npm manually on OpenBSD (#995) bd5bdaaf build(deps): bump github.com/aws/aws-sdk-go from 1.38.21 to 1.38.22 (#996) 74f926cb content: added content.Info.OriginalLength (#989) a2d31adc build(deps): bump github.com/aws/aws-sdk-go from 1.38.18 to 1.38.21 (#994) 28b21ac9 be consistent in $(MAKE) usage (#992) d235672e Remove “KB” label from processed files count (#991) 2062c072 mechanical field renames (#988) 643c7690 build(deps): bump electron-is-dev from 1.2.0 to 2.0.0 in /app (#984) 922bc7cb ci: refactored credentials handling (#987) e0e7f66a simple version of ospath to get default paths for openbsd systems. Based on the XDG version (#986) 2b6af551 Allow to compile Kopia cli on OpenBSD (#983) a50d3149 build(deps): bump @fortawesome/free-regular-svg-icons in /htmlui (#978) bbb65e97 build(deps): bump uuid from 7.0.3 to 8.3.2 in /app (#977) 0d12cdc5 cleanup a few nits (#980) 2c3dfb78 encryption: removed old, non-authenticated encryption methods (#979) 85aa5e67 build: added freebsd build (untested) with FUSE mounting disabled (#964) 721c04c9 build(deps): bump google.golang.org/grpc from 1.36.1 to 1.37.0 (#965) 51ca0189 build(deps): bump github.com/aws/aws-sdk-go from 1.38.17 to 1.38.18 (#973) 52240b77 build(deps): bump @fortawesome/free-regular-svg-icons in /htmlui (#971) 85e62e90 build(deps): bump electron-log from 4.3.2 to 4.3.4 in /app (#972) 2faf72c5 build(deps-dev): bump concurrently from 5.3.0 to 6.0.2 in /app (#975) 667d1717 build(deps): bump google.golang.org/api from 0.43.0 to 0.44.0 (#966) 18c302ec build(deps): bump github.com/aws/aws-sdk-go from 1.38.13 to 1.38.17 (#967) a223b6f7 testing: run retrying_storage_test as part of regular unit tests b59a1131 ci: publish scoop and homebrew from tags