Commit Graph

68 Commits

Author SHA1 Message Date
nabbar
9e8179374b README:
- Add some README file to give missing documentations or update existing documentation file

Package Archive:
- Add some comments to godoc information
- Moving NopWriterCloser interface to ioutils package

Package IOUtils:
- New package NopWriterCloser to implement interfac like NopReader

Package Database:
- KVMap: fix missing function following update of kvdriver

Package Duration:
- Rename BDD testing

Package Context/Gin:
- Moving function New between model & interface file

Package AWS:
- rework Walk function to use more generic with standard walk caller function
- func walk will now no more return and include error (can be catched into the given func)
- func walk will now return a bool to continue or stop the loop
- func walk with many input function will now stop when all given function return false
- func walk will now return error only about main process and not given function

Package errors:
- Add interface error into interface Error

Package IOUtils:
- Moving IOWrapper as subPackage and optimize process + allow thread safe
2025-05-25 06:29:25 +02:00
Nicolas JUHEL
3d9eee878f Package AWS:
- fix bug whos aws config checksum not loaded from config struct, using callback on config process
- bump to latest sdk release

Other:
- bump dependencies
2025-03-13 16:47:17 +01:00
Nicolas JUHEL
90dd383441 Package AWS/Objects:
- add metadata function to walk objects / walk object version function, who's contain the len of result (objects, version, deletemarkers)
2025-03-07 09:41:01 +01:00
Nicolas JUHEL
82d32d81b0 Package AWS:
- bump dependencies aws sdk go v2 with checksum calulation when supported by default
- add function in config interface to allow changing checksum calculation
- set checksum calculation as 'when required' with custom config
- set checksum calcultation as 'when supported' with aws config

Other:
- bump dependencies
2025-01-28 16:42:02 +01:00
Nicolas JUHEL
8746c37784 Package aws/pusher:
- fix defer called before triggered
- fix info called without data when loaded source is closed
2024-10-08 12:09:33 +02:00
Nicolas JUHEL
6213eeb560 Package AWS:
- fix anonymousCredential with retrieve following bump of aws sdk

Package Logger:
- add check on io interface to prevent closing instance and allowing gracefull closing

Other:
- bump dependencies
2024-10-02 15:07:59 +02:00
nabbar
07a63f9636 Package aws:
- add new sub package pusher: pusher is an helper who's allow to push
  single put object or multipart upload and implement io.writer &
  io.readFrom interface

Other:
- bump dependencies
2024-08-11 15:07:00 +02:00
Nicolas JUHEL
39df39938e Package AWS:
- fix double import with different tag for validator in config model
- applys MR about fix on bucket validator to config AWS model
2024-05-30 16:34:02 +02:00
Valentin Morel
88421f1aac Package AWS : Validator tag for s3 bucket (#184)
Package AWS:
- fix validator tag for s3 bucket: use custom function instead of hostname
2024-05-30 16:23:39 +02:00
Nicolas JUHEL
dbeb385267 Package Encoding/RandReader:
- optimize code
- remplace some pointer by atomic to prevent RACE

Other:
- bump dependencies
2024-05-29 16:39:59 +02:00
Nicolas JUHEL
00dee5e6e6 Fix aws group 2024-04-30 17:39:09 +02:00
Salim Amine BOU ARAM
33db3b3453 fixing the empty prefix issue 2024-04-29 13:09:18 +02:00
Salim Amine BOU ARAM
055026ade4 refactoring aws 2024-04-29 10:41:10 +02:00
Nicolas JUHEL
ae0a6b62c9 Package HTTP Client:
- Add DNS Mapper to force destination for a fqdn source
- Allow wildcard and multi wildcard for fqdn source
- DNS Mapper create Transport, Dialer & http client
- DNS Mapper allow a config input to customize timeout, limit and TLS config
- DNS Mapper use a gloabl transport connection poller
- DNS Mapper implement DialContext & Dial method for transport
- DNS Mapper use cache to accelerate process
- DNS Mapper cache is only dnsmapper, not DNS cache
- Replace old helper for http client with DNSMapper
- Add default DNSMapper into the main lib of http client
- Allow to overide the default DNS Mapper with a new one
- Add ticker to force clean idle connection every given duration
- "Compatible" with old config, as the default config will be used instead of old client build
- Clean code and fix minor bugs
- Add config component to allow use a global config
- Config Component httpcli can overide default httpcli dns mapper when updated

Package Certificates
- Add function type to impose function that return a slice of rootCA string
- update config/component/tls by replacing mutex with atomic
- optimize some code

Package Config
- expand errors code index to add component httpcli

Package Config/component:
- database: add new validation, that config key existing into viper
- head: add new validation, that config key existing into viper
- http: add new validation, that config key existing into viper
- ldap: add new validation, that config key existing into viper
- log: add new validation, that config key existing into viper
- mail: add new validation, that config key existing into viper
- smtp: add new validation, that config key existing into viper

Package AWS:
- update package following update of httpcli
- use interface for http cli that implement the Do function
- update following the config/component/aws
- rework config/component/aws to use atomic instead of mutex
- update test unit following change

Package Request:
- update following http client update
- use interface of HTTP Client that implement DO function, instead of http client struct pointer
- update config & code following
- apply same modification into config/component/request
- update config/component/request by replacing mutex to atomic

Package Server
- add function Uptime to model to expose the duration since last start

Package Semaphore
- apply change name of constant following bump of lib associated

Package Crypt
- fix bug into reader to remove suffix of EOF if prevent

Package Errors:
- expand index module to add DNS Mapper client

Package HTTP Server:
- update monitor to apply changes

Package Socket:
- add additional check if reading error: if buffer has data, send it to handler before break run

Other:
- bump dependencies
2024-02-16 15:42:13 +01:00
nabbar
c66a00917a Package Socket:
- fix tcp,udp,unix socket
- add socket unixgram for session less unix socket
- optmize process & code
- add some standalone test for server / client (add example server config for
  rsyslog)

Package Network/Protocol:
- add unixgram protocol

Package Errors:
- fix bug in loop to prevent circular include
2024-01-28 15:18:56 +01:00
Nicolas JUHEL
77c7ebfe83 Package AWS:
- Allow to use config anonymous with no access key / secret key
- Replace *http.client to interface of Do function
2023-12-14 13:40:36 +01:00
nabbar
82be8c0c89 Pakcage AWS:
- add subpackage to sign http.request with AWS V4 signature and parse response based on http.response and given model

Other:
- bump dependencies
2023-11-24 13:22:57 +01:00
Nicolas JUHEL
97e70d063f Package Archives:
- fix security arbitrary path
- fix linter

Package AWS:
- implement resolver v2

Package Cobra:
- fix linter

Package Config/component:
- fix linter

Package Context/Config:
- Add function to set context

Package Database/KV...:
- Fix error
- Fix collision pointer
- Fix models
- Fix circular dependencies
- Add function Delete on driver, table and item
- Add function new on drvier to prevent collision data when create item on table get / walk

Package Duration:
- Add type Duration based on time.Duration to allow transform duration to string instead of int64 nanosecond
- Add function to parse in json, yaml, toml, text, cbor
- Add function to allow convert type into mapstructure (spf13 viper, cobra...)

Package File/Perm:
- Add type Perm based on os.FileMode to allow marshall / unmashall it into octal form instead of string representation (-rwxrwxrwx)
- Add function to marshall / unmarshall in json, yaml, toml, text, cbor
- Add function to allow convert type into mapstructure (spf13 viper, cobra...)

Package File/progress:
- Fix linter

Package HTTPServer :
- Fix linter
- Fix security by adding a default value if not set on config

Package ioutils:
- Fix Linter

Package LDAP:
- Add Clone function

Package logger/hookfile:
- Fix linter

Package nats:
- Fix linter

Package Network/Protocol:
- Fix bug with quote / Dbl Quote on unmarshall

Package Password:
- Replace password with crypto rand instead of math rand

Package Size:
- Fix potential overflow
- Add function to format value into Int32, Int, Uint32, Uint, Float32
- Add function to parse Float64 into type Size

Package Socket:
- change config uint32 to golib Size, time.Duration to golib Duration
- add TLS managment to server TCP, discard for UDP & Unix file Local Domain
- add function Info Server to print information of server when listen is starting
2023-11-21 20:21:37 +01:00
Nicolas JUHEL
b0fd08792c Package AWS
Add feature to change http timeout directly in httpcli store in aws client
  Add copy/MPUCopy function
  Update mpu to allow copy with MPU

Package Cobra
  Fix println by fmt.Fprint to os.stdout

Package Crypt
  Refactor package to use instance of crypt instead of a global crypt unique instance
  Allow to gen key/nonce
  Add io stream reader/writer instead of only buffer slice encoder decoder

Package file/bandwidth
  Add new subpackage bandwith to limit a amount of byte read/write for a duraction
  Use file/progress function increment/reset
  Allow to add custom function increment / reset

Package IOUtils
  Remove useless file error.go, tempFile.go (replaced by package file)
  Move fileDescriptor to a subpackage fileDescriptor: allow to change nb of FD for linux/windows os
  Move Buffer ReadCloser to a new subpackage: allow to add close function to buffer, and expose read / write io interface
  Add sub package multiplexer to allow send/receive multiple io stream into on io stream

Package Socket
  Rework to expose io stream instead of buffer
  Fix minor bugs

Other:
  Bump dependencies
  Update code following bump dependencies
2023-10-03 12:03:36 +02:00
Nicolas JUHEL
aed9d98203 Add Aws Function & Fix Validotor
Package aws
- fix dive, use it only for slice
- add function to version delete with bypass governance policy

Package cluster
- fix dive, use it only for slice

Package config
- fix missing close channel for waitnotify function

Package errors
- fix pattern %s with uint16

Package ioutils
- fix liberr / Error type return

Package nats
- fix dive, use it only for slice

Package nutsdb
- fix dive, use it only for slice

Package request
- fix dive, use it only for slice

Other
- bump dependencies (validator v10.15.2)
2023-09-04 16:37:19 +02:00
Nicolas JUHEL
984ba51587 Rework Error interface
Package Errors:
- add function to check & cast error interface into golib Error interface
- update CodeError type to simplify management & error creation
- add function to simplify call of Error function from a generic error interface
- remove some useless function from Error interface

All Other Packages:
- apply change of package Errors into all other packages
2023-08-28 11:22:08 +02:00
nabbar
c4b5f11efc Rework ioutils/gzipreader/fileprogress
Package ioutils:
- remove file progress from ioutils and rework it to package
  file/progress

Package file/progress:
- simplify call / use of file progress
- optimize code
- use atomic to function progress
- isolation part of code
- make interface more compatible with *os/File / io interface

Package archive/gzipreader
- create package to expose a io.reader interface from a no gzipped io.reader
- add interface GZipReader to expose metrics like rate of compression

Package archive:
- apply following change
- add minor internal change into errors files

Package artifact:
- apply following change
- add minor internal change into errors files

Package aws:
- apply following change
- removing minio server from repo

Package mail:
- apply following change
- add minor internal change into errors files

Package nutsdb:
- apply following change
- add minor internal change into errors files

Package static:
- apply following change

Other:
- bump dependencies
- ci/cd : add a wget command to dl minio server for testing
- add aws/minio to gitignore
2023-08-22 19:42:46 +02:00
nabbar
ffb5724c2c Minor Fix & Go1.21
Package Socket:
- fix bug with timeout

Package AWS:
- update config model following validator update

Other:
- bump dependencies
- go1.21
2023-08-22 19:35:16 +02:00
Nicolas JUHEL
7cd4eadc50 Package Status:
- Rework for the management of mandatory components to qualify the global status.
- Added qualification to components: "should", "must", "one of".
- These qualifications will influence the overall state but not the unitary state of each component.

Package Socket:
- Creation package to open and listen local unix file socket
- Allow to specify a handler who's needing a conn instance as entry
- Create a client sub package to allow negociate with the local unix file socket

Package Shell:
- adding sub package for command and command collection

Package Monitor:
- add function to expose monitor command for shell

Package Config:
- add function to expose monitor command for shell

Package Network:
- Add "IP" protocol

Pakcage Httpcli:
- Rework the construction of the client
- Add capability to specify proxy url

Package Cobra:
- add function to add flag to global command like verbose or config

Package Log:
- fix seg fault when calling a nil instance
- remove println in hookfile / hooksyslog
2023-07-26 21:11:39 +02:00
nabbar
f31950daef Package Logger:
- Rework logger
- Remove deprecated functions
- Split Logger to sub package (fields, level, config, ...)
- Optimize dependencies
- Rework Hookfile: fix error like FD still opened
- Rework Hooksyslog: use same model like Hookfile, use network/protocol instead of self lib
- Rework HookStd: use independent hook for std out & std err
- Fix std config make generic options for files & syslog
- Apply formatter to hook instead of main logger entry
- optimize code

Package ioutils:
- rework PathCheckCreate funct: optimize code & fix some error

Package Network:
- create sub package protocol for all network protocl use
- add encode function

Package httpcli:
- remove file network
- use package network/protocol instead of network file

Package archive:
- apply change following rework of logger

Package aws:
- apply change following rework of logger

Package cluster:
- apply change following rework of logger

Package cobra:
- apply change following rework of logger

Package Config Component:
- apply change following rework of logger to component log
- fix logger for monitoring
- fix component following fix of package request / monitoring

Package context:
- apply change following rework of logger

Package database:
- apply change following rework of logger

Package httpserver:
- apply change following rework of logger

Package ldap:
- apply change following rework of logger

Package monitor:
- apply change following rework of logger
- fix logger for monitoring
- fix minor bugs

Package nats:
- apply change following rework of logger

Package nutsdb:
- apply change following rework of logger

Package request:
- apply change following rework of logger
- fix minor bug
- fix missing logger for monitoring
- add one line for healthcheck (info or error)

Package router:
- apply change following rework of logger

Package static:
- apply change following rework of logger

Package status:
- apply change following rework of logger
- fix bug with mandatory component

Package viper:
- apply change following rework of logger

Other:
- bump dependencies
- github action workflow fix
2023-06-26 08:36:29 +02:00
nabbar
1ab80f338e Package AWS:
- rework MultipartUpload process & helper
- update test to use lib size
- update object multipart to use new helper

Package IO Utils :
- add truncate & sync to FileProgress
- fix error on open file mode for FileProgress

Package Console :
- fix interface used for color buffer

Package Cobra :
- add function to print message on write config to use custom message instead of internal message. If the function is not set, the default message will be print.

Other:
- fix golangci-lint config to remove crazy linter (use only golang group compliance linter)
- bump dependencies
2023-06-02 10:26:35 +02:00
nabbar
5d447406d6 Package AWS:
- objects / list Access All : add username param to allow list access and return a slice of access struct for the specify user, if username is empty, send the request without username
- sizepart : remove sizepart struct and replace it by package golib/size

Package Logger:
- fields : fix error with field & sync/map

Bump dependencies
2023-04-21 10:08:48 +02:00
nabbar
c72199d029 Fix 2023-03-01
Package Config/Component AWS :
- rework monitoring for status config to use Golib Request health system & config
- if Enable is true on the health config, use a request instance for healtcheck, otherwise use the aws check as healthcheck

Package AWS/Config :
- adjust code for config request health

Package Request :
- expose healthcheck into interface

Bump dependencies
2023-03-01 12:08:12 +01:00
nabbar
e3239db998 Rework Monitoring, Prometheus, Status system
Package Monitoring :
- use packag dedicated to monitor component
- each monitor work as standalone server to monitor health
- collect metrics to export them to prometheus exporter

Package Prometheus :
- review to simplify use for API and not API metrics
- optimize code

Package Status :
- Rework to use Monitor package
- Rework to use native json / text Marshaller interface

Context :
- rework context config (context var) to use sync map and sync RWMutex (WORM)
- move gin context to dedicated sub package (dependancies of logger make circular dependencies)
- optimize code

Config :
- rework to optimize sync / collect of component
- rework status to monitor
- remove monitor managment from config to each component
- add a func to set default logger to implement inherit default logger options
- optimize code

IOUtils :
- isolate logger / closer interface as a usable & public interface & instance
- this interface / instance allow to collect io.closer over a context to close all if context is done

Logger :
- rework to use context.config map
- rework to use ioutils closer
- rework to allow options to inherit a default options, or the last version of options
- optimize code

Size :
- Add package Size to calculate and manipulate size Byte or bit
- Add encoding : Text/JSON/Yaml/Toml...
- Add option to défine default unit : Byte or bit

Other :
- adjust following code
- optimize code
- limit use of atomic value
- rework to use RWMutex instead of sync.Mutex to maximize capabilities of read instead of write
- remove 32bit build for CI/CD
- add darwin/arm64 build for CI/CD

Bump Dependencies
2023-02-21 16:30:40 +01:00
nabbar
771a66eff4 - Package AWS : fix error with context while cloning
- Package Errors : fix error with slice of error not returning all parent
- Package Logger :
    - fix log error slice
    - following bump hclog : implement GetLevel + adjust SetLevel to report different between hc log debug & trace following EnableTrace in options
- Bump dependencies
2022-12-07 12:41:17 +01:00
nabbar
980012d921 Package AWS :
- Bucket ADD CreateWithLock : create bucket with Object Lock enable
  - Bucket ADD GetLock : retrieve object lock bucket configuration
  - Bucket ADD SetLock : set object lock bucket configuration
  - Object ADD GetLegalHold : retrieve object lock status for object/version
  - Object ADD SetLegalHold : set object lock flag for object/version
  - Interface ADD SetClientS3 : replace current client S3 with a new one given as parameters
  - Interface ADD SetClientIam : replace current client IAM with a new one given as parameters
  - Interface ADD HTTPCli : return the current setting of func to return a http.client
  - Interface ADD mutex : lock any accessing struct properties to prevent race

Other :
  - bump dependencies
2022-12-02 09:35:29 +01:00
nabbar
eb8097fad8 Fix local path operations by using a wrong package (diff between package path and path/filepath)
Bump dependencies
2022-11-15 14:37:49 +01:00
Nicolas JUHEL
5187e7eae2 Package AWS :
- User : Add/update policy managment (list, walk, attach, detach)
- Group : Add/update policy managment (list, walk, attach, detach)
- Role : Add/update policy managment (list, walk, attach, detach)
- Policy : add capability to not list default verison policy
2022-10-28 15:16:53 +02:00
Nicolas JUHEL
7139797df2 Package AWS :
- Fix bug while check key len : min size was 3, AWS reference API S3 say min len is 1
Update :
- Bump dependencies
2022-10-25 15:31:31 +02:00
Nicolas JUHEL
573af6ce3a Package AWS :
- Config Model : add a config model with a golib RouerStatus Config Model to use a AWS connection for API with a request status health check
  - Config Interface : add function GetAccessKey to retrieve the accesskey value currently used into the current connection
  - function Walk (Object, version, ...) : fix bug with pointer of string not initialized

Package Config :
  - interface Component : add status router pointer into the Init function to allow used a global router status for all component status registration as router status component

Package Status :
  - move Status Config as an sub package of Package Status

Package Errors :
  - add CamelCase const

Package Logger :
  - fix following bump dependencies

Global :
  - bump dependencies
  - change init of errors files : change function never call vy panic to prevent an error code collision
2022-09-13 12:31:04 +02:00
Nicolas JUHEL
55aa0c3637 Package AWS:
- Bucket : add function Walk based on list but run a given function for each found bucket
- Object : add function Walk & WalkPrefix based on ListPrefix to run a given function for each returned object
- Object : add function WalkVersion & WalkVersionPrefix based on ListVersion to run a given function for each returned object version
Package logger:
- Fix bug with compat : a nil error not clean when sent to logger
Package Context:
- Fix bug with context/config : the load key return an atomic value instead of the interface stored
2022-08-10 11:11:34 +02:00
Nicolas JUHEL
f0a5a894f2 Pakcage AWS :
+ Add Tag function for object
  + Add Lock function for object
  + Add Attributes function for object
  + Add lifecycle function for bucket
  + Add function to get client in AWS interface
Package Certificates :
  + Add clean certifs function in config
Bump dependencies
2022-08-03 12:13:15 +02:00
Nicolas JUHEL
90ceb19d2f - Chg :
- Package AWS : extend aws helper to manage accessKey
	- Package Cobra : refactor configure function to allow use independant part of the command
	- Package ldap : fix error
	- Package config
		- interface : add shutdown method
		- components:
			- request : syntax error in default json
			- mail : syntax error default json
			- ldap :
				- allow to access loaded config
				- mutex lock circular
			- tls : allow to access loaded config
	- Package request :
		- add error managment
		- fix error with circular mutex
		- fix bug with empty response body
		- fix some other errors
	- Package status : default label in component default config
	- Bump dependancies
	- Bump yaml to v3 (in waiting fix for CVE-2022-28948)
	- force dependancies gogo/protobuff to v1.3.2+
2022-07-25 16:00:36 +02:00
Nicolas JUHEL
01308cc8f2 Optimize errors for package aws & logger
Bump dependancies
2022-04-28 16:19:55 +02:00
Nicolas JUHEL
48b47ec97e Package AWS :
- fix missing commit : version function
- fix missing commut : deleteall function
- add function to load bucket replication configuration
- add AWS feature to create new client for a config based on current client
- add Config feature to change the credentials of current config
2022-04-22 14:37:59 +02:00
Nicolas JUHEL
017c24f2df - Package AWS : fix missing field for clone function
- Bump dependancies
2022-04-21 15:51:08 +02:00
Nicolas JUHEL
f073bca2d3 Package AWS :
- Fix issue #136 :
  - add function ForceSignerOptions to pass options funct to signer
  - if not option exist, use default signer v4, else specify the options function list to the new signer
- fix error with multiple ptr identifier on model
- add function to list and cancel multipart uploads
- add function to list, get and delete version object
- add function to delete all objects following the S3 API DeleteObjects function (to send array of object key / version) in one request

Package Config :
- modify component interface :
    - add cpt as custom function parameters to prevent new component still not stored when running custom function
    - re-order mutex on start / reload process

Change :
- Bump dependancies
2022-04-20 12:21:26 +02:00
Nicolas JUHEL
b1f5e42db7 Package HttpCli :
- slipt package between pure httpcli and request to become new package Request
- Add options to httpcli allowed to create an http client based on options

Package AWS :
- add option for delete object : check boolean to make or not the check (head) before delete
- add exclusion of NoSuchKey error if check is false

Package Request :
- integer option capabilities to configure the request inteface
- use httpcli option to get httpcli (keep custom function to get an *http.client)
- extend function to mode manage the request
2022-04-01 13:33:21 +02:00
Nicolas JUHEL
6645bcc05e Refactor Package HTTPCli:
- Reword Const (timeout + errors)
- Remove old request interface
- Add new request interface to make a request, send it, parse a result model and check response status
- Add function to skip DNS resolution for http client connection

Change
- Bump dependancies
2022-03-31 13:11:00 +02:00
Nicolas JUHEL
312ce42dc3 Fix issue #132 2022-03-29 08:36:19 +02:00
Nicolas JUHEL
425962f8c0 Add packages :
- package databse : use gorm lib to  implement this helpers
- package fpt client : expose a helper to post / read file from ftp server

Add component into package Config :
- add AWS custom / standard
- add database
- add Nats cluster
- add NutsDB cluster
- add smtp client
- fix some errors into tls, http component

Fix / Update :
- bump dependancies
- adapt some fix into aws config, smtp, http server, nats, nutsdb, status packages
- add gorm logger into logger pacakge
- fix issue #125
- fix issue #126
- fix issue #127
2022-03-16 15:08:33 +01:00
Nicolas JUHEL
7fbcf39c6a AWS : add method to update metadata (copy object)
AWS : add method for website object (update redirection & acl)
AWS : fix bug with missing content-type (all defined as application/octet-stream)
2022-01-17 18:06:28 +01:00
Nicolas JUHEL
74be9a73c4 AWS Test suite : fix cancel context 2022-01-14 13:10:52 +01:00
Nicolas JUHEL
10b2d5e06b - Add Bucket ACL
- Add Sub method for Set Bucket ACL to simplify small call
2022-01-13 17:29:56 +01:00
Nicolas JUHEL
032ddb30a4 PKG AWS :
- add bucket website configuration method
  - add bucket CORS configuration method
  - chg bucket method structure file
  - bump ginkgo to v2

PKG Logger :
  - bump ginkgo to v2

Change :
  - bump dependancies
  - change unit test command to be compliance with ginkgo v2
2022-01-13 10:45:43 +01:00