Added first draft of OLS model with sample data and helper tools

.gitignore

@@ -52,3 +52,7 @@ Mkfile.old
 dkms.conf
 __pycache__
 venv/
+sftp-config.json
+
+*.sublime-project
+*.sublime-workspace

README.md

@@ -166,6 +166,13 @@ Field               | Meaning
 
 It is imporant to not that we use pid 0 as the whole system and not the idle process.
 
+## Test
+
+There is a test script you can run that you need to run on a host that will build the kernel extension and then see if everythig works out. Just run
+```
+sudo ./test.sh
+```
+
 ## Data Collection Helper
 The repository ships with **`energy-logger.sh`** which periodically dumps `/proc/energy/all` to disk for offline analysis (e.g. weight regression).
 
@@ -180,32 +187,69 @@ The repository ships with **`energy-logger.sh`** which periodically dumps `/proc
 
 ## Model
 
-We use a linear model to calculate the energy score for each process. You should train this model yourself by
+To get energy values per process we use a ML Model to estimate weights for the respective hardware component metrics.
 
-1) `src`
-1) running the data collection helper
-```
-echo 100000000 > /sys/module/energy_proc/parameters/sample_ns
-sudo ./energy-logger.sh
+This model is fitted on the machine power and the machine metrics of the system and then mapped on the processes via their individual hardware component metrics.
+
+### Install requirements
+
+1) `$ cd tools`
+2) `$ python3 -m venv venv`
+3) `$ source venv/bin/activate`
+4) `$ pip install -r requirements.txt`
+5) `$ sudo apt install stress-ng` - Or equivalent for your distribution
+
+### Capture energy data while running a workload
+
+```bash
+# optionally set sampling before
+echo 99000000 | sudo tee /sys/module/energy_proc/parameters/sample_ns
+
+## run workload sandwiched between energy logger and start and end
+sudo pkill -f energy-logger.sh
+sudo ./energy-logger.sh & # will output the file it will write to
+python3 run-workload.py mixed
+sudo pkill -f energy-logger.sh
 ```
-1) `python3 -m venv venv`
-1) `source venv/bin/activate`
-1) `pip install -r requirements.txt`
-1) `python3 model.py tmp/energy-XXXX.log`
 
-You can then add the weights to your kernel module by
+By default it will run the *mixed* workload. Check `$ python3 run-workload.py -h` for alternative implemented workloads.
+
+### Estimating the weights
+
+This estimation will give you an estimate to how your system is currently behaving.
+This means it will produce very matching weights for idle workloads if your system is idle.
+But these will not be very fit for compute workloads. So effectively you need to run different workloads and create a mash-up model.
+The mixed model is already an approach in this direction, but the more you know your future workloads, the better the predicitions will be.
+
+Run: `$ python model.py LOG_FILE_NAME --model ols --no-validate`
+
+Example on sample data: `python3 model.py --model ols ../sample_data/energy_logs/mixed_workload.log --no-validate`
+
+*OLS* is the default model for now.
+
+Also you have multiple switches that help with transformation to achieve better numerical stability. Try
+altering the model to use transformed input variables (`--fit OLS-IPS` for instructions per second) or
+try transforming the data as a whole to a new space (`--log` to transform to log space).
+
+### Validating the weights through a prediction
+
+Run: `$ python model.py LOG_FILE_NAME --no-validate --model ols --predict LOG_FILE_NAME_2`
+
+Example on sample data: `python3 model.py ../sample_data/energy_logs/mixed_workload.log --model ols --no-validate --predict ../sample_data/energy_logs/mixed_workload_2.log --scale`
+
+Here you will get descriptive values like the *R2*, *MAE* and *MAPE* to evalute the goodness of the fit for your machine.
+
+
+### Adding estimated weights to kernel module
+
+For each weight you can just send it to the kernel module file endpoint via:
+
 ```bash
 echo 1231232 > /sys/module/energy_proc/parameters/PARAM   # 100 ms
 ```
 
 Please remember that you can not add floats. We use fix decimal values with 3 decimal points.
 
-## Test
-
-There is a test script you can run that you need to run on a host that will build the kernel extension and then see if everythig works out. Just run
-```
-sudo ./test.sh
-```
 
 ## Troubleshooting
 | Symptom                                     | Fix |