![](https://steemitimages.com/0x0/https://steemit-production-imageproxy-thumbnail.s3.amazonaws.com/U5dsojBZ9qVcpeYg2MQmyN1gUrxa7qP_1680x8400)

Every couple of days, depending on my own mood and enthusiasm more or less frequent, I go through the process of copying the data from the VGTU server to my own excel sheets to then produce the charts I post from time to time. Because I get questions about this process from time to time I decided to explain this process in this blog post for future reference.

#### Copy the task details
The first step is to check the timestamp of the last task available in the spreadsheet and copy all new task details of finished tasks from the website. Depending on how long it has been this can span multiple pages because only a limited amount of tasks is shown on a page. This is the case in below screenshot.

<center>![1.png](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387208/ynltwevlj6pixifub0zf.png)
</center>

#### Format and verify
Next I verify the data to be sure it is valid for the experiment. The status needs to be 'completed and validated' and the application must not change. I then remove unneeded data and reformat the needed data to the correct format.

<center>![2.PNG](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387219/ayrdrznqzkjtmzyvvyte.png)
</center>

#### Insert the data
Once I have the data formatted I make room for the new data and insert it at the correct location.
<center>![3.png](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387234/n5h1drbbze80rq7ilk5r.png)
</center>

#### Copy some other data
I need to copy the RAC from [here](https://boinc.vgtu.lt/vtuathome/show_host_detail.php?hostid=23189), the team RAC from[here](https://boinc.vgtu.lt/vtuathome/top_teams.php), the GRC price from [here](https://www.coingecko.com/en/price_charts/gridcoin-research/eur)

<center>![5.png](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387256/sqcotvi60w8zyvvikckf.png)
</center>

#### Verifications
Once I have all the input for the first chart I verify the data. 

*From time to time the number of whitelisted projects changes so I verify if the calculated magnitude matches.
*Sometimes the data on the project server is not chronological. This is because the samples are shown in the order they were sent to the host which doesn't necessarily match the order in which they finish. For RAC calculation however they must be placed chronologically in order of completion. If that happens I need to switch some rows.
*If everything is correct the error should be very small.
 
<center>![6.png](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387266/srgu6vgrhzr2dklp85tt.png)
</center>

#### Points per day
Once the data is verified I use it to complete the daily series.

<center>![7.PNG](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387277/ulsl1cehmntmqapbl70z.png)
</center>

#### Update charts
Now all data is available for the first two charts. The last step is to update the range for the charts:

<center>![8.PNG](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387287/h4gj76ytjeouqt1wbyen.png)
</center>

The end result is two updated charts:

<center>![9.PNG](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387297/khh0qhrfmfulniopqotw.png)</center>

#### Top 10 users
Then it's time to move over to the second sheet of the top 10 users. Before:

<center>![](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387323/arhicbhvlqty5q5lf3c8.png)</center>

For these charts I gather the data from [here](https://boinc.vgtu.lt/vtuathome/top_users.php). I select only the users that are in team gridcoin. Sometime new users enter the top 10 or the positions inside the top 10 switch. In such a case I move the columns around for my convenience. After:

<center>![](https://res.cloudinary.com/hpiynhbhq/image/upload/v1519387336/uveb9ra9bpvuvwexwshn.png)
</center>

#### Update the top 10 chart
For these charts as well the data range of the chart needs to be adapted.

#### Result
The result is twofold. A lot of data is available to try new formulas on. And some charts out of which we have been able to learn quite a few things. 

<br /><hr/><em>Posted on <a href="https://utopian.io/utopian-io/@jefpatat/gridcoin-mining-experiment-data-gathering-details">Utopian.io -  Rewarding Open Source Contributors</a></em><hr/>