ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Having problems installing that new stick of memory? Found some great software or having issues with something? Or maybe want to chat about your PlayStation, X-Box, Nintendo, Sega, even your old Spectrum 48k....! Or maybe something you want to sell or acquire (computing related of course!). Let us know here...
User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Tue Jun 13, 2017 12:15 pm

The funny thing is personally I think it would be easier/better/ and less hassle if one were to write a C++ windows app that does all the downloading, unpacking, extracting, collating, and (local) DB storage - as well as the writing of the UBT stat XML files to a remote webserver. Then the new UBT website would just use these local (webserver stored) XML files.

That way you have the full resources of a workstation, and practically none of the issues as given above.

There are not many webservers that have 14-16 Xeon cores, 32 GB Ram, and 1+ TB storage, available 100% to one account.

For all we know perhaps that is how the original software was done.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Tue Jun 13, 2017 12:37 pm

Hi both,

What I know about what is going on:

a) All the current project stats are available in a compressed form and each "dump" contains stats for everyone who is/has participated. So, individual team stats would need to be extracted each time from each data dump.

b) Closed project stats could be "scraped" from ubtstats or any other stats site and then introduced into whatever storage is being used for current stats (if said project website is no longer live). Otherwise, see above.

c) The SETI project does seem to be the largest project although WCG could be in the running too. In both cases, data dumps from them would be pretty large and hence, in the scheme of things, I would look to introduce those later on in this "project" so that initially just the smaller (and easier to process) projects would be targeted first. The project can then be upscaled with other BOINC projects once the basics have been configured successfully.

d) The code on the ubtstats website is NOT available, so we have no access to any of the coding and/or we have no way of updating said code. I have asked the question, on more than one occasion, but I have not had any response :-(

e) Bandwidth wise, there are a number of options at my disposal: One option could be to do all the data mining on a fast broadband connection, using my own "local" hardware, and then to simply upload the data via FTP, onto a web-facing server, which I already use anyway. This then saves the cost of renting an expensive multi-CPU server and hard drive and paying for any excessive bandwidth costs. Alternatively, it could all be run natively on one of my servers, but to save costs, it might be updated say every 6 hours or so. If you look at boincstats.com, they also schedule their updates at more leisurely intervals than "per hour".

f) I'm still in the process of finding some people with php experience to at least make a start on this. I have found one person, but they run their own business and hence they don't have much time for taking on a non-paying job such as this. Another contact hasn't yet lead to any one else with the right experience.

In the meantime, I've bought myself a nice thick tome on php/mysql with a view to at least getting a handle on what's involved. ;-)

regards
Tim

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Sat Jul 08, 2017 8:40 am

http://boinc.berkeley.edu/trac/wiki/Cre ... #rac-decay

I just came across these which I think will help!!! :wink:

The first is that BOINC actually recommends rather than pull the individual stat XMl files, to instead use an aggregation "service".

They have done a nice graphic to explain this!

Image

A data aggregator is a service that collects XML files from several projects, and computes the totals for participants and hosts (based on cross-project IDs?) across these projects. This aggregate data is then exported in two ways:

    As a set of downloadable files (in a compressed XML format)
    As a set of Web RPCs that return an XML-format description of a given participant or host's credit, based on the cross-project ID

Application developers are encouraged to concentrate on aggregate rather than per-project data. This will encourage participants to join multiple projects.


And finally, the cherry on the icing..

Code examples

Developing a system that collects, analyzes, and stores credit data from all BOINC projects is a large development task. The code for ​BOINC Combined Stats, a web site/service that provides per-project and aggregate data, can be downloaded via git using

git clone git://boinc.berkeley.edu/boinc-combined-stats.git
or
git clone http://boinc.berkeley.edu/git/boinc-combined-stats.git


--

This is what the code example creates.. I have sorted it on UK members! As you can see, you now have 99% of what you need already done.

http://boinc.netsoft-online.com/e107_pl ... d&opt2=236
Last edited by chriscambridge on Sat Jul 08, 2017 9:17 am, edited 5 times in total.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Sat Jul 08, 2017 9:09 am

They are using Java to compile the stats, MySQL to store the stats, and the E-107 CMS (content management system) to display the stats.

So you need someone who knows Java, which is not me as I know very little about this programming language both in syntax and semantics.

This file (readme.txt) explains the files/app/code in detail, taken from that GIT

http://boinc.berkeley.edu/gitweb/?p=boi ... xt;hb=HEAD

This SVN repository contains the source code for the BOINC Combined Statistics web site
(http://boinc.netsoft-online.com/).

Here is a quick guide to the components of the repository:


bsimport - This directory contains the java source code of the import and statistic calculation program. This is the program that is run daily to import the statistics from the various BOINC projects into the stats database.

database - This directory contains the MySQL table creation and update scripts to create the underlying database for the statistics.

e107 - This directory contains the theme and plugin modules for the e107 web site (http://www.e107.org). When these are added and activated to an e107 system, will produce the viewable results of the stats.The site currently uses version 0.7.8 of e107.

scripts - This directory contains other various scripts used by BCS.

The general flow of how BOINC Combined Statistics runs is:

1) The fetch_files.sh script runs twice a day to grab the export files from various projects. The scheduled time of the run is set so it should finish fetching the files before the import process runs. It utilizes wget, and only downloads the file if it has changed from the previous time the script ran. The idea behind running the fetch multiple times is to avoid temporary connection failures to projects.

2) The bsimport java application runs. It runs once a day. This application will take the data files from each project and import the data into the mysql database. After a sucessful import, it creates a backup file of the data. If any of the three import files (team,host,user) has a problem, the program will go through and decay the RAC values so all values in the DB are calculated to the same time.

There are a number of optimizations in the import process. For example, if the host hasn't been heard from in some time, and its RAC is already decayed to zero, then the import process just skips past it - not doing any database transactions for that particular host.

Once all the data has been imported, bsimport creates the stats. It runs through and calculates things like world rank, etc. Often these are done via SQL statments, letting the DB do the work. Other times - in particular the credit per cpu second analysis, the program does the full calculation as it can be done far faster by reading all the host data in order of host cpid than the database can do it.

After the stats are calculated, bsimport exports a sorted list of the combined user data (currently only user data, not host or team data) for use by others to import. The file is a gzip'd xml file that is made available for download on the web site.


3) The statistics are made available via the e107 plugins and other various web RPC pages. These web pages access the data in the mysql database for display via XML, web pages and graphs.

Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Sat Jul 08, 2017 2:15 pm

Hi Chris

Thanks for the detective work on this.

I guess I won't be needing the huge tome on php I bought a couple of weeks ago ;-)

Will be interesting to see how it can be made to work for UBT.

Plus, I'd also like to know why, in the example, I'm MISSING !!!!

regards
Tim

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Sat Jul 08, 2017 5:19 pm

I would not throw away your PHP book just yet as E-107 is built in PHP and that is what displays the "stats output" done by Java.

I would concentrate on trying to find someone to understand the Java portion as that is the actual "engine"; the DB/E-107 side of things is just formatting and display the data via a website. And in truth you may be able to just re-use E-107 and customize it slightly, such as adding Team info and restricting members from just UBT.

There is a chance that all the Java still holds up and hence you could just "run it", which believe is not like PHP. Java (and C/C++) are "thick client" programming languages and hence need to be compiled/interpreted, rather than 'visiting a webpage to run.

[NOTE: Java is NOT javascript!]

What I would say though is E-107 is Version 3, whereas that code int the GIT is version 0.7.8. So I guess what will be of most importance is if BOINC have updated the GIT on any changes (such as when a project changes something), or not. If not then the code probably will not even run..

However if you give it to a Java/LAMP expert they have 99% of what they need already done.

--

I did a quick check, and there are 2 UBT-Timbos. One is UK based and one is International.

https://boincstats.com/en/stats/search/#UBT%20-%20Timbo

That is the only reason why I could think of why you would not be listed int he UK. But from the link above both accounts seem to have different UID (unique IDentifiers). Although I have not gone and checked the CPID's to double check this.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Sat Jul 08, 2017 6:06 pm

chriscambridge wrote:I did a quick check, and there are 2 UBT-Timbos. One is UK based and one is International.

https://boincstats.com/en/stats/search/#UBT%20-%20Timbo

That is the only reason why I could think of why you would not be listed int he UK. But from the link above both accounts seem to have different UID (unique IDentifiers). Although I have not gone and checked the CPID's to double check this.


Hi Chris

It seems that an OLD project I joined, SLinCA, has me as "International" and even though that is a "former" project, it clearly is "scuppering" my UK listing. :-(

I doubt anyone can do much about it, except maybe Willie at BOINCstats ;-)

regards
Tim


Return to “Hardware, Software and Gaming”

Who is online

Users browsing this forum: No registered users and 1 guest