ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Having problems installing that new stick of memory? Found some great software or having issues with something? Or maybe want to chat about your PlayStation, X-Box, Nintendo, Sega, even your old Spectrum 48k....! Or maybe something you want to sell or acquire (computing related of course!). Let us know here...
User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Sun Apr 16, 2017 1:07 pm

Hi all

I have been asking for the above website to be updated or for John, its author, (aka Temujin), to allow me to either license the software or to have admin access to the site so it can be updated.

The last email I had back was about 6 months ago and, although I made another email request recently, I've had no reply.


So, this is a call out to anyone with PHP skills.

Basically, using the ubtstats.co.uk website as a template, can anyone build a new stats website, from the ground up, that can be hosted on one of the teams domain names?

Sadly, there is no likelihood of any payment for your time and effort...and I know from what John has created that it might not be a simple task especially given the many different functions built into the existing stats website.

But I am sure that it can be done, and maybe a start can be made to the point where some basic stats can be displayed. And then, in time, other functions might be incorporated?

So, if you fancy putting something together, please get in touch, either on here or by PM.

regards and thanks
Tim

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Thu Apr 20, 2017 8:02 am

Just to put what you asking into context, this is what you will need to do:

0) Create a master Database with appropriate Database Schema/Tables/Referential Integrity, using say MySQL etc

1) Parse the XML stats files for each project storing the data into a DB, using PHP
2) Continue to parse the XML stat files on a say, hourly basis, storing this data
3) Create a website that preferably is responsive, using XHTML/HTML5/CSS
4) Implement/use some kind of graphing display API in order to present the stats information from the DB, either PHP, HTML5/XHTML, or perhaps JS/etc.

5) Make sure you have a server that handle/run hourly scripts, and create Database scheme/Ref Intg.
6) Have a hosting account that is going to let you run scripts that are downloading massive files/amounts of data, as well as are running scripts for some considerable amount of time processing this data.

Its actually quite an involved project which has a cross range of skills required.

Also, I think it may overload most "shared hosting" accounts which normally have quite short timeouts on scripts running.

I could be wrong about all this, as it has been a year or two since doing LAMP development and perhaps things have changed somewhat, however I very much doubt it as things do not normally change that quickly. Perhaps a current LAMP and XHTML/CSS developer can comment on this.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Thu Apr 20, 2017 10:22 am

Hi Chris

Thanks for the message.

a) On the hosting side, it's not an issue as I host quite a few websites already. If I need to change servers in order to provide more resources then I can do so easily enough.

b) On the data processing side, then, initially, I would minimise the requirements, by simply importing from one XML stats link perhaps once every day.

c) Then as time went on, more projects could be added and the frequency of grabbing the XML data could be increased, depending on server load.

d) All that is needed is some way of coding the scripts and creating/managing the DB, importing, processing and then displaying the stats on a public URL.

I get stuck on d) as this is way above my skill set.

regards
Tim

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Thu Apr 20, 2017 10:56 am

Hi Tim,

a) I am not 100% sure as clearly it depends on the server load put on by the application, and your current provider, but a worse case scenario may be you would need a dedicated server. In truth until you started/finished coding or at least worked out 100% the procedure, I think you would find it hard to tell.

b) Probably the simplest way to begin with would be to manually download the stats.xml (or whatever it is called) and just save it on the server (eg hosting account); that at least stops you running into issues of saving to /tmp/ or what ever mechanism your hosting provider has implemented. Trying to get it working on one project to begin with would make sense, and then just scale it later.

c) The amount of projects, and the timing are different factors. You must have an auto-run script which runs every N hours/days/months or whatever. it does have a specific name but I have forgotten at this time. Some hosting providers allow you to run them, others do not.

d) You would create the DB using PHPMyAdmin, which is normally with your hosting control panel.

However before you did anything, you would have to take a look at the stats.xml (etc) file to find out what data/information it contained. For instance, you are going to have to look (via code) at every record and see if the user is a UBT member. Also, to create the DB you are going to have to work out what columns/fields are required, not only for information to be shown on the website, but also in terms of structure of how the XML file 'works' in terms of identifying each specific user and their stats.

--

I would offer but I do not really have the time, and in truth do not want to get involved with any projects at this time. I did think about it in relation to my own stats a few months ago and decided against it.

Personally I would go back to the original developer and try and buy it from them. Then come back to the members and do a donation drive to get the capital. If I had to put a price on the code, I would definitely say over £2K. However before I made the sale, if possible, I would definitely check what the server requirements are, as if it needs a dedicated server then these can be quite costly; a managed one could cost you £500+ per month, with a minimum contact length.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Thu Apr 20, 2017 4:56 pm

chriscambridge wrote:Hi Tim,

a) I am not 100% sure as clearly it depends on the server load put on by the application, and your current provider, but a worse case scenario may be you would need a dedicated server. In truth until you started/finished coding or at least worked out 100% the procedure, I think you would find it hard to tell.


Hi Chris

Thanks for the extra input.

Dedicated servers are no problem for me - I run a couple already !!

b) Probably the simplest way to begin with would be to manually download the stats.xml (or whatever it is called) and just save it on the server (eg hosting account); that at least stops you running into issues of saving to /tmp/ or what ever mechanism your hosting provider has implemented. Trying to get it working on one project to begin with would make sense, and then just scale it later.


Yup - each project outputs these regularly, so it's just a question of getting the script to access each project in turn and then "import" each set into the DB.

c) The amount of projects, and the timing are different factors. You must have an auto-run script which runs every N hours/days/months or whatever. it does have a specific name but I have forgotten at this time. Some hosting providers allow you to run them, others do not.


I think you're talking about chrontab ?

d) You would create the DB using PHPMyAdmin, which is normally with your hosting control panel.

However before you did anything, you would have to take a look at the stats.xml (etc) file to find out what data/information it contained. For instance, you are going to have to look (via code) at every record and see if the user is a UBT member. Also, to create the DB you are going to have to work out what columns/fields are required, not only for information to be shown on the website, but also in terms of structure of how the XML file 'works' in terms of identifying each specific user and their stats.


My guess is that each stats file, from each project, will basically have the same generic info within it. This can then by imported into the DB, and then "querys" can be run to export the required data (using both historical and up to date info) into a table which can then be ordered and displayed accordingly. That's the simple explanation at least...

I have a couple of people I know who might be able to give this out as a project to someone, like a student :-) At least it'll be a start :-)

--

I would offer but I do not really have the time, and in truth do not want to get involved with any projects at this time. I did think about it in relation to my own stats a few months ago and decided against it.


Don't worry - I appreciate the input for now...and I know this could be a tough ask...but if one doesn't ask :-(

Personally I would go back to the original developer and try and buy it from them. Then come back to the members and do a donation drive to get the capital. If I had to put a price on the code, I would definitely say over £2K. However before I made the sale, if possible, I would definitely check what the server requirements are, as if it needs a dedicated server then these can be quite costly; a managed one could cost you £500+ per month, with a minimum contact length.


I've tried this. John (Temujin) was our Crunch Leader for some time, and even today, he is the still credit leader for our team on a number of projects. And he was also a valuable member of the forum, in terms of his input and helpfulness. But by April 2015, he had stopped running BOINC and had lost all interest in it. I got in touch with him and my last contact with him was in Sept 2016, where he said he'd try to update the website. Nothing appears to have happened since then and my last email to him 2 weeks ago has gone unanswered.

I have offered to license his work, for a fee, so I can host it, and I've also asked to at least be given admin access to the stats "back office" so that updates can be made. Neither of these options were acceptable.

So, I've now given up trying. We're 2 years down the line and no updates have been made so as nice as they are, if they aren't updated, they will eventually prove to be pointless, as old projects stop (or update their websites, such as Einstein and CPDN) and new ones cannot be included.

regards
Tim

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Thu Apr 20, 2017 5:36 pm

Hi Tim,

Working backwards.

I am not surprised he is not interested in licensing it for your own server. As soon as you get the code, you only have to rewrite it, and it then becomes "yours", legally anyway. You may find his interest becomes apparent on the offer of a sale. Its just a suggestion mind you.

****

I agree, every time a project changes URL, it then drops off the stats. CPDN have just changed their URL now so I guess that will no longer update anymore.

****

In terms of the stat XML files this is what I found out when I looked into this..

Each project has a: user.gz file which holds the XML file (which is the structure and data). It is located at: project_url.extension/boinc/stats/

Take for instance, Asteroids:

http://asteroidsathome.net/boinc/stats/user.gz (5.5 MB)

These XML files have the following structure:

Code: Select all

<user>
 <id>somenumber</id>
 <name>someone</name>
 <country>United Kingdom</country>
 <create_time>1471310252</create_time>
 <total_credit>943680.000000</total_credit>
 <expavg_credit>14537.941172</expavg_credit>
 <expavg_time>1489585535.487145</expavg_time>
 <cpid>1f409e6b69a8290c8419ab06f5bb8727</cpid>
 <url>someone.com</url>
 <teamid>35</teamid>
 <has_profile/>
</user>


--

In terms of importing these XML files into a DB, and then exporting the information into the website, that is the way you would go if you were going to do this by hand, which in truth would be a complete nightmare.

In code, the way you would do this is:

A) Decide what data you need to display the stats information on the website

B) Create the DB with the correct structure (schema) eg fields/columns, as decided above.

[Referential integrity (primary/foreign keys, normalization, etc) should be used for data consistency, integrity, etc. As a minimum you should definitely use primary and foreign keys; normalization I guess you could ignore for now].

1) open the XML file in code

2) traverse, eg Parse, the XML file building up an array/vector of the data you want to keep from each 'record/row' in the XML file

3) traverse the array/vector writing each 'row' into the DB

You would do these steps every N time period for each project, to make sure the data is updated into the DB. UBStats does this on an hourly basis.

--

You would then code your website to just query the Database (in code) to get the data/information, and display it via the website. Every time the DB gets updated, it automatically updates the website.

Doing it this way means no manual importing or exporting.

****

"Cron Tab" -- that's the one, what we in PHP would call Cron Jobs. Basically running a PHP script every N time period.

****

Having a couple of dedicated servers means you are part of the way there! At least you can definitly run cron jobs and decide if you want scripts to take as long as what ever time they need.

****

I would then look into getting someone who can do the display side of things.

XHTML or HTML5 for a fully responsive "website" (what it looks like), with the PHP or JS skills to be able to display in Graphs/Charts. Once you have this, then the PHP/SQL side of things is not too difficult. You could probably parse the XML and write it into the DB in just a few lines of code (omitting error checking code).

Chris
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Thu Apr 20, 2017 8:20 pm

chriscambridge wrote:Hi Tim,

Working backwards.

I am not surprised he is not interested in licensing it for your own server. As soon as you get the code, you only have to rewrite it, and it then becomes "yours", legally anyway. You may find his interest becomes apparent on the offer of a sale. Its just a suggestion mind you.


Hi Chris

Money has been offered. No actual amounts nor any suggestion of magnitude...but no dice.

Likewise, I offered to license the code on a "non disclosure basis", given that the stats are only for the team members benefits and no commercial activity would be considered. I certainly wouldn't "sell" the code to any third parties, esp given that John should get any rewards, if there are any :-)

I agree, every time a project changes URL, it then drops off the stats. CPDN have just changed their URL now so I guess that will no longer update anymore.


In fact CPDN had a big issue a year or two ago and ubtstats has been in error ever since :-(

One member, jrenals, apparently is scoring billions of credits every month and yet he still has only 12 million in total !!

http://ubtstats.co.uk/ubt/UBT.php?cs=1&sz=2&nf=1&rp=0&project=CPDN


<big snip>

Chris


re: XML code

There are 3 files on each projects stats page: hosts, team and user - all in gz format.

So, it'll be a question of pulling out the user info and then parsing that, to obtain (say) just the UBT members and then displaying this in a worthwhile way.

Just the thought of trying to do that, on one project and then widening out the scope to include all projects (thereby emulating the ubtstats service), fills me with a whole heap of trepidation !!

If I have to go to night school to learn how to do it, then so be it !!

So, thanks again Chris. I'll make a note of all this and if things work out, then I may need to refer to some of this !!

regards
Tim

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Thu Apr 20, 2017 10:07 pm

Hi Tim,

In truth you have got my interest.

Basically because the user.xml file has the team ID in it means you can get away with just using this file's data, unless I am mistaken.

I have found a fairly simple but good charting/graphing JS/PHP component which could work on this.

Give me a week or two to try and get time to have a play about with the XML files and this graph/chart library, and see if I could spin something out without too much bother. Like I said I do not want to get into any heavy projects, but If I can do it fairly simply then I may be up for the challenge.

For now I will leave the DB stuff, and just write it to a CSV file (treating it like a DB with rows and columns) and pull the data from there into the chart/graph.

However if we go down this route, I would do it to display personal stats on each project, say users of the forum, rather than *all UBT member stats; you can just use BOINCstats for that.

E.G. http:// ubtstats .co.uk/ubt/ubt_graph2.php?my_userid=NNNNNN&project=combined&username=

where NNNNN is your "BOINC Cross Project IDentifier"

I have broken the URL as I do not want it to produce a link as that makes it difficult to see the page I am referring to.

--

Let me know what you think? Obviously I will not bother if above is not acceptable! (unless I do it just for myself).

I have a PHP/SQL hosting account so I should be able to play about on that without it affecting anything else.

--

*** PLEASE NOTE I AM NOT PROMISING AS TIME MAY NOT PERMIT, HOWEVER I WILL TRY AND PLAY ABOUT AND SEE HOW EASY IT WILL BE ***
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Thu Apr 20, 2017 10:51 pm

Hi Chris

Thanks for the info.

I'm not holding you to anything...so, don't worry about coding that much. Do it as a "tea break" type thing, rather than an "full on" programmer type of exercise.

I have had a chat with someone I know who has some experience in the sort of area I think I/we need...and he has given me some pointers as to how to get this project moving, which I'll look into in the next week or two. I also have someone else who I might be able to turn too, so I'll see how that pans out.

regards
Tim

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Thu Apr 20, 2017 11:39 pm

What I will do is let you guys continue, then if you get stuck and I have the time then come back to me and I will have a play about.

Chris
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Fri May 05, 2017 11:33 pm

UBT - Timbo wrote:...

I'm not holding you to anything

...
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Sun May 07, 2017 5:34 pm

chriscambridge wrote:
UBT - Timbo wrote:...

I'm not holding you to anything

...


????????

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Sun May 07, 2017 5:50 pm

:?
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Sun May 07, 2017 7:20 pm

chriscambridge wrote::?


Hi Chris

Sorry - but I was just responding to your previous message.

As far as this project goes, I've got someone who is asking around at a local tech college to see if anyone (such as a current student) wants to help out.

If that doesn't work out, then my next step is to contact another local education establishment and see if any of their students want to take on the job, as a way of doing something with an end result, perhaps over the summer.

If that doesn't work out then I've got one other option at a local university.

For now then, I'm just keeping fingers crossed that I can find some help - of course I'm in no rush, but it would be nice to see if I can find someone locally who can do some work on this.

regards
Tim

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Sun May 07, 2017 8:21 pm

That sounds like a good plan.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Fri May 19, 2017 9:09 pm

BTW you should change the title to PHP, SQL, (X)HTML/CSS Skills Request, as they are the skills you need.

Although you could also go down the JS or MEAN, Java, or even Python route too I guess.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
UBT - Timbo
UBT Forum Admin
Posts: 5833
Joined: Mon Mar 13, 2006 12:00 am
Location: NW Midlands
Contact:

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby UBT - Timbo » Fri May 19, 2017 9:20 pm

Done ;-)

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Tue Jun 13, 2017 10:10 am

** update **

I am undecided over trying to get access and manipulate the project stats in code.

I am aware that if we could access the stats in code then the ubtstats site could be fixed, and a lot of analysis could be done automatically - such as the Millionaires list etc.

I also have a couple of my own ideas that I am milling over and both require access to the stats in code.

The thing holding me back at the moment is how large the zip files and their contents are. And obviously you would need to times this by the number of projects..

My current hosting account would go over quota for just one project stats, let alone all of them, both the packed and unpacked files.

Perhaps someone, not me, should work out exactly how large the packed, and unpacked, project TAR stat files are all together?

(For now you would want to keep all the files, both packed and unpacked, available for testing - basically so you did not have to keep having to downloading/unpacking these, as that would be hassle).

At least if we had an idea of how large a hosting storage account you would need, then perhaps someone could offer this up for testing code and playing about with getting access to full UBT stats in code.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.

User avatar
Woodles
Formula-BOINC Sprint contributor
Posts: 6751
Joined: Thu Dec 20, 2007 12:00 am
Location: Cambridgeshire

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby Woodles » Tue Jun 13, 2017 11:49 am

At a quick check,the SETI stats dump is 105M in compressed form, no idea about expanded (I didn't download it :P)

At present UBT are active in 51 projects and have been active in a further 102 now discontinued projects (although stats are still available from some of them and about most of them)

Obviously the discontinued projects data would be a one off download and SETI is by far the project with the most team members (and hence data)

From what I could see, there isn't a separate download for each team, you'd have to download the entire stats file and pick out the data you wanted.

So to an untrained eye which probably doesn't know what it's doing, stats downloads would appear to be ~5Gig for discontinued projects plus ~2-3gig for the active projects.

If you're going to do this daily (or even hourly as Temujin does! :shock:) you're looking at a lot of data/bandwidth.

Of course, there's a very good chance that I don't know what I'm talking about :)

Mark

User avatar
chriscambridge
UBT Contributing Member
Posts: 982
Joined: Mon Aug 08, 2016 1:56 pm
Location: Cambridge, UK

Re: ubtstats.co.uk website - PHP, SQL, (X)HTML/CSS skills request

Postby chriscambridge » Tue Jun 13, 2017 12:06 pm

I just downloaded and unpacked SETI.

On disk:

107 Mb packed
513 Mb Unpacked

Total: 620 Mb

I would guess SETI is the largest project, so potentially, storage wise you could need, if only initially, for all projects current and discontinued:

153 * 620 Mb = 100 Gb (roughly).

Which for a web server is quite a lot.

--

In terms of processing every N hour, that's a whole different story. For instance, if your server has slow processors, perhaps you would not even be able to complete the full processing in less than one hour. If not, then how would you do hourly updates! Same thing goes for your transfer pipe.. If it takes longer than a hour to get every download, then it is clearly not possible to do updates every hour.

and so on.

Right now I am just trying to attack each potential problem as they occur. The first being downloading and unpacking.

Many webserver use /tmp/ files for downloading/unpacking, so you may have enough storage but not have a /tmp/ file big enough for all these downloads, and unpacking. I am no server network admin so I am not really 100% sure how this all works on LAMP; and I guess it depends on who and how they setup the server which will dictate possible problems.
Chris Cambridge is my old account; I am currently crunching through 2 new private account's.


Return to “Hardware, Software and Gaming”

Who is online

Users browsing this forum: No registered users and 2 guests