Majestic-12
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
Majestic-12
OK then another distributed web crawling project, very user configurable, please join in and help
Majestic-12
It runs fine here with DIMES, Red Lib & BOINC, only uses some CPU for a few mins every so often when it is archiving the results to be uploaded.
Works on most platforms is using Windows download the Zip file I had problems with the Installer version couldn't get it to install
Create an account through the client, then once created sign up to UK BOINC Team through this link log-in then go to the team pages
Also set your country through the link above England, Scotland, Wales, Northern Ireland or UK
Only suitable for people with an unlimited Internet connection as uses lots of bandwidth.
Can set profiles for different times of the day, to optimize your experience of using MJ-12 as well as your computer for day to day activities
Any takers to come and join me with this project?
A manual here on how to tweak it
Any Questions either use there forums or just ask here I will help in any way I can
Here are our team stats, plenty of stats available from the projects and also from Free-DC
Majestic-12
It runs fine here with DIMES, Red Lib & BOINC, only uses some CPU for a few mins every so often when it is archiving the results to be uploaded.
Works on most platforms is using Windows download the Zip file I had problems with the Installer version couldn't get it to install
Create an account through the client, then once created sign up to UK BOINC Team through this link log-in then go to the team pages
Also set your country through the link above England, Scotland, Wales, Northern Ireland or UK
Only suitable for people with an unlimited Internet connection as uses lots of bandwidth.
Can set profiles for different times of the day, to optimize your experience of using MJ-12 as well as your computer for day to day activities
Any takers to come and join me with this project?
A manual here on how to tweak it
Any Questions either use there forums or just ask here I will help in any way I can
Here are our team stats, plenty of stats available from the projects and also from Free-DC
Last edited by UBT - Halifax-lad on Sun Oct 08, 2006 1:07 pm, edited 1 time in total.
-
- Marvin the Dalek
- Posts: 4402
- Joined: Wed Mar 15, 2006 12:00 am
- Location: North Wales
Re: Majestic-12
Well that will knock a lot of people out including me!UBT - Halifax--lad wrote:
Only suitable for people with an unlimited Internet connection as uses lots of bandwidth.
Just had a quick scan to see what sort of bandwidth we're talking here - saw a mention of 30 gigs which is a helluva lot especially as I'm capped at 2 gig! Ho hum....
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 73
- Joined: Tue Mar 14, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
Is this normal?
[02/10/2006 17:29:21] Failed inside AsyncBeginGetResponse due to: System.NullReferenceException: Object reference not set to an instance of an object.
at Majestic12.WebCrawler.AsyncBeginGetResponseStream(IAsyncResult oAR)
[02/10/2006 17:32:24] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:32:24] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... estra=3236
[02/10/2006 17:39:31] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:39:31] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... uestra=216
[02/10/2006 17:44:04] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:44:04] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... uestra=375
[02/10/2006 17:46:06] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:46:06] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... uestra=376
[02/10/2006 17:55:14] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:55:14] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... uestra=379

[02/10/2006 17:29:21] Failed inside AsyncBeginGetResponse due to: System.NullReferenceException: Object reference not set to an instance of an object.
at Majestic12.WebCrawler.AsyncBeginGetResponseStream(IAsyncResult oAR)
[02/10/2006 17:32:24] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:32:24] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... estra=3236
[02/10/2006 17:39:31] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:39:31] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... uestra=216
[02/10/2006 17:44:04] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:44:04] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... uestra=375
[02/10/2006 17:46:06] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:46:06] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... uestra=376
[02/10/2006 17:55:14] SaveWorkerData can't operate on oItem that is null!
[02/10/2006 17:55:14] Worker #0 failed to save its data. Original BID=1 Original URL=http://www.cis.es/cis/opencms/CA/1_encu ... uestra=379

-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Marvin the Dalek
- Posts: 4402
- Joined: Wed Mar 15, 2006 12:00 am
- Location: North Wales
Yes but the DVD writing stuff don't say.... Hibernate when done!
Apart from that If I do hibernate then sometimes my BOINC stuff does not recover and it errors but if I shutdown it picks up near where it was.
I can't explain that one.
Anyway is the server accepting updates cus after 105 mins of work & 23000 URL's crawled I can still not update - just get a busy Try later message!
I'm gonna give it another 15 mins and give up i reckon
Apart from that If I do hibernate then sometimes my BOINC stuff does not recover and it errors but if I shutdown it picks up near where it was.
I can't explain that one.
Anyway is the server accepting updates cus after 105 mins of work & 23000 URL's crawled I can still not update - just get a busy Try later message!
I'm gonna give it another 15 mins and give up i reckon
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
I see you chose the right team to join.Dazza wrote:Just trying to find a half decent team to join thats all.

You're showing up fine and dandy now.

PS If I can run this thang without screwing up, it should be a doddle for you. :lol:
Last edited by Martle on Mon Oct 02, 2006 10:45 pm, edited 1 time in total.
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
I've been pestering the guys on the forum about DIMES and the results are just beginning to trickle through again. I don't assume for a moment that my input had anything to do with it but I'm just kidding myself that I must have irritated them so much, they just wanted to get rid of me. :PUBT - Halifax--lad wrote:I concur all is showing up for you now as well, with all this DIMES stuff up in the air at the moment its a great chance to get ahead on this project.
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
Created a signature logo for Majestic-12 users.
http://www.signature.ukboincteam.com/si ... serid>.gif
where <userid> is your userid!!
So my link is
http://www.signature.ukboincteam.com/si ... c/1305.gif
http://www.signature.ukboincteam.com/si ... serid>.gif
where <userid> is your userid!!
So my link is
http://www.signature.ukboincteam.com/si ... c/1305.gif
I've had to make the signature smaller to accomodate forum space so the new one does not show the last few days processing info
if you want one of those then it's
http://signature.ukboincteam.com/sigs/m ... serid>.gif
I have also made the display dynamic (with some clever code) so I don't have to keep creating the 800 graphics every few hours, only draw back is that the pig keeps changing everytime you visit a page.
I am only "allowed" to grab the data once every 24 hours but I am hoping this restriction can be lifted.
if you want one of those then it's
http://signature.ukboincteam.com/sigs/m ... serid>.gif
I have also made the display dynamic (with some clever code) so I don't have to keep creating the 800 graphics every few hours, only draw back is that the pig keeps changing everytime you visit a page.
I am only "allowed" to grab the data once every 24 hours but I am hoping this restriction can be lifted.
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
I think Dazza has converted his agent to Nuclear power. What's the secret for your high numbers? :roll:UBT - Halifax--lad wrote:Well someone has upped there game aint they
Last edited by Martle on Wed Oct 04, 2006 10:13 pm, edited 1 time in total.
-
- Posts: 11
- Joined: Sat May 06, 2006 1:00 am
As long as eveything is working and your agent is processing the buckets, I wouldn't worry about it.Greenways Court wrote:I think i've just created an account too !
Although when i click on the update button i keep getting a DNet error - busy try later?
A great big warm welcome to our cosy little team. :D
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 570
- Joined: Thu Mar 23, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 570
- Joined: Thu Mar 23, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
Its still working but look at the stats there more or less non exsitent they are not crediting at all, so I'm not doing as much DIMES for now no point doing it when I get no reward for itUBT - PiezPiedPy wrote:Dimes is Still working on my computer - No probs apart from a little glitch earlier this week
-
- Posts: 570
- Joined: Thu Mar 23, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 226
- Joined: Mon Mar 13, 2006 12:00 am
Wonder if anyone can enlighten me.
I've got 3 boxes all networked in through a router and obviously all sharing the same network connection. At the moment I'm running this baby on just one machine and was wondering what would happen if I was to also run it on all 3 machines.
Would the shared network connection be able to cope or would they all share a 33% share of the bandwidth therefore the same share of results? Is there any advantage in doing this?
I can't seem to find any answers on the Majestic forum. :?:
I've got 3 boxes all networked in through a router and obviously all sharing the same network connection. At the moment I'm running this baby on just one machine and was wondering what would happen if I was to also run it on all 3 machines.
Would the shared network connection be able to cope or would they all share a 33% share of the bandwidth therefore the same share of results? Is there any advantage in doing this?
I can't seem to find any answers on the Majestic forum. :?:
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
Yea that's what I thought. :roll:UBT - Halifax--lad wrote:AS your on one connection I guess the 3 clients would battle each other for bandwidth, I don't find you will get anything additional out of using 3 comps instead of just one as its all to do with bandwidth.
I suppose you could always give it a try
I'll just experiment in tweaking the agent for max efficiency. As you've been doing this longer me and you do seem to get higher totals, is there anything you can pass on in the way of tweaks?
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
I just took it nice and easy and increased my crawlers to a max level when I stopped getting errors and then increased my buckets ever so slowly. After setting these minor changes I let it run for a couple of hours to see if there was any improvement.UBT - Halifax--lad wrote:Martle what have you done to get your URL's so high, that's a massive increase to before
I wish I could tell you the secret but it's very machine specific and it's just a case of trial and error.
I think the Welsh Dragon has alot to do with my increase:lol:
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
To all those out there with an unrestricted connection, come and join our team and you too could experience the wonderful world of Majestic tweaking. Our team is quickly rising through the ranks (Cue - Halifax Lad :roll: ).
Join today and Dazza will throw in a neat stats sig - any colour you like (as long as it's black) with a picture of your favourite animal (as long as it's a pig).
Don't delay a moment longer as this offer will end very soon.
Join today and Dazza will throw in a neat stats sig - any colour you like (as long as it's black) with a picture of your favourite animal (as long as it's a pig).
Don't delay a moment longer as this offer will end very soon.
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 226
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Marvin the Dalek
- Posts: 4402
- Joined: Wed Mar 15, 2006 12:00 am
- Location: North Wales
It is a country, but under occupation by the English and has been for several hundred years.
(I wrote a load of 'Welsh Nationalist' stuff after that but realised that it was way off subject, probably politically incorrect and would might have provoked a 'flaming' session so scrubbed it.)
Just need to borrow Martles' flag!

(I wrote a load of 'Welsh Nationalist' stuff after that but realised that it was way off subject, probably politically incorrect and would might have provoked a 'flaming' session so scrubbed it.)

Just need to borrow Martles' flag!

-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
I've been tinkering around with a wireless router and have been having endless problems with the connection just ditching me out without warning. I've twiddled and tweaked and put the culprit down to Majestic. It's a big shame as I was really getting into this project.
So it with is great regret that I'm going to have to temporarily give this baby up until I find the cause.
So for the moment, the Dragon's going to have to bow out. :dontknow:

So it with is great regret that I'm going to have to temporarily give this baby up until I find the cause.
So for the moment, the Dragon's going to have to bow out. :dontknow:
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 17206
- Joined: Sat May 06, 2006 1:00 am
I got one a couple of months ago. Came free with a new ISP. It was dead easy to set up, only gripe was the transmission range isn`t very good.UBT - Halifax--lad wrote:I am getting a wireless router today, with any luck it will work fine its pretty advanced but I will soon find out tonight what happens
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am
-
- Posts: 391
- Joined: Sun Apr 30, 2006 1:00 am
I have a wireless router that won't stay connected to my Tiscali account without help.
I have to tell my email program to check every 4 minutes or the line is dropped (5 minutes is too long).
I always thought that it was Tiscali at fault as they supply usb modems that perhaps poll the system on a routine basis, where a router may just sit and wait, but that is just a my guess.
I have to tell my email program to check every 4 minutes or the line is dropped (5 minutes is too long).
I always thought that it was Tiscali at fault as they supply usb modems that perhaps poll the system on a routine basis, where a router may just sit and wait, but that is just a my guess.

Ahh, don't get me started on routers :evil:
I had a Belkin 4 port/wireless router that had 2 PCs wired into it and 2 laptops on wireless.
It worked fine for months, then all of a sudden the wired ports caused the ADSL connection to recycle.
Remove the 2 PCs cables and it would stay up forever but put em back in and it'd recycle.
I did initially put USB wireless on the 2 PCs but ended up buying a Netgear equivalent and everythings been fine since.
Never did find the reason for it.
I had a Belkin 4 port/wireless router that had 2 PCs wired into it and 2 laptops on wireless.
It worked fine for months, then all of a sudden the wired ports caused the ADSL connection to recycle.
Remove the 2 PCs cables and it would stay up forever but put em back in and it'd recycle.
I did initially put USB wireless on the 2 PCs but ended up buying a Netgear equivalent and everythings been fine since.
Never did find the reason for it.
While we're on the subject of wireless routers, if you go up a bit you'll find my thread with my problems concerning my newly installed Netgear wireless. While I was setting it up (and had it hard wired) I was having big probs with it ditching me out and I blamed Majestic 12.
I don't think that was the cause. I noticed that one of my boxes was allocated a different IP for both wireless and wired connection and during the setup, I was getting all sorts of probs with conflicting IP's. Now that I've disconnected it and it's purely on wireless, it seems to be stable now. The biggest headache was the effect that it was having on my BOINC Manager as it was seeing any communication from my other boxes as a security threat.
It all seems to be sorted out now except that the minute I try to change the Network CSS or apply any sort of encryption (WEP), it just stops working and I have to hard wire it again to regain a connection. So at the moment I'm running a completely open network that's probably being used by all my neighbours. :roll:
I'll sort it one day no doubt. 8)
I don't think that was the cause. I noticed that one of my boxes was allocated a different IP for both wireless and wired connection and during the setup, I was getting all sorts of probs with conflicting IP's. Now that I've disconnected it and it's purely on wireless, it seems to be stable now. The biggest headache was the effect that it was having on my BOINC Manager as it was seeing any communication from my other boxes as a security threat.
It all seems to be sorted out now except that the minute I try to change the Network CSS or apply any sort of encryption (WEP), it just stops working and I have to hard wire it again to regain a connection. So at the moment I'm running a completely open network that's probably being used by all my neighbours. :roll:
I'll sort it one day no doubt. 8)
-
- Posts: 3790
- Joined: Mon Mar 13, 2006 12:00 am